Luke, Generally that configuration should be set on the Configuration object passed to Pipeline vs on the individual DoFns. The configure(...) method is called when re-instantiating the DoFn on the Map/Reduce task and at that point those memory settings wouldn't be honored.
On Tue, Oct 13, 2015 at 2:52 PM, Luke Hansen <[email protected]> wrote: > Does anyone know if this is the right way to configure Hadoop from a > Crunch DoFn? This didn't seem to affect anything. > > Thanks! > > @Override > public void configure(Configuration conf) { > conf.set("mapreduce.map.java.opts", "-Xmx3900m"); > conf.set("mapreduce.reduce.java.opts", "-Xmx3900m"); > > conf.set("mapreduce.map.memory.mb", "4096"); > conf.set("mapreduce.reduce.memory.mb", "4096"); > } > >
