That is a correct way to do it, but it can also be affected by settings in other DoFns that may be declared if they run in the same mapper or reducer.
From: Luke Hansen [mailto:[email protected]] Sent: Tuesday, October 13, 2015 3:52 PM To: [email protected] Subject: Hadoop Configuration from DoFn Does anyone know if this is the right way to configure Hadoop from a Crunch DoFn? This didn't seem to affect anything. Thanks! @Override public void configure(Configuration conf) { conf.set("mapreduce.map.java.opts", "-Xmx3900m"); conf.set("mapreduce.reduce.java.opts", "-Xmx3900m"); conf.set("mapreduce.map.memory.mb", "4096"); conf.set("mapreduce.reduce.memory.mb", "4096"); } This email is intended only for the use of the individual(s) to whom it is addressed. If you have received this communication in error, please immediately notify the sender and delete the original email.
