Running into an issue with some early attempts at converting over to MRv2.
I'm submitting a job using an MRv1 client installation, onto an MRv2
cluster. However, the client is displaying initialization log messages
for every single map task, like the log entries pasted below. In
addition to being unnecessary and wasteful, this is also resulting in
the job taking a long time to get submitted. (It's writing log output
for several thousand map tasks before the job started.
MRv1 didn't used to do this. Anyone know if there's any way to disable?
Thanks,
DR
---
14/04/14 16:56:19 INFO mapred.LocalJobRunner: Starting task:
attempt_local438339852_0001_m_000127_0
14/04/14 16:56:19 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
14/04/14 16:56:19 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@245eca5e
14/04/14 16:56:19 INFO mapred.MapTask: Processing split:
s3n://<my-s3-bucket>/merging/eventsBids/20140408/part-r-00157:0+10081976
14/04/14 16:56:19 INFO mapred.MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/04/14 16:56:19 INFO mapred.MapTask: io.sort.mb = 100
14/04/14 16:56:19 INFO mapred.LocalJobRunner: Starting task:
attempt_local438339852_0001_m_000128_0
14/04/14 16:56:19 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
14/04/14 16:56:19 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d59a8ed
14/04/14 16:56:19 INFO mapred.MapTask: Processing split:
s3n://<my-s3-bucket>/merging/eventsBids/20140408/part-r-00021:0+10081963
14/04/14 16:56:19 INFO mapred.MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/04/14 16:56:19 INFO mapred.MapTask: io.sort.mb = 100
...
