[
https://issues.apache.org/jira/browse/MAPREDUCE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964231#comment-15964231
]
Amit commented on MAPREDUCE-6835:
---------------------------------
I frequently keep on getting the same Warning "WARN io.ReadaheadPool: Failed
readahead on ifile
EBADF: Bad file descriptor" . What can be done to fix the same?
> WARN io.ReadaheadPool: Failed readahead on ifile
> ------------------------------------------------
>
> Key: MAPREDUCE-6835
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6835
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.7.3
> Environment: OS: Ubuntu Desktop 16.04 LTS
> JVM : Oracle Java 8 update 111
> Hadoop: 2.7.3 on pseudo distributed environment
> Hardware: Virtual Machine
> Proceesor - 2 cores
> RAM - 4 Gb
> Swap - 1 Gb
> Reporter: Krishna Kumar Nanjundaprasad
>
> Whenever a MapReduce job is executed, the warning of Bad file desciptor is
> displayed. This mostly happens when the Combiner/Reducer is running.
> This bug is resulting in incorrect results for the reducers where the warning
> is shown.
> The following is the entire job execution log.
> 17/01/24 14:34:25 INFO Configuration.deprecation: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 17/01/24 14:34:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 17/01/24 14:34:25 WARN mapreduce.JobResourceUploader: Hadoop command-line
> option parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
> 17/01/24 14:34:26 INFO input.FileInputFormat: Total input paths to process : 1
> 17/01/24 14:34:26 INFO mapreduce.JobSubmitter: number of splits:5
> 17/01/24 14:34:26 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_local1576831426_0001
> 17/01/24 14:34:26 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 17/01/24 14:34:26 INFO mapreduce.Job: Running job: job_local1576831426_0001
> 17/01/24 14:34:26 INFO mapred.LocalJobRunner: OutputCommitter set in config
> null
> 17/01/24 14:34:26 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:26 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 17/01/24 14:34:26 INFO mapred.LocalJobRunner: Waiting for map tasks
> 17/01/24 14:34:26 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_m_000000_0
> 17/01/24 14:34:27 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:27 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:27 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/krishna/flightinfo/input/2000.csv:0+134217728
> 17/01/24 14:34:27 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 17/01/24 14:34:27 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 17/01/24 14:34:27 INFO mapred.MapTask: soft limit at 83886080
> 17/01/24 14:34:27 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 17/01/24 14:34:27 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 17/01/24 14:34:27 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 17/01/24 14:34:27 INFO mapreduce.Job: Job job_local1576831426_0001 running in
> uber mode : false
> 17/01/24 14:34:27 INFO mapreduce.Job: map 0% reduce 0%
> 17/01/24 14:34:33 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:33 INFO mapreduce.Job: map 9% reduce 0%
> 17/01/24 14:34:35 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:35 INFO mapred.MapTask: Starting flush of map output
> 17/01/24 14:34:35 INFO mapred.MapTask: Spilling map output
> 17/01/24 14:34:35 INFO mapred.MapTask: bufstart = 0; bufend = 17437459;
> bufvoid = 104857600
> 17/01/24 14:34:35 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend =
> 20849028(83396112); length = 5365369/6553600
> 17/01/24 14:34:35 INFO mapred.MapTask: Finished spill 0
> 17/01/24 14:34:35 INFO mapred.Task:
> Task:attempt_local1576831426_0001_m_000000_0 is done. And is in the process
> of committing
> 17/01/24 14:34:35 INFO mapred.LocalJobRunner: map
> 17/01/24 14:34:35 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_m_000000_0' done.
> 17/01/24 14:34:35 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_m_000000_0
> 17/01/24 14:34:35 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_m_000001_0
> 17/01/24 14:34:35 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:35 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:35 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/krishna/flightinfo/input/2000.csv:134217728+134217728
> 17/01/24 14:34:36 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 17/01/24 14:34:36 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 17/01/24 14:34:36 INFO mapred.MapTask: soft limit at 83886080
> 17/01/24 14:34:36 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 17/01/24 14:34:36 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 17/01/24 14:34:36 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 17/01/24 14:34:36 INFO mapreduce.Job: map 100% reduce 0%
> 17/01/24 14:34:41 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:42 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:42 INFO mapred.MapTask: Starting flush of map output
> 17/01/24 14:34:42 INFO mapred.MapTask: Spilling map output
> 17/01/24 14:34:42 INFO mapred.MapTask: bufstart = 0; bufend = 17438733;
> bufvoid = 104857600
> 17/01/24 14:34:42 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend =
> 20848636(83394544); length = 5365761/6553600
> 17/01/24 14:34:42 INFO mapreduce.Job: map 32% reduce 0%
> 17/01/24 14:34:42 INFO mapred.MapTask: Finished spill 0
> 17/01/24 14:34:42 INFO mapred.Task:
> Task:attempt_local1576831426_0001_m_000001_0 is done. And is in the process
> of committing
> 17/01/24 14:34:42 INFO mapred.LocalJobRunner: map
> 17/01/24 14:34:42 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_m_000001_0' done.
> 17/01/24 14:34:42 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_m_000001_0
> 17/01/24 14:34:42 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_m_000002_0
> 17/01/24 14:34:42 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:42 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/krishna/flightinfo/input/2000.csv:268435456+134217728
> 17/01/24 14:34:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 17/01/24 14:34:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 17/01/24 14:34:43 INFO mapred.MapTask: soft limit at 83886080
> 17/01/24 14:34:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 17/01/24 14:34:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 17/01/24 14:34:43 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 17/01/24 14:34:44 INFO mapreduce.Job: map 100% reduce 0%
> 17/01/24 14:34:48 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:49 INFO mapreduce.Job: map 52% reduce 0%
> 17/01/24 14:34:49 INFO mapred.LocalJobRunner: map > map
> 17/01/24 14:34:49 INFO mapred.MapTask: Starting flush of map output
> 17/01/24 14:34:49 INFO mapred.MapTask: Spilling map output
> 17/01/24 14:34:49 INFO mapred.MapTask: bufstart = 0; bufend = 17438941;
> bufvoid = 104857600
> 17/01/24 14:34:49 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend =
> 20848572(83394288); length = 5365825/6553600
> 17/01/24 14:34:50 INFO mapred.MapTask: Finished spill 0
> 17/01/24 14:34:50 INFO mapred.Task:
> Task:attempt_local1576831426_0001_m_000002_0 is done. And is in the process
> of committing
> 17/01/24 14:34:50 INFO mapred.LocalJobRunner: map
> 17/01/24 14:34:50 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_m_000002_0' done.
> 17/01/24 14:34:50 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_m_000002_0
> 17/01/24 14:34:50 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_m_000003_0
> 17/01/24 14:34:50 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:50 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:50 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/krishna/flightinfo/input/2000.csv:402653184+134217728
> 17/01/24 14:34:50 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 17/01/24 14:34:50 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 17/01/24 14:34:50 INFO mapred.MapTask: soft limit at 83886080
> 17/01/24 14:34:50 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 17/01/24 14:34:50 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 17/01/24 14:34:50 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 17/01/24 14:34:51 INFO mapreduce.Job: map 100% reduce 0%
> 17/01/24 14:34:56 INFO mapred.LocalJobRunner:
> 17/01/24 14:34:56 INFO mapred.MapTask: Starting flush of map output
> 17/01/24 14:34:56 INFO mapred.MapTask: Spilling map output
> 17/01/24 14:34:56 INFO mapred.MapTask: bufstart = 0; bufend = 17284761;
> bufvoid = 104857600
> 17/01/24 14:34:56 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend =
> 20896012(83584048); length = 5318385/6553600
> 17/01/24 14:34:56 INFO mapred.LocalJobRunner: map > sort
> 17/01/24 14:34:56 INFO mapred.MapTask: Finished spill 0
> 17/01/24 14:34:56 INFO mapred.Task:
> Task:attempt_local1576831426_0001_m_000003_0 is done. And is in the process
> of committing
> 17/01/24 14:34:56 INFO mapred.LocalJobRunner: map
> 17/01/24 14:34:56 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_m_000003_0' done.
> 17/01/24 14:34:56 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_m_000003_0
> 17/01/24 14:34:56 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_m_000004_0
> 17/01/24 14:34:56 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:56 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:56 INFO mapred.MapTask: Processing split:
> hdfs://localhost:9000/user/krishna/flightinfo/input/2000.csv:536870912+33280701
> 17/01/24 14:34:56 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 17/01/24 14:34:56 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 17/01/24 14:34:56 INFO mapred.MapTask: soft limit at 83886080
> 17/01/24 14:34:56 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 17/01/24 14:34:56 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 17/01/24 14:34:56 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 17/01/24 14:34:58 INFO mapred.LocalJobRunner:
> 17/01/24 14:34:58 INFO mapred.MapTask: Starting flush of map output
> 17/01/24 14:34:58 INFO mapred.MapTask: Spilling map output
> 17/01/24 14:34:58 INFO mapred.MapTask: bufstart = 0; bufend = 4279717;
> bufvoid = 104857600
> 17/01/24 14:34:58 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend =
> 24897564(99590256); length = 1316833/6553600
> 17/01/24 14:34:59 INFO mapred.MapTask: Finished spill 0
> 17/01/24 14:34:59 INFO mapred.Task:
> Task:attempt_local1576831426_0001_m_000004_0 is done. And is in the process
> of committing
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: map
> 17/01/24 14:34:59 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_m_000004_0' done.
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_m_000004_0
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: map task executor complete.
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: Waiting for reduce tasks
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: Starting task:
> attempt_local1576831426_0001_r_000000_0
> 17/01/24 14:34:59 INFO output.FileOutputCommitter: File Output Committer
> Algorithm version is 1
> 17/01/24 14:34:59 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 17/01/24 14:34:59 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle@55bf46b
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: MergerManager:
> memoryLimit=334338464, maxSingleShuffleLimit=83584616,
> mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
> 17/01/24 14:34:59 INFO reduce.EventFetcher:
> attempt_local1576831426_0001_r_000000_0 Thread started: EventFetcher for
> fetching Map Completion Events
> 17/01/24 14:34:59 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
> output of map attempt_local1576831426_0001_m_000003_0 decomp: 17 len: 21 to
> MEMORY
> 17/01/24 14:34:59 INFO reduce.InMemoryMapOutput: Read 17 bytes from
> map-output for attempt_local1576831426_0001_m_000003_0
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
> map-output of size: 17, inMemoryMapOutputs.size() -> 1, commitMemory -> 0,
> usedMemory ->17
> 17/01/24 14:34:59 WARN io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 17/01/24 14:34:59 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
> output of map attempt_local1576831426_0001_m_000002_0 decomp: 17 len: 21 to
> MEMORY
> 17/01/24 14:34:59 INFO reduce.InMemoryMapOutput: Read 17 bytes from
> map-output for attempt_local1576831426_0001_m_000002_0
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
> map-output of size: 17, inMemoryMapOutputs.size() -> 2, commitMemory -> 17,
> usedMemory ->34
> 17/01/24 14:34:59 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
> output of map attempt_local1576831426_0001_m_000001_0 decomp: 17 len: 21 to
> MEMORY
> 17/01/24 14:34:59 WARN io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 17/01/24 14:34:59 INFO reduce.InMemoryMapOutput: Read 17 bytes from
> map-output for attempt_local1576831426_0001_m_000001_0
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
> map-output of size: 17, inMemoryMapOutputs.size() -> 3, commitMemory -> 34,
> usedMemory ->51
> 17/01/24 14:34:59 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
> output of map attempt_local1576831426_0001_m_000004_0 decomp: 17 len: 21 to
> MEMORY
> 17/01/24 14:34:59 INFO reduce.InMemoryMapOutput: Read 17 bytes from
> map-output for attempt_local1576831426_0001_m_000004_0
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
> map-output of size: 17, inMemoryMapOutputs.size() -> 4, commitMemory -> 51,
> usedMemory ->68
> 17/01/24 14:34:59 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle
> output of map attempt_local1576831426_0001_m_000000_0 decomp: 17 len: 21 to
> MEMORY
> 17/01/24 14:34:59 WARN io.ReadaheadPool: Failed readahead on ifile
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
> at
> org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
> at
> org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 17/01/24 14:34:59 INFO reduce.InMemoryMapOutput: Read 17 bytes from
> map-output for attempt_local1576831426_0001_m_000000_0
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: closeInMemoryFile ->
> map-output of size: 17, inMemoryMapOutputs.size() -> 5, commitMemory -> 68,
> usedMemory ->85
> 17/01/24 14:34:59 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: 5 / 5 copied.
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: finalMerge called with 5
> in-memory map-outputs and 0 on-disk map-outputs
> 17/01/24 14:34:59 INFO mapred.Merger: Merging 5 sorted segments
> 17/01/24 14:34:59 INFO mapred.Merger: Down to the last merge-pass, with 5
> segments left of total size: 50 bytes
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: Merged 5 segments, 85 bytes
> to disk to satisfy reduce memory limit
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: Merging 1 files, 81 bytes
> from disk
> 17/01/24 14:34:59 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes
> from memory into reduce
> 17/01/24 14:34:59 INFO mapred.Merger: Merging 1 sorted segments
> 17/01/24 14:34:59 INFO mapred.Merger: Down to the last merge-pass, with 1
> segments left of total size: 70 bytes
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: 5 / 5 copied.
> 17/01/24 14:34:59 INFO Configuration.deprecation: mapred.skip.on is
> deprecated. Instead, use mapreduce.job.skiprecords
> 17/01/24 14:34:59 INFO mapred.Task:
> Task:attempt_local1576831426_0001_r_000000_0 is done. And is in the process
> of committing
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: 5 / 5 copied.
> 17/01/24 14:34:59 INFO mapred.Task: Task
> attempt_local1576831426_0001_r_000000_0 is allowed to commit now
> 17/01/24 14:34:59 INFO output.FileOutputCommitter: Saved output of task
> 'attempt_local1576831426_0001_r_000000_0' to
> hdfs://localhost:9000/user/krishna/flightinfo/flighttotaldistance/_temporary/0/task_local1576831426_0001_r_000000
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: reduce > reduce
> 17/01/24 14:34:59 INFO mapred.Task: Task
> 'attempt_local1576831426_0001_r_000000_0' done.
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: Finishing task:
> attempt_local1576831426_0001_r_000000_0
> 17/01/24 14:34:59 INFO mapred.LocalJobRunner: reduce task executor complete.
> 17/01/24 14:35:00 INFO mapreduce.Job: map 100% reduce 100%
> 17/01/24 14:35:00 INFO mapreduce.Job: Job job_local1576831426_0001 completed
> successfully
> 17/01/24 14:35:00 INFO mapreduce.Job: Counters: 35
> File System Counters
> FILE: Number of bytes read=34276
> FILE: Number of bytes written=1980499
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=2482554234
> HDFS: Number of bytes written=16
> HDFS: Number of read operations=61
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=8
> Map-Reduce Framework
> Map input records=5683048
> Map output records=5683047
> Map output bytes=73879611
> Map output materialized bytes=105
> Input split bytes=625
> Combine input records=5683047
> Combine output records=5
> Reduce input groups=1
> Reduce shuffle bytes=105
> Reduce input records=5
> Reduce output records=1
> Spilled Records=10
> Shuffled Maps =5
> Failed Shuffles=0
> Merged Map outputs=5
> GC time elapsed (ms)=2068
> Total committed heap usage (bytes)=2224029696
> Shuffle Errors
> BAD_ID=0
> CONNECTION=0
> IO_ERROR=0
> WRONG_LENGTH=0
> WRONG_MAP=0
> WRONG_REDUCE=0
> File Input Format Counters
> Bytes Read=570167997
> File Output Format Counters
> Bytes Written=16
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
