[jira] [Created] (MAPREDUCE-5423) Rare deadlock situation when reducers try to fetch map output
Chu Tong created MAPREDUCE-5423: --- Summary: Rare deadlock situation when reducers try to fetch map output Key: MAPREDUCE-5423 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5423 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chu Tong During our cluster deployment, we found there is deadlock situation when reducers try to fetch map output. We had 5 fetchers and log snippet illustrates this problem is below: 2013-07-18 04:32:28,135 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManager: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2013-07-18 04:32:28,138 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0 Thread started: EventFetcher for fetching Map Completion Events 2013-07-18 04:32:28,146 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:28,319 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_17_0 sent hash and receievd reply 2013-07-18 04:32:28,320 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_17_0 decomp: 27 len: 31 to MEMORY 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 27 bytes from map-output for attempt_1373902166027_0622_m_17_0 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 27, inMemoryMapOutputs.size() -gt; 1, commitMemory -gt; 0, usedMemory -gt;27 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 179s 2013-07-18 04:32:33,158 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:33,161 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_16_0 sent hash and receievd reply 2013-07-18 04:32:33,200 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_16_0 decomp: 55841282 len: 55841286 to MEMORY 2013-07-18 04:32:33,322 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 55841282 bytes from map-output for attempt_1373902166027_0622_m_16_0 2013-07-18 04:32:33,323 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 55841282, inMemoryMapOutputs.size() -gt; 2, commitMemory -gt; 27, usedMemory -gt;55841309 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 118022137 bytes from map-output for attempt_1373902166027_0622_m_15_0 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 118022137, inMemoryMapOutputs.size() -gt; 3, commitMemory -gt; 55841309, usedMemory -gt;173863446 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 413s 2013-07-18 04:32:42,188 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1
[jira] [Updated] (MAPREDUCE-5423) Rare deadlock situation when reducers try to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chu Tong updated MAPREDUCE-5423: Description: During our cluster deployment, we found there is deadlock situation when reducers try to fetch map output. We had 5 fetchers and log snippet illustrates this problem is below (all fetchers went into a wait state after they can't acquire more RAM beyond the memoryLimit and no fetcher is releasing memory): 2013-07-18 04:32:28,135 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManager: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2013-07-18 04:32:28,138 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0 Thread started: EventFetcher for fetching Map Completion Events 2013-07-18 04:32:28,146 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:28,319 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_17_0 sent hash and receievd reply 2013-07-18 04:32:28,320 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_17_0 decomp: 27 len: 31 to MEMORY 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 27 bytes from map-output for attempt_1373902166027_0622_m_17_0 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 27, inMemoryMapOutputs.size() -gt; 1, commitMemory -gt; 0, usedMemory -gt;27 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 179s 2013-07-18 04:32:33,158 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:33,161 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_16_0 sent hash and receievd reply 2013-07-18 04:32:33,200 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_16_0 decomp: 55841282 len: 55841286 to MEMORY 2013-07-18 04:32:33,322 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 55841282 bytes from map-output for attempt_1373902166027_0622_m_16_0 2013-07-18 04:32:33,323 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 55841282, inMemoryMapOutputs.size() -gt; 2, commitMemory -gt; 27, usedMemory -gt;55841309 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 118022137 bytes from map-output for attempt_1373902166027_0622_m_15_0 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 118022137, inMemoryMapOutputs.size() -gt; 3, commitMemory -gt; 55841309, usedMemory -gt;173863446 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 413s 2013-07-18 04:32:42,188 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:42,190
[jira] [Updated] (MAPREDUCE-5423) Rare deadlock situation when reducers try to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chu Tong updated MAPREDUCE-5423: Description: During our cluster deployment, we found there is a very rare deadlock situation when reducers try to fetch map output. We had 5 fetchers and log snippet illustrates this problem is below (all fetchers went into a wait state after they can't acquire more RAM beyond the memoryLimit and no fetcher is releasing memory): 2013-07-18 04:32:28,135 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManager: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2013-07-18 04:32:28,138 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0 Thread started: EventFetcher for fetching Map Completion Events 2013-07-18 04:32:28,146 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:28,319 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_17_0 sent hash and receievd reply 2013-07-18 04:32:28,320 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_17_0 decomp: 27 len: 31 to MEMORY 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 27 bytes from map-output for attempt_1373902166027_0622_m_17_0 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 27, inMemoryMapOutputs.size() -gt; 1, commitMemory -gt; 0, usedMemory -gt;27 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 179s 2013-07-18 04:32:33,158 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:33,161 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_16_0 sent hash and receievd reply 2013-07-18 04:32:33,200 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_16_0 decomp: 55841282 len: 55841286 to MEMORY 2013-07-18 04:32:33,322 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 55841282 bytes from map-output for attempt_1373902166027_0622_m_16_0 2013-07-18 04:32:33,323 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 55841282, inMemoryMapOutputs.size() -gt; 2, commitMemory -gt; 27, usedMemory -gt;55841309 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 118022137 bytes from map-output for attempt_1373902166027_0622_m_15_0 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 118022137, inMemoryMapOutputs.size() -gt; 3, commitMemory -gt; 55841309, usedMemory -gt;173863446 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 413s 2013-07-18 04:32:42,188 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:42,188 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18
[jira] [Commented] (MAPREDUCE-5423) Rare deadlock situation when reducers try to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720979#comment-13720979 ] Chu Tong commented on MAPREDUCE-5423: - This is on 2.0.2-alpha Rare deadlock situation when reducers try to fetch map output - Key: MAPREDUCE-5423 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5423 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chu Tong During our cluster deployment, we found there is a very rare deadlock situation when reducers try to fetch map output. We had 5 fetchers and log snippet illustrates this problem is below (all fetchers went into a wait state after they can't acquire more RAM beyond the memoryLimit and no fetcher is releasing memory): 2013-07-18 04:32:28,135 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManager: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2013-07-18 04:32:28,138 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0 Thread started: EventFetcher for fetching Map Completion Events 2013-07-18 04:32:28,146 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:28,319 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_17_0 sent hash and receievd reply 2013-07-18 04:32:28,320 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_17_0 decomp: 27 len: 31 to MEMORY 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 27 bytes from map-output for attempt_1373902166027_0622_m_17_0 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 27, inMemoryMapOutputs.size() -gt; 1, commitMemory -gt; 0, usedMemory -gt;27 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 179s 2013-07-18 04:32:33,158 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:33,161 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_16_0 sent hash and receievd reply 2013-07-18 04:32:33,200 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_16_0 decomp: 55841282 len: 55841286 to MEMORY 2013-07-18 04:32:33,322 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 55841282 bytes from map-output for attempt_1373902166027_0622_m_16_0 2013-07-18 04:32:33,323 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 55841282, inMemoryMapOutputs.size() -gt; 2, commitMemory -gt; 27, usedMemory -gt;55841309 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 118022137 bytes from map-output for attempt_1373902166027_0622_m_15_0 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 118022137, inMemoryMapOutputs.size() -gt; 3, commitMemory -gt; 55841309, usedMemory -gt;173863446 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 413s 2013-07-18 04:32:42,188 INFO [EventFetcher for
[jira] [Commented] (MAPREDUCE-5423) Rare deadlock situation when reducers try to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721207#comment-13721207 ] Chu Tong commented on MAPREDUCE-5423: - I think you are right. I took a look at MAPREDUCE-4842 and I believe this is the issue I experienced. Can you please close this as a duplicate? Thanks Rare deadlock situation when reducers try to fetch map output - Key: MAPREDUCE-5423 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5423 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha Reporter: Chu Tong During our cluster deployment, we found there is a very rare deadlock situation when reducers try to fetch map output. We had 5 fetchers and log snippet illustrates this problem is below (all fetchers went into a wait state after they can't acquire more RAM beyond the memoryLimit and no fetcher is releasing memory): 2013-07-18 04:32:28,135 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManager: MergerManager: memoryLimit=1503238528, maxSingleShuffleLimit=375809632, mergeThreshold=992137472, ioSortFactor=10, memToMemMergeOutputsThreshold=10 2013-07-18 04:32:28,138 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0 Thread started: EventFetcher for fetching Map Completion Events 2013-07-18 04:32:28,146 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:28,146 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:28,319 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_17_0 sent hash and receievd reply 2013-07-18 04:32:28,320 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_17_0 decomp: 27 len: 31 to MEMORY 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 27 bytes from map-output for attempt_1373902166027_0622_m_17_0 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 27, inMemoryMapOutputs.size() -gt; 1, commitMemory -gt; 0, usedMemory -gt;27 2013-07-18 04:32:28,325 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: 101-09-04.sc1.verticloud.com:8080 freed by fetcher#1 in 179s 2013-07-18 04:32:33,158 INFO [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: attempt_1373902166027_0622_r_01_0: Got 1 new map-outputs 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging 101-09-04.sc1.verticloud.com:8080 with 1 to fetcher#1 2013-07-18 04:32:33,158 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to 101-09-04.sc1.verticloud.com:8080 to fetcher#1 2013-07-18 04:32:33,161 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: for url=8080/mapOutput?job=job_1373902166027_0622amp;reduce=1amp;map=attempt_1373902166027_0622_m_16_0 sent hash and receievd reply 2013-07-18 04:32:33,200 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 about to shuffle output of map attempt_1373902166027_0622_m_16_0 decomp: 55841282 len: 55841286 to MEMORY 2013-07-18 04:32:33,322 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 55841282 bytes from map-output for attempt_1373902166027_0622_m_16_0 2013-07-18 04:32:33,323 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 55841282, inMemoryMapOutputs.size() -gt; 2, commitMemory -gt; 27, usedMemory -gt;55841309 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Read 118022137 bytes from map-output for attempt_1373902166027_0622_m_15_0 2013-07-18 04:32:39,594 INFO [fetcher#1] org.apache.hadoop.mapreduce.task.reduce.MergeManager: closeInMemoryFile -gt; map-output of size: 118022137, inMemoryMapOutputs.size() -gt; 3, commitMemory -gt; 55841309, usedMemory -gt;173863446 2013-07-18
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572622#comment-13572622
]
Chu Tong commented on MAPREDUCE-4822:
-
When I look at the code the TaskAttemptID.forName method does not look like a
simple conversion. Therefore, I think it is safer to leave what it is right now.
public static TaskAttemptID forName(String str
) throws IllegalArgumentException {
if(str == null)
return null;
String exceptionMsg = null;
try {
String[] parts = str.split(Character.toString(SEPARATOR));
if(parts.length == 6) {
if(parts[0].equals(ATTEMPT)) {
String type = parts[3];
TaskType t = TaskID.getTaskType(type.charAt(0));
if(t != null) {
return new org.apache.hadoop.mapred.TaskAttemptID
(parts[1],
Integer.parseInt(parts[2]),
t, Integer.parseInt(parts[4]),
Integer.parseInt(parts[5]));
} else
exceptionMsg = Bad TaskType identifier. TaskAttemptId string :
+ str + is not properly formed.;
}
}
} catch (Exception ex) {
//fall below
}
if (exceptionMsg == null) {
exceptionMsg = TaskAttemptId string : + str
+ is not properly formed;
}
throw new IllegalArgumentException(exceptionMsg);
}
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572624#comment-13572624
]
Chu Tong commented on MAPREDUCE-4822:
-
Sorry, I did not put the code in codeblock style.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572749#comment-13572749
]
Chu Tong commented on MAPREDUCE-4822:
-
My primary concern is the error checking to throw the exception at the end of
the function and I do not know if we want to get ride of that.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572867#comment-13572867
]
Chu Tong commented on MAPREDUCE-4822:
-
Ok, I think your point is more solid. I made the changes as you instructed.
Thank you.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572953#comment-13572953
]
Chu Tong commented on MAPREDUCE-4822:
-
right ..., Thanks Jason!
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Changed all affected files under jobhistory directory
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571912#comment-13571912
]
Chu Tong commented on MAPREDUCE-4822:
-
No testcase is included as the change is trivial.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Removed unnecessary conversion, please review.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Priority: Trivial
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong reassigned MAPREDUCE-4822:
---
Assignee: Chu Tong
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Fix Version/s: 0.24.0
Labels: patch (was: )
Target Version/s: 0.23.4
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Open (was: Patch Available)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Open (was: Patch Available)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: (was: MAPREDUCE-4822.patch)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Attachment: MAPREDUCE-4822.patch
No testcase is included as the change is trivial.
For JavaDoc warnings, it is false positive as the same number of warnings are
generated on a clean build under my dev environment.
-1 overall.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated 20 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version )
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events
[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Tong updated MAPREDUCE-4822:
Status: Patch Available (was: Open)
Unnecessary conversions in History Events
-
Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0
Attachments: MAPREDUCE-4822.patch
There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
