[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445997#comment-17445997 ] Ahmed Hussein commented on MAPREDUCE-7169: -- bq. Why is denying racks and hosts should be enabled separately? Can you please elaborate? Currently we try to avoid launching on same rack as old attempt if there are no containers on diff rack then we try choosing node other than old attempt node. Hi [~BilwaST], I second [~jeagles] about separating the rack and the hosts. Considering the block placement policy on Hadoop, launching the speculative attempt on a different rack every-time will be a double edge sword. on same rack will be a double edge sword. leaving it to be configurable will give much more flexibility to work along with block placements. Otherwise, the speculative changes will be "All or nothing: Always run on a different rack or disable it and speculate on the same node ". > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string
[
https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316680#comment-17316680
]
Ahmed Hussein commented on MAPREDUCE-7333:
--
[~Marcono1234] Thanks for spotting this bug.
I suggest to delete {{toHex()}} implementation because it is not being used
anywhere in the Hadoop trunk.
If you find some code redundancy that could be improved by calling {{toHex()}},
then please go ahead with:
* fixing the implementation
* replace redundant code blocks with calls to {{toHex()}}.
> SecureShuffleUtils.toHex(byte[]) creates malformed hex string
> -
>
> Key: MAPREDUCE-7333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.2.2
>Reporter: Marcono1234
>Assignee: Peter Bacsko
>Priority: Major
>
> {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}}
> creates malformed hex strings:
> {code}
> for (byte b : ba) {
> ps.printf("%x", b);
> }
> {code}
> The pattern {{"%x"}} would for bytes < 16 only have on hex char and for
> example both {{1, 0}} and {{16}} would have the result {{"10"}}.
> A correct (and more efficient) implementation would be:
> {code}
> public static String toHex(byte[] ba) {
> StringBuilder sb = new StringBuilder(ba.length * 2);
> for (byte b : ba) {
> int unsignedB = b & 0xFF;
> if (unsignedB < 16) {
> sb.append('0');
> }
> sb.append(Integer.toHexString(unsignedB));
> }
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string
[
https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316390#comment-17316390
]
Ahmed Hussein commented on MAPREDUCE-7333:
--
Thanks [~pbacsko] for the changes.
Sorry for the delay. I was away on a vacation.
I am going to take a look at the changes.
> SecureShuffleUtils.toHex(byte[]) creates malformed hex string
> -
>
> Key: MAPREDUCE-7333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.2.2
>Reporter: Marcono1234
>Assignee: Peter Bacsko
>Priority: Major
>
> {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}}
> creates malformed hex strings:
> {code}
> for (byte b : ba) {
> ps.printf("%x", b);
> }
> {code}
> The pattern {{"%x"}} would for bytes < 16 only have on hex char and for
> example both {{1, 0}} and {{16}} would have the result {{"10"}}.
> A correct (and more efficient) implementation would be:
> {code}
> public static String toHex(byte[] ba) {
> StringBuilder sb = new StringBuilder(ba.length * 2);
> for (byte b : ba) {
> int unsignedB = b & 0xFF;
> if (unsignedB < 16) {
> sb.append('0');
> }
> sb.append(Integer.toHexString(unsignedB));
> }
> return sb.toString();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Attachment: MAPREDUCE-7332.branch-2.10.002.patch > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch, > MAPREDUCE-7332.branch-2.10.002.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309079#comment-17309079 ] Ahmed Hussein commented on MAPREDUCE-7332: -- findbugs is not supported. We need to pull HADOOP-16870 into branch-2.10. https://issues.apache.org/jira/browse/HADOOP-16870?focusedCommentId=17309077=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17309077 > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308954#comment-17308954 ] Ahmed Hussein commented on MAPREDUCE-7332: -- [~Jim_Brennan] Can you please take a look at the fix? > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[
https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308876#comment-17308876
]
Ahmed Hussein commented on MAPREDUCE-7332:
--
To be on the safe side, I preferred to use the guava library which should be
compatible for both JDK7 and JDK8.
To achieve the same level of concurrency, I used {{LoadingCache}} which
guarantees atomicity of the operations.
I experimented {{guava.MapMaker().makeMap()}} but the unit test
{{TestMRIntermediateDataEncryption}} --- with multiple reducers -- failed.
The failure was due to the fact that the {{ConcurrentMap}} returned from
{{guava.MapMaker().makeMap()}} did not guarantee atomicity.
I tested the patch locally and reviewed the checkstyle.
{code:bash}
dev-support/bin/test-patch \
--plugins="maven,checkstyle" \
--test-parallel=true \
--branch=branch-2.10 \
patches/MAPREDUCE-7332.branch-2.10.001.patch
{code}
> Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
> ---
>
> Key: MAPREDUCE-7332
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7332.branch-2.10.001.patch
>
>
> I mistakenly uploaded a patch for branch-2.10 that uses JDK8.
> Yetus did not fail though. It should be investigated why it was not failing
> if JDK8+ is used in the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Attachment: MAPREDUCE-7332.branch-2.10.001.patch Status: Patch Available (was: In Progress) > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7332 started by Ahmed Hussein. > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Priority: Minor (was: Major) > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[
https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308092#comment-17308092
]
Ahmed Hussein edited comment on MAPREDUCE-7332 at 3/24/21, 6:42 PM:
{code:bash}
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0
[ERROR]
/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
[ERROR]
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
java.util.concurrent.ConcurrentHashMap.keySet()
{code}
The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed
between JDK7 and JDK8 . A runtime error is thrown if branch-2.10 is compiled
using JDK8.
branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}}
was (Author: ahussein):
{code:bash}
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0
[ERROR]
/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
[ERROR]
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
java.util.concurrent.ConcurrentHashMap.keySet()
{code}
The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed
between JDK7 and JDK8 .
branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}}
> Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
> ---
>
> Key: MAPREDUCE-7332
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> I mistakenly uploaded a patch for branch-2.10 that uses JDK8.
> Yetus did not fail though. It should be investigated why it was not failing
> if JDK8+ is used in the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[
https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308092#comment-17308092
]
Ahmed Hussein commented on MAPREDUCE-7332:
--
{code:bash}
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0
[ERROR]
/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
[ERROR]
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184:
Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView
java.util.concurrent.ConcurrentHashMap.keySet()
{code}
The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed
between JDK7 and JDK8 .
branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}}
> Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
> ---
>
> Key: MAPREDUCE-7332
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> I mistakenly uploaded a patch for branch-2.10 that uses JDK8.
> Yetus did not fail though. It should be investigated why it was not failing
> if JDK8+ is used in the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
Ahmed Hussein created MAPREDUCE-7332: Summary: Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 Key: MAPREDUCE-7332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Reporter: Ahmed Hussein Assignee: Ahmed Hussein I mistakenly uploaded a patch for branch-2.10 that uses JDK8. Yetus did not fail though. It should be investigated why it was not failing if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307115#comment-17307115
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
Thanks [~Jim_Brennan]!
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch,
> MAPREDUCE-7322.branch-3.2.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306172#comment-17306172
]
Ahmed Hussein commented on MAPREDUCE-7325:
--
[~Jim_Brennan] Can you please take a look at the patch?
The bug was due to an incorrect seek in the local fetcher. It works only if
encryption was disabled.
The changes are:
# The bug fix is just two lines change in LocalFetcher.java
# In {{TestMRIntermediateDataEncryption}}, I added an extra step to verify the
output of the jobs. This is done by comparing the checksum of the output files.
The reference checksum comes from a job that starts at the beginning of the
unit test with encryption disabled.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7325:
-
Attachment: MAPREDUCE-7325.002.patch
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7325:
-
Attachment: MAPREDUCE-7325.001.patch
Status: Patch Available (was: In Progress)
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7325.001.patch
>
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7325 started by Ahmed Hussein.
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned MAPREDUCE-7325:
Assignee: Ahmed Hussein
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
[
https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned MAPREDUCE-7321:
Assignee: Ahmed Hussein
> TestMRIntermediateDataEncryption does not cleanup data folders
> --
>
> Key: MAPREDUCE-7321
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> The data generated by the {{TestMRIntermediateDataEncryption}} does not get
> deleted after Completing the tests. This contributes to Hadoop taking large
> disk spaces to build and run tests.
> The following folders need to be removed:
> * folders of the DFSCluster and the YarnCluster
> * Files used to submit jobs in the test-dir folder
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
[
https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein resolved MAPREDUCE-7321.
--
Resolution: Fixed
> TestMRIntermediateDataEncryption does not cleanup data folders
> --
>
> Key: MAPREDUCE-7321
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, test
>Reporter: Ahmed Hussein
>Priority: Major
>
> The data generated by the {{TestMRIntermediateDataEncryption}} does not get
> deleted after Completing the tests. This contributes to Hadoop taking large
> disk spaces to build and run tests.
> The following folders need to be removed:
> * folders of the DFSCluster and the YarnCluster
> * Files used to submit jobs in the test-dir folder
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302867#comment-17302867
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
Thanks [~Jim_Brennan] for the reviews and committing the patches.
I ran the related tests locally and they were successful.
The Yetus failure has been raised in YARN-10501. Hopefully, it will be fixed
soon.
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch,
> MAPREDUCE-7322.branch-3.2.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.branch-2.10.009.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch,
> MAPREDUCE-7322.branch-3.2.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.branch-3.2.009.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-3.2.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302092#comment-17302092
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
Thanks [~Jim_Brennan] for the review and merging the patch.
In order to port this into branch-3.2 and branch-2.10, we will need to port
MAPREDUCE-7320 because to pull {{GenericTestUtils}} changes.
I can provide patches for both MAPREDUCE-7320 branch-3.2 and branch-2.10 if
they do not pull clean.
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300522#comment-17300522
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
Thanks [~Jim_Brennan] for the reviews.
h4. Unit test Failures:
* TestConfTest failure is unrelated. It is caused by HADOOP-17571. I see that
the failure is being discussed there
h4. List of Changes
* [^MAPREDUCE-7322.009.patch]:
** Address checkstyle error by adding javadoc comment to {{package-info.java}}
** change log level in {{SpillCallBackPathsFinder}} to debug-level.
* [^MAPREDUCE-7322.008.patch]:
** Address checkstyle errors: add {{package-info.java}}, fix lines more than
80 characters.
** Address findbugs warning about ignored return from {{mkdirs}}.
* [^MAPREDUCE-7322.007.patch]:
** Moved {{TestMRIntermediateDataEncryption}} to
{{org.apache.hadoop.mapreduce.}}
** Change the implementation of {{TestMRIntermediateDataEncryption}}
generating textInput, then counting the words. The configuration are set in a
way to guarantee that Spilling is triggered.
** I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers",
2, 4, false}} ) because of the bug reported in MAPREDUCE-7325.
** I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}}
and to expose the random sentence generator.
** Created {{org.apache.hadoop.mapreduce.security.SpillCallBackInjector}}
injecting callbacks while spilling files.
** Instead of changing {{CryptoUtils}} - which can be tricky for backward
compatibility and ecosystem- I preferred adding a new wrapper
{{org.apache.hadoop.mapreduce.security.IntermediateEncryptedStream}}.
** capturing the Spilled files implied changing the calls in: {{MapTask}},
{{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}},
{{BackupStore}}, {{MergeManagerImpl}}
** Added some utility methods in
{{org.apache.hadoop.mapreduce.util.MRJobConfUtil}} to organize output folders:
{{setLocalDirectoriesConfigForTesting}}, and
{{initEncryptedIntermediateConfigsForTesting}}.
** Applied initializations to other unit tests related to intermediate data:
{{TestMROpportunisticMaps}}, {{TestLocalJobSubmission}}, {{TestMerger}},
{{TestRecovery}}.
** Fixed a bug in the {{JarFinder}} where temporary file was not deleted in
test-dir
** Fixed a bug in {{TestMerger}} which was closing the dfs before Yarn cluster.
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.009.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch,
> MAPREDUCE-7322.009.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.008.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.007.patch
Status: Patch Available (was: Open)
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.007.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: (was: MAPREDUCE-7322.007.patch)
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Status: Open (was: Patch Available)
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.006.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch,
> MAPREDUCE-7322.006.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17298877#comment-17298877
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
I think that I should add a configuration to set the class implementation of
the callback injector.
This would allow starting the runtime on a cluster with the injector enabled.
I ruled out making the configuration per Job as this would imply parsing the
JobConf everytime (not a no-op for production).
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297487#comment-17297487
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
List of changes:
* Moved {{TestMRIntermediateDataEncryption}} to
{{org.apache.hadoop.mapreduce}}.
* Change the implementation of {{TestMRIntermediateDataEncryption}} generating
textInput, then counting the words. The configuration are set in a way to
guarantee that Spilling is triggered.
* I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2,
4, false}} ) because of the bug reported in MAPREDUCE-7325.
* I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and
to expose the random sentence generator.
* Created {{SpillCallBackInjector}} injecting callbacks while spilling files.
* Instead of changing {{CryptoUtils}} - which can be tricky for backward
compatibility and ecosystem- I preferred adding a new wrapper
{{IntermediateEncryptedStream}}.
* capturing the Spilled files implied changing the calls in: {{MapTask}},
{{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}},
{{BackupStore}}, {{MergeManagerImpl}}
* Added some utility methods in {{GenericTestUtils}} to organize output folders.
* Applied initializations to other unit tests related to intermediate data.
* Fixed a bug in the {{JarFinder}} where temporary file was not deleted in
{{test-dir}}
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297490#comment-17297490
]
Ahmed Hussein commented on MAPREDUCE-7322:
--
I submitted
[MAPREDUCE-7322.005.patch|https://issues.apache.org/jira/secure/attachment/13021872/MAPREDUCE-7322.005.patch]
fixing the check styles warnings.
[~Jim_Brennan] Can you please take a look at the patch?
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.005.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.004.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.003.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Status: Patch Available (was: In Progress)
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Labels: patch-available (was: )
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.002.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7322:
-
Attachment: MAPREDUCE-7322.001.patch
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7322.001.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7322 started by Ahmed Hussein.
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned MAPREDUCE-7322:
Assignee: Ahmed Hussein
> revisiting TestMRIntermediateDataEncryption
>
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291002#comment-17291002
]
Ahmed Hussein commented on MAPREDUCE-7325:
--
This bug was not detected before because the unit tests did not cover spill
encryption.
Example of these unit test is reported in MAPREDUCE-7322
> Intermediate data encryption is broken in LocalJobRunner
>
>
> Key: MAPREDUCE-7325
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security
>Reporter: Ahmed Hussein
>Priority: Major
>
> With enabling intermediate encryption, running a job using LocalJobRunner
> with multiple reducers fails with the following error:
>
> {code:bash}
> 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
> (LocalJobRunner.java:run(590)) - job_local1344328673_0004
> java.lang.Exception:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in localfetcher#5
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
> Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
> error in shuffle in localfetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
> at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
> at
> org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> at
> org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
> at
> org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
> {code}
> The bug can be reproduced with any test unit like TestLocalJobSubmission.
> Another way to reproduce the bug is to run {{LargeSorter}} with multiple
> reducers.
> {code:java}
> Configuration config = new Configuration();
> // set all the necessary configurations
> config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
> String[] args = new String[] {"output-dir"};
> int res = ToolRunner.run(config, new LargeSorter(), args);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
Ahmed Hussein created MAPREDUCE-7325:
Summary: Intermediate data encryption is broken in LocalJobRunner
Key: MAPREDUCE-7325
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, security
Reporter: Ahmed Hussein
With enabling intermediate encryption, running a job using LocalJobRunner with
multiple reducers fails with the following error:
{code:bash}
2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner
(LocalJobRunner.java:run(590)) - job_local1344328673_0004
java.lang.Exception:
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle
in localfetcher#5
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
in shuffle in localfetcher#5
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error
at
org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229)
at
org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
at
org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
at
org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
{code}
The bug can be reproduced with any test unit like TestLocalJobSubmission.
Another way to reproduce the bug is to run {{LargeSorter}} with multiple
reducers.
{code:java}
Configuration config = new Configuration();
// set all the necessary configurations
config.setInt(LargeSorter.NUM_REDUCE_TASKS,2);
String[] args = new String[] {"output-dir"};
int res = ToolRunner.run(config, new LargeSorter(), args);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7320:
-
Labels: pull-request-available (was: )
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7320:
-
Labels: (was: pull-request-available)
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7320 started by Ahmed Hussein.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work stopped] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7320 stopped by Ahmed Hussein.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288425#comment-17288425
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
You have a reasonable point [~Jim_Brennan]. I agree it would be inconvenient to
modify the code just to look to the output and the logs.
I will disable the cleaning in {{@AfterClass}}.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
Ahmed Hussein created MAPREDUCE-7322:
Summary: revisiting TestMRIntermediateDataEncryption
Key: MAPREDUCE-7322
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, security, test
Reporter: Ahmed Hussein
I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
actually little to do with encryption.
I have the following conclusion:
* Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
change the behavior of the unit test.
* There are no spill files generated by either mappers/reducers
* Wrapping I/O streams with Crypto never happens during the execution of the
unit test.
Unless I misunderstand the purpose of that unit test, I suggest that it gets
re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287438#comment-17287438
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
Thanks [~Jim_Brennan].
I see your point.
My thoughts were that leaving the generated data by default increase the disk
utilization.
I face this problem because local hadoop repos sitting on my local machine are
eating large space, especially when I run long sequence of unit tests.
My intuition was that the common case is to run the unit tests, while debugging
output file is less frequent generally speaking. In that case case, the
developer can omit the {{@AfterClass}} that deletes the directory until he is
done with debugging.
P.S: I also found many unit tests that generate data in the home user directory
{{/tmp/hadoop}} which usually never get cleaned. This probably can be a
separate jira to organize the output.
Let me know if you still prefer to keep the data output after the unit test is
done. As you said, having the top level predictable is a good goal by itself.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286079#comment-17286079
]
Ahmed Hussein edited comment on MAPREDUCE-7320 at 2/19/21, 4:45 PM:
This fix is cleaning up the clusters directories after finishing the unit tests.
* It touches {{MiniYARNCluster.java}} in order to change the base dir to
{{target/test-dir/$TEST_CLASS_NAME}}
* test classes affected
** TestMRJobClient,
** TestStreamingBadRecords,
** TestClusterMapReduceTestCase,
** TestBadRecords.
** TestMRCJCJobClient,
** TestJobName
I tested the TestUnits that use {{MiniYARNCluster.java}} such as:
{code:bash}
Class
MiniYARNCluster
TestOSSMiniYarnCluster (3 usages found)
TestMRTimelineEventHandling (4 usages found)
TestJobHistoryEventHandler (3 usages found)
TestHadoopArchiveLogs (3 usages found)
TestHadoopArchiveLogsRunner (3 usages found)
TestDynamometerInfra (3 usages found)
TestDSTimelineV10
TestDSTimelineV20
TestDSTimelineV15
TestUnmanagedAMLauncher (3 usages found)
TestApplicationMasterServiceProtocolForTimelineV2
TestFederationRMFailoverProxyProvider (3 usages found)
TestHedgingRequestRMFailoverProxyProvider (4 usages found)
TestNoHaRMFailoverProxyProvider (5 usages found)
TestRMFailover (4 usages found)
TestAMRMClient
TestAMRMClientPlacementConstraints
TestAMRMProxy (5 usages found)
TestNMClient (3 usages found)
TestOpportunisticContainerAllocationE2E (3 usages found)
TestYarnClient (3 usages found)
TestYarnClientWithReservation (12 usages found)
TestYarnCLI (7 usages found)
TestContainerManagerSecurity (2 usages found)
TestDiskFailures (2 usages found)
TestMiniYarnCluster (9 usages found)
TestMiniYARNClusterForHA (2 usages found)
TestMiniYarnClusterNodeUtilization (3 usages found)
TestEncryptedShuffle{code}
was (Author: ahussein):
This fix is cleaning up the clusters directories after finishing the unit tests.
* It touches {{MiniYARNCluster.java}} in order to change the base dir to
{{target/test-dir/$TEST_CLASS_NAME}}
* test classes affected
** TestMRJobClient,
** TestStreamingBadRecords,
** TestClusterMapReduceTestCase,
** TestBadRecords.
** TestMRCJCJobClient,
** TestJobName
I tested the TestUnits that use {{MiniYARNCluster.java}} such as:
{code:bash}
Class
MiniYARNCluster
TestOSSMiniYarnCluster (3 usages found)
TestMRTimelineEventHandling (4 usages found)
TestJobHistoryEventHandler (3 usages found)
TestHadoopArchiveLogs (3 usages found)
TestHadoopArchiveLogsRunner (3 usages found)
TestDynamometerInfra (3 usages found)
TestDSTimelineV10
TestDSTimelineV20
TestDSTimelineV15
TestUnmanagedAMLauncher (3 usages found)
TestApplicationMasterServiceProtocolForTimelineV2
TestFederationRMFailoverProxyProvider (3 usages found)
TestHedgingRequestRMFailoverProxyProvider (4 usages found)
TestNoHaRMFailoverProxyProvider (5 usages found)
TestRMFailover (4 usages found)
TestAMRMClient
TestAMRMClientPlacementConstraints
TestAMRMProxy (5 usages found)
TestNMClient (3 usages found)
TestOpportunisticContainerAllocationE2E (3 usages found)
TestYarnClient (3 usages found)
TestYarnClientWithReservation (12 usages found)
TestYarnCLI (7 usages found)
TestContainerManagerSecurity (2 usages found)
TestDiskFailures (2 usages found)
TestMiniYarnCluster (9 usages found)
TestMiniYARNClusterForHA (2 usages found)
TestMiniYarnClusterNodeUtilization (3 usages found)
{code}
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286079#comment-17286079
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
This fix is cleaning up the clusters directories after finishing the unit tests.
* It touches {{MiniYARNCluster.java}} in order to change the base dir to
{{target/test-dir/$TEST_CLASS_NAME}}
* test classes affected
** TestMRJobClient,
** TestStreamingBadRecords,
** TestClusterMapReduceTestCase,
** TestBadRecords.
** TestMRCJCJobClient,
** TestJobName
I tested the TestUnits that use {{MiniYARNCluster.java}} such as:
{code:bash}
Class
MiniYARNCluster
TestOSSMiniYarnCluster (3 usages found)
TestMRTimelineEventHandling (4 usages found)
TestJobHistoryEventHandler (3 usages found)
TestHadoopArchiveLogs (3 usages found)
TestHadoopArchiveLogsRunner (3 usages found)
TestDynamometerInfra (3 usages found)
TestDSTimelineV10
TestDSTimelineV20
TestDSTimelineV15
TestUnmanagedAMLauncher (3 usages found)
TestApplicationMasterServiceProtocolForTimelineV2
TestFederationRMFailoverProxyProvider (3 usages found)
TestHedgingRequestRMFailoverProxyProvider (4 usages found)
TestNoHaRMFailoverProxyProvider (5 usages found)
TestRMFailover (4 usages found)
TestAMRMClient
TestAMRMClientPlacementConstraints
TestAMRMProxy (5 usages found)
TestNMClient (3 usages found)
TestOpportunisticContainerAllocationE2E (3 usages found)
TestYarnClient (3 usages found)
TestYarnClientWithReservation (12 usages found)
TestYarnCLI (7 usages found)
TestContainerManagerSecurity (2 usages found)
TestDiskFailures (2 usages found)
TestMiniYarnCluster (9 usages found)
TestMiniYARNClusterForHA (2 usages found)
TestMiniYarnClusterNodeUtilization (3 usages found)
{code}
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein reassigned MAPREDUCE-7320:
Assignee: Ahmed Hussein
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-7320 started by Ahmed Hussein.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
Ahmed Hussein created MAPREDUCE-7321:
Summary: TestMRIntermediateDataEncryption does not cleanup data
folders
Key: MAPREDUCE-7321
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, test
Reporter: Ahmed Hussein
The data generated by the {{TestMRIntermediateDataEncryption}} does not get
deleted after Completing the tests. This contributes to Hadoop taking large
disk spaces to build and run tests.
The following folders need to be removed:
* folders of the DFSCluster and the YarnCluster
* Files used to submit jobs in the test-dir folder
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
Ahmed Hussein created MAPREDUCE-7320:
Summary: ClusterMapReduceTestCase does not clean directories
Key: MAPREDUCE-7320
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ahmed Hussein
Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
directories and folders without cleaning them up.
For example:
{code:bash}
men test -Dtest=TestMRJobClient{code}
generates the following directories:
{code:bash}
- target
-+ ConfigurableMiniMRCluster_315090884
-+ ConfigurableMiniMRCluster_1335188990
-+ ConfigurableMiniMRCluster_1973037511
-+ test-dir
-+ dfs
-+ hadopp-XYZ-01
-+ hadopp-XYZ-02
-+ hadopp-XYZ-03
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7141: Assignee: Ahmed Hussein (was: Kuhu Shukla) > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276587#comment-17276587 ] Ahmed Hussein commented on MAPREDUCE-7141: -- Thanks [~kshukla] for uploading the patch. I will rebase the patch and try to see how this can be pushed forward. > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251141#comment-17251141 ] Ahmed Hussein commented on MAPREDUCE-7169: -- Hey [~BilwaST], is there anything I can help with to get this change merged? > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175059#comment-17175059 ] Ahmed Hussein commented on MAPREDUCE-7169: -- [~BilwaST] sorry for my late response. I overlooked the notification. (+1 non-binding) for MAPREDUCE-7169.007.patch > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104401#comment-17104401
]
Ahmed Hussein edited comment on MAPREDUCE-7169 at 5/11/20, 12:25 PM:
-
{quote}Actually when task attempt is killed by default Avataar is VIRGIN. this
is defect which needs to be addressed. If speculative task attempt is killed it
is launched as normal task attempt
{quote}
That's interesting.
If speculative task attempt is killed, a new task attempt is launched as
normal. The new attempt will have a new ID I guess, right? in that case, the
map entry would not be relevant and a new entry is created for the new attempt
ID.
{quote}How do you get taskattempt details in RMContainerAllocator??
{quote}
I see your point. It is preferred that "request" object should only be alive if
it is pending or not handled yet. In order to keep that concept, the simplest
work around is to change the field in {{TaskAttemptBlacklistManager}}.
{code:java}
private Map>
taskAttemptToEventMapping =
new HashMap>();
public void addToTaskAttemptBlacklist(ContainerRequestEvent event) {
if (null != event.getBlacklistedNodes()
&& event.getBlacklistedNodes().size() > 0) {
taskAttemptToEventMapping.put(event.getAttemptID(),
event.getBlacklistedNodes());
}
}
{code}
One last thing:
Since we are going to keep {{TaskAttemptBlacklistManager}}, then
{{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be
renamed to something more relevant to its functionality
{{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc.
was (Author: ahussein):
{quote}Actually when task attempt is killed by default Avataar is VIRGIN. this
is defect which needs to be addressed. If speculative task attempt is killed it
is launched as normal task attempt{quote}
That's interesting.
If speculative task attempt is killed, a new task attempt is launched as
normal. The new attempt will have a new ID I guess, right? in that case, the
map entry would not be relevant and a new entry is created for the new attempt
ID.
{quote}How do you get taskattempt details in RMContainerAllocator??{quote}
I see your point. The point is that a "request" object should only be alive if
it is pending or not handled yet. In order to keep that concept, the simplest
work around is to change the field in {{TaskAttemptBlacklistManager}}.
{code:java}
private Map>
taskAttemptToEventMapping =
new HashMap>();
public void addToTaskAttemptBlacklist(ContainerRequestEvent event) {
if (null != event.getBlacklistedNodes()
&& event.getBlacklistedNodes().size() > 0) {
taskAttemptToEventMapping.put(event.getAttemptID(),
event.getBlacklistedNodes());
}
}
{code}
One last thing:
Since we are going to keep {{TaskAttemptBlacklistManager}}, then
{{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be
renamed to something more relevant to its functionality
{{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104401#comment-17104401
]
Ahmed Hussein commented on MAPREDUCE-7169:
--
{quote}Actually when task attempt is killed by default Avataar is VIRGIN. this
is defect which needs to be addressed. If speculative task attempt is killed it
is launched as normal task attempt{quote}
That's interesting.
If speculative task attempt is killed, a new task attempt is launched as
normal. The new attempt will have a new ID I guess, right? in that case, the
map entry would not be relevant and a new entry is created for the new attempt
ID.
{quote}How do you get taskattempt details in RMContainerAllocator??{quote}
I see your point. The point is that a "request" object should only be alive if
it is pending or not handled yet. In order to keep that concept, the simplest
work around is to change the field in {{TaskAttemptBlacklistManager}}.
{code:java}
private Map>
taskAttemptToEventMapping =
new HashMap>();
public void addToTaskAttemptBlacklist(ContainerRequestEvent event) {
if (null != event.getBlacklistedNodes()
&& event.getBlacklistedNodes().size() > 0) {
taskAttemptToEventMapping.put(event.getAttemptID(),
event.getBlacklistedNodes());
}
}
{code}
One last thing:
Since we are going to keep {{TaskAttemptBlacklistManager}}, then
{{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be
renamed to something more relevant to its functionality
{{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101802#comment-17101802
]
Ahmed Hussein edited comment on MAPREDUCE-7169 at 5/7/20, 3:47 PM:
---
[~BilwaST], the speculation, taskAttempts, and allocations code set is not a
straightforward module to tackle. You did a great good job!
I have the following points:
- Address checkstyle report.
*Corner Case scenario:*
* Assuming that a new speculative attempt is created. Following the
implementation, the new attempt X will have blacklisted nodes and skipped racks
relevant to the original taskAttempt Y
* Assuming taskAttempt Y is killed before attempt X gets assigned.
* The RMContainerAllocator would still assign a host to attemptX based on the
dated blacklists.
Is this the expected behavior? or it is supposed to clear attemptX'
blacklisted nodes?
*TaskAttemptBlacklistManager*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to {{concurrentHashMap}} would be sufficient. What do you think?
* In {{taskAttemptToEventMapping}}, the data is only removed when the
taskAttempt is assigned. If taskAttempt is killed before being assigned,
{{taskAttemptToEventMapping}} would still have the taskAttempt.
*{{TaskAttemptBlacklistManager}}*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to concurrentHashMap would be sufficient. What do you think?
* In taskAttemptToEventMapping, the data is only removed when the taskAttempt
is assigned. If taskAttempt is killed before being assigned,
taskAttemptToEventMapping would still have the taskAttempt.
*{{TaskAttemptImpl}}*
* Racks are going to be black listed too. Not just nodes. I believe that the
javadoc and description in default.xml should emphasize that enabling the flag
also avoids the local rack unless no other rack is available for scheduling.
*{{TaskImpl}}*
* why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field
called {{avataar}} ?
*{{ContainerRequestEvent}}*
- That's a design issue. One would expect that RequestEvent's lifetime
should not survive {{handle()}} call. Therefore, the metadata should be
consumed by the handlers. In the patch,
{{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt.
Then you won't need {{TaskAttemptBlacklistManager}} class.
was (Author: ahussein):
[~BilwaST], the speculation, taskAttempts, and allocations code set is not a
straightforward module to tackle. You did a great good job!
I have the following points:
*Corner Case scenario:*
* Assuming that a new speculative attempt is created. Following the
implementation, the new attempt X will have blacklisted nodes and skipped racks
relevant to the original taskAttempt Y
* Assuming taskAttempt Y is killed before attempt X gets assigned.
* The RMContainerAllocator would still assign a host to attemptX based on the
dated blacklists.
Is this the expected behavior? or it is supposed to clear attemptX'
blacklisted nodes?
*TaskAttemptBlacklistManager*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to {{concurrentHashMap}} would be sufficient. What do you think?
* In {{taskAttemptToEventMapping}}, the data is only removed when the
taskAttempt is assigned. If taskAttempt is killed before being assigned,
{{taskAttemptToEventMapping}} would still have the taskAttempt.
*{{TaskAttemptBlacklistManager}}*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to concurrentHashMap would be sufficient. What do you think?
* In taskAttemptToEventMapping, the data is only removed when the taskAttempt
is assigned. If taskAttempt is killed before being assigned,
taskAttemptToEventMapping would still have the taskAttempt.
*{{TaskAttemptImpl}}*
* Racks are going to be black listed too. Not just nodes. I believe that the
javadoc and description in default.xml should emphasize that enabling the flag
also avoids the local rack unless no other rack is available for scheduling.
*{{TaskImpl}}*
* why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field
called {{avataar}} ?
*{{ContainerRequestEvent}}*
- That's a design issue. One would expect that RequestEvent's lifetime
should not survive {{handle()}} call. Therefore, the metadata should be
consumed by the handlers. In the patch,
{{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt.
Then you won't need {{TaskAttemptBlacklistManager}} class.
> Speculative attempts should not run on the same
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101802#comment-17101802
]
Ahmed Hussein commented on MAPREDUCE-7169:
--
[~BilwaST], the speculation, taskAttempts, and allocations code set is not a
straightforward module to tackle. You did a great good job!
I have the following points:
*Corner Case scenario:*
* Assuming that a new speculative attempt is created. Following the
implementation, the new attempt X will have blacklisted nodes and skipped racks
relevant to the original taskAttempt Y
* Assuming taskAttempt Y is killed before attempt X gets assigned.
* The RMContainerAllocator would still assign a host to attemptX based on the
dated blacklists.
Is this the expected behavior? or it is supposed to clear attemptX'
blacklisted nodes?
*TaskAttemptBlacklistManager*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to {{concurrentHashMap}} would be sufficient. What do you think?
* In {{taskAttemptToEventMapping}}, the data is only removed when the
taskAttempt is assigned. If taskAttempt is killed before being assigned,
{{taskAttemptToEventMapping}} would still have the taskAttempt.
*{{TaskAttemptBlacklistManager}}*
* Should that object be synchronized? I believe there are more than one thread
reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}}
to concurrentHashMap would be sufficient. What do you think?
* In taskAttemptToEventMapping, the data is only removed when the taskAttempt
is assigned. If taskAttempt is killed before being assigned,
taskAttemptToEventMapping would still have the taskAttempt.
*{{TaskAttemptImpl}}*
* Racks are going to be black listed too. Not just nodes. I believe that the
javadoc and description in default.xml should emphasize that enabling the flag
also avoids the local rack unless no other rack is available for scheduling.
*{{TaskImpl}}*
* why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field
called {{avataar}} ?
*{{ContainerRequestEvent}}*
- That's a design issue. One would expect that RequestEvent's lifetime
should not survive {{handle()}} call. Therefore, the metadata should be
consumed by the handlers. In the patch,
{{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt.
Then you won't need {{TaskAttemptBlacklistManager}} class.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096874#comment-17096874 ] Ahmed Hussein commented on MAPREDUCE-7169: -- Thanks [~BilwaST], I will take a look to the details of the implementation > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095491#comment-17095491
]
Ahmed Hussein commented on MAPREDUCE-7169:
--
Thanks [~BilwaST] !
{quote}With change like this, there are chances that task attempt will get
launched on same node where it was launched which wouldn't solve the problem.
If we blacklist node then in next iteration of containers assigned it will be
launched on other node.
{quote}
The same node will be picked if there are no other available nodes. In
[^MAPREDUCE-7169.005.patch] , what is the expected behavior if the resources
available to run the taskAttempt are only available on the same node? I do not
see this case in the unit test.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: MAPREDUCE-7261.003.patch
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch,
> MAPREDUCE-7261.003.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: HDFS-10499-branch-2.10.003.patch
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: (was: HDFS-10499-branch-2.10.003.patch)
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272.004.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch,
> MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch, MAPREDUCE-7272.004.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080965#comment-17080965
]
Ahmed Hussein commented on MAPREDUCE-7272:
--
Thanks [~epayne] for the feedback.
I fixed the problems reported by findbugs and uploaded the patches
[^MAPREDUCE-7272.003.patch] and [^MAPREDUCE-7272-branch-2.10.004.patch]
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch,
> MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272-branch-2.10.004.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch,
> MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272.003.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080646#comment-17080646
]
Ahmed Hussein commented on MAPREDUCE-7272:
--
{quote}It looks like findbugs is still complaining about {{ConcurrentHashMap
may not be atomic}}
{quote}
Yes, it is not atomic. Java-7 does not provide an atomic update to concurrent
hash map. it requires using synchronized block but I do not see it is necessary
to add this overhead since the functionality is just for logging.
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272.002.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Status: Patch Available (was: Open)
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Status: Open (was: Patch Available)
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272-branch-2.10.003.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch,
> MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
> the listener will log the progress every 2 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.interval-seconds}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Description:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the task
progress will be logged every 10% of delta progress. Default is 5%.
* {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if
the listener will log the progress every 2 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold. Default is 1
minute.
The listener will long whichever of {{delta.threshold}} and
{{wait.interval-seconds}} is reached first.
Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
two flags and log the task progress on every update.
was:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the task
progress will be logged every 10% of delta progress. Default is 5%.
* {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if the
listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold. Default is 1
minute.
The listener will long whichever of {{delta.threshold}} and {{wait.delta.time}}
is reached first.
Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
two flags and log the task progress on every update.
> TaskAttemptListenerImpl excessive log messages
> --
>
>
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272-branch-2.10.002.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if
> the listener will log the progress every 3 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.delta.time}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272-branch-2.10.001.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272-branch-2.10.001.patch,
> MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if
> the listener will log the progress every 3 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold. Default is
> 1 minute.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.delta.time}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override
> those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Description:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the task
progress will be logged every 10% of delta progress. Default is 5%.
* {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if the
listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold. Default is 1
minute.
The listener will long whichever of {{delta.threshold}} and {{wait.delta.time}}
is reached first.
Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
two flags and log the task progress on every update.
was:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the task
progress will be logged every 10% of delta progress. Default is 5%.
* {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if the
listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold.
The listener will long whichever of {{delta.threshold}} and {{wait.delta.time}}
is reached first.
Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
two flags and log the task progress on every update.
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key:
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Attachment: MAPREDUCE-7272.001.patch
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: MAPREDUCE-7272.001.patch
>
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if
> the listener will log the progress every 3 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.delta.time}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
> two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Status: Patch Available (was: Open)
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files.
> One every call, the listener uses {{LOG.info()}} to printout the progress of
> the {{TaskAttempt}}.
> {code:java}
> taskAttemptStatus.progress = taskStatus.getProgress();
> LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
> + taskStatus.getProgress());
> {code}
>
> {code:bash}
> 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_007783_0 is : 0.40713295
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_020681_0 is : 0.55573714
> 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_024371_0 is : 0.54190344
> 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_033182_0 is : 0.50264555
> 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1586003420099_716645_m_022375_0 is : 0.5495565
> {code}
> After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
> thought that while it is helpful to have a log print of task progress, it is
> still excessive to log the progress in every update.
> This Jira is to suppress the excessive logging from TaskAttemptListener
> without affecting the frequency of progress updates.
> There are two flags:
> * {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the
> task progress will be logged every 10% of delta progress. Default is 5%.
> * {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if
> the listener will log the progress every 3 minutes. This is helpful for long
> running tasks that take long time to achieve the delta threshold.
> The listener will long whichever of {{delta.threshold}} and
> {{wait.delta.time}} is reached first.
> Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
> two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7272:
-
Description:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{-Dmapreduce.task.progress.min.delta.threshold=0.10}}: means that the task
progress will be logged every 10% of delta progress. Default is 5%.
* {{-Dmapreduce.task.progress.wait.delta.time.threshold=3}}: means that if the
listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold.
The listener will long whichever of {{delta.threshold}} and {{wait.delta.time}}
is reached first.
Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override those
two flags and log the task progress on every update.
was:
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{ {{-Dmapreduce.task.progress.min.delta.threshold=0.10: means that the
task progress will be logged every 10% of delta progress. Default is 5%.
* {{ {{-Dmapreduce.task.progress.wait.delta.time.threshold=3: means that
if the listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold.
The listener will long whichever of \{{delta.threshold}} and
\{{wait.delta.time}} is reached first.
Enabling \{{LOG.DEBUG}} for TaskAttemptListenerImpl will override
those two flags and log the task progress on every update.
> TaskAttemptListenerImpl excessive log messages
> --
>
> Key: MAPREDUCE-7272
>
[jira] [Created] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
Ahmed Hussein created MAPREDUCE-7272:
Summary: TaskAttemptListenerImpl excessive log messages
Key: MAPREDUCE-7272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein
{{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. One
every call, the listener uses {{LOG.info()}} to printout the progress of the
{{TaskAttempt}}.
{code:java}
taskAttemptStatus.progress = taskStatus.getProgress();
LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : "
+ taskStatus.getProgress());
{code}
{code:bash}
2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_007783_0 is : 0.40713295
2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_020681_0 is : 0.55573714
2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_024371_0 is : 0.54190344
2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_033182_0 is : 0.50264555
2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1586003420099_716645_m_022375_0 is : 0.5495565
{code}
After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we
thought that while it is helpful to have a log print of task progress, it is
still excessive to log the progress in every update.
This Jira is to suppress the excessive logging from TaskAttemptListener
without affecting the frequency of progress updates.
There are two flags:
* {{ {{-Dmapreduce.task.progress.min.delta.threshold=0.10: means that the
task progress will be logged every 10% of delta progress. Default is 5%.
* {{ {{-Dmapreduce.task.progress.wait.delta.time.threshold=3: means that
if the listener will log the progress every 3 minutes. This is helpful for long
running tasks that take long time to achieve the delta threshold.
The listener will long whichever of \{{delta.threshold}} and
\{{wait.delta.time}} is reached first.
Enabling \{{LOG.DEBUG}} for TaskAttemptListenerImpl will override
those two flags and log the task progress on every update.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7079) JobHistory#ServiceStop implementation is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025417#comment-17025417
]
Ahmed Hussein commented on MAPREDUCE-7079:
--
Thanks [~epayne]!
> JobHistory#ServiceStop implementation is incorrect
> --
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: 2020-01-10-MRApp-stack-dump.txt,
> 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt,
> MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch,
> MAPREDUCE-7079.003.patch, MAPREDUCE-7079.004.patch, MAPREDUCE-7079.005.patch,
> MAPREDUCE-7079.006.patch, MAPREDUCE-7079.007.patch, MAPREDUCE-7079.008.patch,
> MAPREDUCE-7079.009.patch, MAPREDUCE-7079.010.patch
>
>
> {{JobHistory.serviceStop}} skips waiting for the thread pool to terminate.
> The problem is due to incorrect while condition that will evaluate to false
> on the iteration of the loop.
> {code:java}
> scheduledExecutor.shutdown();
> boolean interrupted = false;
> long currentTime = System.currentTimeMillis();
> while (!scheduledExecutor.isShutdown()
> && System.currentTimeMillis() > currentTime + 1000l &&
> !interrupted) {
> try {
> Thread.sleep(20);
> } catch (InterruptedException e) {
> interrupted = true;
> }
> }
> {code}
> The expression "{{System.currentTimeMillis() > currentTime + 1000L}}" is
> false because currentTime was just initialized with
> {{System.currentTimeMillis()}}. As a result the the thread won't wait until
> the executor is terminated. Instead, it will force a shutdown immediately.
> *TestMRIntermediateDataEncryption is failing in precommit builds*
> TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
> which causes the unit tests in jobclient to not pass cleanly during precommit
> builds. From sample precommit console output, note the lack of a test results
> line when the test is run:
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976
> s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
> [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659
> s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [...]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 h
> [INFO] Finished at: 2018-04-12T04:27:06+00:00
> [INFO] Final Memory: 24M/594M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7259) testSpeculateSuccessfulWithUpdateEvents fails Intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7259:
-
Attachment: MAPREDUCE-7259-branch-2.10.005.patch
> testSpeculateSuccessfulWithUpdateEvents fails Intermittently
> --
>
> Key: MAPREDUCE-7259
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7259
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7259-branch-2.10.005.patch,
> MAPREDUCE-7259.001.patch, MAPREDUCE-7259.002.patch, MAPREDUCE-7259.003.patch,
> MAPREDUCE-7259.004.patch, MAPREDUCE-7259.005.patch
>
>
> {{TestSpeculativeExecutionWithMRApp.testSpeculateSuccessfulWithUpdateEvents}}
> fails Intermittently with the exponential estimator. The problem happens
> because assertion fails waiting for the MRApp to stop.
> There maybe a need to redesign the test case because it does not work very
> well because of the racing and the timing between the speculator and the
> tasks. It works fine for the legacy estimator because the estimate is based
> on start-end rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262-branch-2.10.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: MAPREDUCE-7262-branch-2.10.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
>
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: (was: MAPREDUCE-7261.002.patch)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf,
> MAPREDUCE-7262.001.patch, MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Open (was: Patch Available)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7262.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Status: Patch Available (was: Open)
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch,
> MAPREDUCE-7262.002.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275
[jira] [Updated] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7262:
-
Attachment: MAPREDUCE-7261.002.patch
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.002.patch,
> MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
> --
> Method
> waitForState(TaskAttempt, TaskAttemptState)
> Found usages (72 usages found)
> org.apache.hadoop.mapreduce.v2 (2 usages found)
> TestSpeculativeExecutionWithMRApp (2 usages found)
> testSpeculateSuccessfulWithoutUpdateEvents() (1 usage found)
> 212 app.waitForState(taskAttempt.getValue(),
> TaskAttemptState.SUCCEEDED);
> testSpeculateSuccessfulWithUpdateEvents() (1 usage found)
> 275 app.waitForState(taskAttempt.getValue(),
>
[jira] [Commented] (MAPREDUCE-7262) MRApp helpers block for long intervals (500ms)
[
https://issues.apache.org/jira/browse/MAPREDUCE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024627#comment-17024627
]
Ahmed Hussein commented on MAPREDUCE-7262:
--
{quote}waitForState(TaskAttempt attempt, TaskAttemptState...finalStates) was
not updated. I assume this is because we assume MAPREDUCE-7259 will remove this
function. Can you confirm? If it turns out MAPREDUCE-7259 will continue to use
this, it will need to be updated.{quote}
Yes, MAPREDUCE-7259 will remove the method.
{quote}public void waitForState(Task task, TaskState finalState)
Seems to have a few problems. One is that it continues to use
System.out.println. Can you comment on that?.t{quote}
My bad..I did not remove that print statement. I will update the patch.
> MRApp helpers block for long intervals (500ms)
> --
>
> Key: MAPREDUCE-7262
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7262
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7262-elapsedTimes.pdf, MAPREDUCE-7262.001.patch
>
>
> MRApp has a set of methods used as helpers in test cases such as:
> {{waitForInternalState(TA)}}, {{waitForState(TA)}}, {{waitForState(Job)}}..etc
> When the condition fails, the thread sleeps for a minimum of 500ms before
> rechecking the new state of the Job/TA.
> Example:
> {code:java}
> public void waitForState(Task task, TaskState finalState) throws Exception {
> int timeoutSecs = 0;
> TaskReport report = task.getReport();
> while (!finalState.equals(report.getTaskState()) &&
> timeoutSecs++ < 20) {
> System.out.println("Task State for " + task.getID() + " is : "
> + report.getTaskState() + " Waiting for state : " + finalState
> + " progress : " + report.getProgress());
> report = task.getReport();
> Thread.sleep(500);
> }
> System.out.println("Task State is : " + report.getTaskState());
> Assert.assertEquals("Task state is not correct (timedout)", finalState,
> report.getTaskState());
> }
> {code}
> I suggest to reduce the interval 500 to 50, while incrementing the number of
> retries to 200. this will potentially make the test cases run faster. Also,
> the {{System.out}} calls need to be removed because they are not adding
> information dumping the current state on every iteration.
> A tentative list of Junits affected by the change:
> {code:bash}
> Method
> waitForInternalState(JobImpl, JobStateInternal)
> Found usages (12 usages found)
> org.apache.hadoop.mapreduce.v2.app (10 usages found)
> TestJobEndNotifier (3 usages found)
> testNotificationOnLastRetry(boolean) (1 usage found)
> 214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testAbsentNotificationOnNotLastRetryUnregistrationFailure() (1
> usage found)
> 256 app.waitForInternalState(job, JobStateInternal.REBOOT);
> testNotificationOnLastRetryUnregistrationFailure() (1 usage
> found)
> 289 app.waitForInternalState(job, JobStateInternal.REBOOT);
> TestKill (5 usages found)
> testKillJob() (1 usage found)
> 70 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTask() (1 usage found)
> 108 app.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testKillTaskWait() (1 usage found)
> 219 app.waitForInternalState((JobImpl) job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobAfterTA_DONE() (1 usage found)
> 266 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> testKillTaskWaitKillJobBeforeTA_DONE() (1 usage found)
> 316 app.waitForInternalState((JobImpl)job,
> JobStateInternal.KILLED);
> TestMRApp (2 usages found)
> testJobSuccess() (1 usage found)
> 494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
> testJobRebootOnLastRetryOnUnregistrationFailure() (1 usage found)
> 542 app.waitForInternalState((JobImpl) job,
> JobStateInternal.REBOOT);
> org.apache.hadoop.mapreduce.v2.app.rm (2 usages found)
> TestRMContainerAllocator (2 usages found)
> testReportedAppProgress() (1 usage found)
> 1050 mrApp.waitForInternalState((JobImpl) job,
> JobStateInternal.RUNNING);
> testReportedAppProgressWithOnlyMaps() (1 usage found)
> 1202 mrApp.waitForInternalState((JobImpl)job,
> JobStateInternal.RUNNING);
>
[jira] [Updated] (MAPREDUCE-7259) testSpeculateSuccessfulWithUpdateEvents fails Intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7259:
-
Attachment: MAPREDUCE-7259.005.patch
> testSpeculateSuccessfulWithUpdateEvents fails Intermittently
> --
>
> Key: MAPREDUCE-7259
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7259
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7259.001.patch, MAPREDUCE-7259.002.patch,
> MAPREDUCE-7259.003.patch, MAPREDUCE-7259.004.patch, MAPREDUCE-7259.005.patch
>
>
> {{TestSpeculativeExecutionWithMRApp.testSpeculateSuccessfulWithUpdateEvents}}
> fails Intermittently with the exponential estimator. The problem happens
> because assertion fails waiting for the MRApp to stop.
> There maybe a need to redesign the test case because it does not work very
> well because of the racing and the timing between the speculator and the
> tasks. It works fine for the legacy estimator because the estimate is based
> on start-end rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
