[
https://issues.apache.org/jira/browse/MAPREDUCE-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419862#comment-16419862
]
Hadoop QA commented on MAPREDUCE-7057:
--------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 17m
55s{color} | {color:red} Docker failed to build yetus/hadoop:8620d2b. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-7057 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916903/MAPREDUCE-7057.2.2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7375/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> MergeThread Review
> ------------------
>
> Key: MAPREDUCE-7057
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7057
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 3.0.0
> Reporter: BELUGA BEHR
> Priority: Minor
> Attachments: MAPREDUCE-7057.1.patch, MAPREDUCE-7057.2.2.patch,
> MAPREDUCE-7057.2.patch
>
>
> Source:
>
> [MergeThread.java|https://github.com/apache/hadoop/blob/178751ed8c9d47038acf8616c226f1f52e884feb/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeThread.java]
> Update this class to use Java 1.8 concurrent package. There also some
> corner-cases not being addressed with the current implementation:
> {code:java|title=MergeThread.java}
> // There is a scenario here where N threads have submitted inputs and are all
> waiting for the 'pendingToBeMerged' object. At this point, imagine the
> 'close' method is called. The close method will run, see nothing in the
> queue, interrupt the processing thread, and cause it to exit. Afterwards,
> the 'startMerge' threads will all be triggered and add the inputs to a queue
> for which there is no consumer. At this point, the T items have been removed
> from the inputs with no way to recover them. In practice, this may not ever
> be the case, but it can be tightened up.
> public void startMerge(Set<T> inputs) {
> if (!closed) {
> numPending.incrementAndGet();
> List<T> toMergeInputs = new ArrayList<T>();
> Iterator<T> iter=inputs.iterator();
> for (int ctr = 0; iter.hasNext() && ctr < mergeFactor; ++ctr) {
> toMergeInputs.add(iter.next());
> iter.remove();
> }
> LOG.info(getName() + ": Starting merge with " + toMergeInputs.size() +
> " segments, while ignoring " + inputs.size() + " segments");
> synchronized(pendingToBeMerged) {
> pendingToBeMerged.addLast(toMergeInputs);
> pendingToBeMerged.notifyAll();
> }
> }
> }
> public synchronized void close() throws InterruptedException {
> closed = true;
> waitForMerge();
> interrupt();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
