[jira] [Commented] (TEZ-2196) Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput
[
https://issues.apache.org/jira/browse/TEZ-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381458#comment-14381458
]
Hadoop QA commented on TEZ-2196:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12707434/TEZ-2196.4.patch
against master revision 2fe2d63.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/349//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/349//console
This message is automatically generated.
Consider reusing UnorderedPartitionedKVWriter with single output in
UnorderedKVOutput
-
Key: TEZ-2196
URL: https://issues.apache.org/jira/browse/TEZ-2196
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: TEZ-2196.1.patch, TEZ-2196.2.patch, TEZ-2196.3.patch,
TEZ-2196.4.patch
Can possibly get rid of FileBasedKVWriter and reuse
UnorderedPartitionedKVWriter with single partition in UnorderedKVOutput.
This can also benefit from pipelined shuffle changes done in
UnorderedPartitionedKVWriter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2196) Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput
[ https://issues.apache.org/jira/browse/TEZ-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380752#comment-14380752 ] Siddharth Seth commented on TEZ-2196: - Looks good. Minor: getInitialMemoryRequirement - this should return 0 when pipelining is disabled, and numPartitions=1 since buffers aren't being used. Otherwise we unnecessarily penalize other Inputs / Outputs which may exist on the vertex. Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput - Key: TEZ-2196 URL: https://issues.apache.org/jira/browse/TEZ-2196 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2196.1.patch, TEZ-2196.2.patch, TEZ-2196.3.patch Can possibly get rid of FileBasedKVWriter and reuse UnorderedPartitionedKVWriter with single partition in UnorderedKVOutput. This can also benefit from pipelined shuffle changes done in UnorderedPartitionedKVWriter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2196) Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput
[
https://issues.apache.org/jira/browse/TEZ-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375762#comment-14375762
]
Hadoop QA commented on TEZ-2196:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706508/TEZ-2196.3.patch
against master revision aa784be.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/330//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/330//console
This message is automatically generated.
Consider reusing UnorderedPartitionedKVWriter with single output in
UnorderedKVOutput
-
Key: TEZ-2196
URL: https://issues.apache.org/jira/browse/TEZ-2196
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: TEZ-2196.1.patch, TEZ-2196.2.patch, TEZ-2196.3.patch
Can possibly get rid of FileBasedKVWriter and reuse
UnorderedPartitionedKVWriter with single partition in UnorderedKVOutput.
This can also benefit from pipelined shuffle changes done in
UnorderedPartitionedKVWriter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2196) Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput
[
https://issues.apache.org/jira/browse/TEZ-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368852#comment-14368852
]
Hadoop QA commented on TEZ-2196:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705559/TEZ-2196.2.patch
against master revision 9b845f2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/312//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/312//console
This message is automatically generated.
Consider reusing UnorderedPartitionedKVWriter with single output in
UnorderedKVOutput
-
Key: TEZ-2196
URL: https://issues.apache.org/jira/browse/TEZ-2196
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: TEZ-2196.1.patch, TEZ-2196.2.patch
Can possibly get rid of FileBasedKVWriter and reuse
UnorderedPartitionedKVWriter with single partition in UnorderedKVOutput.
This can also benefit from pipelined shuffle changes done in
UnorderedPartitionedKVWriter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2196) Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput
[ https://issues.apache.org/jira/browse/TEZ-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368455#comment-14368455 ] Rajesh Balamohan commented on TEZ-2196: --- [~sseth] - Removed TEZ_RUNTIME_TRANSFER_DATA_VIA_EVENTS_ENABLED in the patch. Can I go ahead and remove the corresponding processing in consumer side as well? (unordered shuffle manager has addCompletedInputWithData). Consider reusing UnorderedPartitionedKVWriter with single output in UnorderedKVOutput - Key: TEZ-2196 URL: https://issues.apache.org/jira/browse/TEZ-2196 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2196.1.patch Can possibly get rid of FileBasedKVWriter and reuse UnorderedPartitionedKVWriter with single partition in UnorderedKVOutput. This can also benefit from pipelined shuffle changes done in UnorderedPartitionedKVWriter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
