[jira] [Commented] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375232#comment-14375232 ] Rajesh Balamohan commented on TEZ-2186: --- It would be good to have it on 0.5 and 0.6. I will add it to both 0.5 and 0.6 soon. OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2219) Should verify the input_name/output_name to be unique
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14374953#comment-14374953 ] Hitesh Shah commented on TEZ-2219: -- Please add more information. Aren't these already enforced to be unique per vertex? These are not meant to be unique across the whole dag. Should verify the input_name/output_name to be unique - Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name / output_name to be unique
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Summary: Should verify the input_name / output_name to be unique (was: Should verify the output name to be unique) Should verify the input_name / output_name to be unique --- Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-2176 PreCommit Build #325
Jira: https://issues.apache.org/jira/browse/TEZ-2176
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/325/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 3598 lines...]
[INFO] Final Memory: 71M/994M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706408/TEZ-2176.2.patch
against master revision a44f5c5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 44 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/325//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/325//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1677780b197cce170f83ff32c28bb8b465e0e9bb logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #324
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2531907 bytes
Compression is 7.2%
Took 1.7 sec
Description set: TEZ-2176
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2176) Move all logging to slf4j
[
https://issues.apache.org/jira/browse/TEZ-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375150#comment-14375150
]
Hadoop QA commented on TEZ-2176:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706408/TEZ-2176.2.patch
against master revision a44f5c5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 44 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/325//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/325//console
This message is automatically generated.
Move all logging to slf4j
-
Key: TEZ-2176
URL: https://issues.apache.org/jira/browse/TEZ-2176
Project: Apache Tez
Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Vasanth kumar RJ
Attachments: TEZ-2176.1.patch, TEZ-2176.2.patch, TEZ-2176.patch
SLF4J supports a more comprehensive set of APIs - MDC, Formatted strings.
Also drop commons-logging from the dependency set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2176) Move all logging to slf4j
[ https://issues.apache.org/jira/browse/TEZ-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasanth kumar RJ updated TEZ-2176: -- Attachment: TEZ-2176.2.patch [~sseth], Attached new patch and changed all fatal to error. But there are some files directly depend on log4j classes, so only those modules has slf4j-log4j12 dependency. Kindly review. Move all logging to slf4j - Key: TEZ-2176 URL: https://issues.apache.org/jira/browse/TEZ-2176 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Vasanth kumar RJ Attachments: TEZ-2176.1.patch, TEZ-2176.2.patch, TEZ-2176.patch SLF4J supports a more comprehensive set of APIs - MDC, Formatted strings. Also drop commons-logging from the dependency set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Summary: Should verify the input_name/output_name to be unique (was: Should verify the input_name / output_name to be unique) Should verify the input_name/output_name to be unique - Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved TEZ-2186. --- Resolution: Fixed Fix Version/s: 0.7.0 Target Version/s: 0.7.0 Hadoop Flags: Reviewed committed to master. Thanks [~sseth] commit cbd7323cd9027e2f1459ba477c7f01c99306ba31 OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375226#comment-14375226 ] Hitesh Shah commented on TEZ-2186: -- Is there a reason why this should not be backported to 0.5 and 0.6? OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2186: - Target Version/s: 0.7.0, 0.5.4, 0.6.1 (was: 0.7.0) OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reopened TEZ-2186: -- OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Description: RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2219) Should verify the output name to be unique
Jeff Zhang created TEZ-2219: --- Summary: Should verify the output name to be unique Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-2219 PreCommit Build #327
Jira: https://issues.apache.org/jira/browse/TEZ-2219
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/327/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2750 lines...]
[INFO] Final Memory: 70M/967M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706422/TEZ-2219-1.txt
against master revision cbd7323.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/327//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/327//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
0b8886377e1da8becdf6398ff5ca701544362789 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #326
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2596320 bytes
Compression is 4.8%
Took 0.82 sec
Description set: TEZ-2219
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[
https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375272#comment-14375272
]
Hadoop QA commented on TEZ-2219:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706422/TEZ-2219-1.txt
against master revision cbd7323.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/327//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/327//console
This message is automatically generated.
Should verify the input_name/output_name to be unique per vertex
Key: TEZ-2219
URL: https://issues.apache.org/jira/browse/TEZ-2219
Project: Apache Tez
Issue Type: Improvement
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2219-1.txt
RuntimeTask try to get the Input/Output using the input_name/output_name, so
input_name/output_name should be unique per vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375284#comment-14375284 ] Hitesh Shah commented on TEZ-2219: -- Mostly looks good. Do we need checks for null and empty names? Also, it is usually good to use Assert.fail(); with a failure message for more clarity if the test fails. Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2220) Compilation failure in branch 0.5
[
https://issues.apache.org/jira/browse/TEZ-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah reassigned TEZ-2220:
Assignee: Hitesh Shah
Compilation failure in branch 0.5
-
Key: TEZ-2220
URL: https://issues.apache.org/jira/browse/TEZ-2220
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Hitesh Shah
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project tez-tests: Compilation failure: Compilation
failure:
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,52]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[543,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,23]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,39]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[577,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,10]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] - [Help 1]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Attachment: TEZ-2219-2.patch Upload new patch to address the review comments. [~hitesh] Please help review it. Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt, TEZ-2219-2.patch RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Issue Type: Improvement (was: Bug) Should verify the input_name/output_name to be unique - Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2209) Fix pipelined shuffle to fetch data from any one attempt
[ https://issues.apache.org/jira/browse/TEZ-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-2209: -- Attachment: TEZ-2209.4.patch Thanks [~sseth]. Addressed the review comments in the latest patch. Will commit it once the precommit build succeeds. Fix pipelined shuffle to fetch data from any one attempt Key: TEZ-2209 URL: https://issues.apache.org/jira/browse/TEZ-2209 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2209.1.patch, TEZ-2209.2.patch, TEZ-2209.3.patch, TEZ-2209.4.patch - Currently, pipelined shuffle will fail-fast the moment it receives data from an attempt other than 0. This was done as an add-on check to prevent data being copied from speculated attempts. - However, in some scenarios (like LLAP), it could be possible that that task attempt gets killed even before generating any data. In such cases, attempt #1 or later attempts, would generate the actual data. - This jira is created to allow pipelined shuffle to download data from any one attempt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-2209 PreCommit Build #326
Jira: https://issues.apache.org/jira/browse/TEZ-2209
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/326/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2751 lines...]
[INFO] Final Memory: 70M/948M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706417/TEZ-2209.4.patch
against master revision a44f5c5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/326//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/326//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
93f5df9daf4884565b4552c9cbffd6212661c2c3 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #325
Archived 44 artifacts
Archive block size is 32768
Received 22 blocks and 2006119 bytes
Compression is 26.4%
Took 0.71 sec
Description set: TEZ-2209
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2209) Fix pipelined shuffle to fetch data from any one attempt
[
https://issues.apache.org/jira/browse/TEZ-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375217#comment-14375217
]
Hadoop QA commented on TEZ-2209:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706417/TEZ-2209.4.patch
against master revision a44f5c5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/326//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/326//console
This message is automatically generated.
Fix pipelined shuffle to fetch data from any one attempt
Key: TEZ-2209
URL: https://issues.apache.org/jira/browse/TEZ-2209
Project: Apache Tez
Issue Type: Improvement
Reporter: Rajesh Balamohan
Assignee: Rajesh Balamohan
Attachments: TEZ-2209.1.patch, TEZ-2209.2.patch, TEZ-2209.3.patch,
TEZ-2209.4.patch
- Currently, pipelined shuffle will fail-fast the moment it receives data
from an attempt other than 0. This was done as an add-on check to prevent
data being copied from speculated attempts.
- However, in some scenarios (like LLAP), it could be possible that that task
attempt gets killed even before generating any data. In such cases, attempt
#1 or later attempts, would generate the actual data.
- This jira is created to allow pipelined shuffle to download data from any
one attempt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Summary: Should verify the input_name/output_name to be unique per vertex (was: Should verify the input_name/output_name to be unique) Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2220) TestTezJobs compile failure in branch 0.5
[
https://issues.apache.org/jira/browse/TEZ-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-2220:
-
Summary: TestTezJobs compile failure in branch 0.5 (was: Compilation
failure in branch 0.5)
TestTezJobs compile failure in branch 0.5
-
Key: TEZ-2220
URL: https://issues.apache.org/jira/browse/TEZ-2220
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Hitesh Shah
Attachments: TEZ-2220.branch-0.5.1.patch
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project tez-tests: Compilation failure: Compilation
failure:
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,52]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[543,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,23]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,39]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[577,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,10]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] - [Help 1]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TEZ-2220) TestTezJobs compile failure in branch 0.5
[
https://issues.apache.org/jira/browse/TEZ-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah resolved TEZ-2220.
--
Resolution: Fixed
Fix Version/s: 0.5.4
Thanks for the review [~rajesh.balamohan]. Committed to branch 0.5
TestTezJobs compile failure in branch 0.5
-
Key: TEZ-2220
URL: https://issues.apache.org/jira/browse/TEZ-2220
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Hitesh Shah
Fix For: 0.5.4
Attachments: TEZ-2220.branch-0.5.1.patch
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project tez-tests: Compilation failure: Compilation
failure:
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,52]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[543,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[542,23]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,39]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[577,11]
cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class org.apache.tez.test.TestTezJobs
[ERROR]
./tez/tez-tests/src/test/java/org/apache/tez/test/TestTezJobs.java:[576,10]
no suitable method found for
run(org.apache.tez.dag.api.TezConfiguration,java.lang.String[],nulltype)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[]) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.tez.examples.SimpleSessionExample.run(java.lang.String[],java.lang.String[],org.apache.hadoop.conf.Configuration,int)
is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] - [Help 1]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[
https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375354#comment-14375354
]
Hadoop QA commented on TEZ-2219:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706438/TEZ-2219-2.patch
against master revision cbd7323.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/328//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/328//console
This message is automatically generated.
Should verify the input_name/output_name to be unique per vertex
Key: TEZ-2219
URL: https://issues.apache.org/jira/browse/TEZ-2219
Project: Apache Tez
Issue Type: Improvement
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2219-1.txt, TEZ-2219-2.patch
RuntimeTask try to get the Input/Output using the input_name/output_name, so
input_name/output_name should be unique per vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2219 PreCommit Build #328
Jira: https://issues.apache.org/jira/browse/TEZ-2219
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/328/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2750 lines...]
[INFO] Final Memory: 71M/981M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12706438/TEZ-2219-2.patch
against master revision cbd7323.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/328//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/328//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
b3130cf601cf755b4f6f0bfe83a830f30644a7c7 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #327
Archived 44 artifacts
Archive block size is 32768
Received 24 blocks and 1936047 bytes
Compression is 28.9%
Took 2.3 sec
Description set: TEZ-2219
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375379#comment-14375379 ] Hitesh Shah commented on TEZ-2219: -- Mostly looks good. Minor nit: Should fail due to inputName is null and Should fail due to ouputName is null is used for all test cases. Maybe change to Should fail due to invalid ... so that the sentence makes sense for null, and cases. Feel free to commit after addressing the above comment. +1 pending above comment being addressed. Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt, TEZ-2219-2.patch RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-2221) VertexGroup name should be unqiue
Jeff Zhang created TEZ-2221: --- Summary: VertexGroup name should be unqiue Key: TEZ-2221 URL: https://issues.apache.org/jira/browse/TEZ-2221 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang VertexGroupCommitStartedEvent VertexGroupCommitFinishedEvent use vertex group name to identify the vertex group commit, the same name of vertex group will conflict. While in the current equals hashCode of VertexGroup, vertex group name and members name are used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2186) OOM with a simple scatter gather job with re-use
[ https://issues.apache.org/jira/browse/TEZ-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved TEZ-2186. --- Resolution: Fixed Thanks [~hitesh]. Committed to branch-0.5 and branch-0.6 as well. OOM with a simple scatter gather job with re-use Key: TEZ-2186 URL: https://issues.apache.org/jira/browse/TEZ-2186 Project: Apache Tez Issue Type: Bug Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.7.0 Attachments: TEZ-2186.1.patch, TEZ-2186.2.patch, noopexample.txt With a no-op scatter gather job, 20K x 2K, on a 20 node cluster with 20 2GB containers per node - reducers end up failing with OOM errors. Haven't been able to generate a heap dump yet. Will add details as they're found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1421) MRCombiner throws NPE in MapredWordCount on master branch
[
https://issues.apache.org/jira/browse/TEZ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375411#comment-14375411
]
Tsuyoshi Ozawa commented on TEZ-1421:
-
[~jeagles], thank you for your help! I'm dealing with this problem, but it
takes more time to solve it essentially. I can attach a workaround for this
problem.
MRCombiner throws NPE in MapredWordCount on master branch
-
Key: TEZ-1421
URL: https://issues.apache.org/jira/browse/TEZ-1421
Project: Apache Tez
Issue Type: Bug
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Priority: Blocker
I tested MapredWordCount against 70GB generated by RandowTextWriter. When a
Combiner runs, it throws NPE. It looks setCombinerClass doesn't work
correctly.
{quote}
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:122)
at org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:112)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.runCombineProcessor(MergeManager.java:472)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager$InMemoryMerger.merge(MergeManager.java:605)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeThread.run(MergeThread.java:89)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2219: Attachment: TEZ-2219-3.patch Upload new patch. Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt, TEZ-2219-2.patch, TEZ-2219-3.patch RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2219) Should verify the input_name/output_name to be unique per vertex
[ https://issues.apache.org/jira/browse/TEZ-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375436#comment-14375436 ] Jeff Zhang edited comment on TEZ-2219 at 3/23/15 5:58 AM: -- Upload new patch. Commit after the pre-build is done. was (Author: zjffdu): Upload new patch. Should verify the input_name/output_name to be unique per vertex Key: TEZ-2219 URL: https://issues.apache.org/jira/browse/TEZ-2219 Project: Apache Tez Issue Type: Improvement Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-2219-1.txt, TEZ-2219-2.patch, TEZ-2219-3.patch RuntimeTask try to get the Input/Output using the input_name/output_name, so input_name/output_name should be unique per vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
