[jira] [Commented] (TEZ-3698) UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
[
https://issues.apache.org/jira/browse/TEZ-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047281#comment-16047281
]
TezQA commented on TEZ-3698:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872781/TEZ-3698.6.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2523//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2523//console
This message is automatically generated.
> UnorderedKV writer should be able to honor
> tez.runtime.enable.final-merge.in.output without pipelinedshuffle
>
>
> Key: TEZ-3698
> URL: https://issues.apache.org/jira/browse/TEZ-3698
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: TEZ-3698.1.patch, TEZ-3698.4.patch, TEZ-3698.5.patch,
> TEZ-3698.6.patch
>
>
> Final merge can be disabled with "tez.runtime.enable.final-merge.in.output"
> setting. Currently this works with UnorderedKV writer only with pipelined
> shuffle. It should be able to honor this parameter, without pipelined shuffle
> as well to avoid final merge.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Success: TEZ-3698 PreCommit Build #2523
Jira: https://issues.apache.org/jira/browse/TEZ-3698
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2523/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 343.20 KB...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 51:23 min
[INFO] Finished at: 2017-06-13T00:46:12Z
[INFO] Final Memory: 99M/1394M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872781/TEZ-3698.6.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2523//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2523//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
e35c64ce7433525b17eb3788c6f09f7f098a2ee0 logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.50 MB of artifacts by 28.5% relative to #2521
[description-setter] Description set: TEZ-3698
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3760) Tez AUX Services: Shading needs to filter SIG files
[
https://issues.apache.org/jira/browse/TEZ-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-3760:
-
Description:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project
tez-aux-services: Error creating shaded jar: Invalid signature file digest for
Manifest main attributes -> [Help 1]
{code}
{code}
$ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
{code}
Needs the equivalent of
https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
was:
{code}
$ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
{code}
Needs the equivalent of
https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
> Tez AUX Services: Shading needs to filter SIG files
> ---
>
> Key: TEZ-3760
> URL: https://issues.apache.org/jira/browse/TEZ-3760
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Priority: Blocker
> Fix For: 0.9.0
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project
> tez-aux-services: Error creating shaded jar: Invalid signature file digest
> for Manifest main attributes -> [Help 1]
> {code}
> {code}
> $ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
> 34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
> 6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
> {code}
> Needs the equivalent of
> https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3760) Tez AUX Services: Shading needs to filter SIG files
[
https://issues.apache.org/jira/browse/TEZ-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047255#comment-16047255
]
Gopal V commented on TEZ-3760:
--
[~kshukla]: can you take a look at this issue with the Tez AUX build?
> Tez AUX Services: Shading needs to filter SIG files
> ---
>
> Key: TEZ-3760
> URL: https://issues.apache.org/jira/browse/TEZ-3760
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Priority: Blocker
> Fix For: 0.9.0
>
>
> {code}
> $ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
> 34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
> 6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
> {code}
> Needs the equivalent of
> https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3760) Tez AUX Services: Shading needs to filter SIG files
[
https://issues.apache.org/jira/browse/TEZ-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-3760:
-
Fix Version/s: 0.9.0
> Tez AUX Services: Shading needs to filter SIG files
> ---
>
> Key: TEZ-3760
> URL: https://issues.apache.org/jira/browse/TEZ-3760
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Priority: Blocker
> Fix For: 0.9.0
>
>
> {code}
> $ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
> 34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
> 6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
> {code}
> Needs the equivalent of
> https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TEZ-3760) Tez AUX Services: Shading needs to filter SIG files
Gopal V created TEZ-3760:
Summary: Tez AUX Services: Shading needs to filter SIG files
Key: TEZ-3760
URL: https://issues.apache.org/jira/browse/TEZ-3760
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Gopal V
Priority: Blocker
{code}
$ jar tvf target/tez-aux-services-0.9.0-SNAPSHOT.jar | grep SIG
34677 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.SF
6875 Mon Jun 12 16:45:18 EDT 2017 META-INF/MSFTSIG.RSA
{code}
Needs the equivalent of
https://github.com/apache/hive/blob/master/jdbc/pom.xml#L183
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3698) UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
[ https://issues.apache.org/jira/browse/TEZ-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3698: -- Attachment: TEZ-3698.6.patch Thanks [~sseth] for the review. Fixed minor spelling mistake in mayBeSendEventsForSpill method in .6 patch. Will commit shortly. > UnorderedKV writer should be able to honor > tez.runtime.enable.final-merge.in.output without pipelinedshuffle > > > Key: TEZ-3698 > URL: https://issues.apache.org/jira/browse/TEZ-3698 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-3698.1.patch, TEZ-3698.4.patch, TEZ-3698.5.patch, > TEZ-3698.6.patch > > > Final merge can be disabled with "tez.runtime.enable.final-merge.in.output" > setting. Currently this works with UnorderedKV writer only with pipelined > shuffle. It should be able to honor this parameter, without pipelined shuffle > as well to avoid final merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (TEZ-3757) Integer overflow in PipelinedSorter
[
https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047190#comment-16047190
]
Rajesh Balamohan edited comment on TEZ-3757 at 6/12/17 11:07 PM:
-
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end
up using {{16 *1024 * 1024}} as the metasize instead of reduced metasize for
larger items. This is more of a memory wastage for metadata.
Changing the datatype in sortSpan constructor for dataSize would fix the issue.
was (Author: rajesh.balamohan):
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end
up using {{16 *1024 * 1024}} as the metasize instead of reduced metasize for
larger items. Changing the datatype in sortSpan constructor for dataSize would
fix the issue.
> Integer overflow in PipelinedSorter
> ---
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4, 0.8.5
>Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
> the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
> (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
> (usually?) ends up causing the capacity check to not work correctly, which
> causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (TEZ-3757) Integer overflow in PipelinedSorter
[
https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047190#comment-16047190
]
Rajesh Balamohan edited comment on TEZ-3757 at 6/12/17 11:06 PM:
-
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end
up using {{16 *1024 * 1024}} as the metasize instead of reduced metasize for
larger items. Changing the datatype in sortSpan constructor for dataSize would
fix the issue.
was (Author: rajesh.balamohan):
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end
up using 16 *1024 *1024 as the metasize instead of reduced metasize for larger
items. Changing the datatype in sortSpan constructor for dataSize would fix the
issue.
> Integer overflow in PipelinedSorter
> ---
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4, 0.8.5
>Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
> the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
> (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
> (usually?) ends up causing the capacity check to not work correctly, which
> causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3757) Integer overflow in PipelinedSorter
[
https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047190#comment-16047190
]
Rajesh Balamohan commented on TEZ-3757:
---
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end
up using 16 *1024 *1024 as the metasize instead of reduced metasize for larger
items. Changing the datatype in sortSpan constructor for dataSize would fix the
issue.
> Integer overflow in PipelinedSorter
> ---
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4, 0.8.5
>Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
> the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
> (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
> (usually?) ends up causing the capacity check to not work correctly, which
> causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3698) UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
[ https://issues.apache.org/jira/browse/TEZ-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047017#comment-16047017 ] Siddharth Seth commented on TEZ-3698: - +1. Thanks [~rajesh.balamohan] > UnorderedKV writer should be able to honor > tez.runtime.enable.final-merge.in.output without pipelinedshuffle > > > Key: TEZ-3698 > URL: https://issues.apache.org/jira/browse/TEZ-3698 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-3698.1.patch, TEZ-3698.4.patch, TEZ-3698.5.patch > > > Final merge can be disabled with "tez.runtime.enable.final-merge.in.output" > setting. Currently this works with UnorderedKV writer only with pipelined > shuffle. It should be able to honor this parameter, without pipelined shuffle > as well to avoid final merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3718) Better handling of 'bad' nodes
[ https://issues.apache.org/jira/browse/TEZ-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047013#comment-16047013 ] Siddharth Seth commented on TEZ-3718: - bq. it. For not accessing nodeTracker from TaskAttempt, are you suggesting notifying TaskAttempt about node health? That would be more overhead than pulling info from node tracker since every TaskAttempt need note down the node health status. Isn't their enough information available in the message itself. Node->Container->TaskAttempt - Whatever state change triggered the node event to go out, that information can be retained in the event sent out by the container. > Better handling of 'bad' nodes > -- > > Key: TEZ-3718 > URL: https://issues.apache.org/jira/browse/TEZ-3718 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Zhiyuan Yang > Attachments: TEZ-3718.1.patch > > > At the moment, the default behaviour in case of a node being marked bad is to > do nothing other than not schedule new tasks on this node. > The alternate, via config, is to retroactively kill every task which ran on > the node, which causes far too many unnecessary re-runs. > Proposing the following changes. > 1. KILL fragments which are currently in the RUNNING state (instead of > relying on a timeout which leads to the attempt being marked as FAILED after > the timeout interval. > 2. Keep track of these failed nodes, and use this as input to the failure > heuristics. Normally source tasks require multiple consumers to report > failure for them to be marked as bad. If a single consumer reports failure > against a source which ran on a bad node, consider it bad and re-schedule > immediately. (Otherwise failures can take a while to propagate, and jobs get > a lot slower). > [~jlowe] - think you've looked at this in the past. Any thoughts/suggestions. > What I'm seeing is retroactive failures taking a long time to apply, and > restart sources which ran on a bad node. Also running tasks being counted as > FAILURES instead of KILLS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3758) Vertex can hang in RUNNING state when two task attempts finish very closely and have retroactive failures
[ https://issues.apache.org/jira/browse/TEZ-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047011#comment-16047011 ] Siddharth Seth commented on TEZ-3758: - cc [~aplusplus], [~harishjp] > Vertex can hang in RUNNING state when two task attempts finish very closely > and have retroactive failures > - > > Key: TEZ-3758 > URL: https://issues.apache.org/jira/browse/TEZ-3758 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.1, 0.9.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: TEZ-3758.001.patch > > > A vertex's count of what tasks are done can go off in a case where two task > attempts finish very closely, say within a millisecond of each other. We had > a case where this task, which was marked successful, never scheduled another > attempt upon getting a retroactive failure since it thought it had one > uncompleted task attempt already. This is because the attempt that finished 1 > ms later transitioned to SUCCEEDED but we don't take any action on the > taskAttempStatus data structure and it stays false. This JIRA will attempt to > solve that race. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3757) Integer overflow in PipelinedSorter
[
https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047000#comment-16047000
]
Siddharth Seth commented on TEZ-3757:
-
cc [~gopalv], [~rajesh.balamohan]
> Integer overflow in PipelinedSorter
> ---
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4, 0.8.5
>Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to
> the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(),
> (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which
> (usually?) ends up causing the capacity check to not work correctly, which
> causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (TEZ-3617) TestHistoryParser#testParserWithSuccessfulJob fails intermittently
[ https://issues.apache.org/jira/browse/TEZ-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046998#comment-16046998 ] Siddharth Seth commented on TEZ-3617: - [~jeagles], is there a way to do this without the upgrade to 2.7.2?, especially since this is a test only change. Tez itself should work fine on 2.7.0. (That said, 2.7.2 has been around for almost 1.5 years - Jan 2016) > TestHistoryParser#testParserWithSuccessfulJob fails intermittently > -- > > Key: TEZ-3617 > URL: https://issues.apache.org/jira/browse/TEZ-3617 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Ubuntu 14.04 >Reporter: Sonia Garudi >Assignee: Jonathan Eagles > Labels: ppc64le, x86 > Attachments: org.apache.tez.history.TestHistoryParser-output.txt, > TEZ-3617.1.patch > > > The TestHistoryParser#testParserWithSuccessfulJob test fails intermittently > in tez-history-parser project. > Error message : > testParserWithSuccessfulJob(org.apache.tez.history.TestHistoryParser) Time > elapsed: 29.952 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.tez.history.TestHistoryParser.verifyJobSpecificInfo(TestHistoryParser.java:266) > at > org.apache.tez.history.TestHistoryParser.testParserWithSuccessfulJob(TestHistoryParser.java:212) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3693) ControlledClock is not used
[
https://issues.apache.org/jira/browse/TEZ-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046962#comment-16046962
]
TezQA commented on TEZ-3693:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872755/TEZ-3693.001.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2522//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2522//console
This message is automatically generated.
> ControlledClock is not used
> ---
>
> Key: TEZ-3693
> URL: https://issues.apache.org/jira/browse/TEZ-3693
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Muhammad Samir Khan
>Priority: Trivial
> Attachments: TEZ-3693.001.patch
>
>
> The org.apache.tez.dag.app.ControlledClock class is not referenced in the
> source. Oddly this is not a test class, like MockClock, as I would have
> expected. If this is not part of the Tez API then it can be removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Failed: TEZ-3693 PreCommit Build #2522
Jira: https://issues.apache.org/jira/browse/TEZ-3693
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2522/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 343.52 KB...]
[INFO]
[INFO] Total time: 51:49 min
[INFO] Finished at: 2017-06-12T19:17:11Z
[INFO] Final Memory: 98M/1372M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872755/TEZ-3693.001.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2522//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2522//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
10d1d940fc7b0a6ea2895d8696fbb34b51197a87 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
Success: TEZ-3283 PreCommit Build #2521
Jira: https://issues.apache.org/jira/browse/TEZ-3283
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2521/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 343.17 KB...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 54:47 min
[INFO] Finished at: 2017-06-12T18:41:54Z
[INFO] Final Memory: 93M/1376M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872748/tez-3283.001.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2521//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2521//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
273475d2ba21b107720853cfdd950d9da052bdcf logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.51 MB of artifacts by 28.5% relative to #2520
[description-setter] Description set: TEZ-3283
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3283) Add a max cap to the heap computation done by Tez
[
https://issues.apache.org/jira/browse/TEZ-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046923#comment-16046923
]
TezQA commented on TEZ-3283:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12872748/tez-3283.001.patch
against master revision 29b45bc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2521//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2521//console
This message is automatically generated.
> Add a max cap to the heap computation done by Tez
> -
>
> Key: TEZ-3283
> URL: https://issues.apache.org/jira/browse/TEZ-3283
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Muhammad Samir Khan
> Labels: newbie
> Attachments: tez-3283.001.patch
>
>
> Change the calculation to not use the fraction blindly. For large containers
> - .e.g 10GB - this ends up leaving 2G of the container unused / outside of
> the heap.
> Instead we can add another parameter which indicates the max amount of memory
> to leave outside the heap - defaulting to 1G
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3693) ControlledClock is not used
[ https://issues.apache.org/jira/browse/TEZ-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3693: - Attachment: TEZ-3693.001.patch Removed ControlledClock > ControlledClock is not used > --- > > Key: TEZ-3693 > URL: https://issues.apache.org/jira/browse/TEZ-3693 > Project: Apache Tez > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Muhammad Samir Khan >Priority: Trivial > Attachments: TEZ-3693.001.patch > > > The org.apache.tez.dag.app.ControlledClock class is not referenced in the > source. Oddly this is not a test class, like MockClock, as I would have > expected. If this is not part of the Tez API then it can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3693) ControlledClock is not used
[ https://issues.apache.org/jira/browse/TEZ-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3693: Assignee: Muhammad Samir Khan > ControlledClock is not used > --- > > Key: TEZ-3693 > URL: https://issues.apache.org/jira/browse/TEZ-3693 > Project: Apache Tez > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Muhammad Samir Khan >Priority: Trivial > > The org.apache.tez.dag.app.ControlledClock class is not referenced in the > source. Oddly this is not a test class, like MockClock, as I would have > expected. If this is not part of the Tez API then it can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (TEZ-3283) Add a max cap to the heap computation done by Tez
[ https://issues.apache.org/jira/browse/TEZ-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated TEZ-3283: - Attachment: tez-3283.001.patch Added a new parameter to specify the max java non heap memory, with a default of 1GB. > Add a max cap to the heap computation done by Tez > - > > Key: TEZ-3283 > URL: https://issues.apache.org/jira/browse/TEZ-3283 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Muhammad Samir Khan > Labels: newbie > Attachments: tez-3283.001.patch > > > Change the calculation to not use the fraction blindly. For large containers > - .e.g 10GB - this ends up leaving 2G of the container unused / outside of > the heap. > Instead we can add another parameter which indicates the max amount of memory > to leave outside the heap - defaulting to 1G -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (TEZ-3283) Add a max cap to the heap computation done by Tez
[ https://issues.apache.org/jira/browse/TEZ-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan reassigned TEZ-3283: Assignee: Muhammad Samir Khan > Add a max cap to the heap computation done by Tez > - > > Key: TEZ-3283 > URL: https://issues.apache.org/jira/browse/TEZ-3283 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Muhammad Samir Khan > Labels: newbie > > Change the calculation to not use the fraction blindly. For large containers > - .e.g 10GB - this ends up leaving 2G of the container unused / outside of > the heap. > Instead we can add another parameter which indicates the max amount of memory > to leave outside the heap - defaulting to 1G -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TEZ-3759) Consider writing index and out file (for shuffle) in the same disk
Rajesh Balamohan created TEZ-3759: - Summary: Consider writing index and out file (for shuffle) in the same disk Key: TEZ-3759 URL: https://issues.apache.org/jira/browse/TEZ-3759 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan Priority: Minor Currently, it is possible to have index and out file in different directories. It would be helpful to store them in same disk if possible to reduce disk load. This could save some ms during high load and especially be useful for shorter jobs (e.g sub second). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
