[jira] [Commented] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614254#comment-15614254
]
TezQA commented on TEZ-3097:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835723/TEZ-3097.01.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2070//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2070//console
This message is automatically generated.
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3097.01.patch
>
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3097 PreCommit Build #2070
Jira: https://issues.apache.org/jira/browse/TEZ-3097
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2070/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4814 lines...]
[INFO] Tez SUCCESS [ 0.025 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 59:39 min
[INFO] Finished at: 2016-10-28T04:20:52+00:00
[INFO] Final Memory: 81M/1488M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835723/TEZ-3097.01.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2070//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2070//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
dc5929ca3015763fc709fd713c603b15085d0d05 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3097
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3097:
---
Attachment: TEZ-3097.01.patch
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3097.01.patch
>
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3477) MRInputHelpers generateInputSplitsToMem public API modified
[
https://issues.apache.org/jira/browse/TEZ-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613988#comment-15613988
]
Ming Ma commented on TEZ-3477:
--
[~jeagles] sorry about that.
* The patch added protected createMRInputPayloadWithGrouping and
createMRInputPayload which seem unnecessary.
* Do you need to add the annotation to the mapreduce version of
createSplitProto? Same for createNewFormatSplitFromUserPayload.
> MRInputHelpers generateInputSplitsToMem public API modified
> ---
>
> Key: TEZ-3477
> URL: https://issues.apache.org/jira/browse/TEZ-3477
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3477.1.patch, TEZ-3477.2.patch
>
>
> Pig and Hive directly rely on specific APIs in MRInputHelpers. I would like
> to ensure these signature are prevented from being modified.
> - MRInputHelpers.generateInputSplitsToMem
> - MRInputHelpers.parseMRInputPayload
> - MRInputHelpers.createSplitProto
> - MRInputHelpers.createOldFormatSplitFromUserPayload
> - MRInputHelpers.configureMRInputWithLegacySplitGeneration
> A recent fixed jira TEZ-3430 modified generateInputSplitsToMem
> {code}
> java.lang.NoSuchMethodError:
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(Lorg/apache/hadoop/conf/Configuration;ZI)Lorg/apache/tez/mapreduce/hadoop/InputSplitInfoMem;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3496) Tez UI: Optimize display of all tasks table
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613645#comment-15613645 ] Hitesh Shah commented on TEZ-3496: -- How many webservice calls are being made to display a table having 10 task rows? > Tez UI: Optimize display of all tasks table > --- > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch, TEZ-3496.1.patch, TEZ-3496.2.patch > > > Display attempt log links in all task tables. The column could be named > "Final Attempt Log" > - When attempts are available, show View & Download links for the last > attempt log > - If no attempts are available, just display "Not Available!" message. > The change must be made in tasks table under DAG & Vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3271) Provide mapreduce failures.maxpercent equivalent
[
https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613573#comment-15613573
]
TezQA commented on TEZ-3271:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835664/TEZ-3271.10.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2069//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2069//console
This message is automatically generated.
> Provide mapreduce failures.maxpercent equivalent
>
>
> Key: TEZ-3271
> URL: https://issues.apache.org/jira/browse/TEZ-3271
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: Succeeded with Failures.png, TEZ-3271.1.patch,
> TEZ-3271.10.patch, TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch,
> TEZ-3271.5.patch, TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch,
> TEZ-3271.9.patch
>
>
> There is a certain category of work that need not have 100% of tasks succeed
> to cause the work to be considered a success. To meet that end, I propose we
> provide a tez equivalent of mapreduce.map.failures.maxpercent and
> mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered
> a success if the number of failures is below a configured threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3271 PreCommit Build #2069
Jira: https://issues.apache.org/jira/browse/TEZ-3271
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2069/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4821 lines...]
[INFO] Tez SUCCESS [ 0.057 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 58:53 min
[INFO] Finished at: 2016-10-27T23:03:36+00:00
[INFO] Final Memory: 82M/1461M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835664/TEZ-3271.10.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2069//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2069//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
06231f53dc6602fd8d1c5bb94f33d883395161f2 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3271
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3271) Provide mapreduce failures.maxpercent equivalent
[ https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3271: - Attachment: TEZ-3271.10.patch > Provide mapreduce failures.maxpercent equivalent > > > Key: TEZ-3271 > URL: https://issues.apache.org/jira/browse/TEZ-3271 > Project: Apache Tez > Issue Type: New Feature >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: Succeeded with Failures.png, TEZ-3271.1.patch, > TEZ-3271.10.patch, TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch, > TEZ-3271.5.patch, TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch, > TEZ-3271.9.patch > > > There is a certain category of work that need not have 100% of tasks succeed > to cause the work to be considered a success. To meet that end, I propose we > provide a tez equivalent of mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered > a success if the number of failures is below a configured threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3479) DAG AM does not schedule any more containers in corner cases
[ https://issues.apache.org/jira/browse/TEZ-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612706#comment-15612706 ] Hitesh Shah commented on TEZ-3479: -- \cc [~harishjp] as this is related to recovery > DAG AM does not schedule any more containers in corner cases > > > Key: TEZ-3479 > URL: https://issues.apache.org/jira/browse/TEZ-3479 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.7.1 >Reporter: Rajesh Balamohan > Attachments: application_1476667862449_0031_not_complete.1.log.tar.gz > > > Env: 3 node AWS cluster with data residing in S3. Tez version is 0.7. > Some workloads end up generating lots of data that the tasks start throwing > "No space available" in local disks (e.g Q29 in TPCDS). DAG should fail after > enough number of retries which happens most of the time. Once in a while (~ > once in 20-30 runs), DAG AM gets into hung state and does not schedule any > more containers for the failed task attempts. Will attach the logs shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3418) Tez UI/UI2 doesn't support kerberized cluster
[ https://issues.apache.org/jira/browse/TEZ-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612704#comment-15612704 ] Hitesh Shah commented on TEZ-3418: -- [~BigDataOrange] Sorry for the late reply. A few points: - the tez-ui today is mostly html/javascript/css. There is no backend hosting server built out today given that there are existing layers which can support the serving aspects. For example, if you have a tomcat server that can be configured to handle kerberos/spnego auth, you can serve the UI using the setup. - The timeline server can already act as a hosting server and already supports all the std. hadoop auth mechanisms already. This feature was introduced as part of YARN-2513 and is in use by folks such as Yahoo to serve the Tez UI on their secure clusters. bq. Hosting the ui war within the timeline server could be another idea, but I'm not sure it would work. Could you let us know what you see as issues with this approach? > Tez UI/UI2 doesn't support kerberized cluster > - > > Key: TEZ-3418 > URL: https://issues.apache.org/jira/browse/TEZ-3418 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Affects Versions: 0.8.3 > Environment: Tez 0.8.3, Hadoop 2.7.2 >Reporter: Alexandre Linte > > Today it is not possible to use Tez UI/UI2 in a Kerberized cluster without > Ambari. > This feature request asks for an improvement of the Tez UI2 in order to fully > support Kerberos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3473) Tez UI: Build is failing in Ubuntu
[ https://issues.apache.org/jira/browse/TEZ-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612678#comment-15612678 ] Hitesh Shah commented on TEZ-3473: -- [~Sreenath] Can you provide more details on the failure? > Tez UI: Build is failing in Ubuntu > -- > > Key: TEZ-3473 > URL: https://issues.apache.org/jira/browse/TEZ-3473 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Sreenath Somarajapuram > > wro4j-maven-plugin that was use for minifying javascript is hitting an API > incompatibility when we try to build in Ubuntu. > \cc [~hitesh] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3485) Tez UI: Improve "Context ID" label in All DAGs page
[ https://issues.apache.org/jira/browse/TEZ-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612676#comment-15612676 ] Hitesh Shah commented on TEZ-3485: -- [~Sreenath] Can you provide more details on this? Is this a regression from earlier behavior? > Tez UI: Improve "Context ID" label in All DAGs page > > > Key: TEZ-3485 > URL: https://issues.apache.org/jira/browse/TEZ-3485 > Project: Apache Tez > Issue Type: Task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > > The label must be set to “Context ID (Hive Query ID or Pig Script ID)” -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3484) Tez UI: Remove .travis.yml from webapp folder
[ https://issues.apache.org/jira/browse/TEZ-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612669#comment-15612669 ] Hitesh Shah commented on TEZ-3484: -- +1 after the rat exclude is removed. > Tez UI: Remove .travis.yml from webapp folder > - > > Key: TEZ-3484 > URL: https://issues.apache.org/jira/browse/TEZ-3484 > Project: Apache Tez > Issue Type: Task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram >Priority: Trivial > Attachments: TEZ-3484.1.patch > > > As of now, travis work only form root folder. Hence the current .travis.yml > in webapp folder would be of no use. > It might be better to get that removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3432) Add unit test for session timeout
[ https://issues.apache.org/jira/browse/TEZ-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612668#comment-15612668 ] Hitesh Shah commented on TEZ-3432: -- [~Sreenath] Any luck in creating a new patch? > Add unit test for session timeout > - > > Key: TEZ-3432 > URL: https://issues.apache.org/jira/browse/TEZ-3432 > Project: Apache Tez > Issue Type: Bug >Reporter: Sushmitha Sreenivasan >Assignee: Sreenath Somarajapuram > Labels: newbie > Attachments: TEZ-3432.1.patch, TEZ-3432.2.patch, TEZ-3432.3.patch, > TEZ-3432.4.patch, TEZ-3432.5.patch > > > Add unit test which sets tez.session.am.dag.submit.timeout.secs to say 5 secs > and checking if dag submission timeouts after the configured time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3477) MRInputHelpers generateInputSplitsToMem public API modified
[
https://issues.apache.org/jira/browse/TEZ-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612665#comment-15612665
]
Hitesh Shah commented on TEZ-3477:
--
[~mingma] Mind helping with the review?
> MRInputHelpers generateInputSplitsToMem public API modified
> ---
>
> Key: TEZ-3477
> URL: https://issues.apache.org/jira/browse/TEZ-3477
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3477.1.patch, TEZ-3477.2.patch
>
>
> Pig and Hive directly rely on specific APIs in MRInputHelpers. I would like
> to ensure these signature are prevented from being modified.
> - MRInputHelpers.generateInputSplitsToMem
> - MRInputHelpers.parseMRInputPayload
> - MRInputHelpers.createSplitProto
> - MRInputHelpers.createOldFormatSplitFromUserPayload
> - MRInputHelpers.configureMRInputWithLegacySplitGeneration
> A recent fixed jira TEZ-3430 modified generateInputSplitsToMem
> {code}
> java.lang.NoSuchMethodError:
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(Lorg/apache/hadoop/conf/Configuration;ZI)Lorg/apache/tez/mapreduce/hadoop/InputSplitInfoMem;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3488) Tez UI Vertex 404
[ https://issues.apache.org/jira/browse/TEZ-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612638#comment-15612638 ] Premal Shah commented on TEZ-3488: -- Hi [~Sreenath] Can you give any ideas to debug this issue? > Tez UI Vertex 404 > - > > Key: TEZ-3488 > URL: https://issues.apache.org/jira/browse/TEZ-3488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: George Liaw > > When we're trying to view vertices for a running job, the Vertex page will > show a 404 error saying: > "Could not retrieve expected data from @ " > And the Vertex Counter page will show: > "Counters unavailable" > and > "Error code: Unknown, message: Resource Manager (RM) out of reach. Either > it's down or CORS is not enabled" > The RMs are up and CORS is enabled. If we go to the displayed url, we get the > expected XML with the application info. When the DAG finishes running, those > errors go away and the pages display fine. Any idea what's going on here? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3271) Provide mapreduce failures.maxpercent equivalent
[ https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612523#comment-15612523 ] Hitesh Shah commented on TEZ-3271: -- bq. The above it true, but one important this I didn't mention above is that edgeManager.getNumDestinationConsumerTasks throws of type Exception. Throwing TezException, IOException, and Exception seems redundant. I will catch the Exception and rethrow TezException to explicitly give a throws TezException, IOException signature. Sorry - my bad. Missed that. Lets stick to Exception in that case - no need to do an additional wrap around. > Provide mapreduce failures.maxpercent equivalent > > > Key: TEZ-3271 > URL: https://issues.apache.org/jira/browse/TEZ-3271 > Project: Apache Tez > Issue Type: New Feature >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: Succeeded with Failures.png, TEZ-3271.1.patch, > TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch, TEZ-3271.5.patch, > TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch, TEZ-3271.9.patch > > > There is a certain category of work that need not have 100% of tasks succeed > to cause the work to be considered a success. To meet that end, I propose we > provide a tez equivalent of mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered > a success if the number of failures is below a configured threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3470) Tez UI: Make the build work in IBM PPC
[ https://issues.apache.org/jira/browse/TEZ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612517#comment-15612517 ] Hitesh Shah commented on TEZ-3470: -- Ignore the above question. Just looked at https://github.com/ibmsoe/phantomjs-1/commit/b61f149c6d0ec443ea1a9ee4edb34674ed45063f > Tez UI: Make the build work in IBM PPC > -- > > Key: TEZ-3470 > URL: https://issues.apache.org/jira/browse/TEZ-3470 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3470.1.patch, TEZ-3470.2.patch, TEZ-3470.3.patch, > TEZ-3470.wip.1.patch > > > Current versions of frontend-maven-plugin, node & npm used by the build is > not functioning as expected in IBM PPC. Following version works, and the > build must be changed to used the same in a PPC. > - frontend-maven-plugin : v1.1 > - node : v5.7.0 > - npm : 3.6.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3470) Tez UI: Make the build work in IBM PPC
[ https://issues.apache.org/jira/browse/TEZ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612510#comment-15612510 ] Hitesh Shah commented on TEZ-3470: -- Thanks for the clarifications [~Ayappan] bq. npm wrapper for phantomjs is maintained by different community. The wrapper just checks the platform/architecture and downloads the phantomjs binary tar.bz2 from the official site. Doesn't the wrapper respect the CDNURL export variable? If yes, couldn't we modify the pom to have the CDNURL var to point to the ibmsoe location on ppc64 environments? Or are you implying that the wrapper does not yet recognize the ppc64 arch and therefore fails to download the correct phantomjs even though it may be available? > Tez UI: Make the build work in IBM PPC > -- > > Key: TEZ-3470 > URL: https://issues.apache.org/jira/browse/TEZ-3470 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3470.1.patch, TEZ-3470.2.patch, TEZ-3470.3.patch, > TEZ-3470.wip.1.patch > > > Current versions of frontend-maven-plugin, node & npm used by the build is > not functioning as expected in IBM PPC. Following version works, and the > build must be changed to used the same in a PPC. > - frontend-maven-plugin : v1.1 > - node : v5.7.0 > - npm : 3.6.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3470) Tez UI: Make the build work in IBM PPC
[ https://issues.apache.org/jira/browse/TEZ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612510#comment-15612510 ] Hitesh Shah edited comment on TEZ-3470 at 10/27/16 5:08 PM: Thanks for the clarifications [~Ayappan] bq. npm wrapper for phantomjs is maintained by different community. The wrapper just checks the platform/architecture and downloads the phantomjs binary tar.bz2 from the official site. Doesn't the wrapper respect the CDNURL export variable? If yes, couldn't we modify the pom to have the CDNURL var to point to the ibmsoe location on ppc64 environments? Or are you implying that the existing wrapper does not yet recognize the ppc64 arch and therefore fails to download the correct phantomjs even though it may be available? was (Author: hitesh): Thanks for the clarifications [~Ayappan] bq. npm wrapper for phantomjs is maintained by different community. The wrapper just checks the platform/architecture and downloads the phantomjs binary tar.bz2 from the official site. Doesn't the wrapper respect the CDNURL export variable? If yes, couldn't we modify the pom to have the CDNURL var to point to the ibmsoe location on ppc64 environments? Or are you implying that the wrapper does not yet recognize the ppc64 arch and therefore fails to download the correct phantomjs even though it may be available? > Tez UI: Make the build work in IBM PPC > -- > > Key: TEZ-3470 > URL: https://issues.apache.org/jira/browse/TEZ-3470 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3470.1.patch, TEZ-3470.2.patch, TEZ-3470.3.patch, > TEZ-3470.wip.1.patch > > > Current versions of frontend-maven-plugin, node & npm used by the build is > not functioning as expected in IBM PPC. Following version works, and the > build must be changed to used the same in a PPC. > - frontend-maven-plugin : v1.1 > - node : v5.7.0 > - npm : 3.6.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3271) Provide mapreduce failures.maxpercent equivalent
[ https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612432#comment-15612432 ] Jonathan Eagles commented on TEZ-3271: -- bq. The function could do explicit throws for both of the exceptions. On the VertexImpl, a generic Exception catch would make sense to ensure that we dont hit a dispatcher error. The above it true, but one important this I didn't mention above is that edgeManager.getNumDestinationConsumerTasks throws of type Exception. Throwing TezException, IOException, and Exception seems redundant. I will catch the Exception and rethrow TezException to explicitly give a throws TezException, IOException signature. bq. Relies on TaskImpl's impl of addAndScheduleAttempt() but you could just use size of the map to decipher the last attempt. Changed the logic to always ensure the highest attempt is chosen. bq. "Assert.assertEquals(2, v6.numSuccessSourceAttemptCompletions);" - shoudnt this get 4 completions given that v6 expects task attempt completions for both tasks in v4 and v5? Likewise for "testFailuresMaxPercentExceededSourceTaskAttemptCompletionEvents" - I would assume this will get 2 for tasks of v5? Will address this bq. Sorry for the last minute review comment on testVertexFailuresMaxPercent - should we be using a 2-vertex DAG with the first vertex having a threshold and verifying that the second vertex ( using a shuffle edge ) completes successfully ? Think the main thrust is that the shuffle is missing between these vertices and should be added. Will do that. > Provide mapreduce failures.maxpercent equivalent > > > Key: TEZ-3271 > URL: https://issues.apache.org/jira/browse/TEZ-3271 > Project: Apache Tez > Issue Type: New Feature >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: Succeeded with Failures.png, TEZ-3271.1.patch, > TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch, TEZ-3271.5.patch, > TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch, TEZ-3271.9.patch > > > There is a certain category of work that need not have 100% of tasks succeed > to cause the work to be considered a success. To meet that end, I propose we > provide a tez equivalent of mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered > a success if the number of failures is below a configured threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612351#comment-15612351
]
Jason Lowe commented on TEZ-3097:
-
Sorry, I should have been more clear. The unit test is trying to test DAG
commit. In order for a DAG to commit, all the vertices need to succeed.
Therefore we should not be trying to schedule tasks beyond the DAG completion
in the context of this test, since that means a vertex succeeded before it
started scheduling tasks. In other words, the unit test is not doing a good
job of emulating what a real DAG does when it tries to commit.
In practice, yes, a DAG can enter a terminal state while vertices could still
be asynchronously trying to schedule, but I don't think that's an error.
Eventually the vertices will receive kill events from the dag and they in turn
will issue kill events to any tasks that were scheduled. The history handler
should have no issue recording these events which may occur after the dag
completion event. It only causes an issue in this test because the test is not
prepared to receive history events after the dag completes (again because it
poorly emulates what a DAG does to get to the commit stage).
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3498) Tez UI: Optimize display of Task details page
[
https://issues.apache.org/jira/browse/TEZ-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612348#comment-15612348
]
TezQA commented on TEZ-3498:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835597/TEZ-3498.1.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2068//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2068//console
This message is automatically generated.
> Tez UI: Optimize display of Task details page
> -
>
> Key: TEZ-3498
> URL: https://issues.apache.org/jira/browse/TEZ-3498
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3498.1.patch
>
>
> Considering we could only have a limited set of task attempts, it would be
> better to display the attempts table in the task details pages itself instead
> of a separate tab. This could decrease one level click.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3498 PreCommit Build #2068
Jira: https://issues.apache.org/jira/browse/TEZ-3498
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2068/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4828 lines...]
[INFO] Tez SUCCESS [ 0.026 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56:51 min
[INFO] Finished at: 2016-10-27T16:06:54+00:00
[INFO] Final Memory: 81M/1488M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835597/TEZ-3498.1.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2068//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2068//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7b7d0c7f040c1a13d96809a9329219d346eb94b7 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3498
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612324#comment-15612324
]
Harish Jaiprakash commented on TEZ-3097:
Thanks [~jlowe]. Lets say during execution of a dag, a failure in one vertex
caused the dag to abort and another vertex is the state shown above. Would the
same thing happen, initialize gets called after dag finished and tasks would
get scheduled and dag does not exists?
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3496 PreCommit Build #2067
Jira: https://issues.apache.org/jira/browse/TEZ-3496
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2067/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4832 lines...]
[INFO] Tez SUCCESS [ 0.038 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:01 h
[INFO] Finished at: 2016-10-27T14:53:11+00:00
[INFO] Final Memory: 75M/1149M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835587/TEZ-3496.2.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2067//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2067//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
d84e17d69e9ebefe89190f2570a93cf71ca6b8b0 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3496
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3496) Tez UI: Optimize display of all tasks table
[
https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612116#comment-15612116
]
TezQA commented on TEZ-3496:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835587/TEZ-3496.2.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2067//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2067//console
This message is automatically generated.
> Tez UI: Optimize display of all tasks table
> ---
>
> Key: TEZ-3496
> URL: https://issues.apache.org/jira/browse/TEZ-3496
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3496.1.WIP.patch, TEZ-3496.1.patch, TEZ-3496.2.patch
>
>
> Display attempt log links in all task tables. The column could be named
> "Final Attempt Log"
> - When attempts are available, show View & Download links for the last
> attempt log
> - If no attempts are available, just display "Not Available!" message.
> The change must be made in tasks table under DAG & Vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3496 PreCommit Build #2066
Jira: https://issues.apache.org/jira/browse/TEZ-3496
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2066/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4832 lines...]
[INFO] Tez SUCCESS [ 0.026 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 59:30 min
[INFO] Finished at: 2016-10-27T14:19:29+00:00
[INFO] Final Memory: 79M/1330M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835572/TEZ-3496.1.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2066//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2066//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1fd16e1755a3b590fad512e372bf2d219c1e4100 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3496
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3496) Tez UI: Optimize display of all tasks table
[
https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612031#comment-15612031
]
TezQA commented on TEZ-3496:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12835572/TEZ-3496.1.patch
against master revision 6cf4378.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2066//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2066//console
This message is automatically generated.
> Tez UI: Optimize display of all tasks table
> ---
>
> Key: TEZ-3496
> URL: https://issues.apache.org/jira/browse/TEZ-3496
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-3496.1.WIP.patch, TEZ-3496.1.patch, TEZ-3496.2.patch
>
>
> Display attempt log links in all task tables. The column could be named
> "Final Attempt Log"
> - When attempts are available, show View & Download links for the last
> attempt log
> - If no attempts are available, just display "Not Available!" message.
> The change must be made in tasks table under DAG & Vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3498) Tez UI: Optimize display of Task details page
[ https://issues.apache.org/jira/browse/TEZ-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3498: Attachment: TEZ-3498.1.patch Attaching a patch that moved Attempts tab under a task into the details page. [~hitesh] [~harishjp] Please help with the patch. > Tez UI: Optimize display of Task details page > - > > Key: TEZ-3498 > URL: https://issues.apache.org/jira/browse/TEZ-3498 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3498.1.patch > > > Considering we could only have a limited set of task attempts, it would be > better to display the attempts table in the task details pages itself instead > of a separate tab. This could decrease one level click. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611904#comment-15611904
]
Jason Lowe commented on TEZ-3097:
-
bq. Is this a bug? Should we handle case where dag can be finished before the
above schedule is done.
I don't think it's a bug. If the DAG believes it is complete and ready to
commit, then why are we just getting around to telling a vertex manager that
the vertex started and it should start scheduling tasks? I think this is an
artifact of the test trying to fake the completion of the DAG.
For the purposes of this test, I suspect the concurrent queue fix would work
since it's not interested in task-level events. It only cares about commit
events and dag completion events, and those appear to be issued synchronously
with the direct handler invocation.
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3496) Tez UI: Optimize display of all tasks table
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3496: Attachment: TEZ-3496.2.patch Attaching a second patch, that would display a "Not Available!" message, instead of a waiting message with a loading animation. > Tez UI: Optimize display of all tasks table > --- > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch, TEZ-3496.1.patch, TEZ-3496.2.patch > > > Display attempt log links in all task tables. The column could be named > "Final Attempt Log" > - When attempts are available, show View & Download links for the last > attempt log > - If no attempts are available, just display "Not Available!" message. > The change must be made in tasks table under DAG & Vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3496) Tez UI: Optimize display of all tasks table
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3496: Attachment: TEZ-3496.1.patch [~hitesh] [~harishjp] Please help with this patch. > Tez UI: Optimize display of all tasks table > --- > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch, TEZ-3496.1.patch > > > Display attempt log links in all task tables. The column could be named > "Final Attempt Log" > - When attempts are available, show View & Download links for the last > attempt log > - If no attempts are available, just display "Not Available!" message. > The change must be made in tasks table under DAG & Vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611534#comment-15611534
]
Harish Jaiprakash commented on TEZ-3097:
I was able to approximately replicate the issue: (Not the concurrent
modification but get events after dag finished).
* delay enqueueAndScheduleNextEvent(new
VertexManagerEventOnVertexStarted(completions)) in
VertexManager.onVertexStarted by 20-30 ms in a different thread.
* This is to simulate a case where the executerService would not have invoked
the callback for onVertexStarted, but the task complete events for the vertex
has been handled and dag is finished.
* Now we have a finished dag, and suddenly
ShuffleVertexManagerBase.OnVertexStarted is scheduled by the VertexManager and
which caused new tasks to be scheduled and events sent to HistoryEventHandler.
Followup:
* Is this a bug? Should we handle case where dag can be finished before the
above schedule is done. (No test fix required in this case).
* Is there a way to fire the task finished events only after Vertex has fully
started. More like, fire task finished only after task start is called? Only
way I can think of is to wait for task started events to arrive in
MockHistoryEventHandler (make it thread safe too).
* These extra events do not affect the outcome, use a concurrent queue instead
of the list to make MockHistoryEventHandler thread safe.
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (TEZ-3097) Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
[
https://issues.apache.org/jira/browse/TEZ-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash reassigned TEZ-3097:
--
Assignee: Harish Jaiprakash
> Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess
> -
>
> Key: TEZ-3097
> URL: https://issues.apache.org/jira/browse/TEZ-3097
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.3
>Reporter: Jeff Zhang
>Assignee: Harish Jaiprakash
>
> {noformat}
> testDAGCommitStartedEventFail_OnDAGSuccess(org.apache.tez.dag.app.dag.impl.TestCommit)
> Time elapsed: 0.072 sec <<< ERROR!
> java.util.ConcurrentModificationException: null
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit$MockHistoryEventHandler.verifyVertexCommitStartedEvent(TestCommit.java:2033)
> at
> org.apache.tez.dag.app.dag.impl.TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess(TestCommit.java:1880)
> {noformat}
> https://builds.apache.org/job/Tez-Build-Hadoop-2.4/245/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3496) Tez UI: Optimize display of all tasks table
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3496: Description: Display attempt log links in all task tables. The column could be named "Final Attempt Log" - When attempts are available, show View & Download links for the last attempt log - If no attempts are available, just display "Not Available!" message. The change must be made in tasks table under DAG & Vertex. was: Display attempt log links in all task tables. The column could be named "Final Attempt Log" - When attempts are available, show View & Download links for the last attempt log - If no attempts are available, just display "Not Available!" message. Issue Type: Improvement (was: Bug) Summary: Tez UI: Optimize display of all tasks table (was: Tez UI: Optimize display of all tasks tab) > Tez UI: Optimize display of all tasks table > --- > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch > > > Display attempt log links in all task tables. The column could be named > "Final Attempt Log" > - When attempts are available, show View & Download links for the last > attempt log > - If no attempts are available, just display "Not Available!" message. > The change must be made in tasks table under DAG & Vertex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3498) Tez UI: Optimize display of Task details page
[ https://issues.apache.org/jira/browse/TEZ-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3498: Issue Type: Improvement (was: Bug) > Tez UI: Optimize display of Task details page > - > > Key: TEZ-3498 > URL: https://issues.apache.org/jira/browse/TEZ-3498 > Project: Apache Tez > Issue Type: Improvement >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > > Considering we could only have a limited set of task attempts, it would be > better to display the attempts table in the task details pages itself instead > of a separate tab. This could decrease one level click. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3498) Tez UI: Optimize display of Task details page
Sreenath Somarajapuram created TEZ-3498: --- Summary: Tez UI: Optimize display of Task details page Key: TEZ-3498 URL: https://issues.apache.org/jira/browse/TEZ-3498 Project: Apache Tez Issue Type: Bug Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Considering we could only have a limited set of task attempts, it would be better to display the attempts table in the task details pages itself instead of a separate tab. This could decrease one level click. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3496) Tez UI: Optimize display of all tasks tab
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3496: Description: Display attempt log links in all task tables. The column could be named "Final Attempt Log" - When attempts are available, show View & Download links for the last attempt log - If no attempts are available, just display "Not Available!" message. was:For tasks with one attempt, show the log View & Download links. > Tez UI: Optimize display of all tasks tab > - > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch > > > Display attempt log links in all task tables. The column could be named > "Final Attempt Log" > - When attempts are available, show View & Download links for the last > attempt log > - If no attempts are available, just display "Not Available!" message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3496) Tez UI: Optimize display of all tasks tab
[ https://issues.apache.org/jira/browse/TEZ-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3496: Summary: Tez UI: Optimize display of all tasks tab (was: Tez UI: Optimize display of all tasks tab and task details tab for the common case of one task attempt) > Tez UI: Optimize display of all tasks tab > - > > Key: TEZ-3496 > URL: https://issues.apache.org/jira/browse/TEZ-3496 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3496.1.WIP.patch > > > For tasks with one attempt, show the log View & Download links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3470) Tez UI: Make the build work in IBM PPC
[ https://issues.apache.org/jira/browse/TEZ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611009#comment-15611009 ] Ayappan commented on TEZ-3470: -- The build will first check whether phantomjs is globally installed or not. If not, then it will try to get it from the official npm repository. The npm repository doesn't have ppc64 supported npm wrapper for phantomjs so it fails. Two things here 1) npm wrapper for phantomjs is maintained by different community. The wrapper just checks the platform/architecture and downloads the phantomjs binary tar.bz2 from the official site. 2)Phantomjs official site itself doesn't have PowerPC built binary. > Tez UI: Make the build work in IBM PPC > -- > > Key: TEZ-3470 > URL: https://issues.apache.org/jira/browse/TEZ-3470 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.3 >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Labels: TezUI > Attachments: TEZ-3470.1.patch, TEZ-3470.2.patch, TEZ-3470.3.patch, > TEZ-3470.wip.1.patch > > > Current versions of frontend-maven-plugin, node & npm used by the build is > not functioning as expected in IBM PPC. Following version works, and the > build must be changed to used the same in a PPC. > - frontend-maven-plugin : v1.1 > - node : v5.7.0 > - npm : 3.6.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
