[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517387#comment-14517387 ] Hitesh Shah commented on TEZ-2226: -- Created TEZ-2380 for [~pramachandran]'s comment. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Fix For: 0.6.1 > > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.12.patch, > TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, > TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, > TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517354#comment-14517354 ] Hitesh Shah commented on TEZ-2226: -- Good catch [~pramachandran]. Will file a separate jira for that. Committing shortly. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.12.patch, > TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, > TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, > TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517127#comment-14517127 ] Prakash Ramachandran commented on TEZ-2226: --- +1 LGTM. one minor item (separate JIRA) could be in DAGClientImpl to not read from timeline if writes are disabled. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.12.patch, > TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, > TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, > TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516003#comment-14516003
]
TezQA commented on TEZ-2226:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12728605/TEZ-2226.12.patch
against master revision 9e9cf99.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/556//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/556//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.12.patch,
> TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch,
> TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch,
> TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515843#comment-14515843
]
TezQA commented on TEZ-2226:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12728523/TEZ-2226.addon-for-patch10-combined.full.patch
against master revision aa87a14.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/555//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/555//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.12.patch,
> TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch,
> TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch,
> TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515185#comment-14515185 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks so much for the addon patch ! It looks very good. +1 (non-binding) > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.2.patch, > TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, > TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, > TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515186#comment-14515186
]
TezQA commented on TEZ-2226:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12728518/TEZ-2226.11.patch
against master revision aa87a14.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/554//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/554//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.2.patch,
> TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch,
> TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch,
> TEZ-2226.addon-for-patch10, TEZ-2226.addon-for-patch10-combined.full.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515125#comment-14515125 ] Hitesh Shah commented on TEZ-2226: -- Thanks for the patch 11 [~lichangleo]. I started making some minor mods over patch 10 in addition to recovery support. Mostly cleanup but also handling one case where history events are generated that are not related to a dag ( app launched etc ). Will upload an add-on patch for .10 shortly in addition to a combined patch. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.2.patch, > TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, > TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515120#comment-14515120 ] Chang Li commented on TEZ-2226: --- Thanks a lot for help [~zjffdu], [~hitesh]! I updated my latest patch to handle the am crash and recover scenario, have tested in my single node cluster. Could you please help review, thanks! > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.11.patch, TEZ-2226.2.patch, > TEZ-2226.3.patch, TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, > TEZ-2226.7.patch, TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513602#comment-14513602
]
Jeff Zhang commented on TEZ-2226:
-
I saw the TEZ_DAG_HISTORY_LOGGING is set in the dag's configuration. So it
should be able to restore this value when recovering.
[~lichangleo] I think you need to update the following code in
RecoveryParser.java when recovering from DAGSubmittedEvent. (Also update the
skippedDAGs of ATSHistoryLoggingService in this place)
{code}
case DAG_SUBMITTED:
{
DAGSubmittedEvent submittedEvent = (DAGSubmittedEvent) event;
LOG.info("Recovering from event"
+ ", eventType=" + eventType
+ ", event=" + event.toString());
recoveredDAGData.recoveredDAG =
dagAppMaster.createDAG(submittedEvent.getDAGPlan(),
lastInProgressDAG);
recoveredDAGData.cumulativeAdditionalResources = submittedEvent
.getCumulativeAdditionalLocalResources();
recoveredDAGData.recoveredDagID =
recoveredDAGData.recoveredDAG.getID();
dagAppMaster.setCurrentDAG(recoveredDAGData.recoveredDAG);
if (recoveredDAGData.nonRecoverable) {
skipAllOtherEvents = true;
}
break;
{code}
BTW there's no apache header for HistoryACLPolicyException.java
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.10.patch, TEZ-2226.2.patch, TEZ-2226.3.patch,
> TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch,
> TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch,
> TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511852#comment-14511852
]
TezQA commented on TEZ-2226:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12728021/TEZ-2226.10.patch
against master revision 2935ef4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/534//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/534//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.10.patch, TEZ-2226.2.patch, TEZ-2226.3.patch,
> TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch,
> TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch,
> TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511769#comment-14511769 ] Hitesh Shah commented on TEZ-2226: -- Not exactly. Based on your implementation, the history service only decides to skip if flag is set in submitted event. On recovery, the submitted is not seen as the dag was submitted previously. There is dag recovered event which could be leveraged but it would need to find the value of the flag from the dag plan > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.2.patch, TEZ-2226.3.patch, > TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, > TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511723#comment-14511723 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks for valuable review and feedback! I also feel that I had abuse using timeline_enabled. I have created a new config setting to ensure all dags within that session will have dag logging turned off. Have also corrected other places you point out. Did manually run jobs and verified that logging is turned on and off as expected. As for the recovery issue, I pass the TEZ_DAG_HISTORY_LOGGING flag from client to am as part of dagplan, even though am crashes and then restarts, I believe if DAGAppMaster called startDAG with the same dagplan it can still see the flag and function correctly. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.10.patch, TEZ-2226.2.patch, TEZ-2226.3.patch, > TEZ-2226.4.patch, TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, > TEZ-2226.8.patch, TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511393#comment-14511393 ] Hitesh Shah commented on TEZ-2226: -- One more point - recovery. Take this case: - dag submitted with logging false. - recovery is enabled. - AM crashes while dag is running. - New AM attempt launches. - DAG recovers and continues to run. - Logging flag is lost. DAG starts writing to history again. \cc [~zjffdu] in case he has any ideas. I will think about this and get back to you on potential options. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, > TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511360#comment-14511360
]
TezQA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727946/TEZ-2226.9.patch
against master revision 778c1f5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/530//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/530//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/530//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch,
> TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511340#comment-14511340 ] Hitesh Shah commented on TEZ-2226: -- Thinking more about it - one option could be to introduce a new field/config setting which can be set to let subsequent code paths know that logging should be disabled for future dags. Currently, using config.timeline_enabled or historyaclmanager is null might get confusing and lead to bugs down the line where another contributor may not understand what is being done and why. In that light, it would be good to document the changes carefully too. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, > TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511335#comment-14511335
]
Hitesh Shah commented on TEZ-2226:
--
Comments:
{code}
} else {
dag.setConf(TezConfiguration.TEZ_DAG_HISTORY_LOGGING, "false");
}
{code}
- does this work correctly for simple history logger where there are no acl
managers involved? I am guessing this might be disable logging for all dags
when a simple history logger is in place?
{code}
if (!amConfig.getTezConfiguration().
359 getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
360
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
361 historyACLPolicyManager = null;
362 }
{code}
- why is this needed? the acl manager should not be created when timeline is
disabled - correct?
{code}
dag.setConf(TezConfiguration.TEZ_DAG_HISTORY_LOGGING, "true");
{code}
- this should not be needed as default true should always kick in if the
config is not present.
Minor nit: rename dagSubmittedEvent.getDagLogging() to something along the
lines of logging enabled.
Mostly looking good. Getting close to a final patch. Will try out the patch for
manual testing. You can try the same by leveraging TestOrderedWordCount to run
multiple dags and manually bring ATS up/down between dags. Take a look at the
INTER_JOB_SLEEP_INTERVAL param in the code to control the time interval between
2 dag submissions.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch,
> TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511303#comment-14511303 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks a lot for this very comprehensive review! I have updated my patch according to your suggestions. I have also added three unit tests to test those scenarios you mentioned. Let me know if there is anything else I need to fix. Thanks. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, > TEZ-2226.9.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503875#comment-14503875
]
Hitesh Shah commented on TEZ-2226:
--
Comments:
Replace TezDomainException with something more generic like
HistoryACLPolicyException. Should be in same package as HistoryAclPolicyManager
" dag.setConf(TezConfiguration.TEZ_DAG_HISTORY_LOGGING,
Boolean.toString(false));" - can just use a string "false" instead?
{code}
426 LOG.warn("Turn Off ATS Logging for DAG " +
427 dag.getName() + " Due To Fail To Create Domain");
{code}
- Rephrase to ("Disabling history logging for dag " + dagName + " due to
error in setting up history acls", e);
- Domains and ATS should not be called as any history logger with its own
history acl manager could be used in the future.
- Please change other locations too. Also, please add "for session +
sessionName" in the case where the logging for full session is being disabled.
{code}
523 try {
524 if (dag == null) {
525 aclConfigs =
historyACLPolicyManager.setupSessionACLs(amConfig.getTezConfiguration(),
526 appId);
527 } else {
528 // Non-session mode
529 // As only a single DAG is support, we should combine AM and
DAG ACLs under the same
530 // acl management layer
531 aclConfigs =
historyACLPolicyManager.setupNonSessionACLs(amConfig.getTezConfiguration(),
532 appId, dag.getDagAccessControls());
533 dag.setConf(TezConfiguration.TEZ_DAG_HISTORY_LOGGING,
Boolean.toString(true));
534 }
535 } catch (TezDomainException e) {
536 boolean aclsEnabled =
amConfig.getTezConfiguration().getBoolean(TezConfiguration.TEZ_AM_ACLS_ENABLED,
537
TezConfiguration.TEZ_AM_ACLS_ENABLED_DEFAULT);
538 if (aclsEnabled) {
539 if (dag == null) {
540 LOG.warn("Turn Off ATS Logging Due To Fail To Create
Domain");
541
amConfig.getTezConfiguration().setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,false);
542 } else {
543 LOG.warn("Turn Off ATS Logging for DAG " +
544 dag.getName() + " Due To Fail To Create Domain");
545 dag.setConf(TezConfiguration.TEZ_DAG_HISTORY_LOGGING,
Boolean.toString(false));
546 }
547 }
{code}
- code might be cleaner to have 2 separate catch blocks instead of one catch
with if/else.
{code}
*/
212 @ConfigurationScope(Scope.DAG)
213 public static final String TEZ_DAG_HISTORY_LOGGING = TEZ_PREFIX +
"dag.history.logging";
214 public static final boolean TEZ_DAG_HISTORY_LOGGING_DEFAULT = true;
{code}
- this should be marked private.
- class TezDomainException needs javadocs
{code}
boolean dagLogging =
newDAG.getConf().getBoolean(TezConfiguration.TEZ_DAG_HISTORY_LOGGING,
2031TezConfiguration.TEZ_DAG_HISTORY_LOGGING_DEFAULT);
2032historyEventHandler.setDagLoggingState(newDAG.getID().toString(),
dagLogging);
{code}
- why not pass in a flag as part of the DAGSubmittedEvent?
- this can be looked at and handled in the relevant history logging service.
See how pre-warm dags are handled in ATS logging.
{code}
throw new TezDomainException("Fail to create domain due to fail to post
domain");
{code}
- Change to "throw new TezDomainException("Fail to create domain due to fail
to post domain", e);
- the underlying root cause is getting lost.
For unit tests:
- one case is missing. Using the mock history acl manager before calling tez
session start and verifying that all dags submitted with this session have dag
logging disabled.
- also it would be good to add a mini cluster based test where when
setDagConf for logging false is done, the test checks that no data got pushed
to ATS. Likewise that not setting logging config via dag conf ( i.e. default
true ), the data gets pushed to ATS.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502903#comment-14502903 ] Chang Li commented on TEZ-2226: --- [~hitesh], [~jeagles] could you please help review the latest patch, thanks. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495510#comment-14495510
]
TezQA commented on TEZ-2226:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725431/TEZ-2226.8.patch
against master revision 11b5843.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/465//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/465//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495196#comment-14495196 ] Chang Li commented on TEZ-2226: --- [~jeagles] Thanks a lot! I just tried it on the new patch. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.8.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495134#comment-14495134
]
TezQA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725406/TEZ-2226.7.patch
against master revision 11b5843.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/464//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/464//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/464//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.patch,
> TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495103#comment-14495103 ] Jonathan Eagles commented on TEZ-2226: -- [~lichangleo]. I haven't tried, but putting the synchronized keyword on the setUpHistoryAclManager helper function will likely get rid of this. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495047#comment-14495047 ] Chang Li commented on TEZ-2226: --- [~hitesh] The latest patch should fix the broken tests. Not sure how to get rid of that findbugs warning, it should be introduced by a helper function I wrote in tezclient to set historyACLPolicyManager. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.7.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494945#comment-14494945
]
TezQA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725378/TEZ-2226.6.patch
against master revision 11b5843.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/462//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/462//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/462//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch,
> TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494894#comment-14494894 ] Hitesh Shah commented on TEZ-2226: -- Looked at the latest patch. Looks to be on the right track. Thanks for taking this on [~lichangleo]. Will add comments once the final patch with unit tests is in. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494856#comment-14494856 ] Chang Li commented on TEZ-2226: --- [~hitesh] Have implemented two tests, which test multiple dag submission in session mode and non session mode respectively. The current code already check for TEZ_AM_ACLS_ENABLED in ATSHistoryACLPolicyManager, and createTimelineDomain will not be called if TEZ_AM_ACLS_ENABLED is set false. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.6.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494263#comment-14494263 ] Chang Li commented on TEZ-2226: --- [~jeagles] Thanks for identifying this issue. I will fix this issue in the next patch. I am currently working on the unit tests and making some progress, should be able to submit a patch with tests today. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494254#comment-14494254 ] Jonathan Eagles commented on TEZ-2226: -- [~lichangleo], there is a problem with the TezDomainException class not having the proper Apache Header. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490431#comment-14490431
]
TezQA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724655/TEZ-2226.5.patch
against master revision c8ef244.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/443//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/443//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/443//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.5.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490330#comment-14490330 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks a lot for suggesting ways of writing tests and insightful review! I just posted a work in progress patch to show the progress I have made. I have updated my patch to handle disabling logging on a per dag basis. I am just beginning to write unit tests, will try to deliver them soon. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.5.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486198#comment-14486198
]
Hitesh Shah commented on TEZ-2226:
--
bq. However I am having some difficulties to write unit tests to test those
scenarios.
Try taking a look at overridden the historyACLPolicyManager class used in
TezClient. It could either be provided via config or through a diff ctor ( that
is only for testing ). You can create a mock acl policy manager which can then
be used to behave in different ways to throw errors as needed.
Other comments on the patch:
{code}
TezConfiguration conf = amConfig.getTezConfiguration();
388 conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
loggingDefault);
389 amConfig.setTezConfiguration(conf);
{code}
This will override the original value. The same change could be done by just
amConfig.getTezConfiguration().set(...)
{code}
} catch (TezUncheckedException e) {
if (e.getMessage().contains("Fail to create domain due to fail
to post domain")) {
{code}
The above is quite error prone especially if the message changes later. Maybe
change the policy manager to throw an explicit exception that can be caught and
handled correctly?
When domain creation fails for a DAG, can you use "public DAG setConf(String
property, String value)" instead to disable logging on a per dag basis? Also,
this might be a bit tricky as there is a single logging service per AM but
there are multiple dags. Furthermore, events can get backlogged so while DAG
1's events are still in the queue waiting to be pushed to ATS, dag 2 can come
in ( with a flag saying history disabled ). This also needs to be accounted
for. WIth this in mind, you will really need to consider having a flag per dag
which can then control whether logging is enabled or not on a per dag basis.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483324#comment-14483324 ] Chang Li commented on TEZ-2226: --- TestMRRJobsDAGApi pass on my local machine > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482260#comment-14482260
]
Hadoop QA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12723485/TEZ-2226.4.patch
against master revision b87a36f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.mapreduce.TestMRRJobsDAGApi
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/399//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/399//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch,
> TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482196#comment-14482196 ] Chang Li commented on TEZ-2226: --- [~hitesh] thanks for pointing out the problems. I have been reading more relevant codes and gained better understanding of this issue now. I have changed the method to handle this issue and updated my patch. The current fix will check acl disable and ignore timeline domain creation failure acl disable is set. For session mode, intermittent failure of dag submission will not affect following dag submissions, we only disable writing history for dags that fail to create domain. The same principle applies to non-session case if the same tezclient is used to submit multiple dags. However I am having some difficulties to write unit tests to test those scenarios. I am having troubles to introduce putdomain failure in test. I was trying to follow the existing test in TestATSHistoryWithACLs, but if I test from tezclient I am not able to mock the timelineClient in HistoryAclPolicyManager to throw exception. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.4.patch, > TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391163#comment-14391163 ] Hitesh Shah commented on TEZ-2226: -- The patch does seem to have some inconsistencies in terms of how it handles various cases. It might be good if you first wrote unit tests to test the various scenarios and verify the patch against them: - if acls disabled, writing to timeline should continue to work - what if there is an intermittent error i.e. within a session, domain creation fails for 1 dag but succeeds for the next one, will we stop writing history for all dags or just one of them? - Likewise for the non-session case, where the same TezClient is being used to submit multiple dags. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390645#comment-14390645 ] Chang Li commented on TEZ-2226: --- [~hitesh] could you kindly help review the latest patch? Thanks > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388782#comment-14388782 ] Chang Li commented on TEZ-2226: --- These two tests pass on my machine > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388724#comment-14388724
]
Hadoop QA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708442/TEZ-2226.3.patch
against master revision 3bfe003.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleInputEventHandlerImpl
org.apache.tez.runtime.library.common.shuffle.TestFetcher
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/372//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/372//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.patch,
> TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388705#comment-14388705 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks for suggestion of proper changes. I have changed my fix. Now the putDomain failure will not throw exception but only set timelineClinet to null. Then createSessionDomain or createDAGDomain will check null for timelineClient, and return null for aclConfig. Within createApplicationSubmissionContext, aclConfig will be check, and turn off timelineservice if aclConfig is null. Does this work? > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.3.patch, TEZ-2226.patch, > TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387516#comment-14387516
]
Hadoop QA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708246/TEZ-2226.2.patch
against master revision 3d5e13f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/367//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/367//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.2.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch,
> TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387500#comment-14387500 ] Hitesh Shah commented on TEZ-2226: -- [~lichangleo] Should have clarified a bit more. I am trying to avoid making the call TezClientUtils.createApplicationSubmissionContext() twice. And especially not from inside the catch block. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387492#comment-14387492 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks for point of correction. I just updated my patch to target my change specifically to timeline domain creation failure. Inside createTimelineDomain(), if putDomain fails I make it throw a TezUncheckedException with message "Could not post timeline domain". Then in TezClient, after I catch the TezUncheckedException I check if it contains that message. If it does I will ignore the error and turn off the timeline. Does this work? > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.2.patch, TEZ-2226.patch, TEZ-2226.wip.2.patch, > TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387438#comment-14387438
]
Hitesh Shah commented on TEZ-2226:
--
The approach to set timeline off if the error occurs during submission is
correct. However:
{code}
} catch (TezUncheckedException e) {
LOG.warn("Failed To Create TimelineDomain. "
+atsHistoryLoggingServiceClassName
+" is disabled.");
{code}
The above does not work. There is no guarantee that the exception is due to a
timeline domain issue. Isn't there a way to find the root location of where
this error will occur and then figure out how to bubble that info up so that
the config is turned off as needed?
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387148#comment-14387148
]
Hadoop QA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708194/TEZ-2226.patch
against master revision 1c9b2fd.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/366//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/366//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387047#comment-14387047 ] Chang Li commented on TEZ-2226: --- [~hitesh] Thanks for helping me understand the problem better. My current way to fix is adding exception throw when fail to createDomain. Tez client will catch that exception and set TIMELINE_SERVICE_ENABLED to false in amconfig. I have verified that in the scenario of failing to create domain, AM code will stop publishing data to ATS. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.patch, TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384343#comment-14384343 ] Hitesh Shah commented on TEZ-2226: -- [~lichangleo] I am not sure how this patch fixes the issue. Consider a situation: there was an intermittent n/w error on the client where the domain creation failed. In this scenario, does the dag submission fail or is the domain creation error ignored? Now, if the domain creation error is ignored and the dag is submitted, how does the AM code prevent publishing of data to ATS via the default domain? > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > Attachments: TEZ-2226.wip.2.patch, TEZ-2226.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384200#comment-14384200
]
Hadoop QA commented on TEZ-2226:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12707798/TEZ-2226.wip.2.patch
against master revision 17d2388.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/361//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/361//console
This message is automatically generated.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[
https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384108#comment-14384108
]
Chang Li commented on TEZ-2226:
---
[~hitesh] Currently within createTimelineDomain, the failure of putDomain
merely put out a warn message. It should've throw a TezUncheckedException so
that both TezClient and ATSHistoryLoggingService can deal with the failure of
correctly. Right now both TezClient and ATSHistoryLoggingService deal with the
TezUncheckedException when trying to instantiate their ACLManager by
{code}
try {
historyACLPolicyManager = ReflectionUtils.createClazzInstance(
atsHistoryACLManagerClassName);
historyACLPolicyManager.setConf(conf);
} catch (TezUncheckedException e) {
LOG.warn("Could not instantiate object for " +
atsHistoryACLManagerClassName
+ ". ACLs cannot be enforced correctly for history data in Timeline",
e);
if
(!conf.getBoolean(TezConfiguration.TEZ_AM_ALLOW_DISABLED_TIMELINE_DOMAINS,
TezConfiguration.TEZ_AM_ALLOW_DISABLED_TIMELINE_DOMAINS_DEFAULT)) {
throw e;
}
historyACLPolicyManager = null;
timelineClient = null;
}
{code}
Therefore, by integrating my changes, it should solve the problem of this jira.
> Disable writing history to timeline if domain creation fails.
> -
>
> Key: TEZ-2226
> URL: https://issues.apache.org/jira/browse/TEZ-2226
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Hitesh Shah
>Assignee: Chang Li
>Priority: Blocker
> Attachments: TEZ-2226.wip.2.patch, TEZ-2226.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383888#comment-14383888 ] Chang Li commented on TEZ-2226: --- [~hitesh] no problem, I will work on this > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Chang Li >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2226) Disable writing history to timeline if domain creation fails.
[ https://issues.apache.org/jira/browse/TEZ-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383042#comment-14383042 ] Hitesh Shah commented on TEZ-2226: -- [~lichangleo] Would you mind taking a look at this too? The gist is that if ACLs are enabled but for some reason if domain creation fails, then there should not be any data written to timeline as it opens up data to users who should not be able to view it. > Disable writing history to timeline if domain creation fails. > - > > Key: TEZ-2226 > URL: https://issues.apache.org/jira/browse/TEZ-2226 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
