[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382623#comment-14382623 ] Hitesh Shah commented on TEZ-2205: -- Comments: Minor nit: {code} LOG.warn(ATSLogging service/acl manager is disabled + due to the Timeline Service is disabled); {code} Maybe use: atsHistoryLoggingServiceClassName + is disabled due to Timeline Service being disabled, + YarnConfiguration.TIMELINE_SERVICE_ENABLED + set to false bq. The changes in ATSHistoryLoggingService could be more optimal. If timeline is disabled, why even bother queueing up events? Previous review comment still not addressed. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383025#comment-14383025 ] Hitesh Shah commented on TEZ-2205: -- +1 Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382751#comment-14382751 ] Hitesh Shah commented on TEZ-2205: -- bq. Is there something more beyond this I need to take care? Yes - should a thread even be started to poll for events? In serviceStart and serviceStop, it might be simpler to just see if timeline is disabled either by checking client ==null or a different boolean and become no-ops. The service started/stopped log message should not even show up if timeline is disabled. This should be done in addition to whether events are queued. TestContainerReuse is not related so you can ignore it. Being addressed in a different jira. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382932#comment-14382932 ] Hadoop QA commented on TEZ-2205: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707633/TEZ-2205.5.patch against master revision d42a3c7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.dag.app.rm.TestContainerReuse Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/357//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/357//console This message is automatically generated. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382905#comment-14382905 ] Chang Li commented on TEZ-2205: --- [~hitesh] Thanks for finding out those problems. I have updated my patch by adding back the check of whether to add the event to the queue. But due to the eventQueue is a private variable of ATSHistoryLoggingService class, I can't have access to it in the test. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382843#comment-14382843 ] Chang Li commented on TEZ-2205: --- [~hitesh] Thanks for suggestions! Have updated my patch. Does my current patch address the issue of queueing events correctly? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382860#comment-14382860 ] Hitesh Shah commented on TEZ-2205: -- Comments: Minor nit: Assert.assertEquals(historyACLPolicyManager.timelineClient, null) ; - use assertNull. Also, can you modify the test to check the size of the queue at the end of the run should be 0. I am guessing you will catch the issue I mention below by adding this check. bq. Does my current patch address the issue of queueing events correctly? I think the latest patch accidentally removed the check of whether to add the event to the queue. Currently, this will add events into the queue and never empty the queue. Rest looks good. I think we are getting very close to a final patch. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.4.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382884#comment-14382884 ] Hadoop QA commented on TEZ-2205: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707626/TEZ-2205.4.patch against master revision d42a3c7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.dag.app.rm.TestContainerReuse Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/356//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/356//console This message is automatically generated. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, TEZ-2205.4.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379951#comment-14379951 ] Chang Li commented on TEZ-2205: --- Thanks a lot for patient discussion and explanations [~hitesh], [~jeagles], [~zjshen]! Understand the implementation requirement now, will provide a patch soon Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380268#comment-14380268 ] Hadoop QA commented on TEZ-2205: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707237/TEZ-2205.1.patch against master revision d1b4bd4. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/347//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/347//console This message is automatically generated. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380270#comment-14380270 ] Hitesh Shah commented on TEZ-2205: -- Adding a test for this would be useful. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380267#comment-14380267 ] Hitesh Shah commented on TEZ-2205: -- {code} LOG.warn(Timeline service is not enabled); {code} - the log should be more clear about the ATSLogging service/acl manager being disabled as the timeline service being disabled The changes in ATSHistoryLoggingService could be more optimal. If it is disabled, why even bother queueing up events? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377562#comment-14377562 ] Rohini Palaniswamy commented on TEZ-2205: - I prefer 3). For jobs launched through Oozie it is easy to turn off ATS via Oozie server side setting and this might be required in the near future now and then considering the issues we are facing with ATS. Since tez-site.xml for those jobs come from HDFS, it is not easy to change the tez ATS Logger easily (replacing the file on HDFS is more manual and can cause running jobs to fail as LocalResource time has changed) and so do not like 1). Also having to change multiple settings to turn off something is cumbersome. 2) is what is happening now but the problem I see is that it impacts performance as time is wasted trying to connect to ATS and failing due to lack of authentication. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378034#comment-14378034 ] Chang Li commented on TEZ-2205: --- [~hitesh] So should I implemnt 3) by switching to simplehistory or wrap each post call with an if statement or just not to add post events to eventqueue? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379013#comment-14379013 ] Hitesh Shah commented on TEZ-2205: -- A log.warn should be sufficient for now. SimpleHistory comes with its own problems of configuring an hdfs location and cleaning it on a regular basis. Looks like we have agreement. [~lichangleo] Does the above clarify the implemenation requirements for this jira? Thanks for the patience in handling the various run-arounds with respect to the design :) Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379007#comment-14379007 ] Jonathan Eagles commented on TEZ-2205: -- I think what zhijie has posted is similar to what I am thinking as well. This will give the on/off flag to users and keep the client in control. A log WARN should be sufficient to alert clients there is a mismatch in configuration. Whether we fall back to SimpleHistory or have a hollow ATS History isn't much of a difference to me. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378996#comment-14378996 ] Zhijie Shen commented on TEZ-2205: -- bq. i.e option 3's impl would be: LGTM. For your reference, this is what we did in MR: {code} if (conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA, MRJobConfig.DEFAULT_MAPREDUCE_JOB_EMIT_TIMELINE_DATA)) { if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) { timelineClient = TimelineClient.createTimelineClient(); timelineClient.init(conf); LOG.info(Timeline service is enabled); LOG.info(Emitting job history data to the timeline server is enabled); } else { LOG.info(Timeline service is not enabled); } } else { LOG.info(Emitting job history data to the timeline server is not enabled); } {code} And only when {{timelineClient != null}}, MR will publish the history info to the timeline server. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378199#comment-14378199 ] Hitesh Shah commented on TEZ-2205: -- bq. For jobs launched through Oozie it is easy to turn off ATS via Oozie server side setting Could you clarify a bit more on how this is being done? [~jeagles] [~zjshen] [~lichangleo] If (3) is the approach that works the best in Yahoo environments, should the eventual fix be in YARN given that other applications will face the same issue i.e. option (2)? i.e. if yarn timeline enabled flag is set to false, all the relevant yarn client and timeline client libs should automatically not make calls to the server related to timeline ( i.e. do not retrieve delegation tokens, treat all calls as no-op) ? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378245#comment-14378245 ] Rohini Palaniswamy commented on TEZ-2205: - OOZIE-2133 is the one that handles getting delegation tokens for ATS for tez jobs. If oozie.action.launcher.yarn.timeline-service.enabled is set to true on the Oozie server configuration, it adds yarn.timeline-service.enabled=true to conf of JobClient that submits the launcher job if the tez-site.xml is part of the distributed cache. JobClient (YARN) fetches ATS delegation token if that setting is set before the job is submitted and adds it to the job. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377268#comment-14377268 ] Hitesh Shah commented on TEZ-2205: -- [~rohini] [~hagleitn] any comments/concerns on the approach that we plan to take? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372109#comment-14372109 ] Hadoop QA commented on TEZ-2205: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706018/TEZ-2205.wip.patch against master revision 6e15b2f. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/322//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/322//console This message is automatically generated. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372045#comment-14372045 ] Hitesh Shah commented on TEZ-2205: -- [~rohini] At the crux of this, there are effectively a couple of issues/questions in terms of what is the logical behavior and what is the expectation? The fix for any approach is probably trivial. Consider the fact that the user configured both YARN and Tez in a conflicting manner. i.e. Configured YARN to disable timeline but made Tez use Timeline. Should Tez: 1) error out due to a conflicting configuration i.e YARN timeline disabled but Tez ATS logger enabled. 2) should Tez try and use Timeline (even though YARN flag is set to false ) and ignore its failures as needed? This should be ok for the most part except that I think there are some cases in YARN which are not handled cleanly and end up causing the app to error out. Also, there were some behavioral changes in YARN-2375 - see below. 3) Should Tez look for the YARN configuration property and silently ignore the fact that TimelineATSLogger has been configured but it should not be used? Also, FWIW, earlier ( before YARN-2375 ), even though Tez invoked Timeline::postEntities, if the YARN flag was set to false, the YARN library silently dropped the call. (2) is probably something that YARN needs to address. As for Tez, we can go with either (1) or (3). (1) is more clear-cut in terms of making it very clear to the user in terms of how to configure Tez. (3) merely hides the fact that something is wrongly configured. Also, to clarify, part of this stems from what is the yarn.timeline-service.enabled flag meant to be used for? Is it a admin flag to control where timeline is enabled or disabled for the whole cluster? It currently is a client-side flag that cannot be enforced at all. Furthermore, if it is meant to be used on a per job basis, should it then be a tez-specific setting ( which we already have in the form of the class setting ). Last question for [~rohini]: does the issue of disabling ATS stem from the fact that it is a bit hard to disable ATS logging via the service class name property? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371983#comment-14371983 ] Jonathan Eagles commented on TEZ-2205: -- This seems like a patch that will get the conversation going for what to do for this jira. [~rohini], can you comment from a pig perspective. Essentially, we are deeming it illegal to specify ATSHistoryLogger when timeline service is disabled. This approach gives clients the control of which history logger to use but leaves the yarn-timeline-service.enabled a flag that describes a feature that is available on the cluster. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371984#comment-14371984 ] Chang Li commented on TEZ-2205: --- [~hitesh] I post an attempt patch which checks if tez has its logging service class set to atsHistoryLoggingService but not enable timelie-service. In that case it will throw an exception with alert stopping the tez job from launching. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372005#comment-14372005 ] Rohini Palaniswamy commented on TEZ-2205: - {code} public void handle(DAGHistoryEvent event) { eventQueue.add(event); } {code} It would be a simple check to not add to the queue if timeline service is disabled. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372025#comment-14372025 ] Jonathan Eagles commented on TEZ-2205: -- [~hitesh], do you want to comment on this? [~rohini], there are many ways to approach this. We have all be discussing the pros and cons. I wanted to loop you into the conversation since you may have better insight as to what users can expect. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371996#comment-14371996 ] Rohini Palaniswamy commented on TEZ-2205: - Wouldn't it be better if ATSHistoryLogger checked the value of the timeline setting and do nothing if it is false. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li Attachments: TEZ-2205.wip.patch when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367561#comment-14367561 ] Hitesh Shah commented on TEZ-2205: -- Added a comment on YARN-2375. Let us see what happens there :) Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367518#comment-14367518 ] Chang Li commented on TEZ-2205: --- Thanks for clarification [~jeagles]. [~hitesh] which way should I proceed to solve this problem Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366073#comment-14366073 ] Hitesh Shah commented on TEZ-2205: -- This seems like a misconfiguration issue. Is there a reason why tez.history.logging.service.class is configured to use ATS but ATS itself is disabled? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366111#comment-14366111 ] Hitesh Shah commented on TEZ-2205: -- [~jeagles] Not exactly but I think I understand the context better now. To clarify, if yarn.timeline-service.enabled=false, then should all the YARN client libs silently ignore all calls made to them i.e do not pull the delegation token, do not create a domain, do not post entities, etc? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366150#comment-14366150 ] Hitesh Shah commented on TEZ-2205: -- That is what I do not understand. If yarn.timeline-service.enabled=false, TimelineClient::putDomain and TimelineClient::postEntities should both be a no-op ( looked at the code ). Is for some reason, the configuration value getting reset when the TimelineClient objects are being instantiated? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366103#comment-14366103 ] Jonathan Eagles commented on TEZ-2205: -- [~hitesh], I think it is correct to say that there is currently a mismatch in usage of this flag at least in secure environments. In a secure environment, tez relies on the timeline server delegation token retrieved by the YarnClientImp (yarn.timeline-service.enabled), in order to postEntities from the AM. The client doesn't have this issue (it is able to retrieve its own delegation token) while putting the domain unless the job was run from within oozie (but that is another story). So as I understand it, tez either needs to completely ignore yarn.timeline-service.enabled and fetch its own timeline server delegation token... or it needs to pay attention to this flag and fallback to simple history when yarn.timeline-service.enabled=false. Seems like you are suggesting the first case. Jon Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366131#comment-14366131 ] Jonathan Eagles commented on TEZ-2205: -- Exactly. In this case, the operations still go through it is only the YarnClient that looks at this flag and doesn't get the delegation token. So tez still tries to putDomain (succeeds) from client and postEntities from AM (fails since there is not a valid delegation token). Again seems like we either need to fall back to simple history or fetch our own delegation token. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366076#comment-14366076 ] Hitesh Shah commented on TEZ-2205: -- Also, firstly yarn.timeline-service.enabled is a YARN config and should not be respected by Tez. Now, if it is set to false, why is the yarn client/timeline client trying to even connect to ATS or trying to obtain the token? Shouldn't that be a bug in YARN? Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false
[ https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366477#comment-14366477 ] Jonathan Eagles commented on TEZ-2205: -- I see the confusion now. Starting with 2.6.1 (see YARN-2375), the behavior changed so that the containing object of the timeline client enforces the enabled flag. Tez still tries to post to ATS when yarn.timeline-service.enabled=false --- Key: TEZ-2205 URL: https://issues.apache.org/jira/browse/TEZ-2205 Project: Apache Tez Issue Type: Sub-task Affects Versions: 0.6.1 Reporter: Chang Li Assignee: Chang Li when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, but hits error as token is not found. Does not fail the job because of the fix to not fail job when there is error posting to ATS. But it should not be trying to post to ATS in the first place. -- This message was sent by Atlassian JIRA (v6.3.4#6332)