[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382623#comment-14382623
 ] 

Hitesh Shah commented on TEZ-2205:
--

Comments: 

Minor nit:

{code}
LOG.warn(ATSLogging service/acl manager is disabled 
 + due to the Timeline Service is disabled);
{code}
Maybe use: atsHistoryLoggingServiceClassName +  is disabled due to Timeline 
Service being disabled, 
+ YarnConfiguration.TIMELINE_SERVICE_ENABLED +  set to false

bq. The changes in ATSHistoryLoggingService could be more optimal. If timeline 
is disabled, why even bother queueing up events?

Previous review comment  still not addressed. 






 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383025#comment-14383025
 ] 

Hitesh Shah commented on TEZ-2205:
--

+1

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382751#comment-14382751
 ] 

Hitesh Shah commented on TEZ-2205:
--

bq. Is there something more beyond this I need to take care?

Yes - should a thread even be started to poll for events? In serviceStart and 
serviceStop, it might be simpler to just see if timeline is disabled either by 
checking client ==null or a different boolean and become no-ops. The service 
started/stopped log message should not even show up if timeline is disabled. 
This should be done in addition to whether events are queued.

TestContainerReuse is not related so you can ignore it. Being addressed in a 
different jira.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382932#comment-14382932
 ] 

Hadoop QA commented on TEZ-2205:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12707633/TEZ-2205.5.patch
  against master revision d42a3c7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.dag.app.rm.TestContainerReuse

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/357//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/357//console

This message is automatically generated.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382905#comment-14382905
 ] 

Chang Li commented on TEZ-2205:
---

[~hitesh] Thanks for finding out those problems. I have updated my patch by 
adding back the check of whether to add the event to the queue. But due to the 
eventQueue is a private variable of ATSHistoryLoggingService class, I can't 
have access to it in the test.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.4.patch, TEZ-2205.5.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382843#comment-14382843
 ] 

Chang Li commented on TEZ-2205:
---

[~hitesh] Thanks for suggestions! Have updated my patch. Does my current patch 
address the issue of queueing events correctly?

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382860#comment-14382860
 ] 

Hitesh Shah commented on TEZ-2205:
--

Comments: 

Minor nit: Assert.assertEquals(historyACLPolicyManager.timelineClient, null) ; 
- use assertNull. 
Also, can you modify the test to check the size of the queue at the end of the 
run should be 0. I am guessing you will catch the issue I mention below by 
adding this check.

bq. Does my current patch address the issue of queueing events correctly?

I think the latest patch accidentally removed the check of whether to add the 
event to the queue. Currently, this will add events into the queue and never 
empty the queue. 

Rest looks good. I think we are getting very close to a final patch. 


 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.4.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-26 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382884#comment-14382884
 ] 

Hadoop QA commented on TEZ-2205:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12707626/TEZ-2205.4.patch
  against master revision d42a3c7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.dag.app.rm.TestContainerReuse

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/356//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/356//console

This message is automatically generated.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.2.patch, TEZ-2205.3.patch, 
 TEZ-2205.4.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-25 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379951#comment-14379951
 ] 

Chang Li commented on TEZ-2205:
---

Thanks a lot for patient discussion and explanations [~hitesh], [~jeagles], 
[~zjshen]! Understand the implementation requirement now, will provide a patch 
soon

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-25 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380268#comment-14380268
 ] 

Hadoop QA commented on TEZ-2205:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12707237/TEZ-2205.1.patch
  against master revision d1b4bd4.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/347//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/347//console

This message is automatically generated.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-25 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380270#comment-14380270
 ] 

Hitesh Shah commented on TEZ-2205:
--

Adding a test for this would be useful. 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-25 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380267#comment-14380267
 ] 

Hitesh Shah commented on TEZ-2205:
--

{code}
LOG.warn(Timeline service is not enabled);
{code}
  - the log should be more clear about the ATSLogging service/acl manager being 
disabled as the timeline service being disabled

The changes in ATSHistoryLoggingService could be more optimal. If it is 
disabled, why even bother queueing up events? 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.1.patch, TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Rohini Palaniswamy (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377562#comment-14377562
 ] 

Rohini Palaniswamy commented on TEZ-2205:
-

I prefer 3). For jobs launched through Oozie it is easy to turn off ATS via 
Oozie server side setting and this might be required in the near future now and 
then considering the issues we are facing with ATS. Since tez-site.xml for 
those jobs come from HDFS, it is not easy to change the tez ATS Logger easily 
(replacing the file on HDFS is more manual and can cause running jobs to fail 
as LocalResource time has changed) and so do not like 1). Also having to change 
multiple settings to turn off something is cumbersome.  2) is what is happening 
now but the problem I see is that it impacts performance as time is wasted 
trying to connect to ATS and failing due to lack of authentication. 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378034#comment-14378034
 ] 

Chang Li commented on TEZ-2205:
---

[~hitesh] So should I implemnt 3) by switching to simplehistory or wrap each 
post call with an if statement or just not to add post events to eventqueue?

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379013#comment-14379013
 ] 

Hitesh Shah commented on TEZ-2205:
--

A log.warn should be sufficient for now. SimpleHistory comes with its own 
problems of configuring an hdfs location and cleaning it on a regular basis. 
Looks like we have agreement. 

[~lichangleo] Does the above clarify the implemenation requirements for this 
jira? Thanks for the patience in handling the various run-arounds with respect 
to the design :)




 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379007#comment-14379007
 ] 

Jonathan Eagles commented on TEZ-2205:
--

I think what zhijie has posted is similar to what I am thinking as well. This 
will give the on/off flag to users and keep the client in control. A log WARN 
should be sufficient to alert clients there is a mismatch in configuration. 
Whether we fall back to SimpleHistory or have a hollow ATS History isn't much 
of a difference to me.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378996#comment-14378996
 ] 

Zhijie Shen commented on TEZ-2205:
--

bq. i.e option 3's impl would be:

LGTM. For your reference, this is what we did in MR:

{code}
if (conf.getBoolean(MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA,
MRJobConfig.DEFAULT_MAPREDUCE_JOB_EMIT_TIMELINE_DATA)) {
  if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {
timelineClient = TimelineClient.createTimelineClient();
timelineClient.init(conf);
LOG.info(Timeline service is enabled);
LOG.info(Emitting job history data to the timeline server is enabled);
  } else {
LOG.info(Timeline service is not enabled);
  }
} else {
  LOG.info(Emitting job history data to the timeline server is not 
enabled);
}
{code}
And only when {{timelineClient != null}}, MR will publish the history info to 
the timeline server.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378199#comment-14378199
 ] 

Hitesh Shah commented on TEZ-2205:
--

bq. For jobs launched through Oozie it is easy to turn off ATS via Oozie server 
side setting

Could you clarify a bit more on how this is being done? 

[~jeagles] [~zjshen] [~lichangleo] If (3) is the approach that works the best 
in Yahoo environments, should the eventual fix be in YARN given that other 
applications will face the same issue i.e. option (2)? i.e. if yarn timeline 
enabled flag is set to false, all the relevant yarn client and timeline client 
libs should automatically not make calls to the server related to timeline ( 
i.e. do not retrieve delegation tokens, treat all calls as no-op) ? 

 






 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-24 Thread Rohini Palaniswamy (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378245#comment-14378245
 ] 

Rohini Palaniswamy commented on TEZ-2205:
-

 OOZIE-2133 is the one that handles getting delegation tokens for ATS for tez 
jobs. If oozie.action.launcher.yarn.timeline-service.enabled is set to true on 
the Oozie server configuration, it adds yarn.timeline-service.enabled=true to 
conf of JobClient that submits the launcher job if the tez-site.xml is part of 
the distributed cache. JobClient (YARN) fetches ATS delegation token if that 
setting is set before the job is submitted and adds it to the job.  

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-23 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377268#comment-14377268
 ] 

Hitesh Shah commented on TEZ-2205:
--

[~rohini] [~hagleitn] any comments/concerns on the approach that we plan to 
take? 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372109#comment-14372109
 ] 

Hadoop QA commented on TEZ-2205:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12706018/TEZ-2205.wip.patch
  against master revision 6e15b2f.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/322//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/322//console

This message is automatically generated.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372045#comment-14372045
 ] 

Hitesh Shah commented on TEZ-2205:
--

[~rohini] At the crux of this, there are effectively a couple of 
issues/questions in terms of what is the logical behavior and what is the 
expectation? The fix for any approach is probably trivial. 

Consider the fact that the user configured both YARN and Tez in a conflicting 
manner. i.e. Configured YARN to disable timeline but made Tez use Timeline. 
Should Tez:

1) error out due to a conflicting configuration i.e YARN timeline disabled but 
Tez ATS logger enabled.
2) should Tez try and use Timeline  (even though YARN flag is set to false ) 
and ignore its failures as needed? This should be ok for the most part except 
that I think there are some cases in YARN which are not handled cleanly and end 
up causing the app to error out. Also, there were some behavioral changes in 
YARN-2375 - see below. 
3) Should Tez look for the YARN configuration property and silently ignore the 
fact that TimelineATSLogger has been configured but it should not be used? 

Also, FWIW, earlier ( before YARN-2375 ), even though Tez invoked 
Timeline::postEntities, if the YARN flag was set to false, the YARN library 
silently dropped the call. 

(2) is probably something that YARN needs to address. 

As for Tez, we can go with either (1) or (3). (1) is more clear-cut in terms of 
making it very clear to the user in terms of how to configure Tez. (3) merely 
hides the fact that something is wrongly configured. Also, to clarify, part of 
this stems from what is the yarn.timeline-service.enabled flag meant to be 
used for? Is it a admin flag to control where timeline is enabled or disabled 
for the whole cluster? It currently is a client-side flag that cannot be 
enforced at all. Furthermore, if it is meant to be used on a per job basis, 
should it then be a tez-specific setting ( which we already have in the form of 
the class setting ).

Last question for [~rohini]: does the issue of disabling ATS stem from the fact 
that it is a bit hard to disable ATS logging via the service class name 
property? 

 








 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371983#comment-14371983
 ] 

Jonathan Eagles commented on TEZ-2205:
--

This seems like a patch that will get the conversation going for what to do for 
this jira. [~rohini], can you comment from a pig perspective. Essentially, we 
are deeming it illegal to specify ATSHistoryLogger when timeline service is 
disabled. This approach gives clients the control of which history logger to 
use but leaves the yarn-timeline-service.enabled a flag that describes a 
feature that is available on the cluster.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371984#comment-14371984
 ] 

Chang Li commented on TEZ-2205:
---

[~hitesh] I post an attempt patch which checks if tez has its logging service 
class set to atsHistoryLoggingService but not enable timelie-service. In that 
case it will throw an exception with alert stopping the tez job from launching.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Rohini Palaniswamy (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372005#comment-14372005
 ] 

Rohini Palaniswamy commented on TEZ-2205:
-

{code}
public void handle(DAGHistoryEvent event) {
eventQueue.add(event);
  }
{code}

It would be a simple check to not add to the queue if timeline service is 
disabled.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372025#comment-14372025
 ] 

Jonathan Eagles commented on TEZ-2205:
--

[~hitesh], do you want to comment on this? [~rohini], there are many ways to 
approach this. We have all be discussing the pros and cons. I wanted to loop 
you into the conversation since you may have better insight as to what users 
can expect.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-20 Thread Rohini Palaniswamy (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371996#comment-14371996
 ] 

Rohini Palaniswamy commented on TEZ-2205:
-

Wouldn't it be better if ATSHistoryLogger checked the value of the timeline 
setting and do nothing if it is false. 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li
 Attachments: TEZ-2205.wip.patch


 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-18 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367561#comment-14367561
 ] 

Hitesh Shah commented on TEZ-2205:
--

Added a comment on YARN-2375. Let us see what happens there :)

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-18 Thread Chang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367518#comment-14367518
 ] 

Chang Li commented on TEZ-2205:
---

Thanks for clarification [~jeagles].  [~hitesh] which way should I proceed to 
solve this problem

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366073#comment-14366073
 ] 

Hitesh Shah commented on TEZ-2205:
--

This seems like a misconfiguration issue. Is there a reason why 
tez.history.logging.service.class is configured to use ATS but ATS itself is 
disabled? 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366111#comment-14366111
 ] 

Hitesh Shah commented on TEZ-2205:
--

[~jeagles] Not exactly but I think I understand the context better now. 

To clarify, if yarn.timeline-service.enabled=false, then should all the YARN 
client libs silently ignore all calls made to them i.e do not pull the 
delegation token, do not create a domain, do not post entities, etc?  

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366150#comment-14366150
 ] 

Hitesh Shah commented on TEZ-2205:
--

That is what I do not understand. If yarn.timeline-service.enabled=false, 
TimelineClient::putDomain and TimelineClient::postEntities should both be a 
no-op ( looked at the code ). 

Is for some reason, the configuration value getting reset when the 
TimelineClient objects are being instantiated? 



 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366103#comment-14366103
 ] 

Jonathan Eagles commented on TEZ-2205:
--

[~hitesh], I think it is correct to say that there is currently a mismatch in 
usage of this flag at least in secure environments. In a secure environment, 
tez relies on the timeline server delegation token retrieved by the 
YarnClientImp (yarn.timeline-service.enabled), in order to postEntities from 
the AM. The client doesn't have this issue (it is able to retrieve its own 
delegation token) while putting the domain unless the job was run from within 
oozie (but that is another story).

So as I understand it, tez either needs to completely ignore 
yarn.timeline-service.enabled and fetch its own timeline server delegation 
token... or it needs to pay attention to this flag and fallback to simple 
history when yarn.timeline-service.enabled=false. Seems like you are suggesting 
the first case.

Jon

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366131#comment-14366131
 ] 

Jonathan Eagles commented on TEZ-2205:
--

Exactly. In this case, the operations still go through it is only the 
YarnClient that looks at this flag and doesn't get the delegation token. So tez 
still tries to putDomain (succeeds) from client and postEntities from AM (fails 
since there is not a valid delegation token). Again seems like we either need 
to fall back to simple history or fetch our own delegation token. 

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Hitesh Shah (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366076#comment-14366076
 ] 

Hitesh Shah commented on TEZ-2205:
--

Also, firstly yarn.timeline-service.enabled is a YARN config and should not be 
respected by Tez. Now, if it is set to false, why is the yarn client/timeline 
client trying to even connect to ATS or trying to obtain the token? Shouldn't 
that be a bug in YARN?  

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (TEZ-2205) Tez still tries to post to ATS when yarn.timeline-service.enabled=false

2015-03-17 Thread Jonathan Eagles (JIRA)

[ 
https://issues.apache.org/jira/browse/TEZ-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366477#comment-14366477
 ] 

Jonathan Eagles commented on TEZ-2205:
--

I see the confusion now. Starting with 2.6.1 (see YARN-2375), the behavior 
changed so that the containing object of the timeline client enforces the 
enabled flag.

 Tez still tries to post to ATS when yarn.timeline-service.enabled=false
 ---

 Key: TEZ-2205
 URL: https://issues.apache.org/jira/browse/TEZ-2205
 Project: Apache Tez
  Issue Type: Sub-task
Affects Versions: 0.6.1
Reporter: Chang Li
Assignee: Chang Li

 when set yarn.timeline-service.enabled=false, Tez still tries posting to ATS, 
 but hits error as token is not found. Does not fail the job because of the 
 fix to not fail job when there is error posting to ATS. But it should not be 
 trying to post to ATS in the first place.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)