[jira] [Commented] (TEZ-3085) In session mode, the credentials passed via the Tez client constructor is not available to all the tasks

2016-03-04 Thread Bikas Saha (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181432#comment-15181432
 ] 

Bikas Saha commented on TEZ-3085:
-

Yes. And looks like its already mentioned in the first comment of this jira.

> In session mode, the credentials passed via the Tez client constructor is not 
> available to all the tasks
> 
>
> Key: TEZ-3085
> URL: https://issues.apache.org/jira/browse/TEZ-3085
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Vinoth Sathappan
>
> The credentials passed through the Tez client constructor isn't available for 
> the tasks in session mode.
> TezClient(String name, TezConfiguration tezConf,
> @Nullable Map localResources,
> @Nullable Credentials credentials)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits

2016-03-04 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181400#comment-15181400
 ] 

TezQA commented on TEZ-3155:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791549/TEZ-3155.3.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.client.TestTezClient

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1546//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1546//console

This message is automatically generated.

> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop 
> ipc limits 
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc 
> limits. One option would be to fall back to local resources if the dag plan 
> is too large. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-3155 PreCommit Build #1546

2016-03-04 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/TEZ-3155
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1546/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4355 lines...]
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :tez-api
[INFO] Build failures were ignored.




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791549/TEZ-3155.3.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.client.TestTezClient

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1546//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1546//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
b1c9b28ee274e127582c13abd4e48b20ecab8cc5 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.tez.client.TestTezClient.testTezclientSession

Error Message:
File already exists: 
file:/tmp/jenkins/tez/staging/.tez/application_0_0001/DAGPlan-1

Stack Trace:
org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: 
file:/tmp/jenkins/tez/staging/.tez/application_0_0001/DAGPlan-1
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:264)
at 
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:257)
at 
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:387)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:428)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:530)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:463)
at 
org.apache.tez.client.TestTezClient.testTezClientSessionWithLargeDAGPlan(TestTezClient.java:173)
at 
org.apache.tez.client.TestTezClient.testTezclientSession(TestTezClient.java:160)

[jira] [Commented] (TEZ-3085) In session mode, the credentials passed via the Tez client constructor is not available to all the tasks

2016-03-04 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181392#comment-15181392
 ] 

Hitesh Shah commented on TEZ-3085:
--

[~vinoths] pointed out TEZ-2886 offline. Is that the jira you are referring to 
[~bikassaha]? 

> In session mode, the credentials passed via the Tez client constructor is not 
> available to all the tasks
> 
>
> Key: TEZ-3085
> URL: https://issues.apache.org/jira/browse/TEZ-3085
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Vinoth Sathappan
>
> The credentials passed through the Tez client constructor isn't available for 
> the tasks in session mode.
> TezClient(String name, TezConfiguration tezConf,
> @Nullable Map localResources,
> @Nullable Credentials credentials)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3085) In session mode, the credentials passed via the Tez client constructor is not available to all the tasks

2016-03-04 Thread Bikas Saha (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181338#comment-15181338
 ] 

Bikas Saha commented on TEZ-3085:
-

IIRC, didn't we recently start passing AM credentials to the DAG?

> In session mode, the credentials passed via the Tez client constructor is not 
> available to all the tasks
> 
>
> Key: TEZ-3085
> URL: https://issues.apache.org/jira/browse/TEZ-3085
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Vinoth Sathappan
>
> The credentials passed through the Tez client constructor isn't available for 
> the tasks in session mode.
> TezClient(String name, TezConfiguration tezConf,
> @Nullable Map localResources,
> @Nullable Credentials credentials)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits

2016-03-04 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180725#comment-15180725
 ] 

Hitesh Shah commented on TEZ-3155:
--

Some general overal comments:
   - patch looks decent for the most part
   - please do not change code lines if they are not needed ( minor edits, new 
lines, etc). Small changes are sometimes ok but numerous modifications make it 
hard for a reviewer. 

Some comments on the patch: 

TezClient
  - why 0.8 for largeDAGPlanRatio ? Why does it need to be so high for a 64 MB 
max limit? Shoudnt it be more like a fixed number to account for the overhead?
  - please consider using DAGPlan.writeTo* to write the output stream instead 
of the objectoutputstream  
  - output stream needs to be closed?
  
{code}
FileSystem fs = dagPlanPath.getFileSystem(dagClientConf);
{code}
  - the above should use amCOnfig.getTezConf() and should be done once per 
TezClient and the fs object re-used. 

{code}
  SubmitDAGRequestProto request = requestBuilder.build();
 SubmitDAGResponseProto response = proxy.submitDAG(null, request);
{code}
  - any reason why this code change was needed? 

  - requestBuilder.build() is getting called twice - should be fixed to be 
called once. 
  - SERIALIZED_DAGPLAN_NAME_PREFIX can be replaced by TEZ_PB_PLAN_BINARY_NAME

{code}
  requestBuilder.setSerializedDagPlanPath(dagPlanPath.toString());
{code}
  - this should be a fully resolved path ( check fs.resolve...) 

TestTezClient:
  - there should be a test for the negative case i.e. one where the dag plan is 
within the defined threshold 

DAGClientServer:
  
{code}
  DAGClientAMProtocolBlockingPBServerImpl service =
DAGClientAMProtocolBlockingPBServerImpl(realInstance, conf);
{code}
  - why was the above needed?

DAGClientAMProtocolBlockingPBServerImpl:
   - it might be better to move this aspect of the code into DAGAppMaster where 
the staging fs object can be reused. 
   - also the stream needs to be closed in a finally block

 


 

> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop 
> ipc limits 
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc 
> limits. One option would be to fall back to local resources if the dag plan 
> is too large. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3145) Reduce message size when empty partitions is high

2016-03-04 Thread Jonathan Eagles (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated TEZ-3145:
-
Attachment: TEZ-3145.SLIM-DME.patch

The SLIM DME idea is to only send the empty partition across in the DME if the 
destination is going to read the source partition.

> Reduce message size when empty partitions is high
> -
>
> Key: TEZ-3145
> URL: https://issues.apache.org/jira/browse/TEZ-3145
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3145.SLIM-DME.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3085) In session mode, the credentials passed via the Tez client constructor is not available to all the tasks

2016-03-04 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180717#comment-15180717
 ] 

Hitesh Shah commented on TEZ-3085:
--

Also, to clarify, these credentials are being passed to the AM but not to the 
tasks as these may have been meant to be the AM credentials. 

> In session mode, the credentials passed via the Tez client constructor is not 
> available to all the tasks
> 
>
> Key: TEZ-3085
> URL: https://issues.apache.org/jira/browse/TEZ-3085
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Vinoth Sathappan
>
> The credentials passed through the Tez client constructor isn't available for 
> the tasks in session mode.
> TezClient(String name, TezConfiguration tezConf,
> @Nullable Map localResources,
> @Nullable Credentials credentials)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits

2016-03-04 Thread Zhiyuan Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated TEZ-3155:
--
Attachment: TEZ-3155.3.patch

> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop 
> ipc limits 
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc 
> limits. One option would be to fall back to local resources if the dag plan 
> is too large. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits

2016-03-04 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180530#comment-15180530
 ] 

TezQA commented on TEZ-3155:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791516/TEZ-3155.2.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 33 javac 
compiler warnings (more than the master's current 31 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Javac warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1545//console

This message is automatically generated.

> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop 
> ipc limits 
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc 
> limits. One option would be to fall back to local resources if the dag plan 
> is too large. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-3155 PreCommit Build #1545

2016-03-04 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/TEZ-3155
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1545/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4569 lines...]
[INFO] 
[INFO] Total time: 56:38 min
[INFO] Finished at: 2016-03-04T21:00:43+00:00
[INFO] Final Memory: 68M/829M
[INFO] 




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791516/TEZ-3155.2.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 33 javac 
compiler warnings (more than the master's current 31 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Javac warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1545//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
02bf14d05a85e1b9418a5bab9efb701d169f1ad3 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Updated] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits

2016-03-04 Thread Zhiyuan Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhiyuan Yang updated TEZ-3155:
--
Attachment: TEZ-3155.2.patch

> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop 
> ipc limits 
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc 
> limits. One option would be to fall back to local resources if the dag plan 
> is too large. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode

2016-03-04 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180363#comment-15180363
 ] 

Hitesh Shah commented on TEZ-1210:
--

Also, please use dev/u...@tez.apache.org for such questions in the future. 

> TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
> -
>
> Key: TEZ-1210
> URL: https://issues.apache.org/jira/browse/TEZ-1210
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Alexander Pivovarov
>  Labels: newbie
> Fix For: 0.5.2
>
> Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch
>
>
> It writes the dagPlan in text form to the same location. Either it should not 
> be invoked in session mode or it should written with a differentiating prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode

2016-03-04 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180359#comment-15180359
 ] 

Hitesh Shah commented on TEZ-1210:
--

You can submit the dag plan via rpc in non-session mode too but from a 
performance view point it is likely to be a bit slower. The client would need 
to submit the app, wait for the AM to come up and then submit the DAG. For 
non-session mode, it is simpler to use local resources as the dag is available 
to be executed as soon as the AM comes up as it is available in its local 
container dir. There are other fault tolerance scenario questions too - what if 
the first AM attempt crashes before the dag could be submitted to it or if the 
cluster is busy and the AM does not come up for quite some time - this would 
require the client process to remain around for a long time to submit the dag 
over rpc. Using local resource in this case allows the client to be a fire and 
forget which is useful for ETL scripts that submit only one DAG. 

> TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
> -
>
> Key: TEZ-1210
> URL: https://issues.apache.org/jira/browse/TEZ-1210
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Alexander Pivovarov
>  Labels: newbie
> Fix For: 0.5.2
>
> Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch
>
>
> It writes the dagPlan in text form to the same location. Either it should not 
> be invoked in session mode or it should written with a differentiating prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-2863 PreCommit Build #1544

2016-03-04 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/TEZ-2863
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1544/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4395 lines...]
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :tez-tests
[INFO] Build failures were ignored.




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791491/TEZ-2863.5.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 33 javac 
compiler warnings (more than the master's current 31 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1544//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1544//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1544//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
244adf7e9ae72d6963e02fc2ee538d89028436e6 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
6 tests failed.
FAILED:  org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit

Error Message:
TezSession has already shutdown. No cluster diagnostics found.

Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No 
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at 
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)


FAILED:  
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices

Error Message:
TezSession has already shutdown. No cluster diagnostics found.

Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No 
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at 
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices(TestFaultTolerance.java:703)


FAILED:  
org.apache.tez.test.TestFaultTolerance.testMultipleInputFailureWithoutExit

Error Message:
TezSession has already shutdown. No cluster diagnostics found.

Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No 
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)

[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch

2016-03-04 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180278#comment-15180278
 ] 

TezQA commented on TEZ-2863:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791491/TEZ-2863.5.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 33 javac 
compiler warnings (more than the master's current 31 warnings).

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1544//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1544//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1544//console

This message is automatically generated.

> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch, 
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch, 
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch, 
> TEZ-2863.3.patch.addendum, TEZ-2863.4-branch-0.7.patch, TEZ-2863.4.patch, 
> TEZ-2863.5-branch-0.7.patch, TEZ-2863.5.patch
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1 
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt 
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id 
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}] 
> |history.HistoryEventHandler|: 
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]: 
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00, 
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread] 
> |rm.YarnTaskSchedulerService|: Allocation request for task: 
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread] 
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager] 
> |rm.YarnTaskSchedulerService|: Assigning container to task: 
> containerId=container_1443711816411_0001_01_02, 
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381, 
> containerPriority= 2, containerResources=, 
> localityMatchType=NodeLocal, matchedLocation=localhost, 
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|: 
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|: 
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0] 
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0] 
> |launcher.ContainerLauncherImpl|: Launching 
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0] 
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0] 
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: 
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback

[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode

2016-03-04 Thread Bikas Saha (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180180#comment-15180180
 ] 

Bikas Saha commented on TEZ-1210:
-

The DAGPlan is downloaded as a local resource to be used to run the DAG. In 
session mode the AM is already running and can accept the DAGPlan over RPC. In 
non-session mode is going to be launched (and there is no connection between it 
and the client) and thus the DAGPlan needs to be provides indirectly as a YARN 
LocalResource via HDFS.

> TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
> -
>
> Key: TEZ-1210
> URL: https://issues.apache.org/jira/browse/TEZ-1210
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Alexander Pivovarov
>  Labels: newbie
> Fix For: 0.5.2
>
> Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch
>
>
> It writes the dagPlan in text form to the same location. Either it should not 
> be invoked in session mode or it should written with a differentiating prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch

2016-03-04 Thread Jonathan Eagles (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated TEZ-2863:
-
Attachment: TEZ-2863.5.patch
TEZ-2863.5-branch-0.7.patch

[~hitesh], [~zjffdu], addressed comments regarding proto conversion. One thing 
to note between the master and branch-0.7 patches is that the proto number 12 
is missing so branch-0.7 keeps the 13,14,15 IDs for those fields.

> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch, 
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch, 
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch, 
> TEZ-2863.3.patch.addendum, TEZ-2863.4-branch-0.7.patch, TEZ-2863.4.patch, 
> TEZ-2863.5-branch-0.7.patch, TEZ-2863.5.patch
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1 
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt 
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id 
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}] 
> |history.HistoryEventHandler|: 
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]: 
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00, 
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread] 
> |rm.YarnTaskSchedulerService|: Allocation request for task: 
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread] 
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread] 
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager] 
> |rm.YarnTaskSchedulerService|: Assigning container to task: 
> containerId=container_1443711816411_0001_01_02, 
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381, 
> containerPriority= 2, containerResources=, 
> localityMatchType=NodeLocal, matchedLocation=localhost, 
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|: 
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|: 
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0] 
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0] 
> |launcher.ContainerLauncherImpl|: Launching 
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0] 
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0] 
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: 
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread] 
> |rm.YarnTaskSchedulerService|: Allocated container 
> completed:container_1443711816411_0001_01_02 last allocated to task: 
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}] 
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02 
> exited with diagnostics set to Container failed, exitCode=1. Exception from 
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1: 
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>   at org.apache.hadoop.util.Shell.run(Shell.java:455)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>   at 
>

[jira] [Updated] (TEZ-3010) Container task acquisition has no retries for errors

2016-03-04 Thread Tsuyoshi Ozawa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated TEZ-3010:

Attachment: TEZ-3010.001.patch

Attaching first patch to add retry semantics to TezChild. This patch is created 
on the patch of TEZ-3009.

> Container task acquisition has no retries for errors
> 
>
> Key: TEZ-3010
> URL: https://issues.apache.org/jira/browse/TEZ-3010
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
> Attachments: TEZ-3010.001.patch
>
>
> There's no retries for errors that occur during task acquisition.  If any 
> error occurs the container will just shut down, resulting in task attempt 
> failures if a task attempt happened to be assigned to the container by the 
> AM.  The container should try harder to obtain the task before giving up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3119) Add missing AM translations in DeprecatedKeys#populateMRToDagParamMap

2016-03-04 Thread Kuhu Shukla (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179989#comment-15179989
 ] 

Kuhu Shukla commented on TEZ-3119:
--

Requesting [~jeagles], [~rohini] for comments/review. Thanks a lot!

> Add missing AM translations in DeprecatedKeys#populateMRToDagParamMap
> -
>
> Key: TEZ-3119
> URL: https://issues.apache.org/jira/browse/TEZ-3119
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.2
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: TEZ-3119.001.patch
>
>
> MRToDagParamMap is missing some of the relevant configs. Some of them include:
> {code}
> TEZ_CREDENTIALS_PATH
> TEZ_AM_LOG_LEVEL
> TEZ_AM_MAX_APP_ATTEMPTS
> TEZ_AM_RESOURCE_MEMORY_MB
> TEZ_AM_RESOURCE_CPU_VCORES
> TEZ_AM_CLIENT_THREAD_COUNT
> TEZ_AM_CLIENT_AM_PORT_RANGE
> TEZ_AM_RM_HEARTBEAT_INTERVAL_MS_MAX
> TASK_HEARTBEAT_TIMEOUT_MS
> TEZ_TASK_AM_HEARTBEAT_INTERVAL_MS
> TEZ_AM_APPLICATION_PRIORITY
> TEZ_AM_VIEW_ACLS
> TEZ_AM_MODIFY_ACLS
> TEZ_CANCEL_DELEGATION_TOKENS_ON_COMPLETION
> TEZ_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT
> TEZ_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT
> TEZ_AM_LEGACY_SPECULATIVE_SLOWTASK_THRESHOLD
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3087) Tez UI 2: Add log links in task & attempt details page

2016-03-04 Thread Sreenath Somarajapuram (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreenath Somarajapuram updated TEZ-3087:

Description: 
Requirement:
- Task attempt details page must have a link to view the attempt's 
inProgressLog file, and a button to download the completedLog on attempt 
completion.
- Task details page have a view link & download button as above to the 
successful attempt if available or the final failed attempt.

View link is currently blocked by YARN-3698.

- Ensure that the functionality is inline with TEZ-3101, and the view link 
protocol (http / https) is taken from yarn.http.policy configuration in 
yarn-site.

PS: The log links were visible in the task and attempts tables of Tez UI 1, but 
wasn't added in Tez UI 2 based on comments.

  was:
Requirement:
- Task attempt details page must have a link to view the attempt's 
inProgressLog file, and a button to download the completedLog on attempt 
completion.
- Task details page have a view link & download button as above to the 
successful attempt if available or the final failed attempt.

View link is currently blocked by YARN-3698.

- Ensure that the functionality is inline with TEZ-3101, and the view link 
protocol (http / https) is taken from yarn.http.policy configuration in 
yarn-site.

PS: The log links were earlier visible in the task and attempts tables, but was 
removed based on user comments.


> Tez UI 2: Add log links in task & attempt details page
> --
>
> Key: TEZ-3087
> URL: https://issues.apache.org/jira/browse/TEZ-3087
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>
> Requirement:
> - Task attempt details page must have a link to view the attempt's 
> inProgressLog file, and a button to download the completedLog on attempt 
> completion.
> - Task details page have a view link & download button as above to the 
> successful attempt if available or the final failed attempt.
> View link is currently blocked by YARN-3698.
> - Ensure that the functionality is inline with TEZ-3101, and the view link 
> protocol (http / https) is taken from yarn.http.policy configuration in 
> yarn-site.
> PS: The log links were visible in the task and attempts tables of Tez UI 1, 
> but wasn't added in Tez UI 2 based on comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3087) Tez UI 2: Add log links in task & attempt details page

2016-03-04 Thread Sreenath Somarajapuram (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179890#comment-15179890
 ] 

Sreenath Somarajapuram commented on TEZ-3087:
-

[~hitesh]
I have updated the description. Hope it is a bit more clear now.

bq. What do you mean by task attempt redirection? Is this for fixing a bug in 
Tez or in Yarn? There are links already to logs for the task attempts - what 
are we missing that needs to be added in the Tez UI?
What I meant was task attempt log redirection. There is a bug in YARN and 
YARN-3698 was filed for the same. 

> Tez UI 2: Add log links in task & attempt details page
> --
>
> Key: TEZ-3087
> URL: https://issues.apache.org/jira/browse/TEZ-3087
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>
> Requirement:
> - Task attempt details page must have a link to view the attempt's 
> inProgressLog file, and a button to download the completedLog on attempt 
> completion.
> - Task details page have a view link & download button as above to the 
> successful attempt if available or the final failed attempt.
> View link is currently blocked by YARN-3698.
> - Ensure that the functionality is inline with TEZ-3101, and the view link 
> protocol (http / https) is taken from yarn.http.policy configuration in 
> yarn-site.
> PS: The log links were earlier visible in the task and attempts tables, but 
> was removed based on user comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3087) Tez UI 2: Add log links in task & attempt details page

2016-03-04 Thread Sreenath Somarajapuram (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreenath Somarajapuram updated TEZ-3087:

Description: 
Requirement:
- Task attempt details page must have a link to view the attempt's 
inProgressLog file, and a button to download the completedLog on attempt 
completion.
- Task details page have a view link & download button as above to the 
successful attempt if available or the final failed attempt.

View link is currently blocked by YARN-3698.

- Ensure that the functionality is inline with TEZ-3101, and the view link 
protocol (http / https) is taken from yarn.http.policy configuration in 
yarn-site.

PS: The log links were earlier visible in the task and attempts tables, but was 
removed based on user comments.

  was:
Requirement:
- Task attempt details page must have a link to view to the attempt's 
inProgressLog file, and a button to download the completedLog on attempt 
completion.
- Task details page have a view link & download button as above to the 
successful attempt if available or the final failed attempt.

View link is currently blocked by 

- Ensure that the functionality is inline with TEZ-3101, and uses the 


> Tez UI 2: Add log links in task & attempt details page
> --
>
> Key: TEZ-3087
> URL: https://issues.apache.org/jira/browse/TEZ-3087
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>
> Requirement:
> - Task attempt details page must have a link to view the attempt's 
> inProgressLog file, and a button to download the completedLog on attempt 
> completion.
> - Task details page have a view link & download button as above to the 
> successful attempt if available or the final failed attempt.
> View link is currently blocked by YARN-3698.
> - Ensure that the functionality is inline with TEZ-3101, and the view link 
> protocol (http / https) is taken from yarn.http.policy configuration in 
> yarn-site.
> PS: The log links were earlier visible in the task and attempts tables, but 
> was removed based on user comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3087) Tez UI 2: Add log links in task & attempt details page

2016-03-04 Thread Sreenath Somarajapuram (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreenath Somarajapuram updated TEZ-3087:

Description: 
Requirement:
- Task attempt details page must have a link to view to the attempt's 
inProgressLog file, and a button to download the completedLog on attempt 
completion.
- Task details page have a view link & download button as above to the 
successful attempt if available or the final failed attempt.

View link is currently blocked by 

- Ensure that the functionality is inline with TEZ-3101, and uses the 

  was:
- UI1 implementation takes you to the container logs and doesn't works as 
expected
- Ensure that the functionality is inline with TEZ-3101


> Tez UI 2: Add log links in task & attempt details page
> --
>
> Key: TEZ-3087
> URL: https://issues.apache.org/jira/browse/TEZ-3087
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>
> Requirement:
> - Task attempt details page must have a link to view to the attempt's 
> inProgressLog file, and a button to download the completedLog on attempt 
> completion.
> - Task details page have a view link & download button as above to the 
> successful attempt if available or the final failed attempt.
> View link is currently blocked by 
> - Ensure that the functionality is inline with TEZ-3101, and uses the 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3087) Tez UI 2: Add log links in task & attempt details page

2016-03-04 Thread Sreenath Somarajapuram (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sreenath Somarajapuram updated TEZ-3087:

Summary: Tez UI 2: Add log links in task & attempt details page  (was: Tez 
UI 2: Log links must be added in task & attempt details page)

> Tez UI 2: Add log links in task & attempt details page
> --
>
> Key: TEZ-3087
> URL: https://issues.apache.org/jira/browse/TEZ-3087
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>
> - UI1 implementation takes you to the container logs and doesn't works as 
> expected
> - Ensure that the functionality is inline with TEZ-3101



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode

2016-03-04 Thread Mai H. Dinh (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179837#comment-15179837
 ] 

Mai H. Dinh commented on TEZ-1210:
--

I am not sure this is the best place for this question but I cannot find a 
better place.

In non-session mode, what is the advantage of writing DAGPlan to hadoop file 
system instead of sending via ProtobufRPC as in session mode?

> TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
> -
>
> Key: TEZ-1210
> URL: https://issues.apache.org/jira/browse/TEZ-1210
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Alexander Pivovarov
>  Labels: newbie
> Fix For: 0.5.2
>
> Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch
>
>
> It writes the dagPlan in text form to the same location. Either it should not 
> be invoked in session mode or it should written with a differentiating prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt

2016-03-04 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179645#comment-15179645
 ] 

TezQA commented on TEZ-3148:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791425/TEZ-3148.001.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1543//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1543//console

This message is automatically generated.

> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> 
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3148.001.patch, am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG. 
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT, 
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1, 
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, 
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt 
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE 
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main" 
> java.lang.AbstractMethodError: 
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at 
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at 
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to 
> reproduce this at 10 TB scale.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-3148 PreCommit Build #1543

2016-03-04 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/TEZ-3148
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1543/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4559 lines...]
[INFO] 
[INFO] Total time: 58:36 min
[INFO] Finished at: 2016-03-04T09:53:53+00:00
[INFO] Final Memory: 73M/1286M
[INFO] 




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791425/TEZ-3148.001.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1543//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1543//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
c115719bf38d343a8469224e39aa6dcf51fdc76d logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

Failed: TEZ-2954 PreCommit Build #1542

2016-03-04 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/TEZ-2954
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1542/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4397 lines...]
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :tez-dag
[INFO] Build failures were ignored.




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791422/TEZ-2954.001.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance
  org.apache.tez.dag.app.dag.impl.TestDAGImpl

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1542//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1542//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1542//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
872c648da40ba2bf62b9f42f17b7b9b71b8c8b4d logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
7 tests failed.
FAILED:  org.apache.tez.dag.app.dag.impl.TestDAGImpl.testCounterLimits

Error Message:
expected: but was:

Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.tez.dag.app.dag.impl.TestDAGImpl.testCounterLimits(TestDAGImpl.java:2290)


FAILED:  org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit

Error Message:
TezSession has already shutdown. No cluster diagnostics found.

Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No 
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at 
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)


FAILED:  
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices

Error Message:
TezSession has already shutdown. No cluster diagnostics found.

Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No 
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at 
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at 
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices(TestFaultTolerance.java:703)

[jira] [Commented] (TEZ-2954) Container launch timeouts should count towards node blacklisting

2016-03-04 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179606#comment-15179606
 ] 

TezQA commented on TEZ-2954:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12791422/TEZ-2954.001.patch
  against master revision 91e24d7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance
  org.apache.tez.dag.app.dag.impl.TestDAGImpl

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1542//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1542//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1542//console

This message is automatically generated.

> Container launch timeouts should count towards node blacklisting
> 
>
> Key: TEZ-2954
> URL: https://issues.apache.org/jira/browse/TEZ-2954
> Project: Apache Tez
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2954.001.patch
>
>
> Currently, only task failures count towards blacklisting. A container timing 
> out should do the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt

2016-03-04 Thread Tsuyoshi Ozawa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated TEZ-3148:

Attachment: TEZ-3148.001.patch

Adding transition after starting task attempt to handle TA_TEZ_EVENT_UPDATE 
without exception. I think this can happen when network delays.

> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> 
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
>  Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: TEZ-3148.001.patch, am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG. 
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT, 
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1, 
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, 
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt 
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE 
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main" 
> java.lang.AbstractMethodError: 
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at 
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at 
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to 
> reproduce this at 10 TB scale.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)