date:20150501

Jira: https://issues.apache.org/jira/browse/TEZ-776
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/605/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2800 lines...]
[INFO] Final Memory: 74M/945M
[INFO] 




{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729843/TEZ-776.10.patch
  against master revision 1a53175.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/605//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/605//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
48a3051d99230a97dc15e368846e24a664edfb9c logged out


==
==
Finished build.
==
==


Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #604
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2586539 bytes
Compression is 7.1%
Took 0.75 sec
Description set: TEZ-776
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Resolved] (TEZ-2399) Tez UI: add proper dependencies for computed properties


 [ 
https://issues.apache.org/jira/browse/TEZ-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran resolved TEZ-2399.
---
Resolution: Fixed

commited to branch 0.6 thanks [~Sreenath]

 Tez UI: add proper dependencies for computed properties
 ---

 Key: TEZ-2399
 URL: https://issues.apache.org/jira/browse/TEZ-2399
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Affects Versions: 0.6.1
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
Priority: Critical
 Attachments: TEZ-2399.1.patch


 some of the computed properties are missing the dependencies, this might 
 cause the ui to cache portions of the page at times between pages. 
 this is a branch 0.6.1 only patch (patch wont apply cleanly on 0.7.0)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2360) per-io counters flag should generate both overall and per-edge counters


 [ 
https://issues.apache.org/jira/browse/TEZ-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran updated TEZ-2360:
--
Fix Version/s: 0.7.0

 per-io counters flag should generate both overall and per-edge counters 
 

 Key: TEZ-2360
 URL: https://issues.apache.org/jira/browse/TEZ-2360
 Project: Apache Tez
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Prakash Ramachandran
 Fix For: 0.7.0

 Attachments: TEZ-2360.1.patch, TEZ-2360.2.patch, TEZ-2360.3.patch, 
 TEZ-2360.4.patch, TEZ-2360.5.patch


 Currently, the per-io flag disables overall per task counters and retains 
 only per edge counters. It would be useful to have both overall and per edge 
 counters. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


 [ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran updated TEZ-2395:
--
Fix Version/s: 0.7.0

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Fix For: 0.7.0

 Attachments: TEZ-2395.1.patch, TEZ-2395.2.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2386) Tez UI: Inconsistent usage of icon colors


 [ 
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran updated TEZ-2386:
--
Attachment: TEZ-2386.3.patch

thanks [~Sreenath] I have attached the patch with the changes. 
additionally I have added proper dependencies on computed properties where they 
were missing. 
can you take a look?

 Tez UI: Inconsistent usage of icon colors
 -

 Key: TEZ-2386
 URL: https://issues.apache.org/jira/browse/TEZ-2386
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.3.patch, 
 TEZ-2386.wip.1.patch


 if there's failed attempts in a DAG, and it succeeds - an orange icon shows 
 up on the DAG page. This is very useful to identify DAGs which may need some 
 debugging.
 However, the color is Green for Vertex / Task views after this - so it's 
 difficult to know which one actually had problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-776) Reduce AM mem usage caused by storing TezEvents


 [ 
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated TEZ-776:
---
Attachment: TEZ-776.8.patch

Patch addresses review comments and Jenkins.

 Reduce AM mem usage caused by storing TezEvents
 ---

 Key: TEZ-776
 URL: https://issues.apache.org/jira/browse/TEZ-776
 Project: Apache Tez
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
 Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch, 
 TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch, 
 TEZ-776.7.patch, TEZ-776.8.patch, TEZ-776.ondemand.1.patch, 
 TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, 
 TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, 
 TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png, 
 With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png, 
 events-problem-solutions.txt, with_patch_jmc_output_of_AM.png, 
 without_patch_jmc_output_of_AM.png


 This is open ended at the moment.
 A fair chunk of the AM heap is taken up by TezEvents (specifically 
 DataMovementEvents - 64 bytes per event).
 Depending on the connection pattern - this puts limits on the number of tasks 
 that can be processed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-776 PreCommit Build #601

Jira: https://issues.apache.org/jira/browse/TEZ-776
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/601/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2558 lines...]




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729787/TEZ-776.8.patch
  against master revision 1dd5f9f.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestAMRecovery

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/601//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/601//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
84ce1295920160eafea3d40bd17b01b245b1d35e logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #600
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2541390 bytes
Compression is 7.2%
Took 1.7 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
2 tests failed.
REGRESSION:  
org.apache.tez.test.TestAMRecovery.testVertexPartiallyFinished_Broadcast

Error Message:
expected:SUCCEEDED but was:FAILED

Stack Trace:
java.lang.AssertionError: expected:SUCCEEDED but was:FAILED
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.tez.test.TestAMRecovery.runDAGAndVerify(TestAMRecovery.java:412)
at 
org.apache.tez.test.TestAMRecovery.testVertexPartiallyFinished_Broadcast(TestAMRecovery.java:206)


REGRESSION:  
org.apache.tez.test.TestAMRecovery.testVertexCompletelyFinished_Broadcast

Error Message:
expected:SUCCEEDED but was:FAILED

Stack Trace:
java.lang.AssertionError: expected:SUCCEEDED but was:FAILED
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.tez.test.TestAMRecovery.runDAGAndVerify(TestAMRecovery.java:412)
at 
org.apache.tez.test.TestAMRecovery.testVertexCompletelyFinished_Broadcast(TestAMRecovery.java:237)

[jira] [Commented] (TEZ-2393) Tez pickup PATH env from gateway machine


[ 
https://issues.apache.org/jira/browse/TEZ-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523423#comment-14523423
 ] 

Hitesh Shah commented on TEZ-2393:
--

Thought of a couple of different problems. Consider: 

{code}
foo=A
bar=B
abc=$def:$bar
def=D

echo abc=$abc
{code}

This will print abc=:B

To get it to print D:B, I think we might need to run 2 passes. 

Also, from a YARN point of view, we need to ensure that all user provided env 
vars are set up after all the framework provided values to ensure that any user 
env var that references a framework based env will be setup properly. I don't 
believe this ordering is being done today but will need to check and confirm.   
\cc [~vinodkv] 

 Tez pickup PATH env from gateway machine
 

 Key: TEZ-2393
 URL: https://issues.apache.org/jira/browse/TEZ-2393
 Project: Apache Tez
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Hitesh Shah
 Attachments: TEZ-2393.1.patch


 I found this issue on Windows. When I do:
 set PATH=C:\dummy;%PATH%
 Then run a tez job. C:\dummy appears in PATH of the vertex container. This 
 is surprising since we don't expect frontend PATH will propagate to backend.
 [~hitesh] tried it on Linux and found the same behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-776) Reduce AM mem usage caused by storing TezEvents


 [ 
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated TEZ-776:
---
Attachment: TEZ-776.8.patch

 Reduce AM mem usage caused by storing TezEvents
 ---

 Key: TEZ-776
 URL: https://issues.apache.org/jira/browse/TEZ-776
 Project: Apache Tez
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
 Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch, 
 TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch, 
 TEZ-776.7.patch, TEZ-776.8.patch, TEZ-776.ondemand.1.patch, 
 TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, 
 TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, 
 TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png, 
 With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png, 
 events-problem-solutions.txt, with_patch_jmc_output_of_AM.png, 
 without_patch_jmc_output_of_AM.png


 This is open ended at the moment.
 A fair chunk of the AM heap is taken up by TezEvents (specifically 
 DataMovementEvents - 64 bytes per event).
 Depending on the connection pattern - this puts limits on the number of tasks 
 that can be processed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-776) Reduce AM mem usage caused by storing TezEvents


 [ 
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated TEZ-776:
---
Attachment: (was: TEZ-776.8.patch)

 Reduce AM mem usage caused by storing TezEvents
 ---

 Key: TEZ-776
 URL: https://issues.apache.org/jira/browse/TEZ-776
 Project: Apache Tez
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
 Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch, 
 TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch, 
 TEZ-776.7.patch, TEZ-776.8.patch, TEZ-776.ondemand.1.patch, 
 TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, 
 TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, 
 TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png, 
 With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png, 
 events-problem-solutions.txt, with_patch_jmc_output_of_AM.png, 
 without_patch_jmc_output_of_AM.png


 This is open ended at the moment.
 A fair chunk of the AM heap is taken up by TezEvents (specifically 
 DataMovementEvents - 64 bytes per event).
 Depending on the connection pattern - this puts limits on the number of tasks 
 that can be processed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher

[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523499#comment-14523499
]

Bikas Saha commented on TEZ-1897:
-

bq. AsyncDispatcherConcurrent(String name, int numThreads) { super(name)
instead of super(dispatcher)
bq. final LinkedBlockingQueueEvent queue;; - Double ;
bq. In AsyncDispatcher - the error checking code for previously registered
dispatchers
bq. aitForDrained.wait(1000);, LO
Fixed

bq. At the same place - do the threads need to
Cannot interrupt executorservice. After waiting shutdownNow is called which
should take care of this. In any case, drained code patch is not used and we
should probably remove it separately.

bq. serviceStop / serviceStart don't need to invoke super.
Thats the way other CompositeServices are written and seems to be the correct
thing to do since this method overrides super.serviceStart().

bq. here's a lot of code duplication between AsyncDispatcher and
AsyncDispatcherConcu
I chose to do so, so that the concurrent version can replace the existing
version in a future release. The concurrent version with thread size set to 1
is the same as the existing version. Manual thread creation is replaced by
using a single thread in the threadpool. So there is no need to maintain 2
classes.

thanks for the reviews. Unless there are further comments, will commit by EOD.

Create a concurrent version of AsyncDispatcher
--

Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch

Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2393) Tez pickup PATH env from gateway machine


[ 
https://issues.apache.org/jira/browse/TEZ-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523412#comment-14523412
 ] 

Hitesh Shah commented on TEZ-2393:
--

[~rohini] Mind taking a look to see how this affects your environment. This is 
a bit of a behavior change but the general impression is that the original 
behavior was actually a bug. 



 Tez pickup PATH env from gateway machine
 

 Key: TEZ-2393
 URL: https://issues.apache.org/jira/browse/TEZ-2393
 Project: Apache Tez
  Issue Type: Bug
Reporter: Daniel Dai
Assignee: Hitesh Shah
 Attachments: TEZ-2393.1.patch


 I found this issue on Windows. When I do:
 set PATH=C:\dummy;%PATH%
 Then run a tez job. C:\dummy appears in PATH of the vertex container. This 
 is surprising since we don't expect frontend PATH will propagate to backend.
 [~hitesh] tried it on Linux and found the same behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents


[ 
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523525#comment-14523525
 ] 

TezQA commented on TEZ-776:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729787/TEZ-776.8.patch
  against master revision 1dd5f9f.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 8 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestAMRecovery

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/601//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/601//console

This message is automatically generated.

 Reduce AM mem usage caused by storing TezEvents
 ---

 Key: TEZ-776
 URL: https://issues.apache.org/jira/browse/TEZ-776
 Project: Apache Tez
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
 Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch, 
 TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch, 
 TEZ-776.7.patch, TEZ-776.8.patch, TEZ-776.ondemand.1.patch, 
 TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, 
 TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, 
 TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png, 
 With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png, 
 events-problem-solutions.txt, with_patch_jmc_output_of_AM.png, 
 without_patch_jmc_output_of_AM.png


 This is open ended at the moment.
 A fair chunk of the AM heap is taken up by TezEvents (specifically 
 DataMovementEvents - 64 bytes per event).
 Depending on the connection pattern - this puts limits on the number of tasks 
 that can be processed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2397) Translation of LocalResources via Tez plan serialization can be lossy


[ 
https://issues.apache.org/jira/browse/TEZ-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523647#comment-14523647
 ] 

Hitesh Shah commented on TEZ-2397:
--

Committing shortly. 

 Translation of LocalResources via Tez plan serialization can be lossy
 -

 Key: TEZ-2397
 URL: https://issues.apache.org/jira/browse/TEZ-2397
 Project: Apache Tez
  Issue Type: Bug
Affects Versions: 0.4.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical
 Attachments: TEZ-2397.1.txt


 Happens when there's no port information. The way we serialize a YarnURL into 
 a string causes the reconstructed path to include the port as -1, which is an 
 invalid URL. Path/URL reconstruction from this causes the hostname to be lost.
 This is problematic on clusters running HDFA HA - since there's no host:port 
 information, only a service name. I'd imaging it'll be a problem for viewfs 
 as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2379) org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: T_ATTEMPT_KILLED at KILLED


[ 
https://issues.apache.org/jira/browse/TEZ-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523666#comment-14523666
 ] 

Bikas Saha commented on TEZ-2379:
-

Please see analysis above. It should actually be invalid for an attempt killed 
to come after the task state is killed because the task is supposed to wait for 
attempts to be complete before entering a final state - thats the whole point 
of the kill_wait state, right?
IMO, the fix is to have the attempt ignore a kill request if its already done. 
Thoughts?

 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 T_ATTEMPT_KILLED at KILLED
 --

 Key: TEZ-2379
 URL: https://issues.apache.org/jira/browse/TEZ-2379
 Project: Apache Tez
  Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Hitesh Shah
Priority: Blocker
 Attachments: TEZ-2379.1.patch


 {noformat}
 2015-04-28 04:49:32,455 ERROR [Dispatcher thread: Central] impl.TaskImpl: 
 Can't handle this event at current state for 
 task_1429683757595_0479_1_03_13
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 T_ATTEMPT_KILLED at KILLED
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
 at 
 org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
 at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:853)
 at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:106)
 at 
 org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1874)
 at 
 org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1860)
 at 
 org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
 at 
 org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:113)
 at java.lang.Thread.run(Thread.java:745)
 {noformat}
 Additional notes:
 
 Hive - latest build 
 Tez - master
 tpch-200 gb scale q_17 (kill the job in the middle of execution)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (TEZ-2400) MiniTezCluster needs to be cleaned up to remove MR dependencies


 [ 
https://issues.apache.org/jira/browse/TEZ-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah moved YARN-2860 to TEZ-2400:


Key: TEZ-2400  (was: YARN-2860)
Project: Apache Tez  (was: Hadoop YARN)

 MiniTezCluster needs to be cleaned up to remove MR dependencies
 ---

 Key: TEZ-2400
 URL: https://issues.apache.org/jira/browse/TEZ-2400
 Project: Apache Tez
  Issue Type: Bug
Reporter: Hitesh Shah

 There is a lot of old MR related code which is likely no longer need. 
 For example, staging dir and job history utils. May be needed for testing MR 
 compatibility but that could be handled explicitly only in cases where MR 
 jobs are being run. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2379) org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: T_ATTEMPT_KILLED at KILLED


 [ 
https://issues.apache.org/jira/browse/TEZ-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated TEZ-2379:
-
Attachment: TEZ-2379.1.patch

 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 T_ATTEMPT_KILLED at KILLED
 --

 Key: TEZ-2379
 URL: https://issues.apache.org/jira/browse/TEZ-2379
 Project: Apache Tez
  Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Hitesh Shah
Priority: Blocker
 Attachments: TEZ-2379.1.patch


 {noformat}
 2015-04-28 04:49:32,455 ERROR [Dispatcher thread: Central] impl.TaskImpl: 
 Can't handle this event at current state for 
 task_1429683757595_0479_1_03_13
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 T_ATTEMPT_KILLED at KILLED
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
 at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
 at 
 org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
 at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:853)
 at org.apache.tez.dag.app.dag.impl.TaskImpl.handle(TaskImpl.java:106)
 at 
 org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1874)
 at 
 org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher.handle(DAGAppMaster.java:1860)
 at 
 org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
 at 
 org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:113)
 at java.lang.Thread.run(Thread.java:745)
 {noformat}
 Additional notes:
 
 Hive - latest build 
 Tez - master
 tpch-200 gb scale q_17 (kill the job in the middle of execution)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher


[ 
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523641#comment-14523641
 ] 

Bikas Saha commented on TEZ-1897:
-

From what I see - AbstractService#start()-serviceStart()-end up calling 
AsyncDispatcher-serviceStart() since it overrides 
CompositeService#serviceStart(). So the derived class must call its parents 
serviceStart(). MRAppMaster is also written that way.
Thanks! We can remove this draining code and other unused code when we remove 
AsyncDispatcher.

 Create a concurrent version of AsyncDispatcher
 --

 Key: TEZ-1897
 URL: https://issues.apache.org/jira/browse/TEZ-1897
 Project: Apache Tez
  Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, 
 TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch


 Currently, it processes events on a single thread. For events that can be 
 executed in parallel, e.g. vertex manager events, allowing higher concurrency 
 may be beneficial.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher


[ 
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523659#comment-14523659
 ] 

TezQA commented on TEZ-1897:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729796/TEZ-1897.7.patch
  against master revision c96eed3.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/602//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/602//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/602//console

This message is automatically generated.

 Create a concurrent version of AsyncDispatcher
 --

 Key: TEZ-1897
 URL: https://issues.apache.org/jira/browse/TEZ-1897
 Project: Apache Tez
  Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, 
 TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch


 Currently, it processes events on a single thread. For events that can be 
 executed in parallel, e.g. vertex manager events, allowing higher concurrency 
 may be beneficial.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents

2015-05-01 Thread Siddharth Seth (JIRA)

[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523656#comment-14523656
]

Siddharth Seth commented on TEZ-776:

bq. 1) The patch in TEZ-2255 is doing e2e Composite event routing (design 1 in
the original design document). So its not creating new DataMovement event
objects in the AM. My profiling shows that new object creation is the biggest
CPU culprit in this code path.
2) The patch in TEZ-2255 is a POC patch while the patch here is taking care of
all cases. A quick look shows at TEZ-2255 shows potential short cuts. Even
though the design in TEZ-2255 envisages the creation of a RoutedEvent the patch
is currently just modifying the CompositeEvent in place with the target index
(which may not be theoretically correct). New object creation eats CPU.
Similarly, the target index is being set in the task by using the tasks id
which is not a real solution (apart from other things it breaks auto-reduce).
It is likely that a full implementation will use more CPU than the currently
attached patch on TEZ-2255.

The entire intent of Option2/3 in the first few comments on this jira is to
allow for Edges to handle events more efficiently. Part of that is storage, the
other aspect is avoiding unnecessary explosions of events.
Setting indices on CompositeEvents in TEZ-2255 makes no difference - the
alternate is to set them in RoutedEvent which is just as efficient. In fact,
the original patch there did exactly this, and avoided duplicate serialization
/ deserialization of events. That gives an even bigger gain - but would not be
an equivalent comparison since ODR does not do that - which is why I removed it.

OneToOne clearly shows that event creation is not the only factor which
contributes to performance. There's just 50K DMEs being created there - the
cost comes from the additional invocations. The same applies to ScatterGather
as well - routing the same event multiple times over contributes to the high
utilization. Avoiding DME creation will help - not to the extent of 40% though.
Running the tasks through in the OneToOne case will make it worse. This runs
with a UnorderedInput - which will complete the moment an event is available.
For a large number of tasks - 50K events aren't scanned in the AM for the
specific benchmark. For a longer running task - all events will be scanned and
will contributed more to the CPU numbers.

bq. However, the numbers are useful because they show how much gain can be
expected to be made after doing e2e composite event routing. I have not done
that in this patch since it increases the scope of work but I will do that as a
follow up since the API allows for it.
Explained above why EtoE composite routing is not what is affecting CPU. Please
hold off on writing this patch - it'll likely be different based on TEZ-2255.

Reduce AM mem usage caused by storing TezEvents
---

Key: TEZ-776
URL: https://issues.apache.org/jira/browse/TEZ-776
Project: Apache Tez
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
TEZ-776.4.patch, TEZ-776.5.patch, TEZ-776.6.A.patch, TEZ-776.6.B.patch,
TEZ-776.7.patch, TEZ-776.8.patch, TEZ-776.9.patch, TEZ-776.ondemand.1.patch,
TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch,
TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch,
TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png,
With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png,
events-problem-solutions.txt, with_patch_jmc_output_of_AM.png,
without_patch_jmc_output_of_AM.png

This is open ended at the moment.
A fair chunk of the AM heap is taken up by TezEvents (specifically
DataMovementEvents - 64 bytes per event).
Depending on the connection pattern - this puts limits on the number of tasks
that can be processed.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-1897 PreCommit Build #602

Jira: https://issues.apache.org/jira/browse/TEZ-1897
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/602/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2802 lines...]




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729796/TEZ-1897.7.patch
  against master revision c96eed3.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 7 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/602//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-TEZ-Build/602//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/602//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
54c5615a0dd09c14bd2cd91beae395ba6791a837 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #600
Archived 45 artifacts
Archive block size is 32768
Received 6 blocks and 2564376 bytes
Compression is 7.1%
Took 1.8 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (TEZ-2231) Create project by-laws


[ 
https://issues.apache.org/jira/browse/TEZ-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523535#comment-14523535
 ] 

Hitesh Shah commented on TEZ-2231:
--

Thanks for the review folks. Committing shortly. 

 Create project by-laws
 --

 Key: TEZ-2231
 URL: https://issues.apache.org/jira/browse/TEZ-2231
 Project: Apache Tez
  Issue Type: Task
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Attachments: TEZ-2231.4.patch, by-laws.2.patch, by-laws.3.patch, 
 by-laws.patch


 Define the Project by-laws.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2392) Have all readers throw an Exception on incorrect next() usage


[ 
https://issues.apache.org/jira/browse/TEZ-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523638#comment-14523638
 ] 

Hitesh Shah commented on TEZ-2392:
--

From a thread safety point of view, is completedProcessing protected 
properly? 

 Have all readers throw an Exception on incorrect next() usage
 -

 Key: TEZ-2392
 URL: https://issues.apache.org/jira/browse/TEZ-2392
 Project: Apache Tez
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Rajesh Balamohan
Priority: Critical
 Attachments: TEZ-2392.1.patch, TEZ-2392.2.patch


 Follow up from TEZ-2348.
 Marking as critical since this is a behaviour change, and we should get it in 
 early.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-1897) Create a concurrent version of AsyncDispatcher


 [ 
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated TEZ-1897:

Attachment: TEZ-1897.7.patch

 Create a concurrent version of AsyncDispatcher
 --

 Key: TEZ-1897
 URL: https://issues.apache.org/jira/browse/TEZ-1897
 Project: Apache Tez
  Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, 
 TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.6.patch, TEZ-1897.7.patch


 Currently, it processes events on a single thread. For events that can be 
 executed in parallel, e.g. vertex manager events, allowing higher concurrency 
 may be beneficial.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TEZ-2231) Create project by-laws


 [ 
https://issues.apache.org/jira/browse/TEZ-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah resolved TEZ-2231.
--
   Resolution: Fixed
Fix Version/s: 0.7.0

 Create project by-laws
 --

 Key: TEZ-2231
 URL: https://issues.apache.org/jira/browse/TEZ-2231
 Project: Apache Tez
  Issue Type: Task
Reporter: Hitesh Shah
Assignee: Hitesh Shah
 Fix For: 0.7.0

 Attachments: TEZ-2231.4.patch, by-laws.2.patch, by-laws.3.patch, 
 by-laws.patch


 Define the Project by-laws.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2392) Have all readers throw an Exception on incorrect next() usage


[ 
https://issues.apache.org/jira/browse/TEZ-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523636#comment-14523636
 ] 

Hitesh Shah commented on TEZ-2392:
--

Not sure I understand this change: 

{code}
boolean hasNext = recordReader.next(key, value);
 if (hasNext) {
  inputRecordCounter.increment(1);  
 } else {
  hasCompletedProcessing();
  completedProcessing = true;
 }
{code}

Why is the hasCompletedProcessing check being done after calling 
recordReader.next() ? Shouldn't be called before? i.e. 

{code} 
hasCompletedProcessing();
boolean hasNext = recordReader.next(key, value);
 if (hasNext) {
  inputRecordCounter.increment(1);  
 } else {
  completedProcessing = true;
 }
{code}


 Have all readers throw an Exception on incorrect next() usage
 -

 Key: TEZ-2392
 URL: https://issues.apache.org/jira/browse/TEZ-2392
 Project: Apache Tez
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Rajesh Balamohan
Priority: Critical
 Attachments: TEZ-2392.1.patch, TEZ-2392.2.patch


 Follow up from TEZ-2348.
 Marking as critical since this is a behaviour change, and we should get it in 
 early.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents

2015-05-01 Thread Siddharth Seth (JIRA)

[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523740#comment-14523740
]

Siddharth Seth commented on TEZ-776:

bq. The code is being optimistic here and its explained in the comments.
Nothing is deleted from this list and only one place for additions. So there is
no concurrent modification issues. Its ok to read the size outside the lock
since we will get a snapshot number to events to look at. Taking a lock is not
going to give any better guarantees about seeing the latest event being added.
I'm not very sure about this. Adding an element elsewhere changes the
underlying structure of the ArrayList. Accessing the size() and elements can be
problematic.
bq. This would need to change the index based lookup from a task to change to
multiple indices instead of single index. That is a change I plan to make in a
follow up jira. This would enable separating edge routing from non-edge routing
or separate routing for different edges. Though I don't want to support the
existing legacy routing in tandem with ODR. Unnecessary complexity and ODR
offloads event processing from the central dispatcher which is a good benefit
by itself.
bq. That is not optimal. In the average small job case, the CPU is not very
relevant. However, for a large vertex with mixed edges we should not fall back
to legacy routing because the broadcast is legacy. In addition it take routing
pressure of the central dispatcher which helps get more useful work done faster
on the central dispatcher. In various scenarios the central dispatcher has
often shown up as a bottleneck. A lot of the CPU overhead will go away once we
stop creating new event objects in the AM. Thats another follow up jira to this
one.
We shouldn't fall back to regular routing just because a single plugin does not
implement ODR. That's the intent of allowing a vertex to work with both. I
maintain that it's a simple change to allow both to run at the same time. Just
add events to the taskEvents list depending on whether the edge uses ODR. The
index lookups should remain the same.
One of the big reasons to do this is to avoid the unnecessary regression in
performance for OneToOne edges that this causes. Broadcast will regress as well
- but to a smaller extent. There's no reason to introduce a regression when it
can be avoided easily.

bq. Partially implementing the API will not work since this enables on demand
routing for all types of events. Like I said, the point of this check is not
identify ODR plugins but to identify legacy plugins. This is only there to
allow older versions of Hive to run with newer versions of Tez transparently.
This check is practically sufficient.
Sufficient only if everyone implements all methods. A separate interface makes
this a lot cleaner, and allows for better evolution in the future. I don't
think ODR is set in stone - and will likely evolve. It helps the ScatterGather
case - so should go in now, but will evolve in future releases. Keeping it as a
separate interface helps with this.

On the B patch, will take a look to see what's changed - but it seems like a
last minute change after the offline discussion on TEZ-2255. Definitely needs
additional review. One quick note on this is to expose the actual event - in
case the user does want to do something with it (not advised).

Reduce AM mem usage caused by storing TezEvents
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


 [ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran resolved TEZ-2395.
---
Resolution: Fixed

thanks [~Sreenath] committed to master

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2395.1.patch, TEZ-2395.2.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


[ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522928#comment-14522928
 ] 

Sreenath Somarajapuram commented on TEZ-2395:
-

Closing td tag is missing.
 td
   {{failedTasks}}
-/td
Other than that looks good.

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2395.1.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2399) Tez UI: add proper dependencies for computed properties


[ 
https://issues.apache.org/jira/browse/TEZ-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523001#comment-14523001
 ] 

Sreenath Somarajapuram commented on TEZ-2399:
-

+1 LGTM

 Tez UI: add proper dependencies for computed properties
 ---

 Key: TEZ-2399
 URL: https://issues.apache.org/jira/browse/TEZ-2399
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Affects Versions: 0.6.1
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
Priority: Critical
 Attachments: TEZ-2399.1.patch


 some of the computed properties are missing the dependencies, this might 
 cause the ui to cache portions of the page at times between pages. 
 this is a branch 0.6.1 only patch (patch wont apply cleanly on 0.7.0)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors


[ 
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522914#comment-14522914
 ] 

TezQA commented on TEZ-2386:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729723/TEZ-2386.3.patch
  against master revision a02a5ea.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/597//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/597//console

This message is automatically generated.

 Tez UI: Inconsistent usage of icon colors
 -

 Key: TEZ-2386
 URL: https://issues.apache.org/jira/browse/TEZ-2386
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.3.patch, 
 TEZ-2386.wip.1.patch


 if there's failed attempts in a DAG, and it succeeds - an orange icon shows 
 up on the DAG page. This is very useful to identify DAGs which may need some 
 debugging.
 However, the color is Green for Vertex / Task views after this - so it's 
 difficult to know which one actually had problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2392) Have all readers throw an Exception on incorrect next() usage


[ 
https://issues.apache.org/jira/browse/TEZ-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522930#comment-14522930
 ] 

TezQA commented on TEZ-2392:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729727/TEZ-2392.1.patch
  against master revision 96efae0.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.runtime.library.common.TestValuesIterator

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/598//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/598//console

This message is automatically generated.

 Have all readers throw an Exception on incorrect next() usage
 -

 Key: TEZ-2392
 URL: https://issues.apache.org/jira/browse/TEZ-2392
 Project: Apache Tez
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Rajesh Balamohan
Priority: Critical
 Attachments: TEZ-2392.1.patch


 Follow up from TEZ-2348.
 Marking as critical since this is a behaviour change, and we should get it in 
 early.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


[ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522928#comment-14522928
 ] 

Sreenath Somarajapuram edited comment on TEZ-2395 at 5/1/15 8:23 AM:
-

Closing td tag is missing.
 td
   {{failedTasks}}
 -/td
Other than that looks good.


was (Author: sreenath):
Closing td tag is missing.
/*
 td
   {{failedTasks}}
-/td
*/
Other than that looks good.

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2395.1.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Failed: TEZ-2392 PreCommit Build #598

Jira: https://issues.apache.org/jira/browse/TEZ-2392
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/598/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2441 lines...]




{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729727/TEZ-2392.1.patch
  against master revision 96efae0.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.runtime.library.common.TestValuesIterator

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/598//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/598//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
7ff6fe1ab1a45436f1b27eabcc3203edac85e2d2 logged out


==
==
Finished build.
==
==


Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #597
Archived 44 artifacts
Archive block size is 32768
Received 25 blocks and 1921139 bytes
Compression is 29.9%
Took 1.2 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
36 tests failed.
REGRESSION:  
org.apache.tez.runtime.library.common.TestValuesIterator.testCountedIteratorWithIFileReader[test[null,
 class org.apache.hadoop.io.Text, class org.apache.hadoop.io.Text, TEXT null 
true {6}]]

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.fail(Assert.java:95)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.getNextFromFinishedIterator(TestValuesIterator.java:221)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.verifyIteratorData(TestValuesIterator.java:304)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.verifyCountedIteratorReader(TestValuesIterator.java:198)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.testCountedIteratorWithIFileReader(TestValuesIterator.java:190)


REGRESSION:  
org.apache.tez.runtime.library.common.TestValuesIterator.testIteratorWithIFileReader[test[null,
 class org.apache.hadoop.io.Text, class org.apache.hadoop.io.Text, TEXT null 
true {6}]]

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.fail(Assert.java:95)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.getNextFromFinishedIterator(TestValuesIterator.java:221)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.verifyIteratorData(TestValuesIterator.java:304)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.testIteratorWithIFileReader(TestValuesIterator.java:180)


REGRESSION:  
org.apache.tez.runtime.library.common.TestValuesIterator.testCountedIteratorWithInmemoryReader[test[null,
 class org.apache.hadoop.io.Text, class org.apache.hadoop.io.Text, TEXT null 
true {6}]]

Error Message:
null

Stack Trace:
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.fail(Assert.java:95)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.getNextFromFinishedIterator(TestValuesIterator.java:221)
at 
org.apache.tez.runtime.library.common.TestValuesIterator.verifyIteratorData(TestValuesIterator.java:304)

[jira] [Comment Edited] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


[ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522928#comment-14522928
 ] 

Sreenath Somarajapuram edited comment on TEZ-2395 at 5/1/15 8:23 AM:
-

Closing td tag is missing.
/*
 td
   {{failedTasks}}
-/td
*/
Other than that looks good.


was (Author: sreenath):
Closing td tag is missing.
 td
   {{failedTasks}}
-/td
Other than that looks good.

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2395.1.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2395) Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you purposefully failed a job.


 [ 
https://issues.apache.org/jira/browse/TEZ-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran updated TEZ-2395:
--
Attachment: TEZ-2395.2.patch

attaching final patch with /td added

 Tez UI: Minimum/Maximum Duration show a empty bracket next to 0 secs when you 
 purposefully failed a job.
 

 Key: TEZ-2395
 URL: https://issues.apache.org/jira/browse/TEZ-2395
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2395.1.patch, TEZ-2395.2.patch


 I set hive.tez.java.opts=-Xmx1m in order to fail a query.
 Vertex Details shows an empty bracket as shown in the attached screenshot:
 Minimum Duration 0 secs [ ] 
 Maximum Duration 0 secs [ ]
 It would look better if the empty bracket is not displayed in a case there is 
 no ask attempts.
 reported by [~taksaito]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Success: TEZ-2392 PreCommit Build #599

Jira: https://issues.apache.org/jira/browse/TEZ-2392
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/599/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2789 lines...]
[INFO] Final Memory: 70M/960M
[INFO] 




{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729736/TEZ-2392.2.patch
  against master revision fcd6bb6.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/599//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/599//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
438e514a894fe8333d5a39dbe54f77068c8ec10c logged out


==
==
Finished build.
==
==


Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #597
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2623410 bytes
Compression is 4.8%
Took 1.8 sec
Description set: TEZ-2392
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors


[ 
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523042#comment-14523042
 ] 

Sreenath Somarajapuram commented on TEZ-2386:
-

The patch is not clean, its failing for vertex_index_controller.

 Tez UI: Inconsistent usage of icon colors
 -

 Key: TEZ-2386
 URL: https://issues.apache.org/jira/browse/TEZ-2386
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.3.patch, 
 TEZ-2386.wip.1.patch


 if there's failed attempts in a DAG, and it succeeds - an orange icon shows 
 up on the DAG page. This is very useful to identify DAGs which may need some 
 debugging.
 However, the color is Green for Vertex / Task views after this - so it's 
 difficult to know which one actually had problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors


[ 
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523136#comment-14523136
 ] 

Sreenath Somarajapuram commented on TEZ-2386:
-

+1 LGTM

 Tez UI: Inconsistent usage of icon colors
 -

 Key: TEZ-2386
 URL: https://issues.apache.org/jira/browse/TEZ-2386
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.3.patch, 
 TEZ-2386.4.patch, TEZ-2386.wip.1.patch


 if there's failed attempts in a DAG, and it succeeds - an orange icon shows 
 up on the DAG page. This is very useful to identify DAGs which may need some 
 debugging.
 However, the color is Green for Vertex / Task views after this - so it's 
 difficult to know which one actually had problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (TEZ-2386) Tez UI: Inconsistent usage of icon colors


 [ 
https://issues.apache.org/jira/browse/TEZ-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Ramachandran updated TEZ-2386:
--
Attachment: TEZ-2386.4.patch

attaching rebased patch

 Tez UI: Inconsistent usage of icon colors
 -

 Key: TEZ-2386
 URL: https://issues.apache.org/jira/browse/TEZ-2386
 Project: Apache Tez
  Issue Type: Bug
  Components: UI
Reporter: Prakash Ramachandran
Assignee: Prakash Ramachandran
 Attachments: TEZ-2386.1.patch, TEZ-2386.2.patch, TEZ-2386.3.patch, 
 TEZ-2386.4.patch, TEZ-2386.wip.1.patch


 if there's failed attempts in a DAG, and it succeeds - an orange icon shows 
 up on the DAG page. This is very useful to identify DAGs which may need some 
 debugging.
 However, the color is Green for Vertex / Task views after this - so it's 
 difficult to know which one actually had problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Success: TEZ-2386 PreCommit Build #600

Jira: https://issues.apache.org/jira/browse/TEZ-2386
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/600/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 2794 lines...]
[INFO] Final Memory: 70M/947M
[INFO] 




{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12729752/TEZ-2386.4.patch
  against master revision fcd6bb6.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/600//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/600//console

This message is automatically generated.


==
==
Adding comment to Jira.
==
==


Comment added.
08cca0e8add04ce22da8db1e6464f67e1e0132c5 logged out


==
==
Finished build.
==
==


Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #599
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2624176 bytes
Compression is 4.8%
Took 0.69 sec
Description set: TEZ-2386
Recording test results
Email was triggered for: Success
Sending email for trigger: Success



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (TEZ-2386) Tez UI: Inconsistent usage of icon colors