[jira] [Commented] (TEZ-2340) TestRecoveryParser fails
[
https://issues.apache.org/jira/browse/TEZ-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507303#comment-14507303
]
Hitesh Shah commented on TEZ-2340:
--
+1
TestRecoveryParser fails
Key: TEZ-2340
URL: https://issues.apache.org/jira/browse/TEZ-2340
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2340-1.patch
Stacktrace
{code}
java.io.IOException: Not supported
at
org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1174)
at
org.apache.tez.dag.history.recovery.RecoveryService.handleSummaryEvent(RecoveryService.java:365)
at
org.apache.tez.dag.history.recovery.RecoveryService.handle(RecoveryService.java:285)
at
org.apache.tez.dag.app.TestRecoveryParser.testSkipAllOtherEvents_1(TestRecoveryParser.java:138)
{code}
Standard Output
{code}
2015-04-17 07:23:55,672 WARN [main] fs.FileUtil
(FileUtil.java:deleteImpl(187)) - Failed to delete file or dir
[D:\w\tez\tez-dag\target\org.apache.tez.dag.app.TestRecoveryParser-tmpDir\recovery\1\.summary.crc]:
it still exists.
2015-04-17 07:23:55,674 WARN [main] fs.FileUtil
(FileUtil.java:deleteImpl(187)) - Failed to delete file or dir
[D:\w\tez\tez-dag\target\org.apache.tez.dag.app.TestRecoveryParser-tmpDir\recovery\1\summary]:
it still exists.
2015-04-17 07:23:55,703 INFO [Thread-5] impl.TestDAGImpl
(TestDAGImpl.java:createTestDAGPlan(446)) - Setting up dag plan
2015-04-17 07:23:55,722 INFO [Thread-5] recovery.RecoveryService
(RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2015-04-17 07:23:55,723 INFO [Thread-5] recovery.RecoveryService
(RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2015-04-17 07:23:55,724 ERROR [Thread-5] recovery.RecoveryService
(RecoveryService.java:handle(314)) - Error handling summary event,
eventType=DAG_SUBMITTED
java.io.IOException: Not supported
at
org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1174)
at
org.apache.tez.dag.history.recovery.RecoveryService.handleSummaryEvent(RecoveryService.java:365)
at
org.apache.tez.dag.history.recovery.RecoveryService.handle(RecoveryService.java:285)
at
org.apache.tez.dag.app.TestRecoveryParser.testSkipAllOtherEvents_1(TestRecoveryParser.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
2015-04-17 07:23:55,724 ERROR [Thread-5] recovery.RecoveryService
(RecoveryService.java:handle(318)) - Adding a flag to ensure next AM attempt
does not start up,
flagFile=target/org.apache.tez.dag.app.TestRecoveryParser-tmpDir/recovery/1/RecoveryFatalErrorOccurred
2015-04-17 07:23:55,725 ERROR [Thread-5] recovery.RecoveryService
(RecoveryService.java:handle(323)) - Recovery failure occurred. Skipping all
events
2015-04-17 07:23:55,756 ERROR [RecoveryEventHandlingThread]
recovery.RecoveryService (RecoveryService.java:run(146)) - Recovery failure
occurred. Stopping recovery thread. Current eventQueueSize=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2348) EOF exception during UnorderedKVReader.next()
[
https://issues.apache.org/jira/browse/TEZ-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-2348:
--
Attachment: TEZ-2348.3.patch
Attaching slightly modified version to reduce conditional checks.
EOF exception during UnorderedKVReader.next()
-
Key: TEZ-2348
URL: https://issues.apache.org/jira/browse/TEZ-2348
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.2
Reporter: Jason Dere
Assignee: Rajesh Balamohan
Attachments: TEZ-2348.1.patch, TEZ-2348.2.patch, TEZ-2348.3.patch,
_tez_session_dir.tgz
{noformat}
Caused by: java.lang.RuntimeException: java.io.IOException: Reached EOF.
Completed reading 516605
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:184)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
... 13 more
Caused by: java.io.IOException: Reached EOF. Completed reading 516605
at
org.apache.tez.runtime.library.common.sort.impl.IFile.checkState(IFile.java:817)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.positionToNextRecord(IFile.java:698)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readRawKey(IFile.java:731)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:727)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.readNextFromCurrentReader(UnorderedKVReader.java:151)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:112)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$KeyValuesFromKeyValue.next(ReduceRecordSource.java:439)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
... 15 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2341) TestMockDAGAppMaster.testBasicCounters fails on windows
[
https://issues.apache.org/jira/browse/TEZ-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated TEZ-2341:
Attachment: TEZ-2341-2.patch
Attach the patch (only verify it in linux platform)
[~bikassaha] [~hitesh] Please help review it.
TestMockDAGAppMaster.testBasicCounters fails on windows
---
Key: TEZ-2341
URL: https://issues.apache.org/jira/browse/TEZ-2341
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2341-1.patch, TEZ-2341-2.patch
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicCounters(TestMockDAGAppMaster.java:323)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2348) EOF exception during UnorderedKVReader.next()
[
https://issues.apache.org/jira/browse/TEZ-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506500#comment-14506500
]
TezQA commented on TEZ-2348:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727124/TEZ-2348.2.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/510//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/510//console
This message is automatically generated.
EOF exception during UnorderedKVReader.next()
-
Key: TEZ-2348
URL: https://issues.apache.org/jira/browse/TEZ-2348
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.2
Reporter: Jason Dere
Assignee: Rajesh Balamohan
Attachments: TEZ-2348.1.patch, TEZ-2348.2.patch, _tez_session_dir.tgz
{noformat}
Caused by: java.lang.RuntimeException: java.io.IOException: Reached EOF.
Completed reading 516605
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:184)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
... 13 more
Caused by: java.io.IOException: Reached EOF. Completed reading 516605
at
org.apache.tez.runtime.library.common.sort.impl.IFile.checkState(IFile.java:817)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.positionToNextRecord(IFile.java:698)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readRawKey(IFile.java:731)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:727)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.readNextFromCurrentReader(UnorderedKVReader.java:151)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:112)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$KeyValuesFromKeyValue.next(ReduceRecordSource.java:439)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
... 15 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2348 PreCommit Build #510
Jira: https://issues.apache.org/jira/browse/TEZ-2348
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/510/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2770 lines...]
[INFO] Final Memory: 70M/958M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727124/TEZ-2348.2.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/510//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/510//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
45dc387506e817145332bd14b3eba017f50657c4 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #509
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2483952 bytes
Compression is 9.5%
Took 0.63 sec
Description set: TEZ-2348
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2342) TestFaultTolerance.testRandomFailingTasks fails due to timeout
[
https://issues.apache.org/jira/browse/TEZ-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506515#comment-14506515
]
Jeff Zhang commented on TEZ-2342:
-
[~hitesh] [~bikassaha] Please help review it.
TestFaultTolerance.testRandomFailingTasks fails due to timeout
--
Key: TEZ-2342
URL: https://issues.apache.org/jira/browse/TEZ-2342
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2342-1.patch, syslog_dag_1429582868137_0001_1
{code}
Error Message
test timed out after 12 milliseconds
Stacktrace
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:126)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingTasks(TestFaultTolerance.java:723)
Standard Output
2015-04-17 07:46:10,952 INFO [main] test.TestFaultTolerance
(TestFaultTolerance.java:setup(65)) - Starting mini clusters
2015-04-17 07:46:11,508 INFO [main] hdfs.MiniDFSCluster
(MiniDFSCluster.java:init(446)) - starting cluster: numNameNodes=1,
numDataNodes=1
Formatting using clusterid: testClusterID
2015-04-17 07:46:12,919 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(716)) - No KeyProvider found.
2015-04-17 07:46:12,920 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(726)) - fsLock is fair:true
2015-04-17 07:46:13,021 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1173)) -
hadoop.configured.node.mapping is deprecated. Instead, use
net.topology.configured.node.mapping
2015-04-17 07:46:13,021 INFO [main] blockmanagement.DatanodeManager
(DatanodeManager.java:init(239)) - dfs.block.invalidate.limit=1000
2015-04-17 07:46:13,022 INFO [main] blockmanagement.DatanodeManager
(DatanodeManager.java:init(245)) -
dfs.namenode.datanode.registration.ip-hostname-check=true
2015-04-17 07:46:13,022 INFO [main] blockmanagement.BlockManager
(InvalidateBlocks.java:printBlockDeletionTime(71)) -
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2015-04-17 07:46:13,025 INFO [main] blockmanagement.BlockManager
(InvalidateBlocks.java:printBlockDeletionTime(76)) - The block deletion will
start around 2015 Apr 17 07:46:13
2015-04-17 07:46:13,029 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(354)) - Computing capacity for map
BlocksMap
2015-04-17 07:46:13,030 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(355)) - VM type = 64-bit
2015-04-17 07:46:13,032 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(356)) - 2.0% max memory 910.3 MB = 18.2
MB
2015-04-17 07:46:13,033 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(361)) - capacity = 2^21 = 2097152
entries
2015-04-17 07:46:13,079 INFO [main] blockmanagement.BlockManager
(BlockManager.java:createBlockTokenSecretManager(365)) -
dfs.block.access.token.enable=false
2015-04-17 07:46:13,080 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(350)) - defaultReplication = 1
2015-04-17 07:46:13,080 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(351)) - maxReplication = 512
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(352)) - minReplication = 1
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(353)) - maxReplicationStreams = 2
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(354)) - shouldCheckForEnoughRacks = false
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(355)) - replicationRecheckInterval = 3000
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(356)) - encryptDataTransfer= false
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(357)) - maxNumBlocksToLog = 1000
2015-04-17 07:46:13,115 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(746)) - fsOwner = jenkins (auth:SIMPLE)
2015-04-17 07:46:13,116 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(747)) - supergroup = supergroup
2015-04-17 07:46:13,116 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(748)) - isPermissionEnabled = true
2015-04-17 07:46:13,116 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(759)) - HA Enabled: false
2015-04-17 07:46:13,120 INFO [main]
[jira] [Commented] (TEZ-2248) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
[ https://issues.apache.org/jira/browse/TEZ-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507563#comment-14507563 ] Bikas Saha commented on TEZ-2248: - Would it make sense to add a finishedState member variable that can be set along with terminationCause in trySetTerminationCause()? That way we could allow different end states for the same termination cause (in case that scenario arises in the future) instead of tying a cause to a state. We may need to check that termiantionCause is being only set via trySetTerminationCause() in all cases. VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks Key: TEZ-2248 URL: https://issues.apache.org/jira/browse/TEZ-2248 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Jeff Zhang Attachments: TEZ-2248-1.patch There is an if check for each termination cause which makes code long and we need to handle each new termination cause with more code. This could be abstracted into a method that gets termination cause string based on the enum and make this method shorter and stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TEZ-2356) TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
[
https://issues.apache.org/jira/browse/TEZ-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikas Saha resolved TEZ-2356.
-
Resolution: Not A Problem
TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
Key: TEZ-2356
URL: https://issues.apache.org/jira/browse/TEZ-2356
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Thejas M Nair
Priority: Blocker
This breaks pig compilation and needs urgent attention.
{code}
src/org/apache/pig/backend/hadoop/executionengine/tez/runtime/PigGraceShuffleVertexManager.java:173:
error: exception TezException is never thrown in body of corresponding try
statement
[javac] } catch (TezException e) {
[javac] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2342) TestFaultTolerance.testRandomFailingTasks fails due to timeout
[
https://issues.apache.org/jira/browse/TEZ-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507543#comment-14507543
]
Bikas Saha commented on TEZ-2342:
-
If this passes with the increased timeout (instead of hanging permanently then
the change looks good. Could you please run this in loop 10-20 times and see if
there are any further issues. If none, then lets commit this. Else lets look
for a code/test bug.
TestFaultTolerance.testRandomFailingTasks fails due to timeout
--
Key: TEZ-2342
URL: https://issues.apache.org/jira/browse/TEZ-2342
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2342-1.patch, syslog_dag_1429582868137_0001_1
{code}
Error Message
test timed out after 12 milliseconds
Stacktrace
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:126)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:114)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingTasks(TestFaultTolerance.java:723)
Standard Output
2015-04-17 07:46:10,952 INFO [main] test.TestFaultTolerance
(TestFaultTolerance.java:setup(65)) - Starting mini clusters
2015-04-17 07:46:11,508 INFO [main] hdfs.MiniDFSCluster
(MiniDFSCluster.java:init(446)) - starting cluster: numNameNodes=1,
numDataNodes=1
Formatting using clusterid: testClusterID
2015-04-17 07:46:12,919 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(716)) - No KeyProvider found.
2015-04-17 07:46:12,920 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(726)) - fsLock is fair:true
2015-04-17 07:46:13,021 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1173)) -
hadoop.configured.node.mapping is deprecated. Instead, use
net.topology.configured.node.mapping
2015-04-17 07:46:13,021 INFO [main] blockmanagement.DatanodeManager
(DatanodeManager.java:init(239)) - dfs.block.invalidate.limit=1000
2015-04-17 07:46:13,022 INFO [main] blockmanagement.DatanodeManager
(DatanodeManager.java:init(245)) -
dfs.namenode.datanode.registration.ip-hostname-check=true
2015-04-17 07:46:13,022 INFO [main] blockmanagement.BlockManager
(InvalidateBlocks.java:printBlockDeletionTime(71)) -
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2015-04-17 07:46:13,025 INFO [main] blockmanagement.BlockManager
(InvalidateBlocks.java:printBlockDeletionTime(76)) - The block deletion will
start around 2015 Apr 17 07:46:13
2015-04-17 07:46:13,029 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(354)) - Computing capacity for map
BlocksMap
2015-04-17 07:46:13,030 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(355)) - VM type = 64-bit
2015-04-17 07:46:13,032 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(356)) - 2.0% max memory 910.3 MB = 18.2
MB
2015-04-17 07:46:13,033 INFO [main] util.GSet
(LightWeightGSet.java:computeCapacity(361)) - capacity = 2^21 = 2097152
entries
2015-04-17 07:46:13,079 INFO [main] blockmanagement.BlockManager
(BlockManager.java:createBlockTokenSecretManager(365)) -
dfs.block.access.token.enable=false
2015-04-17 07:46:13,080 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(350)) - defaultReplication = 1
2015-04-17 07:46:13,080 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(351)) - maxReplication = 512
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(352)) - minReplication = 1
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(353)) - maxReplicationStreams = 2
2015-04-17 07:46:13,083 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(354)) - shouldCheckForEnoughRacks = false
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(355)) - replicationRecheckInterval = 3000
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(356)) - encryptDataTransfer= false
2015-04-17 07:46:13,084 INFO [main] blockmanagement.BlockManager
(BlockManager.java:init(357)) - maxNumBlocksToLog = 1000
2015-04-17 07:46:13,115 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(746)) - fsOwner = jenkins (auth:SIMPLE)
2015-04-17 07:46:13,116 INFO [main] namenode.FSNamesystem
(FSNamesystem.java:init(747)) - supergroup = supergroup
2015-04-17 07:46:13,116 INFO [main] namenode.FSNamesystem
[jira] [Updated] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Attachment: TEZ-1897.4.patch Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Summary: Create a concurrent version of AsyncDispatcher (was: Allow higher concurrency in AsyncDispatcher) Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2356) TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
[
https://issues.apache.org/jira/browse/TEZ-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507493#comment-14507493
]
Bikas Saha commented on TEZ-2356:
-
Please change Pig code. We have discussed this scenario in Tez and decided that
not throwing an exception here is the best choice for us. Since we did not have
a release with either this or the previous API change we did not consider
downstream consumers to be affected. Sorry about that. I am going to resolve
this jira.
TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
Key: TEZ-2356
URL: https://issues.apache.org/jira/browse/TEZ-2356
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Thejas M Nair
Priority: Blocker
This breaks pig compilation and needs urgent attention.
{code}
src/org/apache/pig/backend/hadoop/executionengine/tez/runtime/PigGraceShuffleVertexManager.java:173:
error: exception TezException is never thrown in body of corresponding try
statement
[javac] } catch (TezException e) {
[javac] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507588#comment-14507588
]
TezQA commented on TEZ-1897:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727292/TEZ-1897.4.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.TestMockDAGAppMaster
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/514//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/514//artifact/patchprocess/newPatchFindbugsWarningstez-common.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/514//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2346) TEZ-UI: Load other info / counter data on demand
[ https://issues.apache.org/jira/browse/TEZ-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507608#comment-14507608 ] Sreenath Somarajapuram commented on TEZ-2346: - Hi [~zjshen] I have created a YARN ticket, please find the same @ https://issues.apache.org/jira/browse/YARN-3530. TEZ-UI: Load other info / counter data on demand Key: TEZ-2346 URL: https://issues.apache.org/jira/browse/TEZ-2346 Project: Apache Tez Issue Type: Sub-task Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Attachments: Screen-Shot-2015-04-21-at-1.56.28-AM.jpg, TEZ-2346.wip.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-1897 PreCommit Build #514
Jira: https://issues.apache.org/jira/browse/TEZ-1897
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/514/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2397 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727292/TEZ-1897.4.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.TestMockDAGAppMaster
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/514//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/514//artifact/patchprocess/newPatchFindbugsWarningstez-common.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/514//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
d59d234845e999bf7e24b2428148ae35e6e57ee5 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #513
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2538187 bytes
Compression is 7.2%
Took 2.1 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
1 tests failed.
REGRESSION: org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicEvents
Error Message:
expected:0 but was:1
Stack Trace:
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicEvents(TestMockDAGAppMaster.java:232)
[jira] [Commented] (TEZ-2261) Should add diagnostics in DAGAppMaster when recovery error happens
[
https://issues.apache.org/jira/browse/TEZ-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508286#comment-14508286
]
Jeff Zhang commented on TEZ-2261:
-
Committed to master
Should add diagnostics in DAGAppMaster when recovery error happens
--
Key: TEZ-2261
URL: https://issues.apache.org/jira/browse/TEZ-2261
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2261-1.patch, TEZ-2261-2.patch, TEZ-2261-3.patch
Should add diagnostics in DAGAppMaster when recovery error happens, otherwise
AM is shutdown and the next dag submission will just throw
SessionNotRunningException which would confuse users.
{code}
if (this.historyEventHandler.hasRecoveryFailed()) {
LOG.warn(Recovery had a fatal error, shutting down session after +
DAG completion);
sessionStopped.set(true);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2341) TestMockDAGAppMaster.testBasicCounters fails on windows
[
https://issues.apache.org/jira/browse/TEZ-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated TEZ-2341:
Attachment: TEZ-2341-3.patch
[~hitesh] Thanks for the suggestion. Upload a new patch for that.
TestMockDAGAppMaster.testBasicCounters fails on windows
---
Key: TEZ-2341
URL: https://issues.apache.org/jira/browse/TEZ-2341
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2341-1.patch, TEZ-2341-2.patch, TEZ-2341-3.patch
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicCounters(TestMockDAGAppMaster.java:323)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-391) SharedEdge - Support for passing same output from a vertex as input to two different vertices
[ https://issues.apache.org/jira/browse/TEZ-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-391: --- Attachment: TEZ-391-WIP-6.patch SharedEdge - Support for passing same output from a vertex as input to two different vertices - Key: TEZ-391 URL: https://issues.apache.org/jira/browse/TEZ-391 Project: Apache Tez Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Jeff Zhang Attachments: Shared Edge Design.pdf, TEZ-391-WIP-1.patch, TEZ-391-WIP-2.patch, TEZ-391-WIP-3.patch, TEZ-391-WIP-4.patch, TEZ-391-WIP-5.patch, TEZ-391-WIP-6.patch We need this for lot of usecases. For cases where multi-query is turned off and for optimizing unions. Currently those are BROADCAST or ONE-ONE edges and we write the output multiple times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2357) misc.js.orig is committed by accident
[ https://issues.apache.org/jira/browse/TEZ-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-2357: - Assignee: Prakash Ramachandran misc.js.orig is committed by accident - Key: TEZ-2357 URL: https://issues.apache.org/jira/browse/TEZ-2357 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Jeff Zhang Assignee: Prakash Ramachandran \cc [~pramachandran] [~Sreenath] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2248) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
[ https://issues.apache.org/jira/browse/TEZ-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2248: Attachment: TEZ-2248-2.patch VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks Key: TEZ-2248 URL: https://issues.apache.org/jira/browse/TEZ-2248 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Jeff Zhang Attachments: TEZ-2248-1.patch, TEZ-2248-2.patch There is an if check for each termination cause which makes code long and we need to handle each new termination cause with more code. This could be abstracted into a method that gets termination cause string based on the enum and make this method shorter and stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2248) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
[
https://issues.apache.org/jira/browse/TEZ-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508298#comment-14508298
]
TezQA commented on TEZ-2248:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727493/TEZ-2248-2.patch
against master revision d91eb28.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/517//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/517//console
This message is automatically generated.
VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
Key: TEZ-2248
URL: https://issues.apache.org/jira/browse/TEZ-2248
Project: Apache Tez
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Jeff Zhang
Attachments: TEZ-2248-1.patch, TEZ-2248-2.patch
There is an if check for each termination cause which makes code long and we
need to handle each new termination cause with more code. This could be
abstracted into a method that gets termination cause string based on the enum
and make this method shorter and stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-2357) misc.js.orig is committed by accident
Jeff Zhang created TEZ-2357: --- Summary: misc.js.orig is committed by accident Key: TEZ-2357 URL: https://issues.apache.org/jira/browse/TEZ-2357 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Jeff Zhang \cc [~pramachandran] [~Sreenath] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-391) SharedEdge - Support for passing same output from a vertex as input to two different vertices
[
https://issues.apache.org/jira/browse/TEZ-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508332#comment-14508332
]
TezQA commented on TEZ-391:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727511/TEZ-391-WIP-5.patch
against master revision fe11c5e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/519//console
This message is automatically generated.
SharedEdge - Support for passing same output from a vertex as input to two
different vertices
-
Key: TEZ-391
URL: https://issues.apache.org/jira/browse/TEZ-391
Project: Apache Tez
Issue Type: Sub-task
Reporter: Rohini Palaniswamy
Assignee: Jeff Zhang
Attachments: Shared Edge Design.pdf, TEZ-391-WIP-1.patch,
TEZ-391-WIP-2.patch, TEZ-391-WIP-3.patch, TEZ-391-WIP-4.patch,
TEZ-391-WIP-5.patch
We need this for lot of usecases. For cases where multi-query is turned off
and for optimizing unions. Currently those are BROADCAST or ONE-ONE edges and
we write the output multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508247#comment-14508247 ] Bikas Saha commented on TEZ-1897: - The last patch builds on the previous patch to actually use the concurrent dispatcher to run Task and TaskAttempt events concurrently. There is a configuration to turn this on or off and when it is turned off the code runs exactly the same path as it does today. So this change is very safe. In order to keep things sane, events for a given Task and its attempts are serialized on the same thread. This is done by using a serializing hash determined from the TezTaskID. Different tasks run on different threads. That takes care of a lot of locking issues. Next, Vertex has reference to DAG, Task has reference to Vertex and Attempt has reference to Task and Vertex. This helps remove unnecessary locking issues and delays that occur when they are accessed from the AppContext to get dag/vertex/task etc. and then look-up into their internal maps. This change would be beneficial in general by reducing lock contention compared to today. Added a simulation test that runs 5 tasks at 1000 concurrency which runs up to 30% faster with the change than without. The patch has the config turned on for patch test execution. This will be off by default and is marked private so only advanced users can try this for large clusters where we can get 10-20K running tasks concurrently. Please review. Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-391) SharedEdge - Support for passing same output from a vertex as input to two different vertices
[
https://issues.apache.org/jira/browse/TEZ-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508396#comment-14508396
]
TezQA commented on TEZ-391:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727512/TEZ-391-WIP-6.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 161 javac
compiler warnings (more than the master's current 160 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/520//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/520//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/520//console
This message is automatically generated.
SharedEdge - Support for passing same output from a vertex as input to two
different vertices
-
Key: TEZ-391
URL: https://issues.apache.org/jira/browse/TEZ-391
Project: Apache Tez
Issue Type: Sub-task
Reporter: Rohini Palaniswamy
Assignee: Jeff Zhang
Attachments: Shared Edge Design.pdf, TEZ-391-WIP-1.patch,
TEZ-391-WIP-2.patch, TEZ-391-WIP-3.patch, TEZ-391-WIP-4.patch,
TEZ-391-WIP-5.patch, TEZ-391-WIP-6.patch
We need this for lot of usecases. For cases where multi-query is turned off
and for optimizing unions. Currently those are BROADCAST or ONE-ONE edges and
we write the output multiple times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2305) MR compatibility sleep job fails with IOException: Undefined job output-path
[
https://issues.apache.org/jira/browse/TEZ-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508249#comment-14508249
]
Jeff Zhang commented on TEZ-2305:
-
[~hitesh] Will you commit this ? Or I can help to commit it.
MR compatibility sleep job fails with IOException: Undefined job output-path
Key: TEZ-2305
URL: https://issues.apache.org/jira/browse/TEZ-2305
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Tassapol Athiapinya
Priority: Critical
Attachments: TEZ-2305-3.patch, TEZ-2305-4.patch, TEZ-2305.1.patch,
TEZ-2305.2.patch
Running MR sleep job has an IOException.
{code}
15/04/09 20:52:25 INFO mapreduce.Job: Job job_1428612196442_0002 failed with
state FAILED due to: Vertex failed, vertexName=initialmap,
vertexId=vertex_1428612196442_0002_1_00, diagnostics=[Task failed,
taskId=task_1428612196442_0002_1_00_01, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running task:java.io.IOException:
Undefined job output-path
at
org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
at
org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Failure while running
task:java.io.IOException: Undefined job output-path
at
org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
at
org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Failure while running
task:java.io.IOException: Undefined job output-path
at
org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
at
org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Error: Failure while running
task:java.io.IOException: Undefined job output-path
at
org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
at
[jira] [Commented] (TEZ-2248) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
[ https://issues.apache.org/jira/browse/TEZ-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508254#comment-14508254 ] Jeff Zhang commented on TEZ-2248: - [~bikassaha] I think finishedState member is only valid when terminationCause is set, so I think it would be better to keep it as a member of TerminationCause. And I think we don't need to make that change unless there's a real scenario for that (different end states for the same termination cause), always prefer to do refactoring when it is necessary :) Any thoughts ? Post another patch ( make a little change to use trySetTerminationCause in all cases ) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks Key: TEZ-2248 URL: https://issues.apache.org/jira/browse/TEZ-2248 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Jeff Zhang Attachments: TEZ-2248-1.patch, TEZ-2248-2.patch There is an if check for each termination cause which makes code long and we need to handle each new termination cause with more code. This could be abstracted into a method that gets termination cause string based on the enum and make this method shorter and stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2248 PreCommit Build #517
Jira: https://issues.apache.org/jira/browse/TEZ-2248
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/517/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2773 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727493/TEZ-2248-2.patch
against master revision d91eb28.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/517//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/517//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4d50aee08ddd15d45fa835a17e9b038376eb65db logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #516
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2558722 bytes
Compression is 7.1%
Took 1.6 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-391) SharedEdge - Support for passing same output from a vertex as input to two different vertices
[ https://issues.apache.org/jira/browse/TEZ-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-391: --- Attachment: TEZ-391-WIP-5.patch Rebase the patch. SharedEdge - Support for passing same output from a vertex as input to two different vertices - Key: TEZ-391 URL: https://issues.apache.org/jira/browse/TEZ-391 Project: Apache Tez Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Jeff Zhang Attachments: Shared Edge Design.pdf, TEZ-391-WIP-1.patch, TEZ-391-WIP-2.patch, TEZ-391-WIP-3.patch, TEZ-391-WIP-4.patch, TEZ-391-WIP-5.patch We need this for lot of usecases. For cases where multi-query is turned off and for optimizing unions. Currently those are BROADCAST or ONE-ONE edges and we write the output multiple times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-391 PreCommit Build #519
Jira: https://issues.apache.org/jira/browse/TEZ-391
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/519/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 22 lines...]
==
==
Testing patch for TEZ-391.
==
==
HEAD is now at fe11c5e TEZ-2261. Should add diagnostics in DAGAppMaster when
recovery error happens (zjffdu)
Previous HEAD position was fe11c5e... TEZ-2261. Should add diagnostics in
DAGAppMaster when recovery error happens (zjffdu)
Switched to branch 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use git pull to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to fe11c5e67334f1a08e2b586b68ee4840d8b62763.
TEZ-391 patch is being downloaded at Thu Apr 23 02:30:17 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12727511/TEZ-391-WIP-5.patch
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727511/TEZ-391-WIP-5.patch
against master revision fe11c5e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/519//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
66bbcde1a5eff6ffeeace17d21f1421658bfc8c2 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-2341) TestMockDAGAppMaster.testBasicCounters fails on windows
[
https://issues.apache.org/jira/browse/TEZ-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508343#comment-14508343
]
TezQA commented on TEZ-2341:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727501/TEZ-2341-3.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/518//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/518//console
This message is automatically generated.
TestMockDAGAppMaster.testBasicCounters fails on windows
---
Key: TEZ-2341
URL: https://issues.apache.org/jira/browse/TEZ-2341
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2341-1.patch, TEZ-2341-2.patch, TEZ-2341-3.patch
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicCounters(TestMockDAGAppMaster.java:323)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2341 PreCommit Build #518
Jira: https://issues.apache.org/jira/browse/TEZ-2341
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/518/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2767 lines...]
[INFO] Final Memory: 74M/958M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727501/TEZ-2341-3.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/518//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/518//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7d048a10f6e17b625e2fc2abe10bc218b2278ba5 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #516
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2550043 bytes
Compression is 7.2%
Took 2.4 sec
Description set: TEZ-2341
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
Failed: TEZ-391 PreCommit Build #520
Jira: https://issues.apache.org/jira/browse/TEZ-391
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/520/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2789 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727512/TEZ-391-WIP-6.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 161 javac
compiler warnings (more than the master's current 160 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/520//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/520//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/520//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
972f46ac7a5da8461ca7cc431dedcc7a7138e6cb logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #518
Archived 45 artifacts
Archive block size is 32768
Received 22 blocks and 2035752 bytes
Compression is 26.2%
Took 1.2 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2303) ConcurrentModificationException while processing recovery
[ https://issues.apache.org/jira/browse/TEZ-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2303: Target Version/s: 0.5.4 ConcurrentModificationException while processing recovery - Key: TEZ-2303 URL: https://issues.apache.org/jira/browse/TEZ-2303 Project: Apache Tez Issue Type: Bug Affects Versions: 0.6.0 Reporter: Jason Lowe Assignee: Jeff Zhang Attachments: TEZ-2303-1.patch Saw a Tez AM log a few ConcurrentModificationException messages while trying to recover from a previous attempt that crashed. Exception details to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2303) ConcurrentModificationException while processing recovery
[ https://issues.apache.org/jira/browse/TEZ-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2303: Attachment: TEZ-2303-1.patch Upload patch ( acquire writelock first in restoreFromEvent, this would prevent the ConcurrentModificationException while recovering, but will still cause client get incorrect dag status) ConcurrentModificationException while processing recovery - Key: TEZ-2303 URL: https://issues.apache.org/jira/browse/TEZ-2303 Project: Apache Tez Issue Type: Bug Affects Versions: 0.6.0 Reporter: Jason Lowe Assignee: Jeff Zhang Attachments: TEZ-2303-1.patch Saw a Tez AM log a few ConcurrentModificationException messages while trying to recover from a previous attempt that crashed. Exception details to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2303) ConcurrentModificationException while processing recovery
[ https://issues.apache.org/jira/browse/TEZ-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508428#comment-14508428 ] Jeff Zhang edited comment on TEZ-2303 at 4/23/15 4:18 AM: -- Upload patch ( acquire writelock first in restoreFromEvent, this would prevent the ConcurrentModificationException while recovering, but will still cause client get incorrect dag status) [~hitesh] Please help review it. was (Author: zjffdu): Upload patch ( acquire writelock first in restoreFromEvent, this would prevent the ConcurrentModificationException while recovering, but will still cause client get incorrect dag status) ConcurrentModificationException while processing recovery - Key: TEZ-2303 URL: https://issues.apache.org/jira/browse/TEZ-2303 Project: Apache Tez Issue Type: Bug Affects Versions: 0.6.0 Reporter: Jason Lowe Assignee: Jeff Zhang Attachments: TEZ-2303-1.patch Saw a Tez AM log a few ConcurrentModificationException messages while trying to recover from a previous attempt that crashed. Exception details to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Attachment: TEZ-1897.5.patch Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2294) Add tez-site-template.xml with description of config properties
[ https://issues.apache.org/jira/browse/TEZ-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2294: - Attachment: (was: TezConfiguration.html) Add tez-site-template.xml with description of config properties --- Key: TEZ-2294 URL: https://issues.apache.org/jira/browse/TEZ-2294 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2294.wip.2.patch, TEZ-2294.wip.3.patch, TEZ-2294.wip.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2294) Add tez-site-template.xml with description of config properties
[ https://issues.apache.org/jira/browse/TEZ-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2294: - Attachment: (was: TezRuntimeConfiguration.html) Add tez-site-template.xml with description of config properties --- Key: TEZ-2294 URL: https://issues.apache.org/jira/browse/TEZ-2294 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2294.wip.2.patch, TEZ-2294.wip.3.patch, TEZ-2294.wip.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[ https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1897: Attachment: TEZ-1897.5.patch Create a concurrent version of AsyncDispatcher -- Key: TEZ-1897 URL: https://issues.apache.org/jira/browse/TEZ-1897 Project: Apache Tez Issue Type: Task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch, TEZ-1897.4.patch, TEZ-1897.5.patch Currently, it processes events on a single thread. For events that can be executed in parallel, e.g. vertex manager events, allowing higher concurrency may be beneficial. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2294) Add tez-site-template.xml with description of config properties
[ https://issues.apache.org/jira/browse/TEZ-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507894#comment-14507894 ] Hitesh Shah commented on TEZ-2294: -- At some point, I am hoping our internal docs can get to a point where valid values are populated. This is just a proof of concept currently - still needs some polishing up to integrate with our generated javadocs. Add tez-site-template.xml with description of config properties --- Key: TEZ-2294 URL: https://issues.apache.org/jira/browse/TEZ-2294 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2294.wip.2.patch, TEZ-2294.wip.3.patch, TEZ-2294.wip.patch, TezConfiguration.html, TezRuntimeConfiguration.html, tez-default-template.xml, tez-runtime-default-template.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-776 PreCommit Build #515
Jira: https://issues.apache.org/jira/browse/TEZ-776
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/515/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2792 lines...]
[INFO] Final Memory: 66M/715M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727352/TEZ-776.4.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/515//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/515//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
80b3bca5ed6316253f323793d1888bb7edaa9c99 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #513
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2626403 bytes
Compression is 4.8%
Took 1.4 sec
Description set: TEZ-776
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-776) Reduce AM mem usage caused by storing TezEvents
[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508042#comment-14508042
]
TezQA commented on TEZ-776:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727352/TEZ-776.4.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/515//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/515//console
This message is automatically generated.
Reduce AM mem usage caused by storing TezEvents
---
Key: TEZ-776
URL: https://issues.apache.org/jira/browse/TEZ-776
Project: Apache Tez
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Bikas Saha
Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch,
TEZ-776.4.patch, TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch,
TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch,
TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch,
With_Patch_AM_hotspots.png, With_Patch_AM_profile.png,
Without_patch_AM_CPU_Usage.png, events-problem-solutions.txt,
with_patch_jmc_output_of_AM.png, without_patch_jmc_output_of_AM.png
This is open ended at the moment.
A fair chunk of the AM heap is taken up by TezEvents (specifically
DataMovementEvents - 64 bytes per event).
Depending on the connection pattern - this puts limits on the number of tasks
that can be processed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1897) Create a concurrent version of AsyncDispatcher
[
https://issues.apache.org/jira/browse/TEZ-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508132#comment-14508132
]
TezQA commented on TEZ-1897:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727349/TEZ-1897.5.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/516//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/516//console
This message is automatically generated.
Create a concurrent version of AsyncDispatcher
--
Key: TEZ-1897
URL: https://issues.apache.org/jira/browse/TEZ-1897
Project: Apache Tez
Issue Type: Task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: TEZ-1897.1.patch, TEZ-1897.2.patch, TEZ-1897.3.patch,
TEZ-1897.4.patch, TEZ-1897.5.patch, TEZ-1897.5.patch
Currently, it processes events on a single thread. For events that can be
executed in parallel, e.g. vertex manager events, allowing higher concurrency
may be beneficial.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-1897 PreCommit Build #516
Jira: https://issues.apache.org/jira/browse/TEZ-1897
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/516/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2788 lines...]
[INFO] Final Memory: 70M/932M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727349/TEZ-1897.5.patch
against master revision 9faa54c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/516//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/516//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
2dd07eddae1b37d1a60221a10d4951d663d6433a logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #515
Archived 44 artifacts
Archive block size is 32768
Received 22 blocks and 2030949 bytes
Compression is 26.2%
Took 1.4 sec
Description set: TEZ-1897
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2294) Add tez-site-template.xml with description of config properties
[ https://issues.apache.org/jira/browse/TEZ-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2294: - Attachment: TezRuntimeConfiguration.html TezConfiguration.html tez-runtime-default-template.xml tez-default-template.xml Latest generated config docs. \cc [~jeagles] [~rohini] to review its usefulness. Add tez-site-template.xml with description of config properties --- Key: TEZ-2294 URL: https://issues.apache.org/jira/browse/TEZ-2294 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2294.wip.2.patch, TEZ-2294.wip.3.patch, TEZ-2294.wip.patch, TezConfiguration.html, TezRuntimeConfiguration.html, tez-default-template.xml, tez-runtime-default-template.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-776) Reduce AM mem usage caused by storing TezEvents
[ https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-776: --- Attachment: TEZ-776.4.patch Reduce AM mem usage caused by storing TezEvents --- Key: TEZ-776 URL: https://issues.apache.org/jira/browse/TEZ-776 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Bikas Saha Attachments: TEZ-776.1.patch, TEZ-776.2.patch, TEZ-776.3.patch, TEZ-776.4.patch, TEZ-776.ondemand.1.patch, TEZ-776.ondemand.2.patch, TEZ-776.ondemand.3.patch, TEZ-776.ondemand.4.patch, TEZ-776.ondemand.5.patch, TEZ-776.ondemand.6.patch, TEZ-776.ondemand.7.patch, TEZ-776.ondemand.patch, With_Patch_AM_hotspots.png, With_Patch_AM_profile.png, Without_patch_AM_CPU_Usage.png, events-problem-solutions.txt, with_patch_jmc_output_of_AM.png, without_patch_jmc_output_of_AM.png This is open ended at the moment. A fair chunk of the AM heap is taken up by TezEvents (specifically DataMovementEvents - 64 bytes per event). Depending on the connection pattern - this puts limits on the number of tasks that can be processed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2303 PreCommit Build #521
Jira: https://issues.apache.org/jira/browse/TEZ-2303
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/521/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2770 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727527/TEZ-2303-1.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/521//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/521//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
585e4abd9bb9803f79ba7084fee16667206c944a logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #518
Archived 44 artifacts
Archive block size is 32768
Received 27 blocks and 1860898 bytes
Compression is 32.2%
Took 0.56 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2303) ConcurrentModificationException while processing recovery
[
https://issues.apache.org/jira/browse/TEZ-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508466#comment-14508466
]
TezQA commented on TEZ-2303:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727527/TEZ-2303-1.patch
against master revision fe11c5e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/521//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/521//console
This message is automatically generated.
ConcurrentModificationException while processing recovery
-
Key: TEZ-2303
URL: https://issues.apache.org/jira/browse/TEZ-2303
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Jason Lowe
Assignee: Jeff Zhang
Attachments: TEZ-2303-1.patch
Saw a Tez AM log a few ConcurrentModificationException messages while trying
to recover from a previous attempt that crashed. Exception details to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2294) Add tez-site-template.xml with description of config properties
[ https://issues.apache.org/jira/browse/TEZ-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507884#comment-14507884 ] Rohini Palaniswamy commented on TEZ-2294: - Valid values column is always null and can be removed. Rest looks good. Add tez-site-template.xml with description of config properties --- Key: TEZ-2294 URL: https://issues.apache.org/jira/browse/TEZ-2294 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan Assignee: Rajesh Balamohan Attachments: TEZ-2294.wip.2.patch, TEZ-2294.wip.3.patch, TEZ-2294.wip.patch, TezConfiguration.html, TezRuntimeConfiguration.html, tez-default-template.xml, tez-runtime-default-template.xml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2248 PreCommit Build #511
Jira: https://issues.apache.org/jira/browse/TEZ-2248
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/511/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2770 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727132/TEZ-2248-1.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/511//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/511//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
93ca1ca0569d82b4ae3aa988fe7c1aed46a19378 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #510
Archived 44 artifacts
Archive block size is 32768
Received 6 blocks and 2552855 bytes
Compression is 7.2%
Took 2 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2348) EOF exception during UnorderedKVReader.next()
[
https://issues.apache.org/jira/browse/TEZ-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506552#comment-14506552
]
TezQA commented on TEZ-2348:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727137/TEZ-2348.3.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/512//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/512//console
This message is automatically generated.
EOF exception during UnorderedKVReader.next()
-
Key: TEZ-2348
URL: https://issues.apache.org/jira/browse/TEZ-2348
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.2
Reporter: Jason Dere
Assignee: Rajesh Balamohan
Attachments: TEZ-2348.1.patch, TEZ-2348.2.patch, TEZ-2348.3.patch,
_tez_session_dir.tgz
{noformat}
Caused by: java.lang.RuntimeException: java.io.IOException: Reached EOF.
Completed reading 516605
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:184)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
... 13 more
Caused by: java.io.IOException: Reached EOF. Completed reading 516605
at
org.apache.tez.runtime.library.common.sort.impl.IFile.checkState(IFile.java:817)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.positionToNextRecord(IFile.java:698)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readRawKey(IFile.java:731)
at
org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawKey(IFile.java:727)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.readNextFromCurrentReader(UnorderedKVReader.java:151)
at
org.apache.tez.runtime.library.common.readers.UnorderedKVReader.next(UnorderedKVReader.java:112)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$KeyValuesFromKeyValue.next(ReduceRecordSource.java:439)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232)
... 15 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2348 PreCommit Build #512
Jira: https://issues.apache.org/jira/browse/TEZ-2348
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/512/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2767 lines...]
[INFO] Final Memory: 70M/862M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727137/TEZ-2348.3.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/512//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/512//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
85a68c2d471b930a1d1d66002832588895e049c7 logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #510
Archived 44 artifacts
Archive block size is 32768
Received 24 blocks and 1959571 bytes
Compression is 28.6%
Took 1.9 sec
Description set: TEZ-2348
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2345) Tez UI: Enable cell level loading in all DAGs table
[ https://issues.apache.org/jira/browse/TEZ-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-2345: -- Summary: Tez UI: Enable cell level loading in all DAGs table (was: TEZ-UI: Enable cell level loading in all DAGs table) Tez UI: Enable cell level loading in all DAGs table --- Key: TEZ-2345 URL: https://issues.apache.org/jira/browse/TEZ-2345 Project: Apache Tez Issue Type: Sub-task Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Attachments: TEZ-2345.1.patch, TEZ-2345.2.patch - Enable cell level loading in all DAGs table using basic-ember-table component. - Re-arrange UI element into make it similar to other tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2345) Tez UI: Enable cell level loading in all DAGs table
[ https://issues.apache.org/jira/browse/TEZ-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506635#comment-14506635 ] Prakash Ramachandran commented on TEZ-2345: --- +1 LGTM committing shortly Tez UI: Enable cell level loading in all DAGs table --- Key: TEZ-2345 URL: https://issues.apache.org/jira/browse/TEZ-2345 Project: Apache Tez Issue Type: Sub-task Reporter: Sreenath Somarajapuram Assignee: Sreenath Somarajapuram Attachments: TEZ-2345.1.patch, TEZ-2345.2.patch - Enable cell level loading in all DAGs table using basic-ember-table component. - Re-arrange UI element into make it similar to other tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2248) VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
[
https://issues.apache.org/jira/browse/TEZ-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506534#comment-14506534
]
TezQA commented on TEZ-2248:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727132/TEZ-2248-1.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/511//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/511//console
This message is automatically generated.
VertexImpl/DAGImpl.checkForCompletion have too many termination cause checks
Key: TEZ-2248
URL: https://issues.apache.org/jira/browse/TEZ-2248
Project: Apache Tez
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Jeff Zhang
Attachments: TEZ-2248-1.patch
There is an if check for each termination cause which makes code long and we
need to handle each new termination cause with more code. This could be
abstracted into a method that gets termination cause string based on the enum
and make this method shorter and stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2341) TestMockDAGAppMaster.testBasicCounters fails on windows
[
https://issues.apache.org/jira/browse/TEZ-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506586#comment-14506586
]
TezQA commented on TEZ-2341:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727141/TEZ-2341-2.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/513//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/513//console
This message is automatically generated.
TestMockDAGAppMaster.testBasicCounters fails on windows
---
Key: TEZ-2341
URL: https://issues.apache.org/jira/browse/TEZ-2341
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Priority: Minor
Attachments: TEZ-2341-1.patch, TEZ-2341-2.patch
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.tez.dag.app.TestMockDAGAppMaster.testBasicCounters(TestMockDAGAppMaster.java:323)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2341 PreCommit Build #513
Jira: https://issues.apache.org/jira/browse/TEZ-2341
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/513/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2764 lines...]
[INFO] Final Memory: 70M/947M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12727141/TEZ-2341-2.patch
against master revision ec45c51.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/513//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/513//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7a6ff3453e8957e9e31b92dd04cd42bcbc73af0a logged out
==
==
Finished build.
==
==
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #512
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2485182 bytes
Compression is 9.5%
Took 2.2 sec
Description set: TEZ-2341
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-2161) Support CRDT aggregation models for counters
[ https://issues.apache.org/jira/browse/TEZ-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-2161: - Summary: Support CRDT aggregation models for counters (was: Support different aggregation models for counters ) Support CRDT aggregation models for counters - Key: TEZ-2161 URL: https://issues.apache.org/jira/browse/TEZ-2161 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Some counters such as last event received time need to be handled different to say bytes read counters. Bytes reads requires a summation across all tasks within a vertex. The received time requires doing a max() across all the tasks. First event received time would likely need a min(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2161) Support CRDT aggregation models for counters
[ https://issues.apache.org/jira/browse/TEZ-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506741#comment-14506741 ] Gopal V commented on TEZ-2161: -- The aggregations can be done consistently if and only if any two versions of the counter group can be merged associatively commutatively. Support CRDT aggregation models for counters - Key: TEZ-2161 URL: https://issues.apache.org/jira/browse/TEZ-2161 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Some counters such as last event received time need to be handled different to say bytes read counters. Bytes reads requires a summation across all tasks within a vertex. The received time requires doing a max() across all the tasks. First event received time would likely need a min(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2356) TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
[
https://issues.apache.org/jira/browse/TEZ-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506655#comment-14506655
]
Thejas M Nair commented on TEZ-2356:
Yes, pig is compiling against 0.7.0-SNAPSHOT.
I think it makes sense for this api call to throw TezException. That gives the
api more flexibility. (It can choose not to actually throw as well).
TEZ-2292 breaks VertexManagerPluginContext.reconfigureVertex api
Key: TEZ-2356
URL: https://issues.apache.org/jira/browse/TEZ-2356
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Thejas M Nair
Priority: Blocker
This breaks pig compilation and needs urgent attention.
{code}
src/org/apache/pig/backend/hadoop/executionengine/tez/runtime/PigGraceShuffleVertexManager.java:173:
error: exception TezException is never thrown in body of corresponding try
statement
[javac] } catch (TezException e) {
[javac] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TEZ-2224) EventQueue empty doesn't mean events are consumed in RecoveryService
[ https://issues.apache.org/jira/browse/TEZ-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang resolved TEZ-2224. - Resolution: Fixed Resolve it, the unit test fail is resolve in another ticket. EventQueue empty doesn't mean events are consumed in RecoveryService Key: TEZ-2224 URL: https://issues.apache.org/jira/browse/TEZ-2224 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.7.0, 0.5.4, 0.6.1 Attachments: TEZ-2224-1.patch, TEZ-2224-2-addendum.patch, TEZ-2224-2.patch If the event queue is empty, the event may still been processing. Should fix it like AsyncDispatcher -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2261) Should add diagnostics in DAGAppMaster when recovery error happens
[
https://issues.apache.org/jira/browse/TEZ-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486815#comment-14486815
]
Jeff Zhang edited comment on TEZ-2261 at 4/22/15 8:45 AM:
--
Upload new patch to fix one dead lock issue in the last patch. [~hitesh] Please
help review it.
was (Author: zjffdu):
Upload new patch to fix one dead lock issue in the last patch.
Should add diagnostics in DAGAppMaster when recovery error happens
--
Key: TEZ-2261
URL: https://issues.apache.org/jira/browse/TEZ-2261
Project: Apache Tez
Issue Type: Bug
Reporter: Jeff Zhang
Assignee: Jeff Zhang
Attachments: TEZ-2261-1.patch, TEZ-2261-2.patch, TEZ-2261-3.patch
Should add diagnostics in DAGAppMaster when recovery error happens, otherwise
AM is shutdown and the next dag submission will just throw
SessionNotRunningException which would confuse users.
{code}
if (this.historyEventHandler.hasRecoveryFailed()) {
LOG.warn(Recovery had a fatal error, shutting down session after +
DAG completion);
sessionStopped.set(true);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
