[
https://issues.apache.org/jira/browse/TEZ-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla reopened TEZ-3738:
------------------------------
Reopening this since the test fails intermittently with the same error.
Eg.
https://builds.apache.org/job/PreCommit-TEZ-Build/2671/testReport/junit/org.apache.tez.runtime.library.common.writers/TestUnorderedPartitionedKVWriter/testMergeBuffersAndSpill_test_false__NONE__/
Stack trace:
{code}
java.util.concurrent.RejectedExecutionException: Task
com.google.common.util.concurrent.ListenableFutureTask@5d1a2c96 rejected from
java.util.concurrent.ThreadPoolExecutor@230e3749[Running, pool size = 2, active
threads = 2, queued tasks = 0, completed tasks = 1]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:440)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.scheduleSpill(UnorderedPartitionedKVWriter.java:503)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.mayBeSpill(UnorderedPartitionedKVWriter.java:477)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.getNextAvailableBuffer(UnorderedPartitionedKVWriter.java:541)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.setupNextBuffer(UnorderedPartitionedKVWriter.java:467)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:413)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:365)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:1133)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:1077)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testMergeBuffersAndSpill(TestUnorderedPartitionedKVWriter.java:269)
{code}
Standard output :
{code}
2017-10-21 01:38:20,402 INFO [main] writers.TestUnorderedPartitionedKVWriter
(TestUnorderedPartitionedKVWriter.java:setup(149)) - Setup. Using test dir:
/tmp/TestUnorderedPartitionedKVWriter
2017-10-21 01:38:20,424 INFO [Thread-83]
writers.BaseUnorderedPartitionedKVWriter
(BaseUnorderedPartitionedKVWriter.java:<init>(148)) - Instantiating
Partitioner:
[org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:<init>(285)) - destinationVertexName:
numBuffers=2, sizePerBuffer=1024, skipBuffers=false, numPartitions=10,
availableMemory=2048, maxSingleBufferSizeBytes=2048, pipelinedShuffle=false,
isFinalMergeEnabled=true, numPartitions=10, reportPartitionStats=NONE
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) -
destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,425 INFO [Thread-83] writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) -
destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,428 INFO [UnorderedOutSpiller {destinationVertexName} #0]
writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName:
Finished spill 0
2017-10-21 01:38:20,428 INFO [Thread-83] writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) -
destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,435 INFO [UnorderedOutSpiller {destinationVertexName} #1]
writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName:
Finished spill 1
2017-10-21 01:38:20,437 INFO [Thread-83] writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:setupNextBuffer(460)) -
destinationVertexName: Moving to next buffer. Total filled buffers: 0
2017-10-21 01:38:20,438 INFO [UnorderedOutSpiller {destinationVertexName} #0]
writers.UnorderedPartitionedKVWriter
(UnorderedPartitionedKVWriter.java:callInternal(638)) - destinationVertexName:
Finished spill 2
2017-10-21 01:38:20,438 INFO [main] writers.TestUnorderedPartitionedKVWriter
(TestUnorderedPartitionedKVWriter.java:cleanup(157)) - CleanUp
{code}
I cannot tell if this is broken recently(since most commits don't seem
pertinent to this) or has just been flaky enough to have not failed since June
2017(approx.).
Even though the test fails irrespective of the value of isFinalMergeEnabled,
the following code to me seemed suspect since we don't know if the spilling
phase has finished as the numSpills could change after mergeAll?
{code}
if (isFinalMergeEnabled) {
if (numSpills.get() > 0) {
mergeAll();
} else {
finalSpill();
}
updateTezCountersAndNotify();
eventList.add(generateVMEvent());
eventList.add(generateDMEvent());
}
{code}
Appreciate any thoughts, comments from [~rajesh.balamohan], [~harishjp] to help
understand the issue and if I have missed something. Thanks a lot!
> TestUnorderedPartitionedKVWriter fails due to RejectedExecutionException
> ------------------------------------------------------------------------
>
> Key: TEZ-3738
> URL: https://issues.apache.org/jira/browse/TEZ-3738
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jason Lowe
>
> TestUnorderedPartitionedKVWriter is failing in recent precommit builds.
> Stacktrace to follow.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
