[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781080#comment-16781080
]
Jonathan Eagles commented on TEZ-3664:
--------------------------------------
[~yeshavora], I was able to find recent occurrences of this issue and was able
to possibly analyze the cause.
Similar but not exact test case failing with cause
{code}
FAILED:
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testTextMixedRecordsWithoutFinalMerge[test[false,
PRECISE]]
Error Message:
chmod: cannot access
'/tmp/TestUnorderedPartitionedKVWriter/outDir_1552b697-70ad-4f9a-85bf-c0155af75fd5/output/1552b697-70ad-4f9a-85bf-c0155af75fd5_17/file.out.index':
No such file or directory
Stack Trace:
org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access
'/tmp/TestUnorderedPartitionedKVWriter/outDir_1552b697-70ad-4f9a-85bf-c0155af75fd5/output/1552b697-70ad-4f9a-85bf-c0155af75fd5_17/file.out.index':
No such file or directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
at org.apache.hadoop.util.Shell.run(Shell.java:887)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:777)
at
org.apache.tez.runtime.library.common.sort.impl.TezSpillRecord.writeToFile(TezSpillRecord.java:146)
at
org.apache.tez.runtime.library.common.sort.impl.TezSpillRecord.writeToFile(TezSpillRecord.java:122)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.handleSpillIndex(UnorderedPartitionedKVWriter.java:1147)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.writeLargeRecord(UnorderedPartitionedKVWriter.java:1125)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:412)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:368)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.textTest(TestUnorderedPartitionedKVWriter.java:424)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testTextMixedRecordsWithoutFinalMerge(TestUnorderedPartitionedKVWriter.java:344)
{code}
I think the key take-away is that tests are writing outside of the test
directory and in this case the '/tmp' directory. Many machines may have a temp
cleaner that deletes temporary files written to this directory. A fix is to
write test data within the repository as does Apache Hadoop to keep better
control of the files under test.
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
> ------------------------------------------------------------
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Yesha Vora
> Priority: Major
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:<init>(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:<init>(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true,
> numPartitions=10
> 2017-03-20 19:50:53,871 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) -
> destinationVertexName: Moving to next buffer and triggering spill
> 2017-03-20 19:50:53,913 INFO [UnorderedOutSpiller {destinationVertexName}]
> writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 0
> 2017-03-20 19:50:53,934 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=false),
> spillId=0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName:
> Waiting for all spills to complete : Pending : 0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All
> spills complete
> 2017-03-20 19:50:53,948 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 1
> 2017-03-20 19:50:53,959 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=true),
> spillId=1
> 2017-03-20 19:50:53,972 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
