[
https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988062#comment-15988062
]
Harish Jaiprakash commented on TEZ-3701:
----------------------------------------
Sorry [~jlowe]. Looks like a different issue w.r.t Deflater used to compress
data. Deflater is not thread safe, the following code gets invoked from
multiple threads. deflater.reset() effects might be nullified by another thread
and is causing the thread to fail. I'm not sure what happens if there are 2
writes and a finish. I do not know if this cause the FileNotFound, but its
another race condition. It was introduced recently, when the spillExecutor was
changed from single threaded to use multiple threads.
{code}
public static ByteString compressByteArrayToByteString(byte[] inBytes,
Deflater deflater) throws IOException {
deflater.reset();
ByteString.Output os = ByteString.newOutput();
DeflaterOutputStream compressOs = null;
try {
compressOs = new DeflaterOutputStream(os, deflater);
compressOs.write(inBytes);
compressOs.finish();
ByteString byteString = os.toByteString();
return byteString;
} finally {
if (compressOs != null) {
compressOs.close();
}
}
}
{code}
> UnorderedPartitionedKVWriter does not wait for submitted tasks to finish
> before finalMerge.
> -------------------------------------------------------------------------------------------
>
> Key: TEZ-3701
> URL: https://issues.apache.org/jira/browse/TEZ-3701
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Priority: Blocker
>
> UnorderedPartitionedKVWriter add task to the executor, but does not wait for
> them to finish before starting the final merge. This can cause finalMerge to
> fail or write incorrect data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
