Daniel Dai created PIG-4086:
-------------------------------
Summary: Fix Orc e2e tests for tez
Key: PIG-4086
URL: https://issues.apache.org/jira/browse/PIG-4086
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.14.0
Attachments: PIG-4086-1.patch
All Orc e2e tests fail on tez.
There are two issue:
1. hivelibdir etc is not set in tez.conf
2. OrcStorage produce empty output file
Digging into #2, the problem is in this code in PigProcessor:
{code}
if (fileOutput.isCommitRequired()) {
fileOutput.commit();
}
{code}
fileOutput.commit() invokes both RecordWriter.close() and
committer.commitTask(). However, OrcNewOutputFormate will generate output file
only after RecordWriter.close (if the output file is small),
fileOutput.isCommitRequired will not detect this file, thus skip
fileOutput.commit().
Changing the code to invoke fileOutput.close explicitly fix the issue.
fileOutput.commit will invoke close again, but there is no side effect since
close will check if it has been already called.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
