[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380449#comment-15380449
]
TezQA commented on TEZ-3348:
----------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818284/11.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestTezJobs
org.apache.tez.test.TestSecureShuffle
org.apache.tez.test.TestRecovery
org.apache.tez.mapreduce.TestMRRJobs
org.apache.tez.test.TestLocalMode
org.apache.tez.mapreduce.output.TestMROutput
org.apache.tez.mapreduce.output.TestMROutputLegacy
org.apache.tez.mapreduce.processor.reduce.TestReduceProcessor
org.apache.tez.history.TestHistoryParser
org.apache.tez.tests.TestExternalTezServicesErrors
org.apache.tez.tests.TestExternalTezServices
org.apache.tez.tests.TestExtServicesWithLocalMode
The following test timeouts occurred in :
org.apache.tez.mapreduce.TestMRRJobsDAGApi
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1856//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1856//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1856//console
This message is automatically generated.
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
> --------------------------------------------------------------------------------------------------------
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Piyush Narang
> Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch, 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.<init>(Path.java:105)
> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.<init>(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
