Arnaud Nauwynck created PARQUET-2416:
----------------------------------------
Summary: honor conf "mapreduce.outputcommitter.factory.class" with
PathOutputCommitterFactory in ParquetOutputFormat.getOutputCommitter
Key: PARQUET-2416
URL: https://issues.apache.org/jira/browse/PARQUET-2416
Project: Parquet
Issue Type: Bug
Reporter: Arnaud Nauwynck
The class ParquetOutputFormat extends
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat and override method
getOutputCommitter(TaskAttemptContext), but unfortunatly, it replaces by a
hard-coded new class instead of configurable factory PathOutputCommitterFactory
that use configuration property "mapreduce.outputcommitter.factory.class"
See bugged overriding code :
{code:java}
package org.apache.parquet.hadoop;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter;
public class ParquetOutputFormat<T> extends FileOutputFormat<Void, T> {
public synchronized OutputCommitter getOutputCommitter(TaskAttemptContext
context) throws IOException {
if (this.committer == null) {
Path output = getOutputPath(context);
committer = new ParquetOutputCommitter(output, context); // <==
BUG: hard coded class, ignore propoerty
"mapreduce.outputcommitter.factory.class"
}
return this.committer;
}
{code}
See overriden code:
{code:java}
package org.apache.hadoop.mapreduce.lib.output;
public abstract class FileOutputFormat<K, V> extends OutputFormat<K, V> {
@Override
public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws
IOException {
if (committer == null) {
Path output = getOutputPath(context);
committer = PathOutputCommitterFactory.getCommitterFactory(output,
context.getConfiguration()).createOutputCommitter(output, context); // <== OK:
configurable factory, accept property "mapreduce.outputcommitter.factory.class"
}
return committer;
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
