htran1 commented on a change in pull request #2570: [GOBBLIN-699] Orc
compaction impl.
URL: https://github.com/apache/incubator-gobblin/pull/2570#discussion_r265748952
##########
File path:
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/mapreduce/orc/OrcKeyCompactorOutputFormat.java
##########
@@ -39,4 +49,22 @@ public synchronized OutputCommitter
getOutputCommitter(TaskAttemptContext contex
}
return this.committer;
}
+
+ /**
+ * Required for extension since super method hard-coded file extension as
".orc". To keep flexibility
+ * of extension name, we made it configuration driven.
+ * @param taskAttemptContext The source of configuration that determines the
file extension
+ * @return The {@link RecordWriter} that write out Orc object.
+ * @throws IOException
+ */
+ @Override
+ public RecordWriter getRecordWriter(TaskAttemptContext taskAttemptContext)
throws IOException {
+ Configuration conf = taskAttemptContext.getConfiguration();
+ String extension = "." + conf.get(COMPACTION_OUTPUT_EXTENSION, "orc" );
Review comment:
Do you think empty extension should be supported?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
