mcvsubbu commented on a change in pull request #4800: Pinot Ingestion Jobs
Refactor with Pinot Spark Support
URL: https://github.com/apache/incubator-pinot/pull/4800#discussion_r345930218
##########
File path:
pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
##########
@@ -42,84 +38,31 @@
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.pinot.common.config.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.common.config.TableConfig;
-import org.apache.pinot.common.data.Schema;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper;
-import org.apache.pinot.hadoop.utils.JobPreparationHelper;
-import org.apache.pinot.hadoop.utils.PushLocation;
+import org.apache.pinot.hadoop.utils.PinotHadoopJobPreparationHelper;
+import org.apache.pinot.ingestion.common.JobConfigConstants;
+import org.apache.pinot.ingestion.jobs.SegmentCreationJob;
+import org.apache.pinot.ingestion.utils.JobPreparationHelper;
-public class SegmentCreationJob extends BaseSegmentJob {
- protected static final String APPEND = "APPEND";
-
- protected final Path _inputPattern;
- protected final Path _outputDir;
- protected final Path _stagingDir;
- protected final String _rawTableName;
-
- // Optional
- protected final Path _depsJarDir;
- protected final Path _schemaFile;
- protected final String _defaultPermissionsMask;
- protected final List<PushLocation> _pushLocations;
-
+public class HadoopSegmentCreationJob extends SegmentCreationJob {
// Output Directory FileSystem
protected FileSystem _outputDirFileSystem;
- public SegmentCreationJob(Properties properties) {
+ public HadoopSegmentCreationJob(Properties properties) {
Review comment:
Renaming this class is a backwards-incoompatible. Open source community can
extend (and indeed, we do extend in Linkedin) this class to create their custom
hadoop jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
