danny0405 commented on code in PR #13360:
URL: https://github.com/apache/hudi/pull/13360#discussion_r2126120976
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBulkInsertCommitActionExecutor.java:
##########
@@ -47,47 +35,12 @@ public
DatasetBulkInsertCommitActionExecutor(HoodieWriteConfig config,
super(config, writeClient);
}
- @Override
- protected void preExecute() {
- instantTime = writeClient.startCommit();
- table = writeClient.initTable(getWriteOperationType(),
Option.ofNullable(instantTime));
- }
-
- @Override
- protected Option<HoodieData<WriteStatus>> doExecute(Dataset<Row> records,
boolean arePartitionRecordsSorted) {
- Map<String, String> opts =
writeConfig.getProps().entrySet().stream().collect(Collectors.toMap(
- e -> String.valueOf(e.getKey()),
- e -> String.valueOf(e.getValue())));
- Map<String, String> optsOverrides = Collections.singletonMap(
- HoodieInternalConfig.BULKINSERT_ARE_PARTITIONER_RECORDS_SORTED,
String.valueOf(arePartitionRecordsSorted));
-
- String targetFormat;
- Map<String, String> customOpts = new HashMap<>(1);
- if (HoodieSparkUtils.isSpark3()) {
- targetFormat = "org.apache.hudi.spark.internal";
-
customOpts.put(HoodieInternalConfig.BULKINSERT_INPUT_DATA_SCHEMA_DDL.key(),
records.schema().json());
- } else {
- throw new HoodieException("Bulk insert using row writer is not supported
with current Spark version."
- + " To use row writer please switch to spark 3");
- }
-
- records.write().format(targetFormat)
Review Comment:
can you elaborate why logic is customized before for this executor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
