cshuo commented on code in PR #19576:
URL: https://github.com/apache/hudi/pull/19576#discussion_r3818152095
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BulkInsertInternalPartitionerWithRowsFactory.java:
##########
@@ -30,14 +32,37 @@
*/
public abstract class BulkInsertInternalPartitionerWithRowsFactory {
- public static BulkInsertPartitioner<Dataset<Row>> get(HoodieWriteConfig
config,
+ public static BulkInsertPartitioner<Dataset<Row>> get(HoodieTable table,
Review Comment:
table config.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BulkInsertInternalPartitionerFactory.java:
##########
@@ -45,12 +45,27 @@ public static BulkInsertPartitioner get(HoodieTable table,
return new RDDSimpleBucketBulkInsertPartitioner(table);
}
}
+ if (table.getMetaClient().getTableConfig().isLSMTreeStorageLayout()) {
Review Comment:
Bucket-index support will be solved by another PR and tracked by #19439.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkBulkInsertHelper.java:
##########
@@ -69,13 +71,18 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>>
bulkInsert(final HoodieData<
final
Option<BulkInsertPartitioner> userDefinedBulkInsertPartitioner) {
HoodieWriteMetadata result = new HoodieWriteMetadata();
+ boolean isLsmTable =
table.getMetaClient().getTableConfig().isLSMTreeStorageLayout();
+ checkArgument(!isLsmTable || userDefinedBulkInsertPartitioner.isEmpty(),
Review Comment:
Yes, this is intentional. Flink rejects it because its INSERT/append mode
can be keyless, while Spark LSM writes operate on keyed records. Duplicate
physical keys are allowed by INSERT and later resolved as one logical record
during merge, as defined in #19437.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
