danny0405 commented on code in PR #12245:
URL: https://github.com/apache/hudi/pull/12245#discussion_r1851871655
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BucketIndexBulkInsertPartitioner.java:
##########
@@ -59,12 +62,19 @@ public BucketIndexBulkInsertPartitioner(HoodieTable table,
String sortString, bo
this.sortColumnNames = null;
}
this.preserveHoodieMetadata = preserveHoodieMetadata;
+ // Bulk insert into COW table with bucket index is allowed only once,
otherwise AppendHadleFactory will produce MOR log files
+ this.isAppendAllowed =
!table.getConfig().getTableType().equals(HoodieTableType.COPY_ON_WRITE);
}
@Override
public Option<WriteHandleFactory> getWriteHandleFactory(int idx) {
- return doAppend.get(idx) ? Option.of(new AppendHandleFactory()) :
- Option.of(new
SingleFileHandleCreateFactory(FSUtils.createNewFileId(getFileIdPfx(idx), 0),
this.preserveHoodieMetadata));
+ if (!doAppend.get(idx)) {
+ return Option.of(new
SingleFileHandleCreateFactory(FSUtils.createNewFileId(getFileIdPfx(idx), 0),
this.preserveHoodieMetadata));
+ } else if (isAppendAllowed) {
+ return Option.of(new AppendHandleFactory());
+ } else {
+ throw new HoodieNotSupportedException("Bulk insert into COW table with
bucket index is allowed only once, please, use upsert operation instead");
Review Comment:
I kind of give the wrong info before, I reliazed that we are using bucket
index, bucket index is used only for `UPSERT` semantics, because the same keys
should always be written into one file group, if we allow multiple bulk_inserts
here as inserts, there would be a file slice with multiple base files, this may
work but usually we do not do that, the suggested coposition is: bulk_insert
(one shot for better performance) + multiple upserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
