lxy-9602 commented on code in PR #8355:
URL: https://github.com/apache/paimon/pull/8355#discussion_r3478596062
##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -344,7 +362,13 @@ private RollingFileWriter<InternalRow, DataFileMeta>
createRollingRowWriter() {
FileSource.APPEND,
asyncFileWrite,
statsDenseStore,
- writeCols);
+ writeCols,
+ sharedShredding
+ ? new MapSharedShreddingWritePlanFactory(
Review Comment:
Thanks for the careful review. I agree this is an important gap. This PR
currently focuses on introducing the write-side physical layout, but the normal
scan path still reads with the logical MAP schema, so we should not expose this
as a usable end-to-end feature before the reader-side unshredding path is in
place.
My original thought was to keep this PR relatively small and reviewable by
treating it as internal groundwork. To avoid users accidentally enabling an
incomplete path, I can add schema validation to reject the shared-shredding
option for now, and then enable the option together with the reader conversion,
documentation, and end-to-end logical read tests in a follow-up PR.
While, I am also fine with making this PR larger and adding the reader-side
conversion plus the end-to-end test here if you think the feature should only
land when the full read/write path is included. Which approach would you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
