JingsongLi commented on code in PR #8355:
URL: https://github.com/apache/paimon/pull/8355#discussion_r3475156824
##########
paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java:
##########
@@ -344,7 +362,13 @@ private RollingFileWriter<InternalRow, DataFileMeta>
createRollingRowWriter() {
FileSource.APPEND,
asyncFileWrite,
statsDenseStore,
- writeCols);
+ writeCols,
+ sharedShredding
+ ? new MapSharedShreddingWritePlanFactory(
Review Comment:
This enables writing the shared-shredding physical ROW layout, but I do not
see a matching logical read/unshredding path. The normal table scan / format
reader still requests the logical MAP schema; for Parquet, clipParquetType(MAP)
expects a Parquet map group, so it will not be able to read this physical ROW.
Please add an end-to-end test that writes through this path and reads the table
back with the logical schema, and wire the reader conversion before enabling
the write path.
##########
paimon-core/src/main/java/org/apache/paimon/operation/BaseAppendFileStoreWrite.java:
##########
@@ -155,7 +163,8 @@ protected RecordWriter<InternalRow> createWriter(
options.asyncFileWrite(),
options.statsDenseStore(),
options.dataEvolutionEnabled(),
- blobContext);
+ blobContext,
+ sharedShreddingContext);
Review Comment:
Passing the shared-shredding context only to AppendOnlyWriter leaves the
rewrite paths inconsistent. compactRewrite/clusterRewrite below still create a
plain RowDataRollingFileWriter, so compaction can rewrite shared-shredded
append files back into the default MAP layout without shared-shredding
metadata. Please either route those rewrite writers through the same write plan
or explicitly disable/guard compaction for shared-shredding append tables until
it is supported.
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -695,6 +695,11 @@ private static void validateFileIndex(TableSchema schema) {
+ "Only CHAR/VARCHAR/STRING is supported.",
columnName,
keyType);
+ checkArgument(
+ options.mapStorageLayout(columnName) !=
MapStorageLayout.SHARED_SHREDDING,
Review Comment:
This validation catches nested map indexes, but shared-shredding can still
be configured with formats whose writers do not implement
SupportsWriterMetadata. In that case the table is accepted and the write fails
only when the file is closing. Please reject unsupported file formats during
schema validation, or otherwise fail before any data file write starts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
