JingsongLi commented on code in PR #9370:
URL: https://github.com/apache/paimon/pull/9370#discussion_r3932065251
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala:
##########
@@ -124,6 +124,12 @@ case class PaimonSparkWriter(
}
def write(data: DataFrame): Seq[CommitMessage] = {
+ if (coreOptions.bucketPerPartitionCountEnabled()) {
Review Comment:
[P1] This guard only covers the V1 `PaimonSparkWriter` path. With
`spark.paimon.write.use-v2-write=true`, `PaimonSparkTableBase.newWriteBuilder`
returns `PaimonV2WriteBuilder`, whose `PaimonV2Write.toBatch` writes directly
through `BatchWriteBuilder` and never calls this method. Therefore Spark V2
INSERT/OVERWRITE is still accepted even though the docs say Spark rejects this
table mode; its `PaimonWriteRequirement` also still clusters with the
table-level bucket count, so a rescaled partition can be distributed with the
wrong layout. Please reject this option in the shared Spark write-builder/table
entry point (or in both V1 and V2 paths) and cover both `write.use-v2-write`
settings in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
