JingsongLi opened a new pull request, #8864: URL: https://github.com/apache/paimon/pull/8864
## What changed - Infer the fixed bucket count for new postpone partitions from exact incoming row counts when `postpone.target-row-num-per-bucket` is configured. - Include existing postpone file row counts and cap the inferred count with `postpone.batch-write-fixed-bucket.max-parallelism`. - Preserve known bucket counts and the existing writer-count fallback when the target row count is absent. - Cache the Spark input between counting and writing. - Document the behavior and add Spark regression coverage. ## Why Spark previously used the number of input writers as the bucket count for partitions without real bucket data. Highly parallel jobs could therefore initialize many mostly empty buckets and create excessive small files. ## Validation - `mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PostponeBucketTableTest -Dtest=none test` - `mvn -pl paimon-api,paimon-spark/paimon-spark-common,paimon-spark/paimon-spark-ut -Pspark3 -DskipTests compile` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
