XiaoHongbo-Hope commented on PR #8985:
URL: https://github.com/apache/paimon/pull/8985#issuecomment-5154799933
> Thanks for the update. I re-tested the two original issues: multiple
`write_arrow_batch` calls now produce the same bucket plan as the concatenated
input, and `postpone.target-row-num-per-bucket` now correctly takes precedence.
The Daft legacy fallback also looks reasonable.
>
> I found one remaining API consistency issue: `BatchTableWrite.write_row`
still bypasses the new postpone planning path. It immediately calls
`extract_partition_bucket_row`, and
`PostponeFixedBucketRowKeyExtractor.num_buckets` defaults an unknown partition
to one bucket.
>
> With `postpone.target-row-num-per-bucket=1`,
`postpone.batch-write-fixed-bucket.max-parallelism=8`, and 8 rows in one new
partition, I get:
>
> ```
> write_arrow: {8}
> write_row: {1}
> ```
>
> This means the bucket count depends on which public write API is used.
Calling `write_row` first also records the new partition as one bucket, so
subsequent Arrow batches cannot re-plan it. Please either buffer and plan row
writes together with Arrow inputs, or explicitly reject `write_row` for
postpone fixed-bucket batch writes, and add a regression test.
>
> One additional resource concern: `_postpone_batches` retains all
unknown-partition `RecordBatch` objects until `prepare_commit`, so the chunked
API now has memory usage proportional to the complete input and cannot benefit
from normal incremental file rolling. Spark can persist/spill and Ray can use
object-store spilling; the local Python path has no equivalent. Please at least
document this memory bound, or consider temporary spilling or an explicit
preplanned bucket map.
Thanks for the follow-up. write_row() now joins the deferred bucket planning
with Arrow inputs. Added pure-row and mixed-input tests, and documented the
local buffering memory bound. Existing-bucket partitions still write
incrementally. cc @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
