JingsongLi commented on code in PR #659:
URL: https://github.com/apache/paimon-rust/pull/659#discussion_r3720222892
##########
crates/paimon/src/table/write_builder.rs:
##########
@@ -120,17 +159,42 @@ struct PaimonWriteBuilder<'a> {
table: &'a Table,
commit_user: String,
overwrite: bool,
+ postpone_fixed_bucket: bool,
+ postpone_bucket_plan: Option<PostponeBucketPlan>,
}
impl<'a> PaimonWriteBuilder<'a> {
pub fn new(table: &'a Table) -> Self {
+ let schema = table.schema();
+ let options = CoreOptions::new(schema.options());
+ let postpone_fixed_bucket = options.bucket() == POSTPONE_BUCKET
Review Comment:
[P1] Do not enable fixed-bucket mode before a global plan exists
This makes every ordinary new_write_builder() choose the fixed path by
default, but those builders have no PostponeBucketPlan. For a new partition,
write_batch therefore retains every input RecordBatch in buffered_batches until
prepare_commit. The streaming DataFusion sink feeds this builder without a
plan, so a first large INSERT now holds the whole input in memory, and
partitioned input may be copied by take_rows, instead of flushing
incrementally; memory grows with the complete batch and can reach OOM. Existing
multi-writer Rust/C users also still derive counts independently unless they
adopt the new setter. PyPaimon keeps its ordinary new_batch_write_builder on
the legacy path; Ray first computes one driver-side plan and only then
constructs explicit fixed writers for workers. Please keep the core default on
legacy postpone semantics until an integration can generate and inject a global
plan, or add bounded spill/two-pass planning before enabling this by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
