charlesdong1991 commented on code in PR #3643:
URL: https://github.com/apache/fluss/pull/3643#discussion_r3610794369
##########
fluss-rust/crates/fluss/src/client/table/append.rs:
##########
@@ -141,12 +186,61 @@ impl AppendWriter {
Arc::new(PhysicalTablePath::of(Arc::clone(&self.table_path)))
};
+ let Some(router) = self.bucket_router.as_ref() else {
+ return self.send_arrow_batch(batch, physical_table_path, None);
+ };
+ if batch.num_rows() == 0 {
+ return self.send_arrow_batch(batch, physical_table_path, None);
Review Comment:
so empty batch will result in sticky assigner, but if the first batch is
empty, will this mistake all following keyed appends?
maybe we can add a test to see how it behaves?
##########
fluss-rust/crates/fluss/src/client/table/append.rs:
##########
@@ -141,12 +186,61 @@ impl AppendWriter {
Arc::new(PhysicalTablePath::of(Arc::clone(&self.table_path)))
};
+ let Some(router) = self.bucket_router.as_ref() else {
+ return self.send_arrow_batch(batch, physical_table_path, None);
+ };
+ if batch.num_rows() == 0 {
+ return self.send_arrow_batch(batch, physical_table_path, None);
+ }
+
+ // Group rows by bucket, keeping one key per bucket (it hashes back
there).
+ let num_buckets = self.table_info.get_num_buckets();
+ let batch_arc = Arc::new(batch.clone());
+ let row_type = Arc::new(self.table_info.row_type.clone());
+ let mut groups: HashMap<i32, (Vec<u32>, Bytes)> = HashMap::new();
+ for i in 0..batch.num_rows() {
+ let row = ColumnarRow::new(Arc::clone(&batch_arc),
Arc::clone(&row_type), i, None)?;
+ let (bucket, key) = router.bucket_of(&row, num_buckets)?;
Review Comment:
Nit: it might not be very efficient when row numbers are large given this
will do full typed-batch builds on each iteration, because it will also
downcast each column? maybe can utilize set_row_id in ColumnarRow and move the
cursor instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
