XuQianJin-Stars opened a new pull request, #3572: URL: https://github.com/apache/fluss/pull/3572
Introduce a new `max_bytes_per_batch` Lance table property (default `0`, meaning disabled) that lets the tiering writer flush a batch as soon as its underlying Arrow off-heap allocation reaches the configured number of bytes, in addition to the existing `batch_size` row-count threshold. **Motivation:** - With very wide rows (long strings, large binary/vector columns) the row-count threshold under-flushes, driving peak allocator memory too high. - With very narrow rows the row-count threshold over-flushes and produces many tiny fragments, hurting Lance read performance. **Behavior:** - `batch_size` semantics are preserved (defaults to `512` rows). - `max_bytes_per_batch` defaults to `0`, which fully preserves the previous row-count-only behavior — no change for existing tables. - When `max_bytes_per_batch > 0`, `LanceLakeWriter` flushes as soon as *either* the row count reaches `batch_size` *or* the accumulated Arrow field-vector buffer size for the current row count reaches `max_bytes_per_batch`, whichever comes first. - Negative values are rejected at config parse time with `IllegalArgumentException`. ## Purpose Give operators a byte-oriented knob to control when `LanceLakeWriter` flushes an in-flight row batch to Lance, in addition to the existing row-count threshold. A fixed row-count threshold works poorly across mixed workloads (see Motivation). This aligns Lance tiering with how Iceberg/Paimon writers already expose byte-based flush controls, and bounds peak allocator memory during tiering. Linked issue: close #xxx ## Brief change log - `LanceConfig`: introduce the `max_bytes_per_batch` option and a `getMaxBytesPerBatch(config)` helper. Default is `0` (disabled). Negative values are rejected with `IllegalArgumentException`. - `LanceLakeWriter`: add a `maxBytesPerBatch` field and extract a `shouldFlush()` helper called from `write()`. `shouldFlush()` returns true when the row count reaches `batch_size`, or (when `maxBytesPerBatch > 0`) when the buffer size reported by `VectorSchemaRoot#getBufferSizeFor(recordsCount)` reaches `maxBytesPerBatch`. Uses `getBufferSizeFor(recordsCount)` rather than `getValueCount()` so the check is valid before `finish()` has propagated the row count to the vectors. - Add `LanceConfigTest` with 8 test methods covering default / override / zero-explicit / negative / invalid / missing-warehouse paths for both `batch_size` and `max_bytes_per_batch`. ## Tests - New `LanceConfigTest` (8 tests) covers all parsing branches for the two thresholds. - Existing `LanceTieringTest`, `LanceArrowUtilsTest`, and `ArrowDataConverterTest` continue to pass unchanged. - Full module run: `./mvnw -pl fluss-lake/fluss-lake-lance clean test -Dspotless.check.skip` — 27/27 pass. - Checkstyle: `./mvnw -pl fluss-lake/fluss-lake-lance checkstyle:check` — 0 violations. ## API and Format - No public Java API changes. - New table property `max_bytes_per_batch` on Lance-tiered tables. Default `0` (disabled) keeps existing tables' behavior identical, so this change is backward compatible. - `batch_size` semantics are preserved (default `512` rows). - No wire format, on-disk format, or Lance dataset format changes. ## Documentation - The new property is documented via inline Javadoc on `LanceConfig#getMaxBytesPerBatch`, describing its default, its interaction with `batch_size`, and the sizing rationale (wide-row memory cap vs. narrow-row fragmentation). - No user-facing site docs update is bundled here since the Lance tiering config surface is not yet documented on the Fluss site; a follow-up docs PR can pick this up together with other Lance properties. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
