jianguotian opened a new pull request, #100: URL: https://github.com/apache/paimon-mosaic/pull/100
Wide writers currently build per-column dictionary hash tables while appending and retain their capacity after a row-group reset. Build dictionaries from the buffered values only when encoding each bucket, then release them before preparing the next bucket. This reduces the overlap between dictionaries and the row-group value buffers. `PreparedBucket` borrows the existing values without copying the value cache. Dictionary entry/byte limits, first-occurrence ordering, constant/all-null handling, encoding selection and row-group boundaries are preserved. The transient tables are dropped on successful output and on error. Standalone `BucketWriter` APIs remain available; `MosaicWriter` prepares once per active bucket and reuses that preparation for sizing and encoding. Only `core/src/bucket_writer.rs` and `core/src/writer.rs` change, including adapted dictionary regression tests. There are no logging, FFI, dependency or configuration changes. Validation: - [GitHub CI for this commit](https://github.com/jianguotian/paimon-mosaic/actions/runs/39231167164): Rust, C++, Java, Python and repository checks all passed. - Local workspace: 529 tests passed, two external-fixture readers ignored; formatting, all-target workspace Clippy with warnings denied, binary legal inventory and ASF configuration checks passed. - ARM64 Linux: 184 core tests and dependency license checks passed. Fresh independent baseline/candidate build directories; 48 serialized runs across eight workloads, with all 94 complete output files and row-group counts matching within each workload. Coverage includes paged, monolithic, uncompressed, streaming, constant, late-changing and high-cardinality cases. ARM64 synthetic results (median of five alternating-order runs per version, 8,779 columns, 150 rows per append, 20 appends per file, two files): | Workload | Main peak RSS | Candidate peak RSS | Total time change | | --- | ---: | ---: | ---: | | wide | 81.97 MiB | 55.86 MiB | -3.64% | | late | 82.44 MiB | 50.09 MiB | +8.26% | | high | 75.03 MiB | 66.54 MiB | -5.82% | | constant | 50.38 MiB | 50.11 MiB | -2.12% | Dictionary work moves from append into flush. In the wide case, cumulative close time increased from 252.6 to 483.1 ms. The late-changing case increased total time by 8.26% (CPU +8.33%). Standalone `estimated_raw_size` / `finish` calls each prepare their own transient dictionaries. These are synthetic Linux ARM measurements, not predicted Android customer RSS savings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
