jianguotian opened a new pull request, #98: URL: https://github.com/apache/paimon-mosaic/pull/98
Integer Arrow columns widened by callers currently stay at their logical width throughout row-group buffering, then acquire additional full-size encoded bucket/page buffers at flush. Compacting the integer cache and encoding large units incrementally reduces peak writer memory while preserving the input API and file schema. ### Changes - Cache Int16/Int32/Int64 values in blocks of at most 4,096 values, selecting 1/2/4/8 physical bytes by range. Negative values retain their logical width; a late outlier widens only the current block. Int8 and non-integer caches retain their existing representation. - Preserve logical accounting for dictionary selection, row-group boundaries, statistics and page layout. Keep main's lazy dictionary lifecycle and null bitmap fast paths; constant columns do not create dictionaries, and entry-limit disabling remains effective until reset. - Expand values through a reusable 16 KiB buffer and stream dictionary indexes. Encoded units up to 1 MiB use bounded staging; larger units use one Zstd frame per original bucket/page. Paged output retains compressed slots for its leading directory. - Coalesce uncompressed output through a 16 KiB buffer before file/C/JNI callbacks. Bucket completion drains this buffer without flushing the sink. Failed callbacks and unwinding do not cause buffered bytes to be retried. - Add coverage for promotions, signed extremes, nullable/sliced input, dictionary activation/limits/reset, logical boundaries, stats, output failure and reference-writer parity, plus a synthetic memory benchmark and implementation notes. No C/JNI API or file-schema changes. ### Validation Validated on 2026-10-12 with Rust 1.97.1 and the committed Cargo.lock, against official main `c082354`: - `cargo fmt --all -- --check` and `cargo clippy --locked --all-targets --workspace -- -D warnings` passed. - `python3 tools/verify_binary_artifact.py --source` passed. - Local macOS ARM64: `cargo test --locked --workspace` — **528 passed**, 0 failed; 2 fixture-dependent cross-language readers ignored by the existing upstream annotations. - Native ARM64 Linux: `cargo test --locked -p paimon-mosaic-core --lib` — **183 passed**, 0 failed; release FFI library and benchmark example built. - Additional ARM64 review harness, separate from the committed tests: **1,728 differential comparisons** against the original `c082354` bucket encoder passed across 17 columns, including nested List/Map, nullable/sliced input, dictionary limits and reset. Encoded bytes, page payloads, flags and logical sizes matched. ### ARM64 memory and timing Fresh comparison against `c082354` on ARM64 Linux (Neoverse-N2), using separate build directories. **60 runs**: five repetitions per version/scenario, alternating order and running one workload at a time. Each run writes 20 batches × 250,000 rows, three integer columns, one row group, Zstd level 1, dictionary disabled. The table shows baseline → this change, using medians; peak RSS comes from `/usr/bin/time -v`. | Layout | Input | Peak RSS (MiB) | Append total (ms) | Close (ms) | | --- | --- | ---: | ---: | ---: | | Paged | Widened unsigned | 232.3 → 79.9 | 47.4 → 14.1 | 271.4 → 231.7 | | Paged | Full-range signed | 276.7 → 142.6 | 46.7 → 23.4 | 136.8 → 75.4 | | Paged | Late outlier | 232.3 → 79.9 | 47.3 → 14.7 | 270.3 → 232.9 | | Monolithic | Widened unsigned | 180.7 → 42.3 | 46.8 → 14.2 | 233.1 → 219.4 | | Monolithic | Full-range signed | 210.1 → 75.7 | 47.2 → 23.4 | 90.4 → 50.9 | | Monolithic | Late outlier | 180.7 → 42.3 | 46.6 → 13.8 | 231.8 → 218.6 | All repetitions produced byte-identical files between versions within each scenario. The six candidate files were also decoded and all **90 million integer values** checked against the generated input. The native production source files and manifests matched the submitted revision `d664c21` by SHA-256. These are synthetic results, not Android customer measurements. Caller-created widened Arrow arrays remain. Zstd streaming is not universally byte-identical to one-shot compression; the parity result applies to this benchmark corpus. The regression matrix separately checks decoded values, schema/statistics, row-group boundaries and file sizes. Related implementation: https://github.com/jianguotian/paimon-mosaic/pull/34. This PR is based directly on official `main` and includes the adaptation to its current dictionary/null handling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
