sundapeng opened a new pull request, #82: URL: https://github.com/apache/paimon-mosaic/pull/82
## Problem A customer reported apparent **"high CPU + memory leak"** when writing sensor data to mosaic on Android (ARMv8-A, Android 16), via the FFI/C++ writer. Process RSS climbed steadily through a long write session and never came back down, eventually risking OOM on the device. ## Root cause: buffering, not a leak This is **not a memory leak**. `WriterOptions::row_group_max_size` defaults to **256 MB** ([`DEFAULT_ROW_GROUP_MAX_SIZE`](https://github.com/apache/paimon-mosaic/blob/main/core/src/spec.rs)). A writer buffers every row of the in-progress row group in memory and flushes (compresses + writes) **only once** that much data has accumulated: ```rust // core/src/writer.rs self.current_buffered_size += size; if self.current_buffered_size >= self.row_group_max_size { self.flush_row_group()?; } ``` For a **long-lived streaming writer** — continuous ingestion, mobile/embedded sensors, anything that writes small batches over a long session — this default holds an unbounded amount of data in RAM: - RSS grows roughly **1:1 with the buffered rows** until the 256 MB cap is reached. - A session **shorter than the cap flushes nothing until `close()`**, so the *entire* session lives in memory. On a phone this looks and behaves like a leak and can trigger OOM. The high-CPU observation is the sustained Arrow-FFI import + bucket append, plus one giant zstd burst when the cap (or `close()`) is finally hit — an ANR risk if the write runs on the UI thread. ## Reproduction (Android 16, API 36 emulator, ARMv8-A target) Two runs of the same workload (3000 batches × 1024 rows = 3,072,000 rows, 10 motion columns + `device_id`), one with the 256 MB default, one with an 8 MB cap: | | 256 MB default (customer condition) | 8 MB cap | |---|---|---| | **peak RSS** | 134 → **330 MB** (unbounded growth) | 156 → **174 MB** (plateaus, then drops) | | **peak native heap** | 15 → **278 MB** | 17 → **31 MB** (**8.9× lower**) | | **row groups** | **1** (one giant flush at close) | **23** (~134k rows each) | | **throughput** | 503 batch/s + giant zstd burst at close | 1409 batch/s, flushes spread out | | **verify** | PASS, 3,072,000 rows | PASS, 3,072,000 rows | Both files read back correctly with the upstream `mosaic` CLI (`schema` / `count` / `cat`). The "leak" disappears entirely with a smaller cap: memory is bounded and the native heap stays flat at ~31 MB. macOS `leaks` (host harness) independently reported **0 leaked bytes** — the write/close path frees correctly; the RSS growth is by-design buffering. The 16 KB page-alignment requirement of Android 15+/16 was ruled out (all 4 ABI `.so` are `0x4000`-aligned). The behavior is OS-independent — it is the Rust core's buffering policy. ## Fix: documentation + recommendation (default unchanged) The 256 MB default is **intentional** for batch/server wide-table workloads, where a large row group maximizes compression ratio and amortizes seek cost. Changing the global default would risk regressing that primary use case, so this PR leaves it unchanged and instead **documents the tradeoff** and recommends a smaller `row_group_max_size` (e.g. 8 MB) for long-lived streaming/mobile writers: - `core/src/spec.rs` — rustdoc on `DEFAULT_ROW_GROUP_MAX_SIZE` - `core/src/writer.rs` — rustdoc on `WriterOptions` + the `row_group_max_size` field - `include/mosaic.hpp` — inline comment on the C++ `WriterOptions::row_group_max_size` - `docs/cpp-api.html`, `docs/java-api.html`, `docs/python-api.html` — table cells + "Memory & long-lived writers" warning callouts with per-language examples No behavior change, no default change — docs only. ## Alternatives considered - **Lower the global default** (e.g. to 8 MB): rejected — would regress compression ratio / row-group count for batch/server workloads, and is a behavior change that warrants broader consensus. Could be revisited as a separate discussion if mobile/streaming becomes a primary target. - **Add a streaming preset / builder helper**: possible follow-up, but additive API surface is out of scope for a docs-first fix. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
