1fanwang opened a new pull request, #10626: URL: https://github.com/apache/arrow-rs/pull/10626
# Which issue does this PR close? - Closes #9386. # Rationale for this change `ArrowWriter::write` splits a batch by calling itself on each half, so it recurses once per row group. A row limit much smaller than the batch turns that into one stack frame per row group, and the process aborts: ``` thread '...' has overflowed its stack fatal runtime error: stack overflow, aborting ``` The limits are caller-supplied, so a large batch written under a small `max_row_group_row_count` takes down the process rather than returning an error. # What changes are included in this PR? `write` now loops over the rows still to be written instead of recursing. Each pass fills the in-progress row group, flushes when a limit is reached, and carries the remainder into the next pass, so stack use is constant no matter how many row groups a batch produces. The split points and flush conditions are unchanged. # Are these changes tested? Yes. `test_row_group_limit_rows_only_many_splits` writes 50,000 rows with `max_row_group_row_count = 1` and checks every row lands in its own row group with no rows lost. It aborts with the stack overflow above on the current code and passes here. The existing `test_row_group_limit_*` cases cover the split points and flush behaviour and are unchanged; the parquet suite passes. The byte limit reaches its split path only once rows are already buffered, and re-enters with an empty row group, so it stays shallow — the row count limit was the one that could run away. # Are there any user-facing changes? No API change. Writes that previously aborted the process now complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
