tolleybot opened a new pull request, #55658:
URL: https://github.com/apache/arrow/pull/55658

   ### Rationale for this change
   
   `min_rows_per_group` is documented as making the dataset writer "only write 
the row groups to the disk
   when sufficient rows have accumulated". In practice the writer stops holding 
rows back once 2,097,152
   rows are staged in total across all files being written (`max_rows_staged` 
in `dataset_writer.cc`), and
   then writes smaller row groups. A value above that limit is never met, and a 
partitioned write can
   produce row groups well below `min_rows_per_group`. See #55642 for 
reproducers and measurements.
   
   This PR documents the current behaviour so that users are not surprised by 
it. It does not change the
   behaviour, and it is not meant to settle whether the limit itself should 
change.
   
   ### What changes are included in this PR?
   
   One paragraph describing the limit, added in three places:
   
   * the `min_rows_per_group` entry in the `pyarrow.dataset.write_dataset` 
docstring
   * the doc comment on `FileSystemDatasetWriteOptions::min_rows_per_group` in 
`file_base.h`
   * the "Configuring rows per group during a write" note in the Python dataset 
user guide
   
   The R documentation for `write_dataset` is not changed.
   
   ### Are these changes tested?
   
   Documentation only, no tests added. I have not built the docs locally.
   
   ### Are there any user-facing changes?
   
   Documentation only.
   
   ### Was AI used for this PR?
   
   In accordance to the [AI generation 
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
 please disclose below whether and how AI was used in this PR.
   
   **PR code and description written by:**
   
   - [x] Human
   - [x] AI
   
   **Reviewed before submission by:**
   
   - [x] Human
   - [x] AI
   - [ ] Not reviewed
   
   * GitHub Issue: #55642
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] GH-55642: [C++][Python][D... tolleybot (via GitHub) via github

Reply via email to