[ https://issues.apache.org/jira/browse/ARROW-16240 ]
Joris Van den Bossche deleted comment on ARROW-16240:
-----------------------------------------------
was (Author: jorisvandenbossche):
cc @westonpace do you remember if this has been discussed before how the
row_group_size/chunk_size setting from Parquet fits into the dataset API?
The dataset API now has a {{max_rows_per_group}}, I see, but that doesn't
necessarily directly relate to Parquet row groups?
It's more generic about how many rows are written in one go, but so effectively
is therefore also a max parquet row group size? (since those need to be written
in one go) In that sense the parquet {{row_group_size}} keyword could be
translated into that keyword to preserve the intended usecase?
> [Python] Support row_group_size/chunk_size keyword in pq.write_to_dataset
> with use_legacy_dataset=False
> -------------------------------------------------------------------------------------------------------
>
> Key: ARROW-16240
> URL: https://issues.apache.org/jira/browse/ARROW-16240
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Alenka Frim
> Priority: Major
> Fix For: 8.0.0
>
>
> The {{pq.write_to_dataset}} (legacy implementation) supports the
> {{row_group_size}}/{{chunk_size}} keyword to specify the row group size of
> the written parquet files.
> Now that we made {{use_legacy_dataset=False}} the default, this keyword
> doesn't work anymore.
> This is because {{dataset.write_dataset(..)}} doesn't support the parquet
> {{row_group_size}} keyword. The {{ParquetFileWriteOptions}} class doesn't
> support this keyword.
> On the parquet side, this is also the only keyword that is not passed to the
> {{ParquetWriter}} init (and thus to parquet's {{WriterProperties}} or
> {{ArrowWriterProperties}}), but to the actual {{write_table}} call. In C++
> this can be seen at
> https://github.com/apache/arrow/blob/76d064c729f5e2287bf2a2d5e02d1fb192ae5738/cpp/src/parquet/arrow/writer.h#L62-L71
> See discussion:
> [https://github.com/apache/arrow/pull/12811#discussion_r845304218]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
