[
https://issues.apache.org/jira/browse/ARROW-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457265#comment-17457265
]
Will Jones commented on ARROW-14701:
------------------------------------
Apologies Adrien. I was looking in the development version of Arrow. You are
right this wasn't documented in Arrow 6.0.0. New docstrings were added recently
in ARROW-13668, and will be available in Arrow 7.0.0. Here is one of them:
[https://github.com/apache/arrow/blob/b1f009ca80ef6f12fbdf56a1b53ed8d0e0571a5a/python/pyarrow/parquet.py#L697-L700]
I can make a minor PR to add this to the [write_table
function|https://github.com/apache/arrow/blob/b1f009ca80ef6f12fbdf56a1b53ed8d0e0571a5a/python/pyarrow/parquet.py#L2096-L2097]
as well.
> [Python] parquet.write_table has an undocumented and silent cap on
> row_group_size
> ---------------------------------------------------------------------------------
>
> Key: ARROW-14701
> URL: https://issues.apache.org/jira/browse/ARROW-14701
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 6.0.0
> Reporter: Adrien Hoarau
> Priority: Minor
>
> {code:java}
> from io import BytesIO
> import pandas as pd
> import pyarrow
> from pyarrow import parquet
> from pyarrow import fs
> print(pyarrow._version_)
> def check_row_groups_created(size: int):
> df = pd.DataFrame({"a": range(size)})
> t = pyarrow.Table.from_pandas(df)
> buffer = BytesIO()
> parquet.write_table(t, buffer, row_group_size=size)
> buffer.seek(0)
> print(parquet.read_metadata(buffer))
>
> check_row_groups_created(50_000_000)
> check_row_groups_created(100_000_000) {code}
> outputs:
> {code:java}
> 6.0.0
> <pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
> created_by: parquet-cpp-arrow version 6.0.0
> num_columns: 1
> num_rows: 50000000
> num_row_groups: 1
> format_version: 1.0
> serialized_size: 1493
> <pyarrow._parquet.FileMetaData object at 0x7f838584ab80>
> created_by: parquet-cpp-arrow version 6.0.0
> num_columns: 1
> num_rows: 100000000
> num_row_groups: 2
> format_version: 1.0
> serialized_size: 1640 {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
