jonashaag opened a new issue, #35859:
URL: https://github.com/apache/arrow/issues/35859

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   In the release notes of Arrow 12 say that the default row group size has 
been lowered from 64 Mi to 1 Mi. But here's what's happening in practice 
(PyArrow from conda-forge):
   
   ```
   In [1]: import pyarrow.parquet as pq
   
   In [2]: import pyarrow as pa
   
   In [3]: pa.__version__
   Out[3]: '12.0.0'
   
   In [12]: pq.write_table(pa.Table.from_pydict({"x": [42] * (64 * 1024 * 1024 
+ 1)}), "/tmp/x")
   
   In [13]: pq.read_metadata("/tmp/x")
   Out[13]:
   <pyarrow._parquet.FileMetaData object at 0x7fa6b81eaa70>
     created_by: parquet-cpp-arrow version 12.0.0
     num_columns: 1
     num_rows: 67108865
     num_row_groups: 2  # <=============
     format_version: 2.6
     serialized_size: 485
   
   In [14]: pq.write_table(pa.Table.from_pydict({"x": [42] * (64 * 1024 * 1024 
+ 0)}), "/tmp/x")
   
   In [15]: pq.read_metadata("/tmp/x")
   Out[15]:
   <pyarrow._parquet.FileMetaData object at 0x7fa6b823d490>
     created_by: parquet-cpp-arrow version 12.0.0
     num_columns: 1
     num_rows: 67108864
     num_row_groups: 1  # <=============
     format_version: 2.6
     serialized_size: 376
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to