cpaulik opened a new issue, #44525:
URL: https://github.com/apache/arrow/issues/44525
### Describe the bug, including details regarding any error messages,
version, and platform.
I'm trying to write a `_metadata` file from a set of existing parquet files
written by duckdb.
This is the minimal example I could come up with that reproduces the issue
```python
import duckdb
import pyarrow.parquet as pq
fname = 'parquet_file.parquet'
con = duckdb.connect()
con.sql(("CREATE TABLE t1 (num INTEGER);"
"INSERT INTO t1 VALUES (1);"
"INSERT INTO t1 VALUES (2);"))
con.sql(f"COPY t1 TO '{fname}' (FORMAT PARQUET)")
metadata = pq.read_metadata(fname)
schema = pq.read_schema(fname)
metadata_path = "_metadata"
pq.write_metadata(
schema, metadata_path,
metadata_collector=[metadata, metadata])
```
leads to
```
Traceback (most recent call last):
File
"/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/pyarrow_error.py",
line 17, in <module>
pq.write_metadata(
File
"/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/.venv/lib/python3.10/site-packages/pyarrow/parquet/core.py",
line 2247, in write_metadata
metadata.append_row_groups(m)
File "pyarrow/_parquet.pyx", line 1047, in
pyarrow._parquet.FileMetaData.append_row_groups
RuntimeError: AppendRowGroups requires equal schemas.
The two columns with index 0 differ.
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: NONE,
logical_type: None,
max_definition_level: 1,
max_repetition_level: 0,
}
column descriptor = {
name: num,
path: num,
physical_type: INT32,
converted_type: INT_32,
logical_type: Int(bitWidth=32, isSigned=true),
max_definition_level: 1,
max_repetition_level: 0,
}
```
Package versions
```
"duckdb==1.1.2",
"pyarrow>=17.0.0"
```
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
