legout commented on issue #31678:
URL: https://github.com/apache/arrow/issues/31678#issuecomment-1688905002
Create toy dataset with parquet files having identical column types, but
different column ordering.
```python
import os
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as pds
t1 = pa.Table.from_pydict({"A": [1, 2, 3], "B": ["a", "b", "c"]})
t2 = pa.Table.from_pydict({"B": ["a", "b", "c"], "A": [1, 2, 3]})
temp_path = tempfile.mkdtemp()
pq.write_table(t1, os.path.join(temp_path, "t1.parquet"))
pq.write_table(t2, os.path.join(temp_path, "t2.parquet"))
ds = pds.dataset(temp_path)
print(ds.to_table())
```
```
pyarrow.Table
A: int64
B: string
----
A: [[1,2,3],[1,2,3]]
B: [["a","b","c"],["a","b","c"]]
```
Collect metadata of the individual files and create the (global) metadata
file.
```python
metadata_collector = [frag.metadata for frag in ds.get_fragments()]
metadata = metadata_collector[0]
metadata.append_row_groups(metadata_collector[1])
```
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[193], line 2
1 metadata = metadata_collector[0]
----> 2 metadata.append_row_groups(metadata_collector[1])
File
~/mambaforge/envs/pydala-dev/lib/python3.11/site-packages/pyarrow/_parquet.pyx:793,
in pyarrow._parquet.FileMetaData.append_row_groups()
RuntimeError: AppendRowGroups requires equal schemas.
The two columns with index 0 differ.
column descriptor = {
name: A,
path: A,
physical_type: INT64,
converted_type: NONE,
logical_type: None,
max_definition_level: 1,
max_repetition_level: 0,
}
column descriptor = {
name: B,
path: B,
physical_type: BYTE_ARRAY,
converted_type: UTF8,
logical_type: String,
max_definition_level: 1,
max_repetition_level: 0,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
