paolosartiprom opened a new issue, #37583:
URL: https://github.com/apache/arrow/issues/37583
### Describe the bug, including details regarding any error messages,
version, and platform.
The write_dataset function doesn't write anything if the table is empty.
This has the consequence that schema information is lost when reading it back
(because it didn't write anything in the first place).
This can be avoided by manually writing an empty table with that schema
inside the destination folder in the parquet format. But I expected this to
already be the intended behaviour.
To reproduce:
```python
import os
import shutil
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as pds
dest = tempfile.mkdtemp(suffix='.parquet')
schema = pa.schema([('a', pa.int64()), ('b', pa.int64())])
empty_table = schema.empty_table()
pds.write_dataset(empty_table, dest, format='parquet')
ds = pds.dataset(dest)
print(ds.schema)
# -> no output
# I expected ds.schema == pa.schema([('a', pa.int64()), ('b', pa.int64())])
print(os.listdir(dest))
# -> []
# I expected to find an empty parquet file in the folder to avoid losing the
schema!
# WORKAROUND
# If I write the empty table with pyarrow.parquet.write_table inside the
folder, then the schema is preserved
pq.write_table(empty_table, os.path.join(dest, 'part-0.parquet'))
ds_with_workaround = pds.dataset(dest)
assert ds_with_workaround.schema == schema
# delete folder recursively
shutil.rmtree(dest, ignore_errors=True)
```
Thank you
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]