paolosartiprom opened a new issue, #37583:
URL: https://github.com/apache/arrow/issues/37583

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   The write_dataset function doesn't write anything if the table is empty. 
This has the consequence that schema information is lost when reading it back 
(because it didn't write anything in the first place).
   
   This can be avoided by manually writing an empty table with that schema 
inside the destination folder in the parquet format. But I expected this to 
already be the intended behaviour.
   
   To reproduce:
   ```python
   import os
   import shutil
   import tempfile
   import pyarrow as pa
   import pyarrow.parquet as pq
   import pyarrow.dataset as pds
   
   dest = tempfile.mkdtemp(suffix='.parquet')
   schema = pa.schema([('a', pa.int64()), ('b', pa.int64())])
   empty_table = schema.empty_table()
   
   pds.write_dataset(empty_table, dest, format='parquet')
   
   ds = pds.dataset(dest)
   print(ds.schema)
   # -> no output
   # I expected ds.schema == pa.schema([('a', pa.int64()), ('b', pa.int64())])
   
   print(os.listdir(dest))
   # -> []
   # I expected to find an empty parquet file in the folder to avoid losing the 
schema!
   
   
   # WORKAROUND
   # If I write the empty table with pyarrow.parquet.write_table inside the 
folder, then the schema is preserved
   pq.write_table(empty_table, os.path.join(dest, 'part-0.parquet'))
   ds_with_workaround = pds.dataset(dest)
   assert ds_with_workaround.schema == schema
   
   # delete folder recursively
   shutil.rmtree(dest, ignore_errors=True)
   ```
   
   Thank you
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to