[
https://issues.apache.org/jira/browse/ARROW-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287565#comment-17287565
]
Russell Jurney edited comment on ARROW-11638 at 2/20/21, 6:44 AM:
------------------------------------------------------------------
Having written [this
post]([https://blog.datasyndrome.com/python-and-parquet-performance-e71da65269ce])
I went through it and already tried both of those and neither works - but if
I'm not mistaken read_table is only for a single file? That is what
ParquetDataset is for.
I'll try again just to be sure and report back.
was (Author: russell.jurney):
Having written [this
post](https://blog.datasyndrome.com/python-and-parquet-performance-e71da65269ce)
I went through it and already tried both of those and neither works - but if
I'm not mistaken read_table is only for a single file? That is what
ParquetDataset is for.
> [Python][Parquet] Can't read directory of Parquet data saved by PySpark via
> ----------------------------------------------------------------------------
>
> Key: ARROW-11638
> URL: https://issues.apache.org/jira/browse/ARROW-11638
> Project: Apache Arrow
> Issue Type: Improvement
> Environment: Ubuntu 18.04
> Reporter: Russell Jurney
> Priority: Major
>
> I saved a large Parquet dataset with 200 partitions in PySpark. I am trying
> to load it to use in Dask. Dask's own utilities for this don't work so I am
> trying pyarrow, which has worked in the past when converted into a Table then
> a dataframe. I have verified that one of the file partitions loads just fine.
> This code:
> {code:java}
> import pyarrow.parquet as pq
> fs_gcs = gcsfs.GCSFileSystem(project="my-space-1234")
> table =
> pq.ParquetDataset("gs://open-corporates/parquet/2020_10/officers.parquet",
> filesystem=fs_gcs)
> table
> {code}
> I get this exception:
> {code:java}
> ---------------------------------------------------------------------------
> ArrowInvalid Traceback (most recent call last)
> <ipython-input-69-234baf611ba5> in <module> 5 6 ----> 7 table =
> pq.ParquetDataset("gs://open-corporates/parquet/2020_10/officers.parquet",
> filesystem=fs_gcs) 8 table
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in __init__(self, path_or_paths, filesystem, schema, metadata,
> split_row_groups, validate_schema, filters, metadata_nthreads,
> read_dictionary, memory_map, buffer_size, partitioning, use_legacy_dataset)
> 1274 1275 if validate_schema: -> 1276 self.validate_schemas() 1277 1278 def
> equals(self, other):
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in validate_schemas(self) 1302 self.schema = self.common_metadata.schema 1303
> else: -> 1304 self.schema = self.pieces[0].get_metadata().schema 1305 elif
> self.schema is None: 1306 self.schema = self.metadata.schema
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in get_metadata(self) 734 metadata : FileMetaData 735 """ --> 736 f =
> self.open() 737 return f.metadata 738
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in open(self) 741 Return instance of ParquetFile. 742 """ --> 743 reader =
> self.open_file_func(self.path) 744 if not isinstance(reader, ParquetFile):
> 745 reader = ParquetFile(reader, **self.file_options)
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in _open_dataset_file(dataset, path, meta) 1111 not isinstance(dataset.fs,
> legacyfs.LocalFileSystem)): 1112 path = dataset.fs.open(path, mode='rb') ->
> 1113 return ParquetFile( 1114 path, 1115 metadata=meta,
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/parquet.py
> in __init__(self, source, metadata, common_metadata, read_dictionary,
> memory_map, buffer_size) 215 read_dictionary=None, memory_map=False,
> buffer_size=0): 216 self.reader = ParquetReader() --> 217
> self.reader.open(source, use_memory_map=memory_map, 218
> buffer_size=buffer_size, 219 read_dictionary=read_dictionary,
> metadata=metadata)
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/_parquet.pyx
> in pyarrow._parquet.ParquetReader.open()
> /opt/conda/envs/deep_discovery/lib/python3.8/site-packages/pyarrow/error.pxi
> in pyarrow.lib.check_status() ArrowInvalid: Parquet file size is 0 bytes
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
