Wes McKinney created ARROW-6678:
-----------------------------------
Summary: [C++] Regression in Parquet file compatibility introduced
by ARROW-3246
Key: ARROW-6678
URL: https://issues.apache.org/jira/browse/ARROW-6678
Project: Apache Arrow
Issue Type: Bug
Components: C++
Reporter: Wes McKinney
Fix For: 0.15.0
I randomly discovered that this script fails after applying the patch for
ARROW-3246
https://github.com/apache/arrow/commit/2ba0566b29312e84fafc987fd8dc9664748be96a
{code}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import fastparquet as fp
df = pd.util.testing.makeDataFrame()
pq.write_table(pa.table(df), 'test.parquet')
fp.ParquetFile('test.parquet')
{code}
with
{code}
Traceback (most recent call last):
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py",
line 110, in __init__
with open_with(fn2, 'rb') as f:
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/util.py",
line 38, in default_open
return open(f, mode)
NotADirectoryError: [Errno 20] Not a directory: 'test.parquet/_metadata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 10, in <module>
fp.ParquetFile('test.parquet')
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py",
line 116, in __init__
self._parse_header(f, verify)
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py",
line 135, in _parse_header
fmd = read_thrift(f, parquet_thrift.FileMetaData)
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/thrift_structures.py",
line 25, in read_thrift
obj.read(pin)
File
"/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py",
line 1929, in read
iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid
start byte
{code}
I don't recall making any metadata-related changes but I'm going to review the
patch to see if I can narrow down where the problem is to see whether it's a
bug with Arrow/parquet-cpp or with the third party library
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
