[
https://issues.apache.org/jira/browse/ARROW-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028802#comment-16028802
]
Brecht Machiels commented on ARROW-1064:
----------------------------------------
I did eventually get it to open the set of parquet files with missing _metadata
file by removing an empty directory ("_impala_insert_staging") that was in the
same directory.
I still am not able to to open a particular single-file parquet dataset though.
It fails with {{ArrowIOError: IOError: Invalid parquet file. Corrupt footer.}}.
It cannot be opened by fastparquet either. Trying to load it using PySpark
fails with a similar error, so there must be something wrong with it:
{code}
py4j.protocol.Py4JJavaError: An error occurred while calling o76.parquet.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in
stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID
4, itsusraedlp08.jnj.com): java.io.IOException: Could not read footer:
java.lang.RuntimeException:
hdfs://<snipped>/<parquet_dir>/e4a415679f64f34-7ac06c0506c56aab_1260025109_data.0.
is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but
found [50, 51, 56, 10]
at
parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:248)
...
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException:
hdfs://<snipped>/<parquet_dir>/e4a415679f64f34-7ac06c0506c56aab_1260025109_data.0.
is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but
found [50, 51, 56, 10]
at
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:423)
...
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
{code}
It is possible to run queries against it using Impala though.
> ModuleNotFoundError: No module named 'pyarrow._parquet'
> -------------------------------------------------------
>
> Key: ARROW-1064
> URL: https://issues.apache.org/jira/browse/ARROW-1064
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.3.0
> Environment: Windows 7 64-bit, conda environment, Python 3.6.1
> pyarrow: 0.3.0.post-np112py36_vc14_1 conda-forge [vc14]
> Reporter: Brecht Machiels
> Assignee: Wes McKinney
> Fix For: 0.4.0
>
>
> {code}
> $ python
> Python 3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 20:11:04)
> [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pyarrow
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "C:\Users\bmachie\AppData\Local\Continuum\Miniconda3\envs\ml_recommender\lib\site-packages\pyarrow\__init__.py",
> line 28, in <module>
> from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: DLL load failed: The specified procedure could not be found.
> >>>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)