[ 
https://issues.apache.org/jira/browse/ARROW-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028802#comment-16028802
 ] 

Brecht Machiels commented on ARROW-1064:
----------------------------------------

I did eventually get it to open the set of parquet files with missing _metadata 
file by removing an empty directory ("_impala_insert_staging") that was in the 
same directory.

I still am not able to to open a particular single-file parquet dataset though. 
It fails with {{ArrowIOError: IOError: Invalid parquet file. Corrupt footer.}}. 
It cannot be opened by fastparquet either. Trying to load it using PySpark 
fails with a similar error, so there must be something wrong with it:

{code}
py4j.protocol.Py4JJavaError: An error occurred while calling o76.parquet.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 
4, itsusraedlp08.jnj.com): java.io.IOException: Could not read footer: 
java.lang.RuntimeException: 
hdfs://<snipped>/<parquet_dir>/e4a415679f64f34-7ac06c0506c56aab_1260025109_data.0.
 is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but 
found [50, 51, 56, 10]
        at 
parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:248)
        ...
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: 
hdfs://<snipped>/<parquet_dir>/e4a415679f64f34-7ac06c0506c56aab_1260025109_data.0.
 is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but 
found [50, 51, 56, 10]
        at 
parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:423)
        ...
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more
{code}

It is possible to run queries against it using Impala though.

> ModuleNotFoundError: No module named 'pyarrow._parquet'
> -------------------------------------------------------
>
>                 Key: ARROW-1064
>                 URL: https://issues.apache.org/jira/browse/ARROW-1064
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.3.0
>         Environment: Windows 7 64-bit, conda environment, Python 3.6.1
> pyarrow: 0.3.0.post-np112py36_vc14_1 conda-forge [vc14]
>            Reporter: Brecht Machiels
>            Assignee: Wes McKinney
>             Fix For: 0.4.0
>
>
> {code}
> $ python
> Python 3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 20:11:04) 
> [MSC v.1900 64 bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pyarrow
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File 
> "C:\Users\bmachie\AppData\Local\Continuum\Miniconda3\envs\ml_recommender\lib\site-packages\pyarrow\__init__.py",
>  line 28, in <module>
>     from pyarrow.lib import cpu_count, set_cpu_count
> ImportError: DLL load failed: The specified procedure could not be found.
> >>>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to