Jesse Lord created ARROW-4874: --------------------------------- Summary: Cannot read parquet from encrypted hdfs Key: ARROW-4874 URL: https://issues.apache.org/jira/browse/ARROW-4874 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.12.0 Environment: cloudera yarn cluster, red hat enterprise 7 Reporter: Jesse Lord
Using pyarrow 0.12 I was able to read parquet at first and then the admins added KMS servers and encrypted all of the files on the cluster. Now I get an error and the file system object can only read objects from the local file system of the edge node. Reproducible example: {{import pyarrow as pa fs = pa.hdfs.connect() with fs.open('/user/jlord/test_lots_of_parquet/', 'rb') as fil: _ = fil.read() }} error: {{19/03/14 10:29:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hdfsOpenFile(/user/jlord/test_lots_of_parquet/): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: FileNotFoundException: File /user/jlord/test_lots_of_parquet does not existjava.io.FileNotFoundException: File /user/jlord/test_lots_of_parquet does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:432) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344) Traceback (most recent call last): File "local_hdfs.py", line 15, in <module> with fs.open(file, 'rb') as fil: File "pyarrow/io-hdfs.pxi", line 431, in pyarrow.lib.HadoopFileSystem.open File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status pyarrow.lib.ArrowIOError: HDFS file does not exist: /user/jlord/test_lots_of_parquet/}} If I specify a specific parquet file in that folder I get the following error: {{19/03/14 10:07:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hdfsOpenFile(/user/jlord/test_lots_of_parquet/part-00000-0f130b19-8c8c-428c-9854-fc76bdee1cfa.snappy.parquet): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: FileNotFoundException: File /user/jlord/test_lots_of_parquet/part-00000-0f130b19-8c8c-428c-9854-fc76bdee1cfa.snappy.parquet does not existjava.io.FileNotFoundException: File /user/jlord/test_lots_of_parquet/part-00000-0f130b19-8c8c-428c-9854-fc76bdee1cfa.snappy.parquet does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:598) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:811) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:588) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:432) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:344) Traceback (most recent call last): File "local_hdfs.py", line 15, in <module> with fs.open(file, 'rb') as fil: File "pyarrow/io-hdfs.pxi", line 431, in pyarrow.lib.HadoopFileSystem.open File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status pyarrow.lib.ArrowIOError: HDFS file does not exist: /user/jlord/test_lots_of_parquet/part-00000-0f130b19-8c8c-428c-9854-fc76bdee1cfa.snappy.parquet}} Not sure if this is relevant: spark can read continue to read the parquet files, but it takes a cloudera specific version that can read the following KMS keys from the core-site.xml and hdfs-site.xml: {{<property> <name>dfs.encryption.key.provider.uri</name> <value>kms://ht...@server1.com;server2.com:16000/kms</valu e> </property>}} Using the open source version of spark requires changing these xml values to: {{<property> <name>dfs.encryption.key.provider.uri</name> <value>kms://ht...@server1.com:16000/kms</value> <value>kms://ht...@server2.com:16000/kms</value> </property>}} Might need to point arrow to separate configuration xmls. -- This message was sent by Atlassian JIRA (v7.6.3#76005)