Re: Review Request 68518: ProxyFileSystem.listStatusIterator function override required once migrated to Hadoop 3.2.0+
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68518/ --- (Updated Aug. 28, 2018, 11:34 a.m.) Review request for hive, Marta Kuczora and Peter Vary. Bugs: HIVE-20465 https://issues.apache.org/jira/browse/HIVE-20465 Repository: hive-git Description --- Few list_bucket_query_oneskew_*.q tests started to fail. It is the side effect of this Hadoop change: http://github.mtv.cloudera.com/CDH/hadoop/commit/073a38ee09f40b25677cc49eff777241c8fb2eba#diff-4bf68d76a459a69bbb0affbf579ebcf3 In Hive there is the ProxyFileSystem class which has the 'swizzleParamPath' method. This method will replace the 'pfile' prefix to 'file' in the path. The ProxyFileSystem will be used as file system during the test execution and this class implements some methods of the hadoop FileSystem class. Before the linked Hadoop patch, the file status are fetched with the 'listStatus' method call which is implemented in the ProxyFileSystem. But the patch changed this code path and now it calls the FileSystem.listStatusIterator method which is not implemented in the ProxyFileSystem, so it will fall back to the Hadoop implementation which will throw error for the 'pfile' prefix. Diffs (updated) - shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java 608c7e0578 shims/common/src/main/test/org/apache/hadoop/fs/TestProxyFileSystem.java PRE-CREATION Diff: https://reviews.apache.org/r/68518/diff/2/ Changes: https://reviews.apache.org/r/68518/diff/1-2/ Testing --- Thanks, denys kuzmenko
Review Request 68518: ProxyFileSystem.listStatusIterator function override required once migrated to Hadoop 3.2.0+
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68518/ --- Review request for hive, Marta Kuczora and Peter Vary. Bugs: HIVE-20465 https://issues.apache.org/jira/browse/HIVE-20465 Repository: hive-git Description --- Few list_bucket_query_oneskew_*.q tests started to fail. It is the side effect of this Hadoop change: http://github.mtv.cloudera.com/CDH/hadoop/commit/073a38ee09f40b25677cc49eff777241c8fb2eba#diff-4bf68d76a459a69bbb0affbf579ebcf3 In Hive there is the ProxyFileSystem class which has the 'swizzleParamPath' method. This method will replace the 'pfile' prefix to 'file' in the path. The ProxyFileSystem will be used as file system during the test execution and this class implements some methods of the hadoop FileSystem class. Before the linked Hadoop patch, the file status are fetched with the 'listStatus' method call which is implemented in the ProxyFileSystem. But the patch changed this code path and now it calls the FileSystem.listStatusIterator method which is not implemented in the ProxyFileSystem, so it will fall back to the Hadoop implementation which will throw error for the 'pfile' prefix. Diffs - shims/common/src/main/java/org/apache/hadoop/fs/ProxyFileSystem.java a82b2f0564 Diff: https://reviews.apache.org/r/68518/diff/1/ Testing --- Thanks, denys kuzmenko
