[
https://issues.apache.org/jira/browse/HADOOP-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850438#comment-17850438
]
ASF GitHub Bot commented on HADOOP-19131:
-----------------------------------------
steveloughran commented on code in PR #6686:
URL: https://github.com/apache/hadoop/pull/6686#discussion_r1619247570
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:
##########
@@ -628,13 +654,17 @@ private OpenFileOptions() {
public static final String FS_OPTION_OPENFILE_READ_POLICY_WHOLE_FILE =
"whole-file";
+
Review Comment:
cut
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsWrappedIO.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contract;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.contract.AbstractFSContract;
+import org.apache.hadoop.io.wrappedio.TestWrappedIO;
+
+public class ITestAbfsWrappedIO extends TestWrappedIO {
Review Comment:
javadocs
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:
##########
@@ -598,12 +598,38 @@ private OpenFileOptions() {
public static final String FS_OPTION_OPENFILE_READ_POLICY_ADAPTIVE =
"adaptive";
+ /**
+ * We are an avro file: {@value}.
+ */
+ public static final String FS_OPTION_OPENFILE_READ_POLICY_AVRO = "avro" ;
+
+ /**
+ * This is a columnar file format.
+ * Do whatever is needed to optimize for it: {@value}.
+ */
+ public static final String FS_OPTION_OPENFILE_READ_POLICY_COLUMNAR =
+ "columnar";
+
/**
* Read policy {@value} -whateve the implementation does by default.
*/
public static final String FS_OPTION_OPENFILE_READ_POLICY_DEFAULT =
"default";
+ /**
+ * This is an ORC file.
+ * Do whatever is needed to optimize for it: {@value}.
+ */
+ public static final String FS_OPTION_OPENFILE_READ_POLICY_ORC =
+ "orc";
+
+ /**
+ * This is a v1 parquet file.
Review Comment:
rename par1, add par3
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/contract/hdfs/TestDFSWrappedIO.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.contract.hdfs;
+
+import java.io.IOException;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.contract.AbstractFSContract;
+import org.apache.hadoop.io.wrappedio.TestWrappedIO;
+
+public class TestDFSWrappedIO extends TestWrappedIO {
Review Comment:
javadocs and a newline after
> WrappedIO to export modern filesystem/statistics APIs in a reflection
> friendly form
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-19131
> URL: https://issues.apache.org/jira/browse/HADOOP-19131
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.4.0, 3.3.6
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> parquet, avro etc are still stuck building with older hadoop releases.
> This makes using new APIs hard (PARQUET-2171) and means that APIs which are 5
> years old such as HADOOP-15229 just aren't picked up.
> This lack of openFIle() adoption hurts working with files in cloud storage as
> * extra HEAD requests are made
> * read policies can't be explicitly set
> * split start/end can't be passed down
> HADOOP-18679 added a new WrappedIO class.
> This jira proposes extending this with
> * more of the filesystem/input stream methods
> * iOStatistics
> * Pull in parquet DynMethods to dynamially wrap and invoke through tests.
> This class, DynamicWrappedIO is intended to be copied into libraries
> (parquet, iceberg) for their own use.
> * existing tests to use the dynamic binding for end-to-end testing.
> +then get into the downstream libraries and use where appropriate
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
