LuciferYang commented on PR #40847: URL: https://github.com/apache/spark/pull/40847#issuecomment-1518931685
> > So if there is an way to build and test Hadoop 3.0/3.1 successfully before this pr, but it loses after this pr, I think we should stop this work because Apache Spark has not previously stated on any occasion that it no longer supports Hadoop 3.0/3.1, right ? > > Yes, I think that's probably a sensible thing to do. > > > @xkrogen @sunchao @pan3793 Can you give a command that can be used for build & test with Hadoop 3.0/3.1? I want to manually check it, thanks ~ > > You can check this JIRA for the command to build: https://issues.apache.org/jira/browse/SPARK-37994 I encountered the following error while compiling `hadoop-cloud` module during build with hadoop 3.1.x: ``` [ERROR] [Error] /${spark-source-dir}/hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala:34: value hasPathCapability is not a member of org.apache.hadoop.fs.FileContext [ERROR] [Error] /${spark-source-dir}/hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala:34: not found: value CommonPathCapabilities [ERROR] [Error] /${spark-source-dir}/hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala:52: value abort is not a member of org.apache.hadoop.fs.FSDataOutputStream [ERROR] [Error] /${spark-source-dir}/hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala:66: value abort is not a member of org.apache.hadoop.fs.FSDataOutputStream ``` Due to the fixed version of HADOOP-15691 being ` 3.3.0, 3.2.2, 3.2.3` and the fixed version of HADOOP-16906 being `3.3.1`, so it is definitely not possible to build hadoop-cloud` module using Hadoop 3.1. x. I would like to remove this module to continue my experiment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
