dongjoon-hyun commented on PR #57141: URL: https://github.com/apache/spark/pull/57141#issuecomment-4921720310
Thank you, @pan3793 . > Requires Hadoop 3.4.0+ makes sense according to the [Hadoop supported-versions](https://github.com/apache/hadoop/blob/trunk/SECURITY.md#supported-versions) declaration. > > > Older release lines are end-of-life and do not receive security updates; the remedy for a vulnerability in an old line is to upgrade. Refer to the [Apache Hadoop release and download policy](https://hadoop.apache.org/releases.html) for which lines are currently maintained. For the following, I also have the same concerns. To be clear, I don't think we provide any guarantee on that although it's documented like that. We don't know because we don't have a CI test coverage. > For official distribution building without `-Phadoop-cloud`, I think it's fine to just change the Hadoop version in the build command via `-Phadoop.version=3.4.0`, but I think users might need to tune the `pom.xml` when they want to build with both `-Phadoop-cloud` and `-Phadoop.version=x.x.x`, not sure if this is worth a document section. For the following, what do you mean by *broken*? It's simply a subset of the official Apache Spark binary distribution, isn't it? If the end users copies the required files to `jars` directory, it is supposed to be the Apache Spark full binary distribution. > One side issue, we currently also release `spark-<x.y.z>-bin-without-hadoop.tgz`; is anyone using it? I think it has not been well maintained and is likely broken. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
