pan3793 commented on PR #57141: URL: https://github.com/apache/spark/pull/57141#issuecomment-4922903896
> If the end users copies the required files to `jars` directory, it is supposed to be identical with the Apache Spark full binary distribution. @dongjoon-hyun, My understanding is, `spark-<x.y.z>-bin-without-hadoop.tgz` is supposed to include `hadoop classpath` (the default output does not contain the hadoop shaded client, but looks like below) into the Spark classpath, like many other Hadoop ecosystem projects do (e.g., Hive, HBase, Flink) ``` /etc/hadoop/conf:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/* ``` the current dependencies controlled by `hadoop.deps.scope` are ``` org.slf4j:slf4j-api org.apache.curator:curator-recipes org.apache.hadoop:hadoop-client-api org.apache.hadoop:hadoop-client-runtime org.apache.zookeeper:zookeeper org.apache.hadoop:hadoop-aws software.amazon.s3.analyticsaccelerator:analyticsaccelerator-s3 joda-time:joda-time com.fasterxml.jackson.core:jackson-databind com.fasterxml.jackson.core:jackson-annotations org.apache.httpcomponents:httpclient org.apache.httpcomponents:httpcore org.apache.hadoop:hadoop-azure org.apache.hadoop:hadoop-cloud-storage com.google.guava:guava com.google.guava:failureaccess ``` I'm not sure if it still makes sense, for example, `slf4j-api` is managed by `hadoop.deps.scope`, but other slf4j jars are not, if users put the hadoop-managed slf4j-api 1.x on the Spark classpath, I don't think it works because Spark requires slf4j 2.x. I haven't seen developers say they verified the `spark-<x.y.z>-bin-without-hadoop.tgz` in the voting thread for years, and haven't seen a user report classpath issues about the `*-without-hadoop.tgz`, so I doubt it's still actually in use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
