shanyu commented on issue #22997: SPARK-25999: make-distribution.sh failure with --r and -Phadoop-provided URL: https://github.com/apache/spark/pull/22997#issuecomment-519361762 I'm sorry this PR completely slipped out of my mind. Sorry for the late reply. I copied hadoop dependencies to $SPARK_HOME/assembly/target/scala-${SCALA_VERSION}/jars, that's where these jars are read during building R packages. This is not where we package the tar ball. The folder where we create the dist tar ball is defined at the top of this file: DISTDIR="$SPARK_HOME/dist" And we've done copying from the assembly folder to the dist folder earlier: cp "$SPARK_HOME"/assembly/target/scala*/jars/* "$DISTDIR/jars/" If we build with hadoop, then hadoop jars will end up in the assembly folder first, then copied to dist folder. If we don't choose to build with hadoop, then before building R packages, we copy the dependencies to assembly folder. These jars won't get in the dist tar ball. I've tested this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
