Martin Nigsch created SPARK-28161: ------------------------------------- Summary: Can't build Spark 2.4.3 based on Ubuntu and Oracle Java 8 SDK v212 Key: SPARK-28161 URL: https://issues.apache.org/jira/browse/SPARK-28161 Project: Spark Issue Type: Bug Components: Build Affects Versions: 2.4.3 Environment: # Pull base image. FROM ubuntu:16.04
RUN apt update --fix-missing RUN apt-get install -y software-properties-common RUN mkdir /usr/java ADD jdk-8u212-linux-x64.tar /usr/java ENV JAVA_HOME=/usr/java/jdk1.8.0_212 RUN update-alternatives --install /usr/bin/java java ${JAVA_HOME%*/}/bin/java 20000 RUN update-alternatives --install /usr/bin/javac javac ${JAVA_HOME%*/}/bin/javac 20000 ENV PATH="${PATH}:/usr/java/jdk1.8.0_212/bin" ENV MAVEN_VERSION 3.6.1 RUN apt-get install -y curl wget RUN curl -fsSL http://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz | tar xzf - -C /usr/share \ && mv /usr/share/apache-maven-${MAVEN_VERSION} /usr/share/maven \ && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn ENV MAVEN_HOME /usr/share/maven # Define commonly used JAVA_HOME variable ENV MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" ENV SPARK_SRC="/usr/src/spark" ENV BRANCH="v2.4.3" RUN apt-get update && apt-get install -y --no-install-recommends \ git python3 python3-setuptools r-base-dev r-cran-evaluate RUN mkdir -p $SPARK_SRC RUN git clone --branch $BRANCH https://github.com/apache/spark $SPARK_SRC WORKDIR $SPARK_SRC RUN ./build/mvn -DskipTests clean package #RUN ./dev/make-distribution.sh -e --name mn-spark-2.3.3 --pip --r --tgz -Psparkr -Phadoop-2.7 -Pmesos -Pyarn -Pkubernetes Reporter: Martin Nigsch Trying to build spark from source based on the Dockerfile attached locally (launched on docker on OSX) fails. Attempts to change/add the following things beyond what's recommended on the build page do not bring improvement: 1. adding ```*RUN ./dev/change-scala-version.sh 2.11```* --> doesn't help 2. editing the pom.xml to exclude zinc as in one of the answers in [https://stackoverflow.com/questions/28004552/problems-while-compiling-spark-with-maven/41223558] --> doesn't help 3. adding options -DrecompileMode=all --> doesn't help I've downloaded the java from Oracle directly ( jdk-8u212-linux-x64.tar ) which is manually put into /usr/java as the Oracle java seems to be recommended. Build fails at project streaming with: ``` [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 2.4.3: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 59.875 s] [INFO] Spark Project Tags ................................. SUCCESS [ 20.386 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 3.026 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 5.654 s] [INFO] Spark Project Networking ........................... SUCCESS [ 7.401 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.400 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 6.306 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 17.471 s] [INFO] Spark Project Core ................................. SUCCESS [02:36 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 50.313 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 21.097 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 52.537 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:44 min] [INFO] Spark Project SQL .................................. FAILURE [10:44 min] [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] Spark Avro ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:15 min [INFO] Finished at: 2019-06-25T09:45:49Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-sql_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-sql_2.11 ``` The command '/bin/sh -c ./build/mvn -DskipTests clean package' returned a non-zero code: 1 Any help? I'm stuck with this since 2 days, hence I'm raising this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org