[ https://issues.apache.org/jira/browse/SPARK-28161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Nigsch updated SPARK-28161: ---------------------------------- Description: Trying to build spark from source based on the Dockerfile attached locally (launched on docker on OSX) fails. Attempts to change/add the following things beyond what's recommended on the build page do not bring improvement: 1. adding ```*RUN ./dev/change-scala-version.sh 2.11```* --> doesn't help 2. editing the pom.xml to exclude zinc as in one of the answers in [https://stackoverflow.com/questions/28004552/problems-while-compiling-spark-with-maven/41223558] --> doesn't help 3. adding options -DrecompileMode=all --> doesn't help I've downloaded the java from Oracle directly ( jdk-8u212-linux-x64.tar ) which is manually put into /usr/java as the Oracle java seems to be recommended. Build fails at project streaming with: {code} [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 2.4.3: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 59.875 s] [INFO] Spark Project Tags ................................. SUCCESS [ 20.386 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 3.026 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 5.654 s] [INFO] Spark Project Networking ........................... SUCCESS [ 7.401 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.400 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 6.306 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 17.471 s] [INFO] Spark Project Core ................................. SUCCESS [02:36 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 50.313 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 21.097 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 52.537 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:44 min] [INFO] Spark Project SQL .................................. FAILURE [10:44 min] [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] Spark Avro ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:15 min [INFO] Finished at: 2019-06-25T09:45:49Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-sql_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-sql_2.11 The command '/bin/sh -c ./build/mvn -DskipTests clean package' returned a non-zero code: 1 {code} Any help? I'm stuck with this since 2 days, hence I'm raising this issue. was: Trying to build spark from source based on the Dockerfile attached locally (launched on docker on OSX) fails. Attempts to change/add the following things beyond what's recommended on the build page do not bring improvement: 1. adding ```*RUN ./dev/change-scala-version.sh 2.11```* --> doesn't help 2. editing the pom.xml to exclude zinc as in one of the answers in [https://stackoverflow.com/questions/28004552/problems-while-compiling-spark-with-maven/41223558] --> doesn't help 3. adding options -DrecompileMode=all --> doesn't help I've downloaded the java from Oracle directly ( jdk-8u212-linux-x64.tar ) which is manually put into /usr/java as the Oracle java seems to be recommended. Build fails at project streaming with: {code} [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 2.4.3: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 59.875 s] [INFO] Spark Project Tags ................................. SUCCESS [ 20.386 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 3.026 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 5.654 s] [INFO] Spark Project Networking ........................... SUCCESS [ 7.401 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.400 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 6.306 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 17.471 s] [INFO] Spark Project Core ................................. SUCCESS [02:36 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 50.313 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 21.097 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 52.537 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:44 min] [INFO] Spark Project SQL .................................. FAILURE [10:44 min] [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] Spark Avro ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:15 min [INFO] Finished at: 2019-06-25T09:45:49Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-sql_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-sql_2.11 {code} The command '/bin/sh -c ./build/mvn -DskipTests clean package' returned a non-zero code: 1 Any help? I'm stuck with this since 2 days, hence I'm raising this issue. > Can't build Spark 2.4.3 based on Ubuntu and Oracle Java 8 SDK v212 > ------------------------------------------------------------------ > > Key: SPARK-28161 > URL: https://issues.apache.org/jira/browse/SPARK-28161 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 2.4.3 > Environment: {code} > # Dockerfile > # Pull base image. > FROM ubuntu:16.04 > RUN apt update --fix-missing > RUN apt-get install -y software-properties-common > RUN mkdir /usr/java > ADD jdk-8u212-linux-x64.tar /usr/java > ENV JAVA_HOME=/usr/java/jdk1.8.0_212 > RUN update-alternatives --install /usr/bin/java java ${JAVA_HOME%*/}/bin/java > 20000 > RUN update-alternatives --install /usr/bin/javac javac > ${JAVA_HOME%*/}/bin/javac 20000 > ENV PATH="${PATH}:/usr/java/jdk1.8.0_212/bin" > ENV MAVEN_VERSION 3.6.1 > RUN apt-get install -y curl wget > RUN curl -fsSL > [http://archive.apache.org/dist/maven/maven-3/$]{MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz > | tar xzf - -C /usr/share \ > && mv /usr/share/apache-maven-${MAVEN_VERSION} /usr/share/maven \ > && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn > ENV MAVEN_HOME /usr/share/maven > ENV MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" > ENV SPARK_SRC="/usr/src/spark" > ENV BRANCH="v2.4.3" > RUN apt-get update && apt-get install -y --no-install-recommends \ > git python3 python3-setuptools r-base-dev r-cran-evaluate > RUN mkdir -p $SPARK_SRC > RUN git clone --branch $BRANCH [https://github.com/apache/spark] $SPARK_SRC > WORKDIR $SPARK_SRC > RUN ./build/mvn -DskipTests clean package > {code} > Reporter: Martin Nigsch > Priority: Minor > > Trying to build spark from source based on the Dockerfile attached locally > (launched on docker on OSX) fails. > Attempts to change/add the following things beyond what's recommended on the > build page do not bring improvement: > 1. adding ```*RUN ./dev/change-scala-version.sh 2.11```* --> doesn't help > 2. editing the pom.xml to exclude zinc as in one of the answers in > [https://stackoverflow.com/questions/28004552/problems-while-compiling-spark-with-maven/41223558] > --> doesn't help > 3. adding options -DrecompileMode=all --> doesn't help > I've downloaded the java from Oracle directly ( jdk-8u212-linux-x64.tar ) > which is manually put into /usr/java as the Oracle java seems to be > recommended. > Build fails at project streaming with: > {code} > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary for Spark Project Parent POM 2.4.3: > [INFO] > [INFO] Spark Project Parent POM ........................... SUCCESS [ 59.875 > s] > [INFO] Spark Project Tags ................................. SUCCESS [ 20.386 > s] > [INFO] Spark Project Sketch ............................... SUCCESS [ 3.026 s] > [INFO] Spark Project Local DB ............................. SUCCESS [ 5.654 s] > [INFO] Spark Project Networking ........................... SUCCESS [ 7.401 s] > [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.400 s] > [INFO] Spark Project Unsafe ............................... SUCCESS [ 6.306 s] > [INFO] Spark Project Launcher ............................. SUCCESS [ 17.471 > s] > [INFO] Spark Project Core ................................. SUCCESS [02:36 > min] > [INFO] Spark Project ML Local Library ..................... SUCCESS [ 50.313 > s] > [INFO] Spark Project GraphX ............................... SUCCESS [ 21.097 > s] > [INFO] Spark Project Streaming ............................ SUCCESS [ 52.537 > s] > [INFO] Spark Project Catalyst ............................. SUCCESS [02:44 > min] > [INFO] Spark Project SQL .................................. FAILURE [10:44 > min] > [INFO] Spark Project ML Library ........................... SKIPPED > [INFO] Spark Project Tools ................................ SKIPPED > [INFO] Spark Project Hive ................................. SKIPPED > [INFO] Spark Project REPL ................................. SKIPPED > [INFO] Spark Project Assembly ............................. SKIPPED > [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED > [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED > [INFO] Spark Project Examples ............................. SKIPPED > [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED > [INFO] Spark Avro ......................................... SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 20:15 min > [INFO] Finished at: 2019-06-25T09:45:49Z > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on > project spark-sql_2.11: Execution scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn <goals> -rf :spark-sql_2.11 > The command '/bin/sh -c ./build/mvn -DskipTests clean package' returned a > non-zero code: 1 > {code} > Any help? I'm stuck with this since 2 days, hence I'm raising this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org