Re: Spark build 1.6.2 error
I think the difference was the -Dscala2.11 to the command line. I have seen this show up when I miss that. Regards, Nachiketa On Sat 3 Sep, 2016, 12:14 PM Diwakar Dhanuskodi, < diwakar.dhanusk...@gmail.com> wrote: > Hi, > > Just re-ran again without killing zinc server process > > /make-distribution.sh --name custom-spark --tgz -Phadoop-2.6 -Phive > -Pyarn -Dmaven.version=3.0.4 -Dscala-2.11 -X -rf :spark-sql_2.11 > > Build is success. Not sure how it worked with just re-running command > again. > > On Sat, Sep 3, 2016 at 11:44 AM, Diwakar Dhanuskodi < > diwakar.dhanusk...@gmail.com> wrote: > >> Hi, >> >> java version 7 >> >> mvn command >> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.6 -Phive >> -Phive-thriftserver -Pyarn -Dmaven.version=3.0.4 >> >> >> yes, I executed script to change scala version to 2.11 >> killed "com.typesafe zinc.Nailgun" process >> >> re-ran mvn with below command again >> >> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.6 -Phive >> -Phive-thriftserver -Pyarn -Dmaven.version=3.0.4 -X -rf :spark-sql_2.11 >> >> Getting same error >> >> [warn] >> /home/cloudera/Downloads/spark-1.6.2/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala:911: >> method isDir in class FileStatus is deprecated: see corresponding Javadoc >> for more information. >> [warn] status.isDir, >> [warn]^ >> [error] missing or invalid dependency detected while loading class file >> 'WebUI.class'. >> [error] Could not access term eclipse in package org, >> [error] because it (or its dependencies) are missing. Check your build >> definition for >> [error] missing or conflicting dependencies. (Re-run with >> `-Ylog-classpath` to see the problematic classpath.) >> [error] A full rebuild may help if 'WebUI.class' was compiled against an >> incompatible version of org. >> [error] missing or invalid dependency detected while loading class file >> 'WebUI.class'. >> [error] Could not access term jetty in value org.eclipse, >> [error] because it (or its dependencies) are missing. Check your build >> definition for >> [error] missing or conflicting dependencies. (Re-run with >> `-Ylog-classpath` to see the problematic classpath.) >> [error] A full rebuild may help if 'WebUI.class' was compiled against an >> incompatible version of org.eclipse. >> [warn] 17 warnings found >> [error] two errors found >> [debug] Compilation failed (CompilerInterface) >> [error] Compile failed at Sep 3, 2016 11:28:34 AM [21.611s] >> [INFO] >> >> [INFO] Reactor Summary: >> [INFO] >> [INFO] Spark Project Parent POM .. SUCCESS >> [5.583s] >> [INFO] Spark Project Test Tags ... SUCCESS >> [4.189s] >> [INFO] Spark Project Launcher SUCCESS >> [12.226s] >> [INFO] Spark Project Networking .. SUCCESS >> [13.386s] >> [INFO] Spark Project Shuffle Streaming Service ... SUCCESS >> [6.723s] >> [INFO] Spark Project Unsafe .. SUCCESS >> [21.231s] >> [INFO] Spark Project Core SUCCESS >> [3:46.334s] >> [INFO] Spark Project Bagel ... SUCCESS >> [7.032s] >> [INFO] Spark Project GraphX .. SUCCESS >> [19.558s] >> [INFO] Spark Project Streaming ... SUCCESS >> [50.452s] >> [INFO] Spark Project Catalyst SUCCESS >> [1:14.172s] >> [INFO] Spark Project SQL . FAILURE >> [23.222s] >> [INFO] Spark Project ML Library .. SKIPPED >> [INFO] Spark Project Tools ... SKIPPED >> [INFO] Spark Project Hive SKIPPED >> [INFO] Spark Project Docker Integration Tests SKIPPED >> [INFO] Spark Project REPL SKIPPED >> [INFO] Spark Project YARN Shuffle Service SKIPPED >> [INFO] Spark Project YARN SKIPPED >> [INFO] Spark Project Assembly SKIPPED >> [INFO] Spark Project External Twitter SKIPPED >> [INFO] Spark Project External Flume Sink . SKIPPED >> [INFO] Spark Project External Flume .
Re: Spark build 1.6.2 error
Hi Diwakar, Could you please share the entire maven command that you are using to build ? And also the JDK version you are using ? Also could you please confirm that you did execute the script for change scala version to 2.11 before starting the build ? Thanks. Regards, Nachiketa On Wed, Aug 31, 2016 at 2:00 AM, Diwakar Dhanuskodi < diwakar.dhanusk...@gmail.com> wrote: > Hi, > > While building Spark 1.6.2 , getting below error in spark-sql. Much > appreciate for any help. > > ERROR] missing or invalid dependency detected while loading class file > 'WebUI.class'. > Could not access term eclipse in package org, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'WebUI.class' was compiled against an > incompatible version of org. > [ERROR] missing or invalid dependency detected while loading class file > 'WebUI.class'. > Could not access term jetty in value org.eclipse, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'WebUI.class' was compiled against an > incompatible version of org.eclipse. > [WARNING] 17 warnings found > [ERROR] two errors found > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Spark Project Parent POM .. SUCCESS [4.399s] > [INFO] Spark Project Test Tags ... SUCCESS [3.443s] > [INFO] Spark Project Launcher SUCCESS > [10.131s] > [INFO] Spark Project Networking .. SUCCESS > [11.849s] > [INFO] Spark Project Shuffle Streaming Service ... SUCCESS [6.641s] > [INFO] Spark Project Unsafe .. SUCCESS > [19.765s] > [INFO] Spark Project Core SUCCESS > [4:16.511s] > [INFO] Spark Project Bagel ... SUCCESS > [13.401s] > [INFO] Spark Project GraphX .. SUCCESS > [1:08.824s] > [INFO] Spark Project Streaming ... SUCCESS > [2:18.844s] > [INFO] Spark Project Catalyst SUCCESS > [2:43.695s] > [INFO] Spark Project SQL . FAILURE > [1:01.762s] > [INFO] Spark Project ML Library .. SKIPPED > [INFO] Spark Project Tools ... SKIPPED > [INFO] Spark Project Hive SKIPPED > [INFO] Spark Project Docker Integration Tests SKIPPED > [INFO] Spark Project REPL SKIPPED > [INFO] Spark Project YARN Shuffle Service SKIPPED > [INFO] Spark Project YARN SKIPPED > [INFO] Spark Project Assembly SKIPPED > [INFO] Spark Project External Twitter SKIPPED > [INFO] Spark Project External Flume Sink . SKIPPED > [INFO] Spark Project External Flume .. SKIPPED > [INFO] Spark Project External Flume Assembly . SKIPPED > [INFO] Spark Project External MQTT ... SKIPPED > [INFO] Spark Project External MQTT Assembly .. SKIPPED > [INFO] Spark Project External ZeroMQ . SKIPPED > [INFO] Spark Project External Kafka .. SKIPPED > [INFO] Spark Project Examples SKIPPED > [INFO] Spark Project External Kafka Assembly . SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 12:40.525s > [INFO] Finished at: Wed Aug 31 01:56:50 IST 2016 > [INFO] Final Memory: 71M/830M > [INFO] > > [ERROR] Failed to execute goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile > (scala-compile-first) on project spark-sql_2.11: Execution > scala-compile-first of goal > net.alchim31.maven:scala-maven-plugin:3.2.2:compile > failed. CompileFailed -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ > PluginExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :spark-sql_2.11 > > > -- Regards, -- Nachiketa
Re: removing header from csv file
Why "without sqlcontext" ? Could you please describe what is it that you are trying to accomplish ? Thanks. Regards, Nachiketa On Wed, Apr 27, 2016 at 10:54 AM, Ashutosh Kumar wrote: > I see there is a library spark-csv which can be used for removing header > and processing of csv files. But it seems it works with sqlcontext only. Is > there a way to remove header from csv files without sqlcontext ? > > Thanks > Ashutosh > -- Regards, -- Nachiketa
Re: Spark 1.4.0, Secure YARN Cluster, Application Master throws 500 connection refused (Resolved)
Setting the yarn.resourcemanager.webapp.address.rm1 and yarn.resourcemanager.webapp.address.rm2 in yarn-site.xml seems to have resolved the issue. Appreciate any comments about the regression from 1.3.1 ? Thanks. Regards, Nachiketa On Fri, Jun 26, 2015 at 1:28 AM, Nachiketa wrote: > A few other observations. > > 1. Spark 1.3.1 (custom built against HDP 2.2) was running fine against the > same cluster and same hadoop configuration (hence seems like regression). > > 2. HA is enabled for YARN RM and HDFS (not sure if this would impact > anything but wanted to share anyway). > > 3. Found this issue : https://issues.apache.org/jira/browse/SPARK-5837 > and multiple references to other YARN issues to the same. Continuing to > understand and explore the possibilities documented there. > > Regards, > Nachiketa > > On Fri, Jun 26, 2015 at 12:52 AM, Nachiketa > wrote: > >> Spark 1.4.0 - Custom built from source against Hortonworks HDP 2.2 >> (hadoop 2.6.0+) >> HDP 2.2 Cluster (Secure, kerberos) >> >> spark-shell (--master yarn-client) launches fine and the prompt shows up. >> Clicking on the Application Master url on the YARN RM UI, throws 500 >> connect error. >> >> The same build works well against a non-secure cluster (same HDP >> distribution). >> >> No debug logs or stack trace is easily visible. Where do I look for what >> is going wrong ? And has anything changed in spark security that could be >> contributing to this ? >> >> Thank you for your help with this. >> >> >> Regards, >> Nachiketa >> > > > > -- > Regards, > -- Nachiketa > -- Regards, -- Nachiketa
Re: Spark 1.4.0, Secure YARN Cluster, Application Master throws 500 connection refused
A few other observations. 1. Spark 1.3.1 (custom built against HDP 2.2) was running fine against the same cluster and same hadoop configuration (hence seems like regression). 2. HA is enabled for YARN RM and HDFS (not sure if this would impact anything but wanted to share anyway). 3. Found this issue : https://issues.apache.org/jira/browse/SPARK-5837 and multiple references to other YARN issues to the same. Continuing to understand and explore the possibilities documented there. Regards, Nachiketa On Fri, Jun 26, 2015 at 12:52 AM, Nachiketa wrote: > Spark 1.4.0 - Custom built from source against Hortonworks HDP 2.2 (hadoop > 2.6.0+) > HDP 2.2 Cluster (Secure, kerberos) > > spark-shell (--master yarn-client) launches fine and the prompt shows up. > Clicking on the Application Master url on the YARN RM UI, throws 500 > connect error. > > The same build works well against a non-secure cluster (same HDP > distribution). > > No debug logs or stack trace is easily visible. Where do I look for what > is going wrong ? And has anything changed in spark security that could be > contributing to this ? > > Thank you for your help with this. > > > Regards, > Nachiketa > -- Regards, -- Nachiketa
Spark 1.4.0, Secure YARN Cluster, Application Master throws 500 connection refused
Spark 1.4.0 - Custom built from source against Hortonworks HDP 2.2 (hadoop 2.6.0+) HDP 2.2 Cluster (Secure, kerberos) spark-shell (--master yarn-client) launches fine and the prompt shows up. Clicking on the Application Master url on the YARN RM UI, throws 500 connect error. The same build works well against a non-secure cluster (same HDP distribution). No debug logs or stack trace is easily visible. Where do I look for what is going wrong ? And has anything changed in spark security that could be contributing to this ? Thank you for your help with this. Regards, Nachiketa
