Hi Marco, I did run in IDE(Intellij) as well. It works fine. VG, make sure the right jar is in classpath.
--Siva On Fri, Jun 17, 2016 at 4:11 PM, Marco Mistroni <mmistr...@gmail.com> wrote: > and your eclipse path is correct? > i suggest, as Siva did before, to build your jar and run it via > spark-submit by specifying the --packages option > it's as simple as run this command > > spark-submit --packages > com.databricks:spark-xml_<scalaversion>:<packageversion> --class <Name of > your class containing main> <path to your jar> > > Indeed, if you have only these lines to run, why dont you try them in > spark-shell ? > > hth > > On Fri, Jun 17, 2016 at 11:32 AM, VG <vlin...@gmail.com> wrote: > >> nopes. eclipse. >> >> >> On Fri, Jun 17, 2016 at 3:58 PM, Siva A <siva9940261...@gmail.com> wrote: >> >>> If you are running from IDE, Are you using Intellij? >>> >>> On Fri, Jun 17, 2016 at 3:20 PM, Siva A <siva9940261...@gmail.com> >>> wrote: >>> >>>> Can you try to package as a jar and run using spark-submit >>>> >>>> Siva >>>> >>>> On Fri, Jun 17, 2016 at 3:17 PM, VG <vlin...@gmail.com> wrote: >>>> >>>>> I am trying to run from IDE and everything else is working fine. >>>>> I added spark-xml jar and now I ended up into this dependency >>>>> >>>>> 6/06/17 15:15:57 INFO BlockManagerMaster: Registered BlockManager >>>>> Exception in thread "main" *java.lang.NoClassDefFoundError: >>>>> scala/collection/GenTraversableOnce$class* >>>>> at >>>>> org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.<init>(ddl.scala:150) >>>>> at >>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:154) >>>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>>> Caused by:* java.lang.ClassNotFoundException: >>>>> scala.collection.GenTraversableOnce$class* >>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>> ... 5 more >>>>> 16/06/17 15:15:58 INFO SparkContext: Invoking stop() from shutdown hook >>>>> >>>>> >>>>> >>>>> On Fri, Jun 17, 2016 at 2:59 PM, Marco Mistroni <mmistr...@gmail.com> >>>>> wrote: >>>>> >>>>>> So you are using spark-submit or spark-shell? >>>>>> >>>>>> you will need to launch either by passing --packages option (like in >>>>>> the example below for spark-csv). you will need to iknow >>>>>> >>>>>> --packages com.databricks:spark-xml_<scala.version>:<package version> >>>>>> >>>>>> hth >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 17, 2016 at 10:20 AM, VG <vlin...@gmail.com> wrote: >>>>>> >>>>>>> Apologies for that. >>>>>>> I am trying to use spark-xml to load data of a xml file. >>>>>>> >>>>>>> here is the exception >>>>>>> >>>>>>> 16/06/17 14:49:04 INFO BlockManagerMaster: Registered BlockManager >>>>>>> Exception in thread "main" java.lang.ClassNotFoundException: Failed >>>>>>> to find data source: org.apache.spark.xml. Please find packages at >>>>>>> http://spark-packages.org >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102) >>>>>>> at >>>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >>>>>>> at >>>>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) >>>>>>> at org.ariba.spark.PostsProcessing.main(PostsProcessing.java:19) >>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>> org.apache.spark.xml.DefaultSource >>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) >>>>>>> at scala.util.Try$.apply(Try.scala:192) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) >>>>>>> at scala.util.Try.orElse(Try.scala:84) >>>>>>> at >>>>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62) >>>>>>> ... 4 more >>>>>>> >>>>>>> Code >>>>>>> SQLContext sqlContext = new SQLContext(sc); >>>>>>> DataFrame df = sqlContext.read() >>>>>>> .format("org.apache.spark.xml") >>>>>>> .option("rowTag", "row") >>>>>>> .load("A.xml"); >>>>>>> >>>>>>> Any suggestions please .. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 17, 2016 at 2:42 PM, Marco Mistroni <mmistr...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> too little info >>>>>>>> it'll help if you can post the exception and show your sbt file (if >>>>>>>> you are using sbt), and provide minimal details on what you are doing >>>>>>>> kr >>>>>>>> >>>>>>>> On Fri, Jun 17, 2016 at 10:08 AM, VG <vlin...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Failed to find data source: com.databricks.spark.xml >>>>>>>>> >>>>>>>>> Any suggestions to resolve this >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
