I am on spark 1.3.1
When I do the following with spark-shell, it works
spark-shell --packages com.databricks:spark-csv_2.10:1.0.3
Then I can create a DF using the spark-csv package
import sqlContext.implicits._
import org.apache.spark.sql._
// Return the dataset specified by data source as a DataFrame, use the
header for column names
val df = sqlContext.load("com.databricks.spark.csv", Map("path" ->
"sfpd.csv", "header" -> "true"))
Now, I want to do the above as part of my package using spark-submit
spark-submit --class "ServiceProject" --master local[4] --packages
com.databricks:spark-csv_2.10:1.0.3
target/scala-2.10/serviceproject_2.10-1.0.jar
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url =
jar:file:/usr/hdp/2.3.0.0-2557/spark/lib/spark-assembly-1.3.1.2.3.0.0-2557-h
adoop2.7.1.2.3.0.0-2557.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.databricks#spark-csv_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-csv_2.10;1.0.3 in central
found org.apache.commons#commons-csv;1.1 in list
:: resolution report :: resolve 366ms :: artifacts dl 11ms
:: modules in use:
com.databricks#spark-csv_2.10;1.0.3 from central in [default]
org.apache.commons#commons-csv;1.1 from list in [default]
---------------------------------------------------------------------
| | modules || artifacts
|
| conf | number| search|dwnlded|evicted||
number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 2 | 0
|
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 2 already retrieved (0kB/19ms)
Error: application failed with exception
java.lang.ArrayIndexOutOfBoundsException: 0
at ServiceProject$.main(ServiceProject.scala:29)
at ServiceProject.main(ServiceProject.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$ru
nMain(SparkSubmit.scala:577)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:174)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
What am I doing wrong?
Subhajit
