that is pretty odd -- toMap not being there would be from scala...but what
is even weirder is that toMap is positively executed on the driver machine,
which is the same when you do spark-shell and spark-submit...does it work
if you run with --master local[*]?
Also, you can try to put a set -x in bin/spark-class right before the
RUNNER gets invoked -- this will show you the exact java command that is
being run, classpath and all
On Thu, Jul 23, 2015 at 3:14 PM, Dan Dong <dongda...@gmail.com> wrote:
> The problem should be "toMap", as I tested that "val maps2=maps.collect"
> runs ok. When I run spark-shell, I run with "--master
> mesos://cluster-1:5050" parameter which is the same with "spark-submit".
> Confused here.
>
>
>
> 2015-07-22 20:01 GMT-05:00 Yana Kadiyska <yana.kadiy...@gmail.com>:
>
>> Is it complaining about "collect" or "toMap"? In either case this error
>> is indicative of an old version usually -- any chance you have an old
>> installation of Spark somehow? Or scala? You can try running spark-submit
>> with --verbose. Also, when you say it runs with spark-shell do you run
>> spark shell in local mode or with --master? I'd try with --master <whatever
>> master you use for spark-submit>
>>
>> Also, if you're using standalone mode I believe the worker log contains
>> the launch command for the executor -- you probably want to examine that
>> classpath carefully
>>
>> On Wed, Jul 22, 2015 at 5:25 PM, Dan Dong <dongda...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a simple test spark program as below, the strange thing is that
>>> it runs well under a spark-shell, but will get a runtime error of
>>>
>>> java.lang.NoSuchMethodError:
>>>
>>> in spark-submit, which indicate the line of:
>>>
>>> val maps2=maps.collect.toMap
>>>
>>> has problem. But why the compilation has no problem and it works well
>>> under spark-shell(==>maps2: scala.collection.immutable.Map[Int,String] =
>>> Map(269953 -> once, 97 -> a, 451002 -> upon, 117481 -> was, 226916 ->
>>> there, 414413 -> time, 146327 -> king) )? Thanks!
>>>
>>> import org.apache.spark.SparkContext._
>>> import org.apache.spark.SparkConf
>>> import org.apache.spark.mllib.feature.HashingTF
>>> import org.apache.spark.mllib.linalg.Vector
>>> import org.apache.spark.rdd.RDD
>>> import org.apache.spark.SparkContext
>>> import org.apache.spark._
>>> import SparkContext._
>>>
>>>
>>> val docs=sc.parallelize(Array(Array("once" ,"upon", "a", "time"),
>>> Array("there", "was", "a", "king")))
>>>
>>> val hashingTF = new HashingTF()
>>>
>>> val maps=docs.flatMap{term=>term.map(ele=>(hashingTF.indexOf(ele),ele))}
>>>
>>> val maps2=maps.collect.toMap
>>>
>>>
>>> Cheers,
>>>
>>> Dan
>>>
>>>
>>
>
