Didn't really edit the configs as much .. but here's what the spark-env.sh
is:
#!/usr/bin/env bash
##
# Generated by Cloudera Manager and should not be modified directly
##
export SPARK_HOME=/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/spark
export STANDALONE_SPARK_MASTER_HOST=cloudera-1.
As Marcelo mentioned, the issue occurs mostly when incompatible classes are
used by executors or drivers. Try out if the output is coming on
spark-shell. If yes, then most probably in your case, there might be some
issue with your configuration files. It will be helpful if you can paste
the conten
Sorry meant cdh 5.2 w/ spark 1.1.
On Wed, Nov 19, 2014, 17:41 Anson Abraham wrote:
> yeah CDH distribution (1.1).
>
> On Wed Nov 19 2014 at 5:29:39 PM Marcelo Vanzin
> wrote:
>
>> On Wed, Nov 19, 2014 at 2:13 PM, Anson Abraham
>> wrote:
>> > yeah but in this case i'm not building any files. j
yeah CDH distribution (1.1).
On Wed Nov 19 2014 at 5:29:39 PM Marcelo Vanzin wrote:
> On Wed, Nov 19, 2014 at 2:13 PM, Anson Abraham
> wrote:
> > yeah but in this case i'm not building any files. just deployed out
> config
> > files in CDH5.2 and initiated a spark-shell to just read and output
On Wed, Nov 19, 2014 at 2:13 PM, Anson Abraham wrote:
> yeah but in this case i'm not building any files. just deployed out config
> files in CDH5.2 and initiated a spark-shell to just read and output a file.
In that case it is a little bit weird. Just to be sure, you are using
CDH's version of
yeah but in this case i'm not building any files. just deployed out config
files in CDH5.2 and initiated a spark-shell to just read and output a file.
On Wed Nov 19 2014 at 4:52:51 PM Marcelo Vanzin wrote:
> Hi Anson,
>
> We've seen this error when incompatible classes are used in the driver
>
Hi Anson,
We've seen this error when incompatible classes are used in the driver
and executors (e.g., same class name, but the classes are different
and thus the serialized data is different). This can happen for
example if you're including some 3rd party libraries in your app's
jar, or changing t
Question ... when you mean different versions, different versions of
dependency files? what are the dependency files for spark?
On Tue Nov 18 2014 at 5:27:18 PM Anson Abraham
wrote:
> when cdh cluster was running, i did not set up spark role. When I did for
> the first time, it was working ie,
when cdh cluster was running, i did not set up spark role. When I did for
the first time, it was working ie, the same load of test file gave me
output. But in this case, how can there be different versions? This is
all done through cloudera manager parcels how does one find out version
installe
It can be a serialization issue.
Happens when there are different versions installed on the same system.
What do you mean by the first time you installed and tested it out?
On Wed, Nov 19, 2014 at 3:29 AM, Anson Abraham
wrote:
> I'm essentially loading a file and saving output to another locatio
I'm essentially loading a file and saving output to another location:
val source = sc.textFile("/tmp/testfile.txt")
source.saveAsTextFile("/tmp/testsparkoutput")
when i do so, i'm hitting this error:
14/11/18 21:15:08 INFO DAGScheduler: Failed to run saveAsTextFile at
:15
org.apache.spark.SparkEx
11 matches
Mail list logo
