Hi,
I'm trying to get Spark 1.5.1 to work with Hive 0.13.1. I set the following
properties in spark-defaults.conf:
spark.sql.hive.metastore.version 0.13.1
spark.sql.hive.metastore.jars
/usr/lib/hadoop/client/*:/opt/hive/current/lib/*
but I get the following exception when launching the shell:
Hi,
in the Spark UI, one of the metrics is shuffle spill (memory). What is it
exactly? Spilling to disk when the shuffle data doesn't fit in memory I get
it, but what does it mean to spill to memory?
Thanks,
- Sebastien
Hi,
I'm having trouble serializing tasks for this code:
val rddC = (rddA join rddB)
.map { case (x, (y, z)) = z - y }
.reduceByKey( { (y1, y2) = Semigroup.plus(y1, y2) }, 1000)
Somehow when running on a small data set the size of the serialized task is
about 650KB, which is very big, and
Hello Federico,
is it working with the 1.0 branch? In either branch, make sure that you
have this commit:
https://github.com/apache/spark/commit/1132e472eca1a00c2ce10d2f84e8f0e79a5193d3
I never saw the behavior you are describing, but that commit is important
if you are running in fine-grained
+ Mesos within my company.
I'm bumping +1 for the request of putting this fix in the 1.0.1 if
possible!
thanks,
Federico
2014-06-20 20:51 GMT+02:00 Sébastien Rainville
sebastienrainvi...@gmail.com:
Hi,
this is just a follow-up regarding this issue. Turns out that it's
caused by a bug
at 2:57 PM, Sébastien Rainville
sebastienrainvi...@gmail.com wrote:
Hi,
I'm having trouble running spark on mesos in fine-grained mode. I'm
running spark 1.0.0 and mesos 0.18.0. The tasks are failing randomly, which
most of the time, but not always, cause the job to fail. The same code
Hi,
I'm having trouble running spark on mesos in fine-grained mode. I'm running
spark 1.0.0 and mesos 0.18.0. The tasks are failing randomly, which most of
the time, but not always, cause the job to fail. The same code is running
fine in coarse-grained mode. I see the following exceptions in the
