Re: Please reply if you use Mesos fine grained mode

[Soren Macbeth]

we use fine-grained mode. coarse-grained mode keeps JVMs around which often
leads to OOMs, which in turn kill the entire executor, causing entire
stages to be retried. In fine-grained mode, only the task fails and
subsequently gets retried without taking out an entire stage or worse.

On Tue, Nov 3, 2015 at 3:54 PM, Reynold Xin  wrote:

> If you are using Spark with Mesos fine grained mode, can you please
> respond to this email explaining why you use it over the coarse grained
> mode?
>
> Thanks.
>
>

java.net.ConnectException: Connection timed out

[Soren Macbeth]

Hello,

I get a lot of these exceptions on my mesos cluster when running spark jobs:

14/07/19 16:29:43 WARN spark.network.SendingConnection: Error finishing
connection to prd-atl-mesos-slave-010/10.88.160.200:37586
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318)
at
org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Handling connection
error on connection to ConnectionManagerId(prd-atl-mesos-slave-010,37586)
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Removing
SendingConnection to ConnectionManagerId(prd-atl-mesos-slave-010,37586)
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Notifying
org.apache.spark.network.ConnectionManager$MessageStatus@4b0472b4
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Notifying
org.apache.spark.network.ConnectionManager$MessageStatus@1106ade6
14/07/19 16:29:43 ERROR
spark.storage.BlockFetcherIterator$BasicBlockFetcherIterator: Could not get
block(s) from ConnectionManagerId(prd-atl-mesos-slave-010,37586)
14/07/19 16:29:43 ERROR
spark.storage.BlockFetcherIterator$BasicBlockFetcherIterator: Could not get
block(s) from ConnectionManagerId(prd-atl-mesos-slave-010,37586)
14/07/19 16:29:43 WARN spark.network.SendingConnection: Error finishing
connection to prd-atl-mesos-slave-004/10.88.160.156:35446
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318)
at
org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Handling connection
error on connection to ConnectionManagerId(prd-atl-mesos-slave-004,35446)
14/07/19 16:29:43 INFO spark.network.ConnectionManager: Removing
SendingConnection to ConnectionManagerId(prd-atl-mesos-slave-004,35446)

I've tried bumping up the spark.akka.timeout, but it doesn't seem to have
much of an effect.

Has anyone else seen these? Is there a spark configuration option that I
should tune? Or perhaps some JVM properties that I should be setting on my
executors?

TIA

Re: Data loading to Parquet using spark

[Soren Macbeth]

I typed spark parquet into google and the top results was this blog post
about reading and writing parquet files from spark

http://zenfractal.com/2013/08/21/a-powerful-big-data-trio/


On Mon, Jul 7, 2014 at 5:23 PM, Michael Armbrust mich...@databricks.com
wrote:

 SchemaRDDs, provided by Spark SQL, have a saveAsParquetFile command.  You
 can turn a normal RDD into a SchemaRDD using the techniques described here:
 http://spark.apache.org/docs/latest/sql-programming-guide.html

 This should work with Impala, but if you run into any issues please let me
 know.


 On Sun, Jul 6, 2014 at 5:30 PM, Shaikh Riyaz shaikh@gmail.com wrote:

 Hi,

 We are planning to use spark to load data to Parquet and this data will
 be query by Impala for present visualization through Tableau.

 Can we achieve this flow? How to load data to Parquet from spark? Will
 impala be able to access the data loaded by spark?

 I will greatly appreciate if someone can help with the example to achieve
 the goal.

 Thanks in advance.

 --
 Regards,

 Riyaz

Re: No FileSystem for scheme: hdfs

[Soren Macbeth]

Are the hadoop configuration files on the classpath for your mesos
executors?


On Thu, Jul 3, 2014 at 6:45 PM, Steven Cox s...@renci.org wrote:

  ...and a real subject line.
  --
 *From:* Steven Cox [s...@renci.org]
 *Sent:* Thursday, July 03, 2014 9:21 PM
 *To:* user@spark.apache.org
 *Subject:*

   Folks, I have a program derived from the Kafka streaming wordcount
 example which works fine standalone.


  Running on Mesos is not working so well. For starters, I get the error
 below No FileSystem for scheme: hdfs.


  I've looked at lots of promising comments on this issue so now I have -

 * Every jar under hadoop in my classpath

 * Hadoop HDFS and Client in my pom.xml


  I find it odd that the app writes checkpoint files to HDFS successfully
 for a couple of cycles then throws this exception. This would suggest the
 problem is not with the syntax of the hdfs URL, for example.


  Any thoughts on what I'm missing?


  Thanks,


  Steve


  Mesos : 0.18.2

 Spark : 0.9.1



  14/07/03 21:14:20 WARN TaskSetManager: Lost TID 296 (task 1514.0:0)

 14/07/03 21:14:20 WARN TaskSetManager: Lost TID 297 (task 1514.0:1)

 14/07/03 21:14:20 WARN TaskSetManager: Lost TID 298 (task 1514.0:0)

 14/07/03 21:14:20 ERROR TaskSetManager: Task 1514.0:0 failed 10 times;
 aborting job

 14/07/03 21:14:20 ERROR JobScheduler: Error running job streaming job
 140443646 ms.0

 org.apache.spark.SparkException: Job aborted: Task 1514.0:0 failed 10
 times (most recent failure: Exception failure: java.io.IOException: No
 FileSystem for scheme: hdfs)

 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)

 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)

 at
 scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

 at
 scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

 at org.apache.spark.scheduler.DAGScheduler.org
 $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)

 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)

 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)

 at scala.Option.foreach(Option.scala:236)

 at
 org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)

 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)

 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)

 at akka.actor.ActorCell.invoke(ActorCell.scala:456)

 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)

[ANNOUNCE] Flambo - A Clojure DSL for Apache Spark

[Soren Macbeth]

Yieldbot is pleased to announce the release of Flambo, our Clojure DSL for
Apache Spark.

Flambo allows one to write spark applications in pure Clojure as an
alternative to Scala, Java and Python currently available in Spark.

We have already written a substantial amount of internal code in clojure
using flambo and we are excited to hear and see what other will come up
with.

As ever, Pull Request and/or Issues on Github are greatly appreciated!

You can find links to source, api docs and literate source code here:

http://bit.ly/V8FmzC

-- @sorenmacbeth

Re: Crazy Kryo Exception

[Soren Macbeth]

Does this perhaps have to do with the spark.closure.serializer?


On Sat, May 3, 2014 at 7:50 AM, Soren Macbeth so...@yieldbot.com wrote:

 Poking around in the bowels of scala, it seems like this has something to
 do with implicit scala - java collection munging. Why would it be doing
 this and where? The stack trace given is entirely unhelpful to me. Is there
 a better one buried in my task logs? None of my tasks actually failed, so
 it seems that it dying while trying to fetch results from my tasks to
 return back to the driver.

 Am I close?


 On Fri, May 2, 2014 at 3:35 PM, Soren Macbeth so...@yieldbot.com wrote:

 Hallo,

 I've getting this rather crazy kryo exception trying to run my spark job:

 Exception in thread main org.apache.spark.SparkException: Job aborted:
 Exception while deserializing and fetching task:
 com.esotericsoftware.kryo.KryoException:
 java.lang.IllegalArgumentException: Can not set final
 scala.collection.convert.Wrappers field
 scala.collection.convert.Wrappers$SeqWrapper.$outer to my.custom.class
 Serialization trace:
 $outer (scala.collection.convert.Wrappers$SeqWrapper)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
 at
 scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
 at
 scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
 at org.apache.spark.scheduler.DAGScheduler.org
 $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
 at scala.Option.foreach(Option.scala:236)
 at
 org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
 at akka.actor.ActorCell.invoke(ActorCell.scala:456)
 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
 at akka.dispatch.Mailbox.run(Mailbox.scala:219)
 at
 akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
 at
 scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
 at
 scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
 at
 scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
 at
 scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

 I have a kryo serializer for my.custom.class and I've registered it using
 a custom registrator on my context object. I've tested the custom
 serializer and the registrator locally and they both function as expected.
 This job is running spark 0.9.1 under mesos in fine grained mode.

 Please help!

using kryo for spark.closure.serializer with a registrator doesn't work

[Soren Macbeth]

Is this supposed to be supported? It doesn't work, at least in mesos fine
grained mode. First it fails a bunch of times because it can't find my
registrator class because my assembly jar hasn't been fetched like so:

java.lang.ClassNotFoundException: pickles.kryo.PicklesRegistrator
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at 
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:63)
at 
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:61)
at scala.Option.foreach(Option.scala:236)
at 
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:61)
at 
org.apache.spark.serializer.KryoSerializerInstance.init(KryoSerializer.scala:116)
at 
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:79)
at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:180)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at 
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

after that it does finally fetch my jar, then it fails with the following
expection:

14/05/04 04:23:57 ERROR executor.Executor: Exception in task ID 79
java.nio.ReadOnlyBufferException
at java.nio.ByteBuffer.array(ByteBuffer.java:961)
at 
org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:136)
at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at 
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at 
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)


Should I file a bug?

Re: Crazy Kryo Exception

[Soren Macbeth]

Poking around in the bowels of scala, it seems like this has something to
do with implicit scala - java collection munging. Why would it be doing
this and where? The stack trace given is entirely unhelpful to me. Is there
a better one buried in my task logs? None of my tasks actually failed, so
it seems that it dying while trying to fetch results from my tasks to
return back to the driver.

Am I close?


On Fri, May 2, 2014 at 3:35 PM, Soren Macbeth so...@yieldbot.com wrote:

 Hallo,

 I've getting this rather crazy kryo exception trying to run my spark job:

 Exception in thread main org.apache.spark.SparkException: Job aborted:
 Exception while deserializing and fetching task:
 com.esotericsoftware.kryo.KryoException:
 java.lang.IllegalArgumentException: Can not set final
 scala.collection.convert.Wrappers field
 scala.collection.convert.Wrappers$SeqWrapper.$outer to my.custom.class
 Serialization trace:
 $outer (scala.collection.convert.Wrappers$SeqWrapper)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
 at
 scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
 at
 scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
 at org.apache.spark.scheduler.DAGScheduler.org
 $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
 at scala.Option.foreach(Option.scala:236)
 at
 org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
 at
 org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
 at akka.actor.ActorCell.invoke(ActorCell.scala:456)
 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
 at akka.dispatch.Mailbox.run(Mailbox.scala:219)
 at
 akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
 at
 scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
 at
 scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
 at
 scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
 at
 scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

 I have a kryo serializer for my.custom.class and I've registered it using
 a custom registrator on my context object. I've tested the custom
 serializer and the registrator locally and they both function as expected.
 This job is running spark 0.9.1 under mesos in fine grained mode.

 Please help!

custom kryoserializer class under mesos

[Soren Macbeth]

Hello,

Is it possible to use a custom class as my spark's KryoSerializer running
under Mesos?

I've tried adding my jar containing the class to my spark context (via
SparkConf.addJars), but I always get:

java.lang.ClassNotFoundException: flambo.kryo.FlamboKryoSerializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at 
org.apache.spark.serializer.SerializerManager.get(SerializerManager.scala:56)
at 
org.apache.spark.serializer.SerializerManager.setDefault(SerializerManager.scala:38)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:146)
at org.apache.spark.executor.Executor.init(Executor.scala:110)
at 
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread Thread-0


Do I need to include this jar containing my serializer class in my
make-distribution executor tgz or something?

Thanks