MMMmmmh good point !!

Before answering, I tried to use callioppe but I got an issue and since the
iteration review was near I quickly switched to the datastax driver. But
I'll get to callioppe soon, with some questions maybe ;-).

Regarding your point (very good one, I've to say), actually I'm creating a
session and a batch per partitions.
Now the shamy part... I haven't set any options for the pool :-/. Is there
some tuning clues? In my case the C* is local (docker image) so maybe
should i do
builder.poolingOptions().setMaxConnectionsPerHost(LOCAL, BIGNUMBER)?

The point is, what about this BIGNUMBER... can it be really big? (Sounds
weird to me, but I don't want to pre-filter options based on feelings).

Thanks for your response

andy

On Fri, Feb 21, 2014 at 10:36 AM, Sourav Chandra <
sourav.chan...@livestream.com> wrote:

> From stacktrace it looks like you are using datstax cassandra driver and
> it tried to create cluster.
>
> How many connections you are creating in poolingOptions()  i.e. builder.
> poolingOptions().setMaxConnectionsPerHost(...)
>
> Are you creating this per rdd? Might be there are lots of connections
> created and at last it failed to create any more.
>
> Thanks,
> Sourav
>
>
> On Fri, Feb 21, 2014 at 3:02 PM, andy petrella <andy.petre...@gmail.com>wrote:
>
>> Hey guyz,
>>
>> I've got this issue (see bottom) with Spark, deployed in Standalone mode
>> on a local docker environment.
>> I know that I need to raise the ulimit (only 1024 now) but in the
>> meantime I was just wondering how this could happen.
>> My gut feeling is because I'm mounting a lot in memory and Spark tries to
>> dump some RDDs on the FS, and then boom.
>>
>> Also, I was wondering if it cannot be a clue that my job is maybe to
>> eager in memory? How is it something quite normal which such a low ulimit
>> on workers?
>>
>> Thanks a lot (in advance ^^)
>>
>> Cheers,
>> andy
>>
>>
>>
>> 14/02/21 08:32:15 ERROR Executor: Exception in task ID 472
>> org.jboss.netty.channel.ChannelException: Failed to create a selector.
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:53)
>> at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
>>  at
>> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
>>  at
>> org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
>>  at
>> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
>> at
>> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
>>  at
>> com.datastax.driver.core.Connection$Factory.<init>(Connection.java:349)
>> at com.datastax.driver.core.Connection$Factory.<init>(Connection.java:360)
>>  at com.datastax.driver.core.Cluster$Manager.<init>(Cluster.java:857)
>> at com.datastax.driver.core.Cluster$Manager.<init>(Cluster.java:806)
>>  at com.datastax.driver.core.Cluster.<init>(Cluster.java:76)
>> at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:132)
>>  at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:771)
>> at
>> com.virdata.core.batch.sample.Timeseries$$anonfun$storeInCassandra$1$1$$anonfun$apply$1$$anonfun$apply$2.apply(Timeseries.scala:45)
>>  at
>> com.virdata.core.batch.sample.Timeseries$$anonfun$storeInCassandra$1$1$$anonfun$apply$1$$anonfun$apply$2.apply(Timeseries.scala:38)
>> at
>> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:595)
>>  at
>> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:595)
>> at
>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
>>  at
>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:884)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
>>  at org.apache.spark.scheduler.Task.run(Task.scala:53)
>> at
>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
>>  at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:46)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:45)
>>  at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>  at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at
>> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:45)
>>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.IOException: Too many open files
>> at sun.nio.ch.IOUtil.makePipe(Native Method)
>> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
>>  at
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
>> at java.nio.channels.Selector.open(Selector.java:227)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
>> ... 37 more
>> 14/02/21 08:32:53 WARN BlockManagerMaster: Error sending message to
>> BlockManagerMaster in 1 attempts
>> java.util.concurrent.TimeoutException: Futures timed out after [30
>> seconds]
>> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>  at
>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>  at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.result(package.scala:107)
>>  at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:162)
>> at
>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>  at org.apache.spark.storage.BlockManager.org
>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>  at
>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>> at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>  at
>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> 14/02/21 08:33:26 WARN BlockManagerMaster: Error sending message to
>> BlockManagerMaster in 2 attempts
>> java.util.concurrent.TimeoutException: Futures timed out after [30
>> seconds]
>> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>  at
>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>  at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.result(package.scala:107)
>>  at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:162)
>> at
>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>  at org.apache.spark.storage.BlockManager.org
>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>  at
>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>> at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>  at
>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> 14/02/21 08:33:59 WARN BlockManagerMaster: Error sending message to
>> BlockManagerMaster in 3 attempts
>> java.util.concurrent.TimeoutException: Futures timed out after [30
>> seconds]
>> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>  at
>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>  at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.result(package.scala:107)
>>  at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:162)
>> at
>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>  at org.apache.spark.storage.BlockManager.org
>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>  at
>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>> at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>  at
>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> 14/02/21 08:34:03 ERROR Executor: Uncaught exception in thread
>> Thread[Connection manager future execution context-0,5,main]
>> java.lang.Error: org.apache.spark.SparkException: Error sending message
>> to BlockManagerMaster [message = HeartBeat(BlockManagerId(1, 172.17.0.4,
>> 52780, 0))]
>>  at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1116)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>  at java.lang.Thread.run(Thread.java:722)
>> Caused by: org.apache.spark.SparkException: Error sending message to
>> BlockManagerMaster [message = HeartBeat(BlockManagerId(1, 172.17.0.4,
>> 52780, 0))]
>>  at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>> at
>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>  at org.apache.spark.storage.BlockManager.org
>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>  at
>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>> at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>  at
>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>  ... 2 more
>> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
>> [30 seconds]
>> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>>  at
>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>>  at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.result(package.scala:107)
>>  at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:162)
>> ... 8 more
>>
>
>
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chan...@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>

Reply via email to