Re: Spark runs only on Mesos v0.21?

[Iulian Dragoș]

Spark cares about Mesos, and it's safe to run it with Mesos 0.27. 0.21 is a
minimum requirement.

On Fri, Feb 12, 2016 at 9:42 AM, Tamas Szuromi <
tamas.szur...@odigeo.com.invalid> wrote:

> Hello Petr,
>
> We're running Spark 1.5.2 and 1.6.0 on Mesos 0.25.0 without any problem.
> We upgraded from 0.21.0 originally.
>
> cheers,
> Tamas
>
>
>
>
> On 12 February 2016 at 09:31, Petr Novak  wrote:
>
>> Hi all,
>> based on documenation:
>>
>> "Spark 1.6.0 is designed for use with Mesos 0.21.0 and does not require
>> any special patches of Mesos."
>>
>> We are considering Mesos for our use but this concerns me a lot. Mesos is
>> currently on v0.27 which we need for its Volumes feature. But Spark locks
>> us to 0.21 only. I understand that it is the problem that Mesos is not 1.0
>> yet and make breaking changes to its API. But when leading frameworks
>> doesn't catch up fast it beats the whole purpose of Mesos - run on one
>> unified platform and share resources, single framework can lock down Mesos
>> upgrade.
>>
>> We don't want to develop our own services against Spark 0.21 API.
>>
>> Is there a reason why Spark is so much behind? Does Spark actually cares
>> about Mesos and its support or the focus moved to YARN?
>>
>> Many thanks,
>> Petr
>>
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Memory problems and missing heartbeats

[Iulian Dragoș]

Regarding your 2nd problem, my best guess is that you’re seeing GC pauses.
It’s not unusual, given you’re using 40GB heaps. See for instance this blog
post


>From conducting numerous tests, we have concluded that unless you are
utilizing some off-heap technology (e.g. GridGain OffHeap), no Garbage
Collector provided with JDK will render any kind of stable GC performance
with heap sizes larger that 16GB. For example, on 50GB heaps we can often
encounter up to 5 minute GC pauses, with average pauses of 2 to 4 seconds.

Not sure if Yarn can do this, but I would try to run with a smaller
executor heap, and more executors per node.

iulian

Re: how to correctly run scala script using spark-shell through stdin (spark v1.0.0)

[Iulian Dragoș]

On Fri, Jan 29, 2016 at 5:22 PM, Iulian Dragoș <iulian.dra...@typesafe.com>
wrote:

> I found the issue in the 2.11 version of the REPL, PR will follow shortly.
>


https://github.com/apache/spark/pull/10984



>
> The 2.10 version of Spark doesn't have this issue, so you could use that
> in the mean time.
>
> iulian
>
> On Wed, Jan 27, 2016 at 3:17 PM, <andres.fernan...@wellsfargo.com> wrote:
>
>> So far, still cannot find a way of running a small Scala script right
>> after executing the shell, and get the shell to remain open. Is there a way
>> of doing this?
>>
>> Feels like a simple/naive question but really couldn’t find an answer.
>>
>>
>>
>> *From:* Fernandez, Andres
>> *Sent:* Tuesday, January 26, 2016 2:53 PM
>> *To:* 'Ewan Leith'; Iulian Dragoș
>> *Cc:* user
>> *Subject:* RE: how to correctly run scala script using spark-shell
>> through stdin (spark v1.0.0)
>>
>>
>>
>> True thank you. Is there a way of having the shell not closed (how to
>> avoid the :quit statement). Thank you both.
>>
>>
>>
>> Andres
>>
>>
>>
>> *From:* Ewan Leith [mailto:ewan.le...@realitymine.com
>> <ewan.le...@realitymine.com>]
>> *Sent:* Tuesday, January 26, 2016 1:50 PM
>> *To:* Iulian Dragoș; Fernandez, Andres
>> *Cc:* user
>> *Subject:* RE: how to correctly run scala script using spark-shell
>> through stdin (spark v1.0.0)
>>
>>
>>
>> I’ve just tried running this using a normal stdin redirect:
>>
>>
>>
>> ~/spark/bin/spark-shell < simple.scala
>>
>>
>>
>> Which worked, it started spark-shell, executed the script, the stopped
>> the shell.
>>
>>
>>
>> Thanks,
>>
>> Ewan
>>
>>
>>
>> *From:* Iulian Dragoș [mailto:iulian.dra...@typesafe.com
>> <iulian.dra...@typesafe.com>]
>> *Sent:* 26 January 2016 15:00
>> *To:* fernandrez1987 <andres.fernan...@wellsfargo.com>
>> *Cc:* user <user@spark.apache.org>
>> *Subject:* Re: how to correctly run scala script using spark-shell
>> through stdin (spark v1.0.0)
>>
>>
>>
>> I don’t see -i in the output of spark-shell --help. Moreover, in master
>> I get an error:
>>
>> $ bin/spark-shell -i test.scala
>>
>> bad option: '-i'
>>
>> iulian
>>
>> ​
>>
>>
>>
>> On Tue, Jan 26, 2016 at 3:47 PM, fernandrez1987 <
>> andres.fernan...@wellsfargo.com> wrote:
>>
>> spark-shell -i file.scala is not working for me in Spark 1.6.0, was this
>> removed or what do I have to take into account? The script does not get
>> run
>> at all. What can be happening?
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/script.png
>> >
>>
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/shell-call.png
>> >
>>
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/no-println.png
>> >
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-correctly-run-scala-script-using-spark-shell-through-stdin-spark-v1-0-0-tp12972p26071.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>>
>>
>>
>> --
>>
>>
>> --
>> Iulian Dragos
>>
>>
>>
>> --
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>>
>>
>
>
>
> --
>
> --
> Iulian Dragos
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: how to correctly run scala script using spark-shell through stdin (spark v1.0.0)

[Iulian Dragoș]

I found the issue in the 2.11 version of the REPL, PR will follow shortly.

The 2.10 version of Spark doesn't have this issue, so you could use that in
the mean time.

iulian

On Wed, Jan 27, 2016 at 3:17 PM, <andres.fernan...@wellsfargo.com> wrote:

> So far, still cannot find a way of running a small Scala script right
> after executing the shell, and get the shell to remain open. Is there a way
> of doing this?
>
> Feels like a simple/naive question but really couldn’t find an answer.
>
>
>
> *From:* Fernandez, Andres
> *Sent:* Tuesday, January 26, 2016 2:53 PM
> *To:* 'Ewan Leith'; Iulian Dragoș
> *Cc:* user
> *Subject:* RE: how to correctly run scala script using spark-shell
> through stdin (spark v1.0.0)
>
>
>
> True thank you. Is there a way of having the shell not closed (how to
> avoid the :quit statement). Thank you both.
>
>
>
> Andres
>
>
>
> *From:* Ewan Leith [mailto:ewan.le...@realitymine.com
> <ewan.le...@realitymine.com>]
> *Sent:* Tuesday, January 26, 2016 1:50 PM
> *To:* Iulian Dragoș; Fernandez, Andres
> *Cc:* user
> *Subject:* RE: how to correctly run scala script using spark-shell
> through stdin (spark v1.0.0)
>
>
>
> I’ve just tried running this using a normal stdin redirect:
>
>
>
> ~/spark/bin/spark-shell < simple.scala
>
>
>
> Which worked, it started spark-shell, executed the script, the stopped the
> shell.
>
>
>
> Thanks,
>
> Ewan
>
>
>
> *From:* Iulian Dragoș [mailto:iulian.dra...@typesafe.com
> <iulian.dra...@typesafe.com>]
> *Sent:* 26 January 2016 15:00
> *To:* fernandrez1987 <andres.fernan...@wellsfargo.com>
> *Cc:* user <user@spark.apache.org>
> *Subject:* Re: how to correctly run scala script using spark-shell
> through stdin (spark v1.0.0)
>
>
>
> I don’t see -i in the output of spark-shell --help. Moreover, in master I
> get an error:
>
> $ bin/spark-shell -i test.scala
>
> bad option: '-i'
>
> iulian
>
> ​
>
>
>
> On Tue, Jan 26, 2016 at 3:47 PM, fernandrez1987 <
> andres.fernan...@wellsfargo.com> wrote:
>
> spark-shell -i file.scala is not working for me in Spark 1.6.0, was this
> removed or what do I have to take into account? The script does not get run
> at all. What can be happening?
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/script.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/shell-call.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/no-println.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-correctly-run-scala-script-using-spark-shell-through-stdin-spark-v1-0-0-tp12972p26071.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
>
>
>
> --
>
>
> --
> Iulian Dragos
>
>
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark streaming flow control and back pressure

[Iulian Dragoș]

Calling `store` should get you there. What version of Spark are you using?
Can you share your code?

iulian

On Thu, Jan 28, 2016 at 2:28 AM, Lin Zhao  wrote:

> I have an actor receiver that reads data and calls "store()" to save data
> to spark. I was hoping spark.streaming.receiver.maxRate and
> spark.streaming.backpressure would help me block the method when needed to
> avoid overflowing the pipeline. But it doesn't. My actor pumps millions of
> lines to spark when backpressure and the rate limit is in effect. Whereas
> these data is slow flowing into the input blocks, the data created sits
> around and creates memory problem.
>
> Is there guideline how to handle this? What's the best way for my actor to
> know it should slow down so it doesn't keep creating millions of messages?
> Blocking store() call seems aptable.
>
> Thanks, Lin
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: how to correctly run scala script using spark-shell through stdin (spark v1.0.0)

[Iulian Dragoș]

I don’t see -i in the output of spark-shell --help. Moreover, in master I
get an error:

$ bin/spark-shell -i test.scala
bad option: '-i'

iulian
​

On Tue, Jan 26, 2016 at 3:47 PM, fernandrez1987 <
andres.fernan...@wellsfargo.com> wrote:

> spark-shell -i file.scala is not working for me in Spark 1.6.0, was this
> removed or what do I have to take into account? The script does not get run
> at all. What can be happening?
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/script.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/shell-call.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/no-println.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-correctly-run-scala-script-using-spark-shell-through-stdin-spark-v1-0-0-tp12972p26071.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: how to correctly run scala script using spark-shell through stdin (spark v1.0.0)

[Iulian Dragoș]

On Tue, Jan 26, 2016 at 4:08 PM, <andres.fernan...@wellsfargo.com> wrote:

> Yes no option –i. Thanks Iulian, but do you know how can I send three
> lines to be executed just after spark-shell has initiated. Please check
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-correctly-run-scala-script-using-spark-shell-through-stdin-spark-v1-0-0-td12972.html#a26071
> .
>

To be honest, I think this might be a regression I introduced, or at least,
it's something that works in the 2.10 version of Spark. By just looking at
the code, it should accept the same arguments as the Scala interpreter.
I'll look into it.

iulian


>
>
> Thank you very much for your time.
>
>
>
> *From:* Iulian Dragoș [mailto:iulian.dra...@typesafe.com]
> *Sent:* Tuesday, January 26, 2016 12:00 PM
> *To:* Fernandez, Andres
> *Cc:* user
> *Subject:* Re: how to correctly run scala script using spark-shell
> through stdin (spark v1.0.0)
>
>
>
> I don’t see -i in the output of spark-shell --help. Moreover, in master I
> get an error:
>
> $ bin/spark-shell -i test.scala
>
> bad option: '-i'
>
> iulian
>
> ​
>
>
>
> On Tue, Jan 26, 2016 at 3:47 PM, fernandrez1987 <
> andres.fernan...@wellsfargo.com> wrote:
>
> spark-shell -i file.scala is not working for me in Spark 1.6.0, was this
> removed or what do I have to take into account? The script does not get run
> at all. What can be happening?
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/script.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/shell-call.png
> >
>
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n26071/no-println.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-correctly-run-scala-script-using-spark-shell-through-stdin-spark-v1-0-0-tp12972p26071.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
>
>
>
> --
>
>
> --
> Iulian Dragos
>
>
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark Streaming on mesos

[Iulian Dragoș]

On Mon, Nov 30, 2015 at 4:09 PM, Renjie Liu <liurenjie2...@gmail.com> wrote:

> Hi, Lulian:
>

Please, it's Iulian, not Lulian.


> Are you sure that it'll be a long running process in fine-grained mode? I
> think you have a misunderstanding about it. An executor will be launched
> for some tasks, but not a long running process. When a group of tasks
> finished, it will get shutdown.
>

Sorry I missed your answer. Yes, I'm pretty sure, and if you have SSH
access to one of the slaves it's pretty easy to check. What makes you think
otherwise?


>
> On Mon, Nov 30, 2015 at 6:25 PM Iulian Dragoș <iulian.dra...@typesafe.com>
> wrote:
>
>> Hi,
>>
>> Latency isn't such a big issue as it sounds. Did you try it out and
>> failed some performance metrics?
>>
>> In short, the *Mesos* executor on a given slave is going to be
>> long-running (consuming memory, but no CPUs). Each Spark task will be
>> scheduled using Mesos CPU resources, but they don't suffer much latency.
>>
>> iulian
>>
>>
>> On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <liurenjie2...@gmail.com>
>> wrote:
>>
>>> Hi, Tim:
>>> Fine grain mode is not suitable for streaming applications since it need
>>> to start up an executor each time. When will the revamp get release? In the
>>> coming 1.6.0?
>>>
>>> On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen <t...@mesosphere.io> wrote:
>>>
>>>> Hi Renjie,
>>>>
>>>> You can set number of cores per executor with spark executor cores in
>>>> fine grain mode.
>>>>
>>>> If you want coarse grain mode to support that it will
>>>> Be supported in the near term as he coarse grain scheduler is getting
>>>> revamped now.
>>>>
>>>> Tim
>>>>
>>>> On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi, Nagaraj:
>>>>  Thanks for the response, but this does not solve my problem.
>>>> I think executor memory should be proportional to number of cores, or
>>>> number of core
>>>> in each executor should be the same.
>>>> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar <
>>>> nchandrashe...@innominds.com> wrote:
>>>>
>>>>> Hi Renjie,
>>>>>
>>>>> I have not setup Spark Streaming on Mesos but there is something
>>>>> called reservations in Mesos.  It supports both Static and Dynamic
>>>>> reservations.  Both types of reservations must have role defined. You may
>>>>> want to explore these options.   Excerpts from the Apache Mesos
>>>>> documentation.
>>>>>
>>>>> Cheers
>>>>> Nagaraj C
>>>>> Reservation
>>>>>
>>>>> Mesos provides mechanisms to reserve resources in specific slaves.
>>>>> The concept was first introduced with static reservation in 0.14.0
>>>>> which enabled operators to specify the reserved resources on slave 
>>>>> startup.
>>>>> This was extended with dynamic reservation in 0.23.0 which enabled
>>>>> operators and authorized frameworks to dynamically reserve resources
>>>>> in the cluster.
>>>>>
>>>>> No breaking changes were introduced with dynamic reservation, which
>>>>> means the existing static reservation mechanism continues to be fully
>>>>> supported.
>>>>>
>>>>> In both types of reservations, resources are reserved for a role.
>>>>> Static Reservation (since 0.14.0)
>>>>>
>>>>> An operator can configure a slave with resources reserved for a role.
>>>>> The reserved resources are specified via the --resources flag. For
>>>>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave
>>>>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role.
>>>>> We start the slave like so:
>>>>>
>>>>> $ mesos-slave \
>>>>>   --master=: \
>>>>>   --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096"
>>>>>
>>>>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave.
>>>>>
>>>>>
>>>>> From: Renjie Liu <liurenjie2...@gmail.com>
>>>>> Date: Friday, November 27, 2015 at 9:57 PM
>>>>> To: "user@spark.apache.org" <user@spark.apache.org>
>>>>> Subject: Spark Streaming on mesos
>>>>>
>>>>> Hi, all:
>>>>> I'm trying to run spark streaming on mesos and it seems that none of
>>>>> the scheduler is suitable for that. Fine grain scheduler will start an
>>>>> executor for each task so it will significantly increase the latency. 
>>>>> While
>>>>> coarse grained mode can only set the max core numbers and executor memory
>>>>> but there's no way to set the number of cores for each executor. Has 
>>>>> anyone
>>>>> deployed spark streaming on mesos? And what's your settings?
>>>>> --
>>>>> Liu, Renjie
>>>>> Software Engineer, MVAD
>>>>>
>>>> --
>>>> Liu, Renjie
>>>> Software Engineer, MVAD
>>>>
>>>> --
>>> Liu, Renjie
>>> Software Engineer, MVAD
>>>
>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> --
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>> --
> Liu, Renjie
> Software Engineer, MVAD
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Mesos scheduler obeying limit of tasks / executor

[Iulian Dragoș]

Hi Charles,

I am not sure I totally understand your issues, but the spark.task.cpus
limit is imposed at a higher level, for all cluster managers. The code is
in TaskSchedulerImpl

.

There is a pending PR to implement spark.executor.cores (and launching
multiple executors on a single worker), but it wasn’t yet merged:
https://github.com/apache/spark/pull/4027

iulian
​

On Wed, Dec 9, 2015 at 7:23 PM, Charles Allen  wrote:

> I have a spark app in development which has relatively strict cpu/mem
> ratios that are required. As such, I cannot arbitrarily add CPUs to a
> limited memory size.
>
> The general spark cluster behaves as expected, where tasks are launched
> with a specified memory/cpu ratio, but the mesos scheduler seems to ignore
> this.
>
> Specifically, I cannot find where in the code the limit of number of tasks
> per executor of "spark.executor.cores" / "spark.task.cpus" is enforced in
> the MesosBackendScheduler.
>
> The Spark App in question has some JVM heap heavy activities inside a
> RDD.mapPartitionsWithIndex, so having more tasks per limited JVM memory
> resource is bad. The workaround planned handling of this is to limit the
> number of tasks per JVM, which does not seem possible in mesos mode, where
> it seems to just keep stacking on CPUs as tasks come in without adjusting
> any memory constraints, or looking for limits of tasks per executor.
>
> How can I limit the tasks per executor (or per memory pool) in the Mesos
> backend scheduler?
>
> Thanks,
> Charles Allen
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: default parallelism and mesos executors

[Iulian Dragoș]

On Wed, Dec 9, 2015 at 4:29 PM, Adrian Bridgett 
wrote:

> (resending, text only as first post on 2nd never seemed to make it)
>
> Using parallelize() on a dataset I'm only seeing two tasks rather than the
> number of cores in the Mesos cluster.  This is with spark 1.5.1 and using
> the mesos coarse grained scheduler.
>
> Running pyspark in a console seems to show that it's taking a while before
> the mesos executors come online (at which point the default parallelism is
> changing).  If I add "sleep 30" after initialising the SparkContext I get
> the "right" number (42 by coincidence!)
>
> I've just tried increasing minRegisteredResourcesRatio to 0.5 but this
> doesn't affect either the test case below nor my code.
>

This limit seems to be implemented only in the coarse-grained Mesos
scheduler, but the fix will be available starting with Spark 1.6.0 (1.5.2
doesn't have it).

iulian


>
> Is there something else I can do instead?  Perhaps it should be seeing how
> many tasks _should_ be available rather than how many are (I'm also using
> dynamicAllocation).
>
> 15/12/02 14:34:09 INFO mesos.CoarseMesosSchedulerBackend: SchedulerBackend
> is ready for scheduling beginning after reached
> minRegisteredResourcesRatio: 0.0
> >>>
> >>>
> >>> print (sc.defaultParallelism)
> 2
> >>> 15/12/02 14:34:12 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 5
> is now TASK_RUNNING
> 15/12/02 14:34:13 INFO mesos.MesosExternalShuffleClient: Successfully
> registered app 20151117-115458-164233482-5050-24333-0126 with external
> shuffle service.
> 
> 15/12/02 14:34:15 INFO mesos.CoarseMesosSchedulerBackend: Registered
> executor:
> AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@ip-10-1-200-147.ec2.internal:41194/user/Executor#-1021429650])
> with ID 20151117-115458-164233482-5050-24333-S22/5
> 15/12/02 14:34:15 INFO spark.ExecutorAllocationManager: New executor
> 20151117-115458-164233482-5050-24333-S22/5 has registered (new total is 1)
> 
> >>> print (sc.defaultParallelism)
> 42
>
> Thanks
>
> Adrian Bridgett
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Scala 2.11 and Akka 2.4.0

[Iulian Dragoș]

As a I mentioned on the akka mailing list, in case others are following
this thread: the issue isn't with dependencies. It's a bug in the
maven-shade-plugin. It breaks classfiles when creating the assembly jar (it
seems to do some constant propagation). `sbt assembly` doesn't suffer from
this issue, probably because it uses another library for jar merging.

iulian

On Tue, Dec 1, 2015 at 7:21 PM, Boavida, Rodrigo  wrote:

> HI Jacek,
>
> Yes I was told that as well but no one gave me release schedules, and I
> have the immediate need to have Spark Applications communicating with Akka
> clusters based on latest version. I'm aware there is an ongoing effort to
> change to the low level netty implementation but AFAIK it's not available
> yet.
>
> Any suggestions are very welcomed.
>
> Tnks,
> Rod
>
> -Original Message-
> From: Jacek Laskowski [mailto:ja...@japila.pl]
> Sent: 01 December 2015 18:17
> To: Boavida, Rodrigo 
> Cc: user 
> Subject: Re: Scala 2.11 and Akka 2.4.0
>
> On Tue, Dec 1, 2015 at 2:32 PM, RodrigoB 
> wrote:
>
> > I'm currently trying to build spark with Scala 2.11 and Akka 2.4.0.
>
> Why? AFAIK Spark's leaving Akka's boat and joins Netty's.
>
> Jacek
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark Streaming on mesos

[Iulian Dragoș]

Hi,

Latency isn't such a big issue as it sounds. Did you try it out and failed
some performance metrics?

In short, the *Mesos* executor on a given slave is going to be long-running
(consuming memory, but no CPUs). Each Spark task will be scheduled using
Mesos CPU resources, but they don't suffer much latency.

iulian


On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu  wrote:

> Hi, Tim:
> Fine grain mode is not suitable for streaming applications since it need
> to start up an executor each time. When will the revamp get release? In the
> coming 1.6.0?
>
> On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen  wrote:
>
>> Hi Renjie,
>>
>> You can set number of cores per executor with spark executor cores in
>> fine grain mode.
>>
>> If you want coarse grain mode to support that it will
>> Be supported in the near term as he coarse grain scheduler is getting
>> revamped now.
>>
>> Tim
>>
>> On Nov 28, 2015, at 7:31 PM, Renjie Liu  wrote:
>>
>> Hi, Nagaraj:
>>  Thanks for the response, but this does not solve my problem.
>> I think executor memory should be proportional to number of cores, or
>> number of core
>> in each executor should be the same.
>> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar <
>> nchandrashe...@innominds.com> wrote:
>>
>>> Hi Renjie,
>>>
>>> I have not setup Spark Streaming on Mesos but there is something called
>>> reservations in Mesos.  It supports both Static and Dynamic reservations.
>>> Both types of reservations must have role defined. You may want to explore
>>> these options.   Excerpts from the Apache Mesos documentation.
>>>
>>> Cheers
>>> Nagaraj C
>>> Reservation
>>>
>>> Mesos provides mechanisms to reserve resources in specific slaves. The
>>> concept was first introduced with static reservation in 0.14.0 which
>>> enabled operators to specify the reserved resources on slave startup. This
>>> was extended with dynamic reservation in 0.23.0 which enabled operators
>>> and authorized frameworks to dynamically reserve resources in the
>>> cluster.
>>>
>>> No breaking changes were introduced with dynamic reservation, which
>>> means the existing static reservation mechanism continues to be fully
>>> supported.
>>>
>>> In both types of reservations, resources are reserved for a role.
>>> Static Reservation (since 0.14.0)
>>>
>>> An operator can configure a slave with resources reserved for a role.
>>> The reserved resources are specified via the --resources flag. For
>>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave
>>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role.
>>> We start the slave like so:
>>>
>>> $ mesos-slave \
>>>   --master=: \
>>>   --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096"
>>>
>>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave.
>>>
>>>
>>> From: Renjie Liu 
>>> Date: Friday, November 27, 2015 at 9:57 PM
>>> To: "user@spark.apache.org" 
>>> Subject: Spark Streaming on mesos
>>>
>>> Hi, all:
>>> I'm trying to run spark streaming on mesos and it seems that none of the
>>> scheduler is suitable for that. Fine grain scheduler will start an executor
>>> for each task so it will significantly increase the latency. While coarse
>>> grained mode can only set the max core numbers and executor memory but
>>> there's no way to set the number of cores for each executor. Has anyone
>>> deployed spark streaming on mesos? And what's your settings?
>>> --
>>> Liu, Renjie
>>> Software Engineer, MVAD
>>>
>> --
>> Liu, Renjie
>> Software Engineer, MVAD
>>
>> --
> Liu, Renjie
> Software Engineer, MVAD
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

[Iulian Dragoș]

Hi Jo,

I agree that there's something fishy with the cluster dispatcher, I've seen
some issues like that.

I think it actually tries to send all properties as part of
`SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:

https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377

Can you please open a Jira ticket and describe also the symptoms? This
might be related, or the same issue: SPARK-11280
 and also SPARK-11327


thanks,
iulian




On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers 
wrote:

>
> Hi all,
>
> I'm running the mesos cluster dispatcher, however when I submit jobs with
> things like jvm args, classpath order and UI port aren't added to the
> commandline executed by the mesos scheduler. In fact it only cares about
> the class, jar and num cores/mem.
>
>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>
> I've made an attempt at adding a few of the args that I believe are useful
> to the MesosClusterScheduler class, which seems to solve my problem.
>
> Please have a look:
>
> https://github.com/apache/spark/pull/9752
>
> Thanks
>
> - Jo Voordeckers
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Conf Settings in Mesos

[Iulian Dragoș]

Hi John,

I don't think this is specific to Mesos.

Note that `spark-defaults.conf` are only defaults. Normally you'd pass your
specific options using `--conf`. Does that work?

iulian


On Thu, Nov 12, 2015 at 3:05 PM, John Omernik  wrote:

> Hey all,
>
> I noticed today that if I take a tgz as my URI for Mesos, that I have to
> repackaged it with my conf settings from where I execute say pyspark for
> the executors to have the right configuration settings.
>
> That is...
>
> If I take a "stock" tgz from makedistribution.sh, unpack it, and then set
> the URI in spark-defaults to be the unmodified tgz as the URI. Change other
> settings in both spark-defaults.conf and spark-env.sh, then run
> ./bin/pyspark from that unpacked directory, I guess I would have thought
> that when the executor spun up, that some sort of magic was happening where
> the conf directory or the conf settings would propagate out to the
> executors (thus making configuration changes easier to manage)
>
> For things to work, I had to unpack the tgz, change conf settings, then
> repackage the tgz with all my conf settings for the tgz in the URI then run
> it. Then it seemed to work.
>
> I have a work around, but I guess, from a usability point of view, it
> would be nice to have tgz that is "binaries" and that when it's run, it
> takes the conf at run time. It would help with managing multiple
> configurations that are using the same binaries (different models/apps etc)
> Instead of having to repackage an tgz for each app, it would just
> propagate...am I looking at this wrong?
>
> John
>
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: ClassNotFoundException even if class is present in Jarfile

[Iulian Dragoș]

Where is the exception thrown (full stack trace)? How are you running your
application, via spark-submit or spark-shell?

On Tue, Nov 3, 2015 at 1:43 AM, hveiga  wrote:

> Hello,
>
> I am facing an issue where I cannot run my Spark job in a cluster
> environment (standalone or EMR) but it works successfully if I run it
> locally using local[*] as master.
>
> I am getting ClassNotFoundException: com.mycompany.folder.MyObject on the
> slave executors. I don't really understand why this is happening since I
> have uncompressed the Jarfile to make sure that the class is present inside
> (both .java and .class) and all the rest of the classes are being loaded
> fine.
>
> Also, I would like to mention something weird that might be related but not
> sure. There are two packages inside my jarfile that are called the same but
> with different casing:
>
> - com.mycompany.folder.MyObject
> - com.myCompany.something.Else
>
> Could that be the reason?
>
> Also, I have tried adding my jarfiles in all the ways I could find
> (sparkConf.setJars(...), sparkContext.addJar(...), spark-submit opt --jars,
> ...) but none of the actually worked.
>
> I am using Apache Spark 1.5.0, Java 7, sbt 0.13.7, scala 2.10.5.
>
> Thanks a lot,
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-even-if-class-is-present-in-Jarfile-tp25254.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Is coalesce smart while merging partitions?

[Iulian Dragoș]

It's smart. Have a look at
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L123

On Thu, Oct 8, 2015 at 4:00 AM, Cesar Flores  wrote:

> It is my understanding that the default behavior of coalesce function when
> the user reduce the number of partitions is to only merge them without
> executing shuffle.
>
> My question is: Is this merging smart? For example does spark try to merge
> the small partitions first or the election of partitions to merge is random?
>
>
> Thanks
> --
> Cesar Flores
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark Streaming: Doing operation in Receiver vs RDD

[Iulian Dragoș]

You can have a look at
http://spark.apache.org/docs/latest/streaming-programming-guide.html#receiver-reliability
for details on Receiver reliability. If you go the receiver way you'll need
to enable Write Ahead Logs to ensure no data loss. In Kafka direct you
don't have this problem.

Regarding where to apply decryption, I'd lean towards doing it as RDD
transformations for the reasons you mentioned. Also, in case only some
fields are encrypted, this way you can delay decryption until really need
(assuming some records would be filtered out, etc.).

iulian

On Wed, Oct 7, 2015 at 9:55 PM, emiretsk  wrote:

> Hi,
>
> I have a Spark Streaming program that is consuming message from Kafka and
> has to decrypt and deserialize each message. I can implement it either as
> Kafka deserializer (that will run in a receiver or the new receiver-less
> Kafka consumer)  or as RDD operations. What are the pros/cons of each?
>
> As I see it, doing the operations on RDDs has the following implications
> Better load balancing, and fault tolerance. (though I'm not quite sure what
> happens when a receiver fails). Also, not sure if this is still true with
> the new Kafka receiver-less consumer as it creates an RDD partition for
> each
> Kafka partition
> All functions that are applied to RDDs need to be either static or part of
> serialzable objects. This makes using standard/3rd party Java libraries
> harder.
> Cheers,
> Eugene
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Doing-operation-in-Receiver-vs-RDD-tp24973.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Optimal way to avoid processing null returns in Spark Scala

[Iulian Dragoș]

On Wed, Oct 7, 2015 at 6:42 PM, swetha  wrote:

Hi,
>
> I have the following functions that I am using for my job in Scala. If you
> see the getSessionId function I am returning null sometimes. If I return
> null the only way that I can avoid processing those records is by filtering
> out null records. I wanted to avoid having another pass for filtering so I
> tried returning "None" . But, it seems to be having issues as it demands
> the
> return type as optional. What is the optimal way to avoid processing null
> records and at the same way avoid having Option as the return type using
> None? The use of Option[] and Some(()) seems to be having type issues in
> subsequent function calls.
>
You should use RDD.flatMap, this way you can map and filter at the same
time. Something like

rdd.flatMap { case (x, y) =>
  val sessionid = getSessionId(y)
  if (sessionId != null)
  Seq(((sessionId, (getTimeStamp(y).toLong,y
  else
  Seq()
}

I didn’t try to compile that method, but you’ll figure out the types, if
need be.

iulian


>
> val sessions = filteredStream.transform(rdd=>getBeaconMap(rdd))
>
>   def getBeaconMap(rdd: RDD[(String, String)]): RDD[(String, (Long,
> String))] = {
> rdd.map[(String, (Long, String))]{ case (x, y) =>
>   ((getSessionId(y), (getTimeStamp(y).toLong,y)))
> }
>   }
>
>   def getSessionId(eventRecord:String): String = {
> val beaconTestImpl: BeaconTestLoader = new BeaconTestImpl//This needs
> to
> be changed.
> val beaconEvent: BeaconEventData =
> beaconTestImpl.getBeaconEventData(eventRecord)
>
> if(beaconEvent!=null){
>beaconEvent.getSessionID //This might be in Set Cookie header
> }else{
>  null
> }
>
>
> val groupedAndSortedSessions =
> sessions.transform(rdd=>ExpoJobCommonNew.getGroupedAndSortedSessions(rdd))
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Optimal-way-to-avoid-processing-null-returns-in-Spark-Scala-tp24972.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
> ​
-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Temp files are not removed when done (Mesos)

[Iulian Dragoș]

It is indeed a bug. I believe the shutdown procedure in #7820 only kicks in
when the external shuffle service is enabled (a pre-requisite of dynamic
allocation). As a workaround you can use dynamic allocation (you can set
spark.dynamicAllocation.maxExecutors and
spark.dynamicAllocation.minExecutors to the same value.

I'll file a Jira ticket.

On Wed, Oct 7, 2015 at 10:14 AM, Alexei Bakanov  wrote:

> Hi
>
> I'm running Spark 1.5.1 on Mesos in coarse-grained mode. No dynamic
> allocation or shuffle service. I see that there are two types of temporary
> files under /tmp folder associated with every executor: /tmp/spark-
> and /tmp/blockmgr-. When job is finished /tmp/spark- is gone,
> but blockmgr directory is left with all gigabytes in it. In Spark 1.4.1
> blockmgr- folder was under /tmp/spark-, so when /tmp/spark
> folder was removed blockmgr was gone too.
> Is it a bug in 1.5.1?
>
> By the way, in fine-grain mode /tmp/spark- folder does not get
> removed in neither 1.4.1 nor 1.5.1 for some reason.
>
> Thanks,
> Alexei
>



-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Temp files are not removed when done (Mesos)

[Iulian Dragoș]

https://issues.apache.org/jira/browse/SPARK-10975

On Wed, Oct 7, 2015 at 11:36 AM, Iulian Dragoș <iulian.dra...@typesafe.com>
wrote:

> It is indeed a bug. I believe the shutdown procedure in #7820 only kicks
> in when the external shuffle service is enabled (a pre-requisite of dynamic
> allocation). As a workaround you can use dynamic allocation (you can set
> spark.dynamicAllocation.maxExecutors and
> spark.dynamicAllocation.minExecutors to the same value.
>
> I'll file a Jira ticket.
>
> On Wed, Oct 7, 2015 at 10:14 AM, Alexei Bakanov <russ...@gmail.com> wrote:
>
>> Hi
>>
>> I'm running Spark 1.5.1 on Mesos in coarse-grained mode. No dynamic
>> allocation or shuffle service. I see that there are two types of temporary
>> files under /tmp folder associated with every executor: /tmp/spark-
>> and /tmp/blockmgr-. When job is finished /tmp/spark- is gone,
>> but blockmgr directory is left with all gigabytes in it. In Spark 1.4.1
>> blockmgr- folder was under /tmp/spark-, so when /tmp/spark
>> folder was removed blockmgr was gone too.
>> Is it a bug in 1.5.1?
>>
>> By the way, in fine-grain mode /tmp/spark- folder does not get
>> removed in neither 1.4.1 nor 1.5.1 for some reason.
>>
>> Thanks,
>> Alexei
>>
>
>
>
> --
>
> --
> Iulian Dragos
>
> --
> Reactive Apps on the JVM
> www.typesafe.com
>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: hdfs-ha on mesos - odd bug

[Iulian Dragoș]

I've seen similar traces, but couldn't track down the failure completely.
You are using Kerberos for your HDFS cluster, right? AFAIK Kerberos isn't
supported in Mesos deployments.

Can you resolve that host name (nameservice1) from the driver machine (ping
nameservice1)? Can it be resolved from the other machines in the cluster?

Does it help if you read using `newAPIHadoopFile` instead of `textFile`?

On Mon, Sep 14, 2015 at 3:55 PM, Adrian Bridgett 
wrote:

> I'm hitting an odd issue with running spark on mesos together with
> HA-HDFS, with an even odder workaround.
>
> In particular I get an error that it can't find the HDFS nameservice
> unless I put in a _broken_ url (discovered that workaround by mistake!).
> core-site.xml, hdfs-site.xml is distributed to the slave node - and that
> file is read since I deliberately break the file then I get an error as
> you'd expect.
>
> NB: This is a bit different to
> http://mail-archives.us.apache.org/mod_mbox/spark-user/201402.mbox/%3c1392442185079-1549.p...@n3.nabble.com%3E
>
>
> Spark 1.5.0:
>
> t=sc.textFile("hdfs://nameservice1/tmp/issue")
> t.count()
> (fails)
>
> t=sc.textFile("file://etc/passwd")
> t.count()
> (errors about bad url - should have an extra / of course)
> t=sc.textFile("hdfs://nameservice1/tmp/issue")
> t.count()
> then it works!!!
>
> I should say that using file:///etc/passwd or hdfs:///tmp/issue both fail
> as well.  Unless preceded by a broken url.I've tried setting
> spark.hadoop.cloneConf to true, no change.
>
> Sample (broken) run:
> 15/09/14 13:00:14 DEBUG HadoopRDD: Creating new JobConf and caching it for
> later re-use
> 15/09/14 13:00:14 DEBUG : address: ip-10-1-200-165/10.1.200.165
> isLoopbackAddress: false, with host 10.1.200.165 ip-10-1-200-165
> 15/09/14 13:00:14 DEBUG BlockReaderLocal:
> dfs.client.use.legacy.blockreader.local = false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal: dfs.client.read.shortcircuit =
> false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal:
> dfs.client.domain.socket.data.traffic = false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal: dfs.domain.socket.path =
> /var/run/hdfs-sockets/dn
> 15/09/14 13:00:14 DEBUG HAUtil: No HA service delegation token found for
> logical URI hdfs://nameservice1
> 15/09/14 13:00:14 DEBUG BlockReaderLocal:
> dfs.client.use.legacy.blockreader.local = false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal: dfs.client.read.shortcircuit =
> false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal:
> dfs.client.domain.socket.data.traffic = false
> 15/09/14 13:00:14 DEBUG BlockReaderLocal: dfs.domain.socket.path =
> /var/run/hdfs-sockets/dn
> 15/09/14 13:00:14 DEBUG RetryUtils: multipleLinearRandomRetry = null
> 15/09/14 13:00:14 DEBUG Server: rpcKind=RPC_PROTOCOL_BUFFER,
> rpcRequestWrapperClass=class
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper,
> rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@6245f50b
> 15/09/14 13:00:14 DEBUG Client: getting client out of cache:
> org.apache.hadoop.ipc.Client@267f0fd3
> 15/09/14 13:00:14 DEBUG NativeCodeLoader: Trying to load the custom-built
> native-hadoop library...
> 15/09/14 13:00:14 DEBUG NativeCodeLoader: Loaded the native-hadoop library
> ...
> 15/09/14 13:00:14 DEBUG Client: Connecting to
> mesos-1.example.com/10.1.200.165:8020
> 15/09/14 13:00:14 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu: starting, having
> connections 1
> 15/09/14 13:00:14 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu sending #0
> 15/09/14 13:00:14 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu got value #0
> 15/09/14 13:00:14 DEBUG ProtobufRpcEngine: Call: getFileInfo took 36ms
> 15/09/14 13:00:14 DEBUG FileInputFormat: Time taken to get FileStatuses: 69
> 15/09/14 13:00:14 INFO FileInputFormat: Total input paths to process : 1
> 15/09/14 13:00:14 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu sending #1
> 15/09/14 13:00:14 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu got value #1
> 15/09/14 13:00:14 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
> 15/09/14 13:00:14 DEBUG FileInputFormat: Total # of splits generated by
> getSplits: 2, TimeTaken: 104
> ...
> 15/09/14 13:00:24 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu: closed
> 15/09/14 13:00:24 DEBUG Client: IPC Client (1739425103) connection to
> mesos-1.example.com/10.1.200.165:8020 from ubuntu: stopped, remaining
> connections 0
> 15/09/14 13:00:24 DEBUG
> AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received
> message
> AkkaMessage(ExecutorRemoved(20150826-133446-3217621258-5050-4064-S1),true)
> from Actor[akka://sparkDriver/temp/$g]
> 15/09/14 13:00:24 DEBUG
> 

Re: Can not allocate executor when running spark on mesos

[Iulian Dragoș]

On Thu, Sep 10, 2015 at 3:35 AM, canan chen  wrote:

> Finally got the answer.  Actually it works fine. The allocation behavior
> on mesos is a little different from yarn/standalone. Seems the executor in
> mesos is lazily allocated (only when job is executed) while executor in
> yarn/standalone is allocated when spark-shell is started.
>

That's in fine-grained mode (the default). You can turn on coarse-grained
mode to acquire executors on startup.

iulian


>
>
>
> On Tue, Sep 8, 2015 at 10:39 PM, canan chen  wrote:
>
>> Yes, I follow the guide in this doc, and run it as mesos client mode
>>
>> On Tue, Sep 8, 2015 at 6:31 PM, Akhil Das 
>> wrote:
>>
>>> In which mode are you submitting your application? (coarse-grained or
>>> fine-grained(default)). Have you gone through this documentation already?
>>> http://spark.apache.org/docs/latest/running-on-mesos.html#using-a-mesos-master-url
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Tue, Sep 8, 2015 at 12:54 PM, canan chen  wrote:
>>>
 Hi all,

 I try to run spark on mesos, but it looks like I can not allocate
 resources from mesos. I am not expert of mesos, but from the mesos log, it
 seems spark always decline the offer from mesos. Not sure what's wrong,
 maybe need some configuration change. Here's the mesos master log

 I0908 15:08:16.515960 301916160 master.cpp:1767] Received registration
 request for framework 'Spark shell' at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
 I0908 15:08:16.520545 301916160 master.cpp:1834] Registering framework
 20150908-143320-16777343-5050-41965- (Spark shell) at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133 with
 checkpointing disabled and capabilities [  ]
 I0908 15:08:16.522307 300843008 hierarchical.hpp:386] Added framework
 20150908-143320-16777343-5050-41965-
 I0908 15:08:16.525845 301379584 master.cpp:4290] Sending 1 offers to
 framework 20150908-143320-16777343-5050-41965- (Spark shell) at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
 I0908 15:08:16.637677 302452736 master.cpp:2884] Processing DECLINE
 call for offers: [ 20150908-143320-16777343-5050-41965-O0 ] for framework
 20150908-143320-16777343-5050-41965- (Spark shell) at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
 I0908 15:08:16.639197 299233280 hierarchical.hpp:761] Recovered
 cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000] (total:
 cpus(*):8; mem(*):15360; disk(*):470842; ports(*):[31000-32000], allocated:
 ) on slave 20150908-143320-16777343-5050-41965-S0 from framework
 20150908-143320-16777343-5050-41965-
 I0908 15:08:21.786932 300306432 master.cpp:4290] Sending 1 offers to
 framework 20150908-143320-16777343-5050-41965- (Spark shell) at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133
 I0908 15:08:21.789979 298696704 master.cpp:2884] Processing DECLINE
 call for offers: [ 20150908-143320-16777343-5050-41965-O1 ] for framework
 20150908-143320-16777343-5050-41965- (Spark shell) at
 scheduler-1ea1c85b-68bd-40b4-8c7c-ddccfd56f82b@192.168.3.3:57133

>>>
>>>
>>
>


-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Windowed stream operations -- These are too lazy for some use cases

[Iulian Dragoș]

On Thu, Aug 20, 2015 at 6:58 PM, Justin Grimes jgri...@adzerk.com wrote:

We are aggregating real time logs of events, and want to do windows of 30
 minutes. However, since the computation doesn't start until 30 minutes have
 passed, there is a ton of data built up that processing could've already
 started on. When it comes time to actually process the data, there is too
 much for our cluster to handle at once.

 The basic idea is this:

  val mergedMain = mergedStream
   .flatMap(r = ) // denormalize data for this particular output
 stream
   .reduceByKey((x:Array[Int],y:Array[Int]) = sumAggregates(x,y)) //
 this would sum over the batches

Could you add a dummy action at this point?

val firstStep = mergedStream
  .flatMap(r = ) // denormalize data for this particular output stream
  .reduceByKey((x:Array[Int],y:Array[Int]) = sumAggregates(x,y))
// this would sum over the batches
  .persist() // this will be reused in windowing operations

firstStep.count() // just to trigger computation

firstStep
  .reduceByKeyAndWindow((x:Array[Int],y:Array[Int]) =
sumAggregates(x,y), 180, 180) // sum over the windows
  .map(rec = ...) // convert data to other format
  .foreachRDD{ (rdd, time) =
rdd.saveAsTextFile(...) // save to text files
  }

  .reduceByKeyAndWindow((x:Array[Int],y:Array[Int]) =
 sumAggregates(x,y), 180, 180) // sum over the windows
   .map(rec = ...) // convert data to other format
   .foreachRDD{ (rdd, time) =
 rdd.saveAsTextFile(...) // save to text files
   }

 I would want the batches to be reduced as soon as they arrive (the first
 reduceByKey), since there isn't any reason to wait. Instead all of the
 unprocessed data has to be processed at the same time (this data is being
 heavily denormalized in some cases, and so generates a bunch of additional
 data).

 Thanks for any help.

​
-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark Mesos task rescheduling

[Iulian Dragoș]

On Thu, Jul 9, 2015 at 12:32 PM, besil sbernardine...@beintoo.com wrote:

 Hi,

 We are experimenting scheduling errors due to mesos slave failing.
 It seems to be an open bug, more information can be found here.

 https://issues.apache.org/jira/browse/SPARK-3289

 According to this  link
 
 https://mail-archives.apache.org/mod_mbox/mesos-user/201310.mbox/%3ccaakwvaxprrnrcdlazcybnmk1_9elyheodaf8urf8ssrlbac...@mail.gmail.com%3E
 
 from mail archive, it seems that Spark doesn't reschedule LOST tasks to
 active executors, but keep trying rescheduling it on the failed host.


Are you running in fine-grained mode? In coarse-grained mode it seems that
Spark will notice a slave that fails repeatedly and would not accept offers
on that slave:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala#L188



 We would like to dynamically resize our Mesos cluster (adding or removing
 machines - using an AWS autoscaling group), but this bug kills our running
 applications if a Mesos slave running a Spark executor is shut down.


I think what you need is dynamic allocation, which should be available soon
(PR: 4984 https://github.com/apache/spark/pull/4984).


 Is any known workaround?

 Thank you



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Mesos-task-rescheduling-tp23740.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Accessing Kerberos Secured HDFS Resources from Spark on Mesos

[Iulian Dragoș]

This is something we (at Typesafe) also thought about, but didn't start
yet. It would be good to pool efforts.

On Sat, Jun 27, 2015 at 12:44 AM, Dave Ariens dari...@blackberry.com
wrote:

  Fair. I will look into an alternative with a generated delegation token.
   However the same issue exists.   How can I have the executor run some
 arbitrary code when it gets a task assignment and before it proceeds to
 process it's resources?

 *From: *Marcelo Vanzin
 *Sent: *Friday, June 26, 2015 6:20 PM
 *To: *Dave Ariens
 *Cc: *Tim Chen; Olivier Girardot; user@spark.apache.org
 *Subject: *Re: Accessing Kerberos Secured HDFS Resources from Spark on
 Mesos

   On Fri, Jun 26, 2015 at 3:09 PM, Dave Ariens dari...@blackberry.com
 wrote:

  Would there be any way to have the task instances in the slaves call
 the UGI login with a principal/keytab provided to the driver?


  That would only work with a very small number of executors. If you have
 many login requests in a short period of time with the same principal, the
 KDC will start to deny logins. That's why delegation tokens are used
 instead of explicit logins.

  --
 Marcelo




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Roadmap for Spark with Kafka on Scala 2.11?

[Iulian Dragoș]

On Tue, May 26, 2015 at 10:09 AM, algermissen1971 
algermissen1...@icloud.com wrote:

 Hi,

 I am setting up a project that requires Kafka support and I wonder what
 the roadmap is for Scala 2.11 Support (including Kafka).

 Can we expect to see 2.11 support anytime soon?


The upcoming 1.4 release (now at RC2) includes support for Kafka and Scala
2.11.6. It'd be great if you could give it a try. You can find the binaries
(and staging repository including 2.11 artifacts) here:

 https://www.mail-archive.com/dev@spark.apache.org/msg09347.html

iulian



 Jan
 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Tasks randomly stall when running on mesos

[Iulian Dragoș]

On Mon, May 25, 2015 at 2:43 PM, Reinis Vicups sp...@orbit-x.de wrote:

  Hello,

 I am using Spark 1.3.1-hadoop2.4 with Mesos 0.22.1 with zookeeper and
 running on a cluster with 3 nodes on 64bit ubuntu.

 My application is compiled with spark 1.3.1 (apparently with mesos 0.21.0
 dependency), hadoop 2.5.1-mapr-1503 and akka 2.3.10. Only with this
 combination I have succeeded to run spark-jobs on mesos at all. Different
 versions are causing class loader issues.

 I am submitting spark jobs with spark-submit with mesos://zk://.../mesos.


Are you using coarse grained or fine grained mode?

sandbox log of slave-node app01 (the one that stalls) shows following:

 10:01:25.815506 35409 fetcher.cpp:214] Fetching URI
 'hdfs://dev-hadoop01/apps/spark-1.3.1-bin-hadoop2.4.tgz'
 10:01:26.497764 35409 fetcher.cpp:99] Fetching URI
 'hdfs://dev-hadoop01/apps/spark-1.3.1-bin-hadoop2.4.tgz' using Hadoop Client
 10:01:26.497869 35409 fetcher.cpp:109] Downloading resource from
 'hdfs://dev-hadoop01/apps/spark-1.3.1-bin-hadoop2.4.tgz' to
 '/tmp/mesos/slaves/20150511-150924-3410235146-5050-1903-S3/frameworks/20150511-150924-3410235146-5050-1903-0249/executors/20150511-150924-3410235146-5050-1903-S3/runs/ec3a0f13-2f44-4952-bb23-4d48abaacc05/spark-1.3.1-bin-hadoop2.4.tgz'
 10:01:32.877717 35409 fetcher.cpp:78] Extracted resource
 '/tmp/mesos/slaves/20150511-150924-3410235146-5050-1903-S3/frameworks/20150511-150924-3410235146-5050-1903-0249/executors/20150511-150924-3410235146-5050-1903-S3/runs/ec3a0f13-2f44-4952-bb23-4d48abaacc05/spark-1.3.1-bin-hadoop2.4.tgz'
 into
 '/tmp/mesos/slaves/20150511-150924-3410235146-5050-1903-S3/frameworks/20150511-150924-3410235146-5050-1903-0249/executors/20150511-150924-3410235146-5050-1903-S3/runs/ec3a0f13-2f44-4952-bb23-4d48abaacc05'
 Using Spark's default log4j profile:
 org/apache/spark/log4j-defaults.properties
 10:01:34 INFO MesosExecutorBackend: Registered signal handlers for [TERM,
 HUP, INT]
 10:01:34.459292 35730 exec.cpp:132] Version: 0.22.0
 *10:01:34 ERROR MesosExecutorBackend: Received launchTask but executor was
 null*
 10:01:34.540870 35765 exec.cpp:206] Executor registered on slave
 20150511-150924-3410235146-5050-1903-S3
 10:01:34 INFO MesosExecutorBackend: Registered with Mesos as executor ID
 20150511-150924-3410235146-5050-1903-S3 with 1 cpus


It looks like an inconsistent state on the Mesos scheduler. It tries to
launch a task on a given slave before the executor has registered. This
code was improved/refactored in 1.4, could you try 1.4.0-RC1?

iulian


 10:01:34 INFO SecurityManager: Changing view acls to...
 10:01:35 INFO Slf4jLogger: Slf4jLogger started
 10:01:35 INFO Remoting: Starting remoting
 10:01:35 INFO Remoting: Remoting started; listening on addresses
 :[akka.tcp://sparkExecutor@app01:xxx]
 10:01:35 INFO Utils: Successfully started service 'sparkExecutor' on port
 xxx.
 10:01:35 INFO AkkaUtils: Connecting to MapOutputTracker:
 akka.tcp://sparkDriver@dev-web01/user/MapOutputTracker
 10:01:35 INFO AkkaUtils: Connecting to BlockManagerMaster:
 akka.tcp://sparkDriver@dev-web01/user/BlockManagerMaster
 10:01:36 INFO DiskBlockManager: Created local directory at
 /tmp/spark-52a6585a-f9f2-4ab6-bebc-76be99b0c51c/blockmgr-e6d79818-fe30-4b5c-bcd6-8fbc5a201252
 10:01:36 INFO MemoryStore: MemoryStore started with capacity 88.3 MB
 10:01:36 WARN NativeCodeLoader: Unable to load native-hadoop library for
 your platform... using builtin-java classes where applicable
 10:01:36 INFO AkkaUtils: Connecting to OutputCommitCoordinator:
 akka.tcp://sparkDriver@dev-web01/user/OutputCommitCoordinator
 10:01:36 INFO Executor: Starting executor ID
 20150511-150924-3410235146-5050-1903-S3 on host app01
 10:01:36 INFO NettyBlockTransferService: Server created on XXX
 10:01:36 INFO BlockManagerMaster: Trying to register BlockManager
 10:01:36 INFO BlockManagerMaster: Registered BlockManager
 10:01:36 INFO AkkaUtils: Connecting to HeartbeatReceiver:
 akka.tcp://sparkDriver@dev-web01/user/HeartbeatReceiver

 As soon as spark-driver is aborted, following log entries are added to the
 sandbox log of slave-node app01:

 10:17:29.559433 35772 exec.cpp:379] Executor asked to shutdown
 10:17:29 WARN ReliableDeliverySupervisor: Association with remote system
 [akka.tcp://sparkDriver@dev-web01] has failed, address is now gated for
 [5000] ms. Reason is: [Disassociated]

 Successful Job shows instead following in spark-driver log:

 08:03:19,862 INFO  o.a.s.s.TaskSetManager - Finished task 3.0 in stage 1.0
 (TID 7) in 1688 ms on app01 (1/4)
 08:03:19,869 INFO  o.a.s.s.TaskSetManager - Finished task 0.0 in stage 1.0
 (TID 4) in 1700 ms on app03 (2/4)
 08:03:19,874 INFO  o.a.s.s.TaskSetManager - Finished task 1.0 in stage 1.0
 (TID 5) in 1703 ms on app02 (3/4)
 08:03:19,878 INFO  o.a.s.s.TaskSetManager - Finished task 2.0 in stage 1.0
 (TID 6) in 1706 ms on app02 (4/4)
 08:03:19,878 INFO  o.a.s.s.DAGScheduler - Stage 1
 (saveAsNewAPIHadoopDataset at ImportSparkJob.scala:90) finished in 1.718 s
 

Re: Multi user setup and saving a DataFrame / RDD to a network exported file system

[Iulian Dragoș]

You could try setting `SPARK_USER` to the user under which your workers are
running. I couldn't find many references to this variable, but at least
Yarn and Mesos take it into account when spawning executors. Chances are
that standalone mode also does it.

iulian

On Wed, May 20, 2015 at 9:29 AM, Tomasz Fruboes tomasz.frub...@fuw.edu.pl
wrote:

 Hi,

  thanks for answer. The rights are

 drwxr-xr-x 3 tfruboes all 5632 05-19 15:40 test19EE/

  I have tried setting the rights to 777 for this directory prior to
 execution. This does not get propagated down the chain, ie the directory
 created as a result of the save call (namesAndAges.parquet2 in the path
 in the dump [1] below) is created with the drwxr-xr-x rights (owned by the
 user submitting the job, ie tfruboes). The temp directories created inside

 namesAndAges.parquet2/_temporary/0/

 (e.g. task_201505200920_0009_r_01) are owned by root, again with
 drwxr-xr-x access rights

  Cheers,
   Tomasz

 W dniu 19.05.2015 o 23:56, Davies Liu pisze:

  It surprises me, could you list the owner information of
 /mnt/lustre/bigdata/med_home/tmp/test19EE/ ?

 On Tue, May 19, 2015 at 8:15 AM, Tomasz Fruboes
 tomasz.frub...@fuw.edu.pl wrote:

 Dear Experts,

   we have a spark cluster (standalone mode) in which master and workers
 are
 started from root account. Everything runs correctly to the point when we
 try doing operations such as

  dataFrame.select(name, age).save(ofile, parquet)

 or

  rdd.saveAsPickleFile(ofile)

 , where ofile is path on a network exported filesystem (visible on all
 nodes, in our case this is lustre, I guess on nfs effect would be
 similar).

   Unsurprisingly temp files created on workers are owned by root, which
 then
 leads to a crash (see [1] below). Is there a solution/workaround for this
 (e.g. controlling file creation mode of the temporary files)?

 Cheers,
   Tomasz


 ps I've tried to google this problem, couple of similar reports, but no
 clear answer/solution found

 ps2 For completeness - running master/workers as a regular user solves
 the
 problem only for the given user. For other users submitting to this
 master
 the result is given in [2] below


 [0] Cluster details:
 Master/workers: centos 6.5
 Spark 1.3.1 prebuilt for hadoop 2.4 (same behaviour for the 2.6 build)


 [1]
 ##
 File

 /mnt/home/tfruboes/2015.05.SparkLocal/spark-1.3.1-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py,
 line 300, in get_return_value
 py4j.protocol.Py4JJavaError: An error occurred while calling o27.save.
 : java.io.IOException: Failed to rename

 DeprecatedRawLocalFileStatus{path=file:/mnt/lustre/bigdata/med_home/tmp/test19EE/namesAndAges.parquet2/_temporary/0/task_201505191540_0009_r_01/part-r-2.parquet;
 isDirectory=false; length=534; replication=1; blocksize=33554432;
 modification_time=1432042832000; access_time=0; owner=; group=;
 permission=rw-rw-rw-; isSymlink=false} to

 file:/mnt/lustre/bigdata/med_home/tmp/test19EE/namesAndAges.parquet2/part-r-2.parquet
  at

 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346)
  at

 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:362)
  at

 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
  at

 parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:43)
  at

 org.apache.spark.sql.parquet.ParquetRelation2.insert(newParquet.scala:690)
  at

 org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:129)
  at
 org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240)
  at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1196)
  at org.apache.spark.sql.DataFrame.save(DataFrame.scala:1181)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at

 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at

 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
  at
 py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
  at py4j.Gateway.invoke(Gateway.java:259)
  at
 py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
  at py4j.commands.CallCommand.execute(CallCommand.java:79)
  at py4j.GatewayConnection.run(GatewayConnection.java:207)
  at java.lang.Thread.run(Thread.java:745)
 ##



 [2]
 ##
 15/05/19 14:45:19 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 3,
 wn23023.cis.gov.pl): 

Re: Spark on Mesos vs Yarn

[Iulian Dragoș]

Hi Ankur,

Just to add a thought to Tim's excellent answer, Spark on Mesos is very
important to us and is the recommended deployment for our customers as
Typesafe.

Thanks for pointing to your PR, I see Tim already went through a round of
reviews. It seems very useful, I'll give it a try as well.

thanks,
iulian



On Fri, May 15, 2015 at 9:53 AM, Ankur Chauhan an...@malloc64.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi Tim,

 Thanks for such a detailed email. I am excited to hear about the new
 features, I had a pull request going for adding attribute based
 filtering in the mesos scheduler but it hasn't received much love -
 https://github.com/apache/spark/pull/5563 . I am a fan of
 mesos/marathon/mesosphere and spark ecosystems and trying to push
 adoption at my workplace.

 It would love to see documentation, tutorials (anything actually) that
 would make mesos + spark a better and more fleshed out solution. Would
 it be possible for you to share some links to the JIRA and pull
 requests so that I can keep track on the progress/features.

 Again, thanks for replying.

 - -- Ankur Chauhan

 On 15/05/2015 00:39, Tim Chen wrote:
  Hi Ankur,
 
  This is a great question as I've heard similar concerns about Spark
  on Mesos.
 
  At the time when I started to contribute to Spark on Mesos approx
  half year ago, the Mesos scheduler and related code hasn't really
  got much attention from anyone and it was pretty much in
  maintenance mode.
 
  As a Mesos PMC that is really interested in Spark I started to
  refactor and check out different JIRAs and PRs around the Mesos
  scheduler, and after that started to fix various bugs in Spark,
  added documentation and also in fix related Mesos issues as well.
 
  Just recently for 1.4 we've merged in Cluster mode and Docker
  support, and there are also pending PRs around framework
  authentication, multi-role support, dynamic allocation, more finer
  tuned coarse grain mode scheduling configurations, etc.
 
  And finally just want to mention that Mesosphere and Typesafe is
  collaborating to bring a certified distribution
  (https://databricks.com/spark/certification/certified-spark-distributi
 on)
  of Spark on Mesos and DCOS, and we will be pouring resources into
  not just maintain Spark on Mesos but drive more features into the
  Mesos scheduler and also in Mesos so stateful services can leverage
  new APIs and features to make better scheduling decisions and
  optimizations.
 
  I don't have a solidified roadmap to share yet, but we will be
  discussing this and hopefully can share with the community soon.
 
  In summary Spark on Mesos is not dead or in maintenance mode, and
  look forward to see a lot more changes from us and the community.
 
  Tim
 
  On Thu, May 14, 2015 at 11:30 PM, Ankur Chauhan
  an...@malloc64.com mailto:an...@malloc64.com wrote:
 
  Hi,
 
  This is both a survey type as well as a roadmap query question. It
  seems like of the cluster options to run spark (i.e. via YARN and
  Mesos), YARN seems to be getting a lot more attention and patches
  when compared to Mesos.
 
  Would it be correct to assume that spark on mesos is more or less
  a dead or something like a maintenance-only feature and YARN is
  the recommended way to go?
 
  What is the roadmap for spark on mesos? and what is the roadmap
  for spark on yarn. I like mesos so as much as I would like to see
  it thrive I don't think spark community is active (or maybe it
  just appears that way).
 
  Another more community oriented question: what do most people use
  to run spark in production or more-than-POC products? Why did you
  make that decision?
 
  There was a similar post form early 2014 where Metei answered that
  mesos and yarn were equally important, but has this changed as
  spark has now reached almost 1.4.0 stage?
 
  -- Ankur Chauhan
 
  -
 
 
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
  mailto:user-unsubscr...@spark.apache.org For additional commands,
  e-mail: user-h...@spark.apache.org
  mailto:user-h...@spark.apache.org
 
 
 -BEGIN PGP SIGNATURE-

 iQEcBAEBAgAGBQJVVaXvAAoJEOSJAMhvLp3LzkIH/RLPVUoOcFo0Gij1NpZDszN4
 xWvesbOZszuqD8H1Dhyndz4RQKnrodyCE+NycFB+utd9epmuyGemmHpTnq18Gek6
 PR5jqmgza94dOy0rfuIVvba14ALZb4tO9SgkjyGujrpMlFYvxTjBYdYCAjfEOTx7
 A/vqaCzPSBRBmO8gWx07GWa4zI70qBSZ9KnV7dgtqfUUgPKdF4NnMZWRJjTO9Bp8
 tTmWMldqYPqI95wdeeqTGMH0XT6JAKAiCskf62DGadRBsOshrhmh5mAQzUFwoTpA
 w4uZ+qMrTsblBvOf9z++v0eY8VBiQpOyXfOBiYiCNRtSsGa0KvqwgF1S/yLeRs0=
 =4Aax
 -END PGP SIGNATURE-

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Error in SparkSQL/Scala IDE

[Iulian Dragoș]

On Thu, May 7, 2015 at 10:18 AM, Iulian Dragoș iulian.dra...@typesafe.com
wrote:

 Got it!

 I'll open a Jira ticket and PR when I have a working solution.


Scratch that, I found SPARK-5281
https://issues.apache.org/jira/browse/SPARK-5281..



 On Wed, May 6, 2015 at 11:53 PM, Michael Armbrust mich...@databricks.com
 wrote:

 Hi Iulian,

 The relevant code is in ScalaReflection
 https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala,
 and it would be awesome if you could suggest how to fix this more
 generally. Specifically, this code is also broken when running from SBT:


 $ build/sbt hive/console
 scala import implicits._
 import implicits._

 scala Seq((1,2)).toDF(a, b)
 scala.reflect.internal.MissingRequirementError: class
 org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with
 java.net.URLClassLoader@752d2e33 of type class java.net.URLClassLoader
 with classpath
 [file:/root/.sbt/boot/scala-2.10.4/lib/jline.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-library.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-compiler.jar,file:/root/.sbt/boot/scala-2.10.4/lib/jansi.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-reflect.jar]
 and parent being xsbt.boot.BootFilteredLoader@625e4591 of type class
 xsbt.boot.BootFilteredLoader with classpath [unknown] and parent being
 sun.misc.Launcher$AppClassLoader@1a8c064 of type class
 sun.misc.Launcher$AppClassLoader with classpath
 [file:/root/spark/build/sbt-launch-0.13.7.jar] and parent being
 sun.misc.Launcher$ExtClassLoader@6deee615 of type class
 sun.misc.Launcher$ExtClassLoader with classpath
 [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/
 nsns.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so]
 and parent being primordial classloader with boot classpath
 [/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes]
 not found.

 Michael

 On Wed, May 6, 2015 at 2:12 AM, Iulian Dragoș iulian.dra...@typesafe.com
  wrote:

 Hi, I just saw this question. I posted my solution to this stack
 overflow question.
 https://stackoverflow.com/questions/29796928/whats-the-most-efficient-way-to-filter-a-dataframe

 Scala reflection can take a classloader when creating a mirror (
 universe.runtimeMirror(loader)). I can have a look, but I didn’t find
 much about mirrors in spark-sql.

 iulian
 ​





 --

 --
 Iulian Dragos

 --
 Reactive Apps on the JVM
 www.typesafe.com




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Error in SparkSQL/Scala IDE

[Iulian Dragoș]

Got it!

I'll open a Jira ticket and PR when I have a working solution.

On Wed, May 6, 2015 at 11:53 PM, Michael Armbrust mich...@databricks.com
wrote:

 Hi Iulian,

 The relevant code is in ScalaReflection
 https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala,
 and it would be awesome if you could suggest how to fix this more
 generally. Specifically, this code is also broken when running from SBT:


 $ build/sbt hive/console
 scala import implicits._
 import implicits._

 scala Seq((1,2)).toDF(a, b)
 scala.reflect.internal.MissingRequirementError: class
 org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with
 java.net.URLClassLoader@752d2e33 of type class java.net.URLClassLoader
 with classpath
 [file:/root/.sbt/boot/scala-2.10.4/lib/jline.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-library.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-compiler.jar,file:/root/.sbt/boot/scala-2.10.4/lib/jansi.jar,file:/root/.sbt/boot/scala-2.10.4/lib/scala-reflect.jar]
 and parent being xsbt.boot.BootFilteredLoader@625e4591 of type class
 xsbt.boot.BootFilteredLoader with classpath [unknown] and parent being
 sun.misc.Launcher$AppClassLoader@1a8c064 of type class
 sun.misc.Launcher$AppClassLoader with classpath
 [file:/root/spark/build/sbt-launch-0.13.7.jar] and parent being
 sun.misc.Launcher$ExtClassLoader@6deee615 of type class
 sun.misc.Launcher$ExtClassLoader with classpath
 [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/
 nsns.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so]
 and parent being primordial classloader with boot classpath
 [/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes]
 not found.

 Michael

 On Wed, May 6, 2015 at 2:12 AM, Iulian Dragoș iulian.dra...@typesafe.com
 wrote:

 Hi, I just saw this question. I posted my solution to this stack
 overflow question.
 https://stackoverflow.com/questions/29796928/whats-the-most-efficient-way-to-filter-a-dataframe

 Scala reflection can take a classloader when creating a mirror (
 universe.runtimeMirror(loader)). I can have a look, but I didn’t find
 much about mirrors in spark-sql.

 iulian
 ​





-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: branch-1.4 scala 2.11

[Iulian Dragoș]

There's an open PR to fix it: https://github.com/apache/spark/pull/5966

On Thu, May 7, 2015 at 6:07 PM, Koert Kuipers ko...@tresata.com wrote:

 i am having no luck using the 1.4 branch with scala 2.11

 $ build/mvn -DskipTests -Pyarn -Dscala-2.11 -Pscala-2.11 clean package

 [error]
 /home/koert/src/opensource/spark/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala:78:
 in object RDDOperationScope, multiple overloaded alternatives of method
 withScope define default arguments.
 [error] private[spark] object RDDOperationScope {
 [error]




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Error in SparkSQL/Scala IDE

[Iulian Dragoș]

Hi, I just saw this question. I posted my solution to this stack overflow
question.
https://stackoverflow.com/questions/29796928/whats-the-most-efficient-way-to-filter-a-dataframe

Scala reflection can take a classloader when creating a mirror (
universe.runtimeMirror(loader)). I can have a look, but I didn’t find much
about mirrors in spark-sql.

iulian
​

Re: Some questions on Multiple Streams

[Iulian Dragoș]

It looks like you’re creating 23 actions in your job (one per DStream). As
far as I know by default Spark Streaming executes only one job at a time.
So your 23 actions are executed one after the other. Try setting
spark.streaming.concurrentJobs to something higher than one.

iulian
​

On Fri, Apr 24, 2015 at 10:44 AM, Laeeq Ahmed laeeqsp...@yahoo.com.invalid
wrote:

 Hi,

 Any comments please.

 Regards,
 Laeeq



   On Friday, April 17, 2015 11:37 AM, Laeeq Ahmed
 laeeqsp...@yahoo.com.INVALID wrote:


 Hi,

 I am working with multiple Kafka streams (23 streams) and currently I am
 processing them separately. I receive one stream from each topic. I have
 the following questions.

 1.Spark streaming guide suggests to union these streams. *Is it
 possible to get statistics of each stream even after they are unioned?*

 2.My calculations are not complex. I use 2 second batch interval and
 if I use 2 streams they get easily processed under 2 seconds by a single
 core. There is some shuffling involved in my application. As I increase the
 number of streams and the number of executors accordingly, the applications
 scheduling delay increases and become unmanageable in 2 seconds. As I
 believe this happens because with that many streams, the number of tasks
 increases thus the shuffling magnifies and also that all streams using the
 same executors. *Is it possible to provide part of executors to
 particular stream while processing streams simultaneously?* E.g. if I
 have 15 cores on cluster and 5 streams, 5  cores will be taken by 5
 receivers and of the rest 10, can I provide 2 cores each to one of the 5
 streams. Just to add, increasing the batch interval does help but I don't
 want to increase the batch size due to application restrictions and delayed
 results (The blockInterval and defaultParallelism does help to a limited
 extent).

 *Please see attach file for CODE SNIPPET*

 Regards,
 Laeeq


 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: A Spark Group by is running forever

[Iulian Dragoș]

On Thu, Apr 23, 2015 at 6:09 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com wrote:


 I have seen multiple blogs stating to use reduceByKey instead of
 groupByKey. Could someone please help me in converting below code to use
 reduceByKey


 Code
 
 some spark processing
 ...

 Below
 val viEventsWithListingsJoinSpsLevelMetric:

  
 org.apache.spark.rdd.RDD[(com.ebay.ep.poc.spark.reporting.process.detail.model.DetailInputRecord,
  com.ebay.ep.poc.spark.reporting.process.detail.viewitem.provider.VISummary,
 Long)]


   val sellerSegments = viEventsWithListingsJoinSpsLevelMetric.groupBy {
   case (viDetail, vi, itemId) =
 (viDetail.get(0), viDetail.get(1).asInstanceOf[Long],
 viDetail.get(2), viDetail.get(8).asInstanceOf[Int])
 }

 We grouby above key so that we get an iterable (list), with list we can
 compute .max values for powersellers and sellerstdlevel.

 val powerSellerLevel = sellerSegments.map {
   case (k, v) =
 val viGrouped = v.toList
 val viPowerSellers = viGrouped.map { viTuple =
 Option(viTuple._2.powerSellerLevel).getOrElse() }
 val viSellerStandardLevels = viGrouped.map { viTuple =
 Option(viTuple._2.sellerStdLevel).getOrElse() }
 val powerSellerLevel = viPowerSellers.max
 val sellerStandardLevel = viSellerStandardLevels.max
 val viEventDetail = viGrouped.head._1
 val viSummary = viGrouped.head._2
 viSummary.powerSellerLevel = powerSellerLevel
 viSummary.sellerStdLevel = sellerStandardLevel
 viSummary.itemId = viGrouped.head._3
 (viEventDetail, viSummary)
 }


 The above groupBy query ran for 6H and does not seem to finish. Hence i
 started thinking of reduceByKey. Now reduceByKey() needs pairs and hence i
 modified viEventsWithListingsJoinSpsLevelMetric ( x,y,z) to
 viEventsWithListingsJoinSpsLevelMetric (A,B).

 I moved the key generated through groupByquery into the processing of
 viEventsWithListingsJoinSpsLevelMetric, so that
 viEventsWithListingsJoinSpsLevelMetric is of type A,B. Hence it is modified
 as

 (((viEventDetail.get(0), viEventDetail.get(1).asInstanceOf[Long],
 viEventDetail.get(2),
 viEventDetail.get(8).asInstanceOf[Int])),(viEventDetail, viSummary,
 itemId)).

 Now i want to compute max values, and i do the next processing using
 reduceByKey

 val powerSellerLevel = viEventsWithListingsJoinSpsLevelMetric.reduceByKey
 {

   case (k, v) =

 val viGrouped = v.toList

  // Some code to compute max needs to go here.

 }


 But i get a compiler error that v.toList is not supported.

 [ERROR]
 /Users/dvasthimal/ebay/projects/ep-spark/ep-spark/src/main/scala/com/ebay/ep/poc/spark/reporting/process/detail/viewitem/provider/VISummaryDataProvider.scala:115:
 error: value toList is not a member of
 (com.ebay.ep.poc.spark.reporting.process.detail.model.DetailInputRecord,
 com.ebay.ep.poc.spark.reporting.process.detail.viewitem.provider.VISummary,
 Long)

 [INFO] val viGrouped = v.toList

 [INFO]   ^

 [ERROR] one error found


As the error points out, your value (`v`) is a Tuple3. It does not have a
toList method. Your argument to reduceByKey should combine two values (in
your case, the two values are two Tuple3s). It's guaranteed that you'll
only see values that belong to the same key, but only 2 at a time. Given
that you need to return the same type as for v (again a Tuple3, with the
same element types), this might not be general enough to compute your max.

You have more freedom with combineByKey which lets the result of your
combine to be different from the value type, but it's a bit more involved.
See the docs
https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions
.

iulian



 Now if you think, groupBy was generating (k, Iterable) and hence the next
 map() could get list and run through that list to compute max. How is that
 possible with reduceByKey because it never generates max.


 Suggestions are appreciated.


 -Deepak















 On Thu, Apr 23, 2015 at 1:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) deepuj...@gmail.com
 wrote:

 I have a groupBy query after a map-side join  leftOuterJoin. And this
 query is running for more than 2 hours.


 asks IndexIDAttemptStatusLocality LevelExecutor ID / HostLaunch Time
 DurationGC TimeShuffle Read Size / RecordsWrite TimeShuffle Write Size /
 RecordsErrors  0 36 0 RUNNING PROCESS_LOCAL 17 /
 phxaishdc9dn1560.stratus.phx.ebay.com 2015/04/22 23:27:00 1.4 h  29 s
 61.8 MB / 63144909  0.0 B / 0



 The input looks to be only 60 MB.
 *Command*
 ./bin/spark-submit -v --master yarn-cluster --driver-class-path
 /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar
 --jars
 

Re: Some questions on Multiple Streams

[Iulian Dragoș]

On Fri, Apr 24, 2015 at 4:56 PM, Laeeq Ahmed laeeqsp...@yahoo.com wrote:

 Thanks Dragos,

 Earlier test shows spark.streaming.concurrentJobs has worked.


Glad to hear it worked!

iulian



 Regards,
 Laeeq




   On Friday, April 24, 2015 11:58 AM, Iulian Dragoș 
 iulian.dra...@typesafe.com wrote:


 It looks like you’re creating 23 actions in your job (one per DStream). As
 far as I know by default Spark Streaming executes only one job at a time.
 So your 23 actions are executed one after the other. Try setting
 spark.streaming.concurrentJobs to something higher than one.
 iulian
 ​

 On Fri, Apr 24, 2015 at 10:44 AM, Laeeq Ahmed 
 laeeqsp...@yahoo.com.invalid wrote:

 Hi,

 Any comments please.

 Regards,
 Laeeq



   On Friday, April 17, 2015 11:37 AM, Laeeq Ahmed
 laeeqsp...@yahoo.com.INVALID wrote:


 Hi,

 I am working with multiple Kafka streams (23 streams) and currently I am
 processing them separately. I receive one stream from each topic. I have
 the following questions.

 1.Spark streaming guide suggests to union these streams. *Is it
 possible to get statistics of each stream even after they are unioned?*

 2.My calculations are not complex. I use 2 second batch interval and
 if I use 2 streams they get easily processed under 2 seconds by a single
 core. There is some shuffling involved in my application. As I increase the
 number of streams and the number of executors accordingly, the applications
 scheduling delay increases and become unmanageable in 2 seconds. As I
 believe this happens because with that many streams, the number of tasks
 increases thus the shuffling magnifies and also that all streams using the
 same executors. *Is it possible to provide part of executors to
 particular stream while processing streams simultaneously?* E.g. if I
 have 15 cores on cluster and 5 streams, 5  cores will be taken by 5
 receivers and of the rest 10, can I provide 2 cores each to one of the 5
 streams. Just to add, increasing the batch interval does help but I don't
 want to increase the batch size due to application restrictions and delayed
 results (The blockInterval and defaultParallelism does help to a limited
 extent).

 *Please see attach file for CODE SNIPPET*

 Regards,
 Laeeq


 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




 --

 --
 Iulian Dragos

 --
 Reactive Apps on the JVM
 www.typesafe.com






-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Building Spark : Building just one module.

[Iulian Dragoș]

One way is to use export SPARK_PREPEND_CLASSES=true. This will instruct the
launcher to prepend the target directories for each project to the spark
assembly. I’ve had mixed experiences with it lately, but in principle
that's the only way I know.
​

On Wed, Apr 22, 2015 at 3:42 PM, zia_kayani zia.kay...@platalytics.com
wrote:

 Hi,

 I've to add custom things into spark SQL and Catalyst Module ... But for
 every time I change a line of code I've to compile the whole spark, if I
 only compile sql/core and sql/catalyst module those changes aren't visible
 when I run the job over that spark, What I'm missing ? Any other way to
 overcome this, as this is time talking ...



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-Building-just-one-module-tp22607.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: Spark on Mesos: Multiple Users with iPython Notebooks

[Iulian Dragoș]

On Thu, Feb 19, 2015 at 2:49 PM, John Omernik j...@omernik.com wrote:

 I am running Spark on Mesos and it works quite well.  I have three
 users, all who setup iPython notebooks to instantiate a spark instance
 to work with on the notebooks. I love it so far.

 Since I am auto instantiating (I don't want a user to have to
 think about instantiating and submitting a spark app to do adhoc
 analysis, I want the environment setup ahead of time) this is done
 whenever an iPython notebook is open.  So far it's working pretty
 good, save one issue:

 Every notebook is a new driver. I.e. every time they open a notebook,
 a new spark submit is called, and the driver resources are allocated,
 regardless if they are used or not.  Yes, it's only the driver, but
 even that I find starts slowing down my queries for the notebooks that
 using spark.  (I am running in Mesos Fined Grained mode).


 I have three users on my system, ideally, I would love to find a way
 so that on the first notebook being opened, a driver is started for
 that user, and then can be used for any notebook the user has open. So
 if they open a new notebook, I can check that yes, the user has a
 spark driver running, and thus, that notebook, if there is a query,
 will run it through that driver. That allows me to understand the
 resource allocation better, and it limits users from running 10
 notebooks and having a lot of resources.

 The other thing I was wondering is could the driver actually be run on
 the mesos cluster? Right now, I have a edge node as an iPython
 server, the drivers all exist on that server, so as I get more and
 more drivers, the box's local resources get depleted with unused
 drivers.  Obviously if I could reuse the drivers per user, on that
 box, that is great first step, but if I could reuse drivers, and run
 them on the cluster, that would be ideal.  looking through the docs I
 was not clear on those options. If anyone could point me in the right
 direction, I would greatly appreciate it!


Cluster mode support for Spark is tracked under [SPARK-5338](
https://issues.apache.org/jira/browse/SPARK-5338). I know Tim Chen is
working on it, so there will be progress soon.

iulian



 John

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com

Re: NPE in Parquet

[Iulian Dragoș]

It’s an array.length, where the array is null. Looking through the code, it
looks like the type converter assumes that FileSystem.globStatus never
returns null, but that is not the case. Digging through the Hadoop
codebase, inside Globber.glob, here’s what I found:

/*
 * When the input pattern looks like just a simple filename, and we
 * can't find it, we return null rather than an empty array.
 * This is a special case which the shell relies on.
 *
 * To be more precise: if there were no results, AND there were no
 * groupings (aka brackets), and no wildcards in the input (aka stars),
 * we return null.
 */

 if ((!sawWildcard)  results.isEmpty() 
(flattenedPatterns.size() = 1)) {
  return null;
}

So, if your file is a concrete filename, without wildcards, you might get a
null back. Seems like a bug in ParquetTypesConverter.

iulian
​

On Tue, Jan 20, 2015 at 5:29 PM, Alessandro Baretta alexbare...@gmail.com
wrote:

 All,

 I strongly suspect this might be caused by a glitch in the communication
 with Google Cloud Storage where my job is writing to, as this NPE exception
 shows up fairly randomly. Any ideas?

 Exception in thread Thread-126 java.lang.NullPointerException
 at
 scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:114)
 at
 scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:114)
 at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:32)
 at
 scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
 at
 scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
 at
 scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:108)
 at
 org.apache.spark.sql.parquet.ParquetTypesConverter$.readMetaData(ParquetTypes.scala:447)
 at
 org.apache.spark.sql.parquet.ParquetTypesConverter$.readSchemaFromFile(ParquetTypes.scala:485)
 at
 org.apache.spark.sql.parquet.ParquetRelation.init(ParquetRelation.scala:65)
 at
 org.apache.spark.sql.SQLContext.parquetFile(SQLContext.scala:190)
 at
 Truven$Stats$anonfun$save_to_parquet$3$anonfun$21$anon$7.run(Truven.scala:957)


 Alex




-- 

--
Iulian Dragos

--
Reactive Apps on the JVM
www.typesafe.com