at I forgot your typical Spark application submits multiple
"jobs", each of which has multiple stages, etc.
So in my case I need to read up more closely about YARN queues
<https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>
since I want to share
t.
On Wed, Apr 5, 2017 at 1:11 PM, Nicholas Chammas <nicholas.cham...@gmail.com
> wrote:
> Hmm, so when I submit an application with `spark-submit`, I need to
> guarantee it resources using YARN queues and not Spark's scheduler pools.
> Is that correct?
>
> When are Spark's
Hmm, so when I submit an application with `spark-submit`, I need to
guarantee it resources using YARN queues and not Spark's scheduler pools.
Is that correct?
When are Spark's scheduler pools relevant/useful in this context?
On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra <m...@clearstorydata.
grrr... s/your/you're/
On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra <m...@clearstorydata.com>
wrote:
> Your mixing up different levels of scheduling. Spark's fair scheduler
> pools are about scheduling Jobs, not Applications; whereas YARN queues with
> Spark are about scheduli
Your mixing up different levels of scheduling. Spark's fair scheduler pools
are about scheduling Jobs, not Applications; whereas YARN queues with Spark
are about scheduling Applications, not Jobs.
On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas <nicholas.cham...@gmail.com>
wrote:
> I
I'm having trouble understanding the difference between Spark fair
scheduler pools
<https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools>
and YARN queues
<https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>.
Do they confli
Hi,
I'm trying to get my head around the different parts of Spark on YARN
architecture with YARN's schedulers and queues as well as Spark's own
schedulers - FAIR and FIFO.
I'd appreciate if you could read how I see things and correct me where
I'm wrong. Thanks!
The default scheduler in YARN
Has anyone implemented Queues using RDDs?
Thank You
