Re: spark shuffle service on yarn

2016-03-21 Thread Marcelo Vanzin
;ko...@tresata.com> wrote: > spark on yarn is nice because i can bring my own spark. i am worried that > the shuffle service forces me to use some "sanctioned" spark version that is > officially "installed" on the cluster. > > so... can i safely install the

Re: spark shuffle service on yarn

2016-03-19 Thread Steve Loughran
quot; on the cluster. so... can i safely install the spark 1.3 shuffle service on yarn and use it with other 1.x versions of spark? That's an interesting question. I don't know the answer there. Normally I'd just say "use the latest, hope they got compatibility right". Spark

spark shuffle service on yarn

2016-03-18 Thread Koert Kuipers
spark on yarn is nice because i can bring my own spark. i am worried that the shuffle service forces me to use some "sanctioned" spark version that is officially "installed" on the cluster. so... can i safely install the spark 1.3 shuffle service on yarn and use it wit