I don't have spark expertise, but here're some points I'm thinking about. IIRC, spark standalone do not support Kerberos. And the benefit of deploying spark on yarn should be that you don't need to maintain packages by your own on hundreds of node cluster. Not sure if there're downsides. Just want to add some points:)
2015-02-20 9:49 GMT+08:00 Konstantin Boudnik <[email protected]>: > They way we're deploying spark is in the standalone mode - I never seen any > value in using YARN for that, but I guess it's just me. > > HDFS use comes with no hassle, AFAIR, the way we setup it up. But my > knowledge > might be a bit outdated... > > Cos > > On Thu, Feb 19, 2015 at 08:45PM, jay vyas wrote: > > hi folks. > > > > is anyone planning to use spark on yarn or spark w/ hdfs in bigtop? I > > havent tried either... > > > > - anyone using spark <-> HDFS in bigtop ? Do we need to update any spark > > configs to do so ? > > - we want spark to run on yarn ? standalone ? > > > > im spinning some VMs up now, ill let folks know if it works. > > -- > > jay vyas >
