Hi Andreas, Yes, seems I can't avoid Zookeeper right now. It would be really nice if we can achieve HA via gossip protocol like Cassandra/Spark DSE does ?
Is this possible ? Cheers On Wed, Nov 25, 2015 at 4:12 PM, Andreas Fritzler < [email protected]> wrote: > Hi Welly, > > you will need Zookeeper if you want to setup the standalone cluster in HA > mode. > http://spark.apache.org/docs/latest/spark-standalone.html#high-availability > > In the YARN case you probably have already Zookeeper in place if you are > running YARN in HA mode. > > Regards, > Andreas > > On Wed, Nov 25, 2015 at 10:02 AM, Welly Tambunan <[email protected]> > wrote: > >> Hi Ufuk >> >> >In failure cases I find YARN more convenient, because it takes care of >> restarting failed task manager processes/containers for you. >> >> So this mean that we don't need zookeeper ? >> >> >> Cheers >> >> On Wed, Nov 25, 2015 at 3:46 PM, Ufuk Celebi <[email protected]> wrote: >> >>> > On 25 Nov 2015, at 02:35, Welly Tambunan <[email protected]> wrote: >>> > >>> > Hi All, >>> > >>> > I would like to know if there any feature differences between using >>> Standalone Cluster vs YARN ? >>> > >>> > Until now we are using Standalone cluster for our jobs. >>> > Is there any added value for using YARN ? >>> > >>> > We don't have any hadoop infrastructure in place right now but we can >>> provide that if there's some value to that. >>> >>> There are no features, which only work on YARN or in standalone >>> clusters. YARN mode is essentially starting a standalone cluster in YARN >>> containers. >>> >>> In failure cases I find YARN more convenient, because it takes care of >>> restarting failed task manager processes/containers for you. >>> >>> – Ufuk >>> >>> >> >> >> -- >> Welly Tambunan >> Triplelands >> >> http://weltam.wordpress.com >> http://www.triplelands.com <http://www.triplelands.com/blog/> >> > > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
