Hi Fabian,

This make sense now.

I would like to avoid zookeeper if possible. Is there any way to avoid this
to achieve HA ?

I see that DataStax Enterprise achieve this availability for Spark Master
without using Zookeeper.
https://academy.datastax.com/demos/how-spark-master-high-availability-works-dse

Is this possible to achieve in Flink also ?


Cheers

On Wed, Nov 25, 2015 at 4:11 PM, Fabian Hueske <[email protected]> wrote:

> YARN is not a replacement for Zookeeper. Zookeeper is mandatory to run
> Flink in high-availability mode and takes care of leader (JobManager)
> election and meta-data persistance.
>
> With YARN, Flink can automatically start new Taskmanagers (and
> JobManagers) to compensate for failures. In cluster mode, you need stand-by
> TMs and JMs and manually take care that these are "filled-up" again in case
> of a failure.
>
> 2015-11-25 10:06 GMT+01:00 Welly Tambunan <[email protected]>:
>
>> Hi Fabian,
>>
>> Interesting !
>>
>> However YARN is still tightly couple to HDFS, is that seems wasteful to
>> use only YARN without Hadoop ?
>>
>> Currently we are using Cassandra and CFS ( cass file system )
>>
>>
>> Cheers
>>
>> On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <[email protected]> wrote:
>>
>>> A strong argument for YARN mode can be the isolation of multiple users
>>> and jobs. You can easily start a new Flink cluster for each job or user.
>>> However, this comes at the price of resource (memory) fragmentation. YARN
>>> mode does not use memory as effective as cluster mode.
>>>
>>> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <[email protected]>:
>>>
>>>> > On 25 Nov 2015, at 02:35, Welly Tambunan <[email protected]> wrote:
>>>> >
>>>> > Hi All,
>>>> >
>>>> > I would like to know if there any feature differences between using
>>>> Standalone Cluster vs YARN ?
>>>> >
>>>> > Until now we are using Standalone cluster for our jobs.
>>>> > Is there any added value for using YARN ?
>>>> >
>>>> > We don't have any hadoop infrastructure in place right now but we can
>>>> provide that if there's some value to that.
>>>>
>>>> There are no features, which only work on YARN or in standalone
>>>> clusters. YARN mode is essentially starting a standalone cluster in YARN
>>>> containers.
>>>>
>>>> In failure cases I find YARN more convenient, because it takes care of
>>>> restarting failed task manager processes/containers for you.
>>>>
>>>> – Ufuk
>>>>
>>>>
>>>
>>
>>
>> --
>> Welly Tambunan
>> Triplelands
>>
>> http://weltam.wordpress.com
>> http://www.triplelands.com <http://www.triplelands.com/blog/>
>>
>
>


-- 
Welly Tambunan
Triplelands

http://weltam.wordpress.com
http://www.triplelands.com <http://www.triplelands.com/blog/>

Reply via email to