Vijay, That is my understanding as well: the HA solution only solves the problem up to the point all job managers fail/restart at the same time. That's where my original concern was.
But to Aleksandar and Yun's point, running in HA with 2 or 3 Job Managers per cluster--as long as they are all deployed to separate GKE nodes--would provide a very high uptime/low failure rate, at least on paper. It's a promising enough option that we're going to run in HA for a month or two and monitor results before we put in any extra work to customize the savepoint start-up behavior. On Fri, Sep 27, 2019 at 2:24 AM Vijay Bhaskar <bhaskar.eba...@gmail.com> wrote: > I don't think HA will help to recover from cluster crash, for that we > should take periodic savepoint right? Please correct me in case i am wrong > > Regards > Bhaskar > > On Fri, Sep 27, 2019 at 11:48 AM Vijay Bhaskar <bhaskar.eba...@gmail.com> > wrote: > >> Suppose my cluster got crashed and need to bring up the entire cluster >> back? Does HA still helps to run the cluster from latest save point? >> >> Regards >> Bhaskar >> >> On Thu, Sep 26, 2019 at 7:44 PM Sean Hester <sean.hes...@bettercloud.com> >> wrote: >> >>> thanks to everyone for all the replies. >>> >>> i think the original concern here with "just" relying on the HA option >>> is that there are some disaster recovery and data center migration use >>> cases where the continuity of the job managers is difficult to preserve. >>> but those are admittedly very edgy use cases. i think it's definitely worth >>> reviewing the SLAs with our site reliability engineers to see how likely it >>> would be to completely lose all job managers under an HA configuration. >>> that small a risk might be acceptable/preferable to a one-off solution. >>> >>> @Aleksander, would love to learn more about Zookeeper-less HA. i think i >>> spotted a thread somewhere between Till and someone (perhaps you) about >>> that. feel free to DM me. >>> >>> thanks again to everyone! >>> >>> On Thu, Sep 26, 2019 at 7:32 AM Yang Wang <danrtsey...@gmail.com> wrote: >>> >>>> Hi, Aleksandar >>>> >>>> Savepoint option in standalone job cluster is optional. If you want to >>>> always recover >>>> from the latest checkpoint, just as Aleksandar and Yun Tang said you >>>> could use the >>>> high-availability configuration. Make sure the cluster-id is not >>>> changed, i think the job >>>> could recover both at exceptionally crash and restart by expectation. >>>> >>>> @Aleksandar Mastilovic <amastilo...@sightmachine.com>, we are also >>>> have an zookeeper-less high-availability implementation[1]. >>>> Maybe we could have some discussion and contribute this useful feature >>>> to the community. >>>> >>>> [1]. >>>> https://docs.google.com/document/d/1Z-VdJlPPEQoWT1WLm5woM4y0bFep4FrgdJ9ipQuRv8g/edit >>>> >>>> Best, >>>> Yang >>>> >>>> Aleksandar Mastilovic <amastilo...@sightmachine.com> 于2019年9月26日周四 >>>> 上午4:11写道： >>>> >>>>> Would you guys (Flink devs) be interested in our solution for >>>>> zookeeper-less HA? I could ask the managers how they feel about >>>>> open-sourcing the improvement. >>>>> >>>>> On Sep 25, 2019, at 11:49 AM, Yun Tang <myas...@live.com> wrote: >>>>> >>>>> As Aleksandar said, k8s with HA configuration could solve your >>>>> problem. There already have some discussion about how to implement such HA >>>>> in k8s if we don't have a zookeeper service: FLINK-11105 [1] and >>>>> FLINK-12884 [2]. Currently, you might only have to choose zookeeper as >>>>> high-availability service. >>>>> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-11105 >>>>> [2] https://issues.apache.org/jira/browse/FLINK-12884 >>>>> >>>>> Best >>>>> Yun Tang >>>>> ------------------------------ >>>>> *From:* Aleksandar Mastilovic <amastilo...@sightmachine.com> >>>>> *Sent:* Thursday, September 26, 2019 1:57 >>>>> *To:* Sean Hester <sean.hes...@bettercloud.com> >>>>> *Cc:* Hao Sun <ha...@zendesk.com>; Yuval Itzchakov <yuva...@gmail.com>; >>>>> user <user@flink.apache.org> >>>>> *Subject:* Re: Challenges Deploying Flink With Savepoints On >>>>> Kubernetes >>>>> >>>>> Can’t you simply use JobManager in HA mode? It would pick up where it >>>>> left off if you don’t provide a Savepoint. >>>>> >>>>> On Sep 25, 2019, at 6:07 AM, Sean Hester <sean.hes...@bettercloud.com> >>>>> wrote: >>>>> >>>>> thanks for all replies! i'll definitely take a look at the Flink k8s >>>>> Operator project. >>>>> >>>>> i'll try to restate the issue to clarify. this issue is specific to >>>>> starting a job from a savepoint in job-cluster mode. in these cases the >>>>> Job >>>>> Manager container is configured to run a single Flink job at start-up. the >>>>> savepoint needs to be provided as an argument to the entrypoint. the Flink >>>>> documentation for this approach is here: >>>>> >>>>> >>>>> https://github.com/apache/flink/tree/master/flink-container/kubernetes#resuming-from-a-savepoint >>>>> >>>>> the issue is that taking this approach means that the job will >>>>> *always* start from the savepoint provided as the start argument in >>>>> the Kubernetes YAML. this includes unplanned restarts of the job manager, >>>>> but we'd really prefer any *unplanned* restarts resume for the most >>>>> recent checkpoint instead of restarting from the configured savepoint. so >>>>> in a sense we want the savepoint argument to be transient, only being used >>>>> during the initial deployment, but this runs counter to the design of >>>>> Kubernetes which always wants to restore a deployment to the "goal state" >>>>> as defined in the YAML. >>>>> >>>>> i hope this helps. if you want more details please let me know, and >>>>> thanks again for your time. >>>>> >>>>> >>>>> On Tue, Sep 24, 2019 at 1:09 PM Hao Sun <ha...@zendesk.com> wrote: >>>>> >>>>> I think I overlooked it. Good point. I am using Redis to save the path >>>>> to my savepoint, I might be able to set a TTL to avoid such issue. >>>>> >>>>> Hao Sun >>>>> >>>>> >>>>> On Tue, Sep 24, 2019 at 9:54 AM Yuval Itzchakov <yuva...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi Hao, >>>>> >>>>> I think he's exactly talking about the usecase where the JM/TM restart >>>>> and they come back up from the latest savepoint which might be stale by >>>>> that time. >>>>> >>>>> On Tue, 24 Sep 2019, 19:24 Hao Sun, <ha...@zendesk.com> wrote: >>>>> >>>>> We always make a savepoint before we shutdown the job-cluster. So the >>>>> savepoint is always the latest. When we fix a bug or change the job graph, >>>>> it can resume well. >>>>> We only use checkpoints for unplanned downtime, e.g. K8S killed JM/TM, >>>>> uncaught exception, etc. >>>>> >>>>> Maybe I do not understand your use case well, I do not see a need to >>>>> start from checkpoint after a bug fix. >>>>> From what I know, currently you can use checkpoint as a savepoint as >>>>> well >>>>> >>>>> Hao Sun >>>>> >>>>> >>>>> On Tue, Sep 24, 2019 at 7:48 AM Yuval Itzchakov <yuva...@gmail.com> >>>>> wrote: >>>>> >>>>> AFAIK there's currently nothing implemented to solve this problem, but >>>>> working on a possible fix can be implemented on top of >>>>> https://github.com/lyft/flinkk8soperator which already has a pretty >>>>> fancy state machine for rolling upgrades. I'd love to be involved as this >>>>> is an issue I've been thinking about as well. >>>>> >>>>> Yuval >>>>> >>>>> On Tue, Sep 24, 2019 at 5:02 PM Sean Hester < >>>>> sean.hes...@bettercloud.com> wrote: >>>>> >>>>> hi all--we've run into a gap (knowledge? design? tbd?) for our use >>>>> cases when deploying Flink jobs to start from savepoints using the >>>>> job-cluster mode in Kubernetes. >>>>> >>>>> we're running a ~15 different jobs, all in job-cluster mode, using a >>>>> mix of Flink 1.8.1 and 1.9.0, under GKE (Google Kubernetes Engine). these >>>>> are all long-running streaming jobs, all essentially acting as >>>>> microservices. we're using Helm charts to configure all of our >>>>> deployments. >>>>> >>>>> we have a number of use cases where we want to restart jobs from a >>>>> savepoint to replay recent events, i.e. when we've enhanced the job logic >>>>> or fixed a bug. but after the deployment we want to have the job resume >>>>> it's "long-running" behavior, where any unplanned restarts resume from the >>>>> latest checkpoint. >>>>> >>>>> the issue we run into is that any obvious/standard/idiomatic >>>>> Kubernetes deployment includes the savepoint argument in the >>>>> configuration. >>>>> if the Job Manager container(s) have an unplanned restart, when they come >>>>> back up they will start from the savepoint instead of resuming from the >>>>> latest checkpoint. everything is working as configured, but that's not >>>>> exactly what we want. we want the savepoint argument to be transient >>>>> somehow (only used during the initial deployment), but Kubernetes doesn't >>>>> really support the concept of transient configuration. >>>>> >>>>> i can see a couple of potential solutions that either involve custom >>>>> code in the jobs or custom logic in the container (i.e. a custom >>>>> entrypoint >>>>> script that records that the configured savepoint has already been used in >>>>> a file on a persistent volume or GCS, and potentially when/why/by which >>>>> deployment). but these seem like unexpected and hacky solutions. before we >>>>> head down that road i wanted to ask: >>>>> >>>>> - is this is already a solved problem that i've missed? >>>>> - is this issue already on the community's radar? >>>>> >>>>> thanks in advance! >>>>> >>>>> -- >>>>> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865 >>>>> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305 >>>>> <http://www.bettercloud.com/> <http://www.bettercloud.com/> >>>>> *Altitude 2019 in San Francisco | Sept. 23 - 25* >>>>> It’s not just an IT conference, it’s “a complete learning and >>>>> networking experience” >>>>> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Yuval Itzchakov. >>>>> >>>>> >>>>> >>>>> -- >>>>> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865 >>>>> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305 >>>>> <http://www.bettercloud.com/> <http://www.bettercloud.com/> >>>>> *Altitude 2019 in San Francisco | Sept. 23 - 25* >>>>> It’s not just an IT conference, it’s “a complete learning and >>>>> networking experience” >>>>> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude> >>>>> >>>>> >>>>> >>> >>> -- >>> *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865 >>> 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305 >>> <http://www.bettercloud.com> <http://www.bettercloud.com> >>> *Altitude 2019 in San Francisco | Sept. 23 - 25* >>> It’s not just an IT conference, it’s “a complete learning and networking >>> experience” >>> <https://altitude.bettercloud.com/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-altitude> >>> >>> -- *Sean Hester* | Senior Staff Software Engineer | m. 404-828-0865 3525 Piedmont Rd. NE, Building 6, Suite 500, Atlanta, GA 30305 <http://www.bettercloud.com> <http://www.bettercloud.com> *Introducing the BetterCloud Integration Center * Automate actions across every app and own SaaSOps <https://www.bettercloud.com/integrations-webinar/?utm_source=gmail&utm_medium=signature&utm_campaign=2019-integration-center>
