Re: Deploying Jobmanager on k8s as a Deployment
For native K8s integration, the Flink ResourceManager will delete the JobManager K8s deployment as well as the HA data once the job reached a globally terminal state. However, it is indeed a problem for standalone mode since the JobManager will be restarted again even the job has finished. I think the flink-kubernetes-operator could handle this situation by doing the cleanup. Best, Yang Austin Cawley-Edwards 于2022年9月8日周四 06:01写道： > Hey Gil, > > I'm referring to when a pod exits on its own, not when being deleted. > Deployments only support the "Always" restart policy [1]. > > In my understanding, the JM only cleans up HA data when it is shutdown[2], > after which the process will exit which leads to the problem with k8s > Deployment restart policies. > > Best, > Austin > > [1]: > https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template > [2]: > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/rest_api/#cluster > > On Wed, Sep 7, 2022 at 4:43 PM Gil De Grove > wrote: > >> Hello Austin, >> >> I'm not aware of any limitations of deployement not letting pod exit >> (correctly or incorrectly). What do you mean by that exactly? Would it be >> possible for you to point out to piece of documentation that make you think >> that ? >> >> A pod, if correctly setup will be exited when receiving it's sigterm or >> sigkill from the orchestrator. >> So when "deleting" the deployment, the pods are quitted correctly. In the >> case flink did triggered a savepoint before, you can then restart from that >> savepoint. >> Usually, when a pod is not being terminated this means that the SIG is >> not transferred to the correct process. >> >> Hopes this helps. >> >> Regards, >> Gil >> >> >> On Wed, Sep 7, 2022, 21:16 Austin CawleyEdwards >> wrote: >> >>> Cool, thanks! How does it clean up the HA data, if the cluster is never >>> able to shut down (due to the k8s Deployment restriction)? >>> >>> Best, >>> Austin >>> >>> On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra wrote: >>> Hi! The operator supports both Flink native and standalone deployment modes and in both cases the JM is deployed as k8s Deployment. During upgrade Flink/operator deletes the deployment after savepoint and waits for termination before it creates a new one with the updated spec. Cheers, Gyula On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Marco, > > Unfortunately there is no built in k8s API that models an application > mode JM exactly but Deployments should be fine, in general. As Gyula > notes, > where they can be difficult is during application upgrades as Deployments > never let their pods exit, even if successful, so there is no way to stop > the cluster gracefully. > > Is stopping your application with a savepoint and redeploying a > workable solution for image upgrades? In this way a Job could still be > used. > > > @Gyula, how are JMs handled in the operator? Job, Deployment, or > something custom? > > > Best, > Austin > > > > On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra > wrote: > >> You can use deployments of course , the operator and native k8s >> integration does exactly that. >> >> Even then job updates can be tricky so I believe you are much better >> off with the operator. >> >> Gyula >> >> On Sun, 4 Sep 2022 at 11:11, marco andreas >> wrote: >> >>> Hello, >>> >>> Thanks for the response, I will take a look at it. >>> >>> But if we aren't able to use the flink operator due to technical >>> constraints is it possible to deploy the JM as deployment without any >>> consequences that I am not aware of? >>> >>> Sincerely, >>> >>> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a >>> écrit : >>> Hi! You should check out the Flink Kubernetes Operator. I think that covers all your needs . https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ Cheers, Gyula On Sat, 3 Sep 2022 at 13:45, marco andreas < marcoandreas...@gmail.com> wrote: > > We are deploying a flink application cluster on k8S. Following the > official documentation the JM is deployed As a job resource , however > we > are deploying a long running flink job that is not supposed to be > terminated and also we need to update the image of the flink job. > > The problem is that the job is an immutable resource, we > cant update it. > > So I'm wondering if it's possible to use a deployment resource for > the jobmanager and if there will be any side effects or repercussions. > > Thanks, >
Re: Deploying Jobmanager on k8s as a Deployment
Hey Gil, I'm referring to when a pod exits on its own, not when being deleted. Deployments only support the "Always" restart policy [1]. In my understanding, the JM only cleans up HA data when it is shutdown[2], after which the process will exit which leads to the problem with k8s Deployment restart policies. Best, Austin [1]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template [2]: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/rest_api/#cluster On Wed, Sep 7, 2022 at 4:43 PM Gil De Grove wrote: > Hello Austin, > > I'm not aware of any limitations of deployement not letting pod exit > (correctly or incorrectly). What do you mean by that exactly? Would it be > possible for you to point out to piece of documentation that make you think > that ? > > A pod, if correctly setup will be exited when receiving it's sigterm or > sigkill from the orchestrator. > So when "deleting" the deployment, the pods are quitted correctly. In the > case flink did triggered a savepoint before, you can then restart from that > savepoint. > Usually, when a pod is not being terminated this means that the SIG is not > transferred to the correct process. > > Hopes this helps. > > Regards, > Gil > > > On Wed, Sep 7, 2022, 21:16 Austin CawleyEdwards > wrote: > >> Cool, thanks! How does it clean up the HA data, if the cluster is never >> able to shut down (due to the k8s Deployment restriction)? >> >> Best, >> Austin >> >> On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra wrote: >> >>> Hi! >>> >>> The operator supports both Flink native and standalone deployment modes >>> and in both cases the JM is deployed as k8s Deployment. >>> >>> During upgrade Flink/operator deletes the deployment after savepoint and >>> waits for termination before it creates a new one with the updated spec. >>> >>> Cheers, >>> Gyula >>> >>> On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards < >>> austin.caw...@gmail.com> wrote: >>> Hey Marco, Unfortunately there is no built in k8s API that models an application mode JM exactly but Deployments should be fine, in general. As Gyula notes, where they can be difficult is during application upgrades as Deployments never let their pods exit, even if successful, so there is no way to stop the cluster gracefully. Is stopping your application with a savepoint and redeploying a workable solution for image upgrades? In this way a Job could still be used. @Gyula, how are JMs handled in the operator? Job, Deployment, or something custom? Best, Austin On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: > You can use deployments of course , the operator and native k8s > integration does exactly that. > > Even then job updates can be tricky so I believe you are much better > off with the operator. > > Gyula > > On Sun, 4 Sep 2022 at 11:11, marco andreas > wrote: > >> Hello, >> >> Thanks for the response, I will take a look at it. >> >> But if we aren't able to use the flink operator due to technical >> constraints is it possible to deploy the JM as deployment without any >> consequences that I am not aware of? >> >> Sincerely, >> >> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a >> écrit : >> >>> Hi! >>> You should check out the Flink Kubernetes Operator. I think that >>> covers all your needs . >>> >>> >>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >>> >>> Cheers, >>> Gyula >>> >>> On Sat, 3 Sep 2022 at 13:45, marco andreas < >>> marcoandreas...@gmail.com> wrote: >>> We are deploying a flink application cluster on k8S. Following the official documentation the JM is deployed As a job resource , however we are deploying a long running flink job that is not supposed to be terminated and also we need to update the image of the flink job. The problem is that the job is an immutable resource, we cant update it. So I'm wondering if it's possible to use a deployment resource for the jobmanager and if there will be any side effects or repercussions. Thanks, >>> Gil De Grove >
Re: Deploying Jobmanager on k8s as a Deployment
Hello Austin, I'm not aware of any limitations of deployement not letting pod exit (correctly or incorrectly). What do you mean by that exactly? Would it be possible for you to point out to piece of documentation that make you think that ? A pod, if correctly setup will be exited when receiving it's sigterm or sigkill from the orchestrator. So when "deleting" the deployment, the pods are quitted correctly. In the case flink did triggered a savepoint before, you can then restart from that savepoint. Usually, when a pod is not being terminated this means that the SIG is not transferred to the correct process. Hopes this helps. Regards, Gil On Wed, Sep 7, 2022, 21:16 Austin CawleyEdwards wrote: > Cool, thanks! How does it clean up the HA data, if the cluster is never > able to shut down (due to the k8s Deployment restriction)? > > Best, > Austin > > On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra wrote: > >> Hi! >> >> The operator supports both Flink native and standalone deployment modes >> and in both cases the JM is deployed as k8s Deployment. >> >> During upgrade Flink/operator deletes the deployment after savepoint and >> waits for termination before it creates a new one with the updated spec. >> >> Cheers, >> Gyula >> >> On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> Hey Marco, >>> >>> Unfortunately there is no built in k8s API that models an application >>> mode JM exactly but Deployments should be fine, in general. As Gyula notes, >>> where they can be difficult is during application upgrades as Deployments >>> never let their pods exit, even if successful, so there is no way to stop >>> the cluster gracefully. >>> >>> Is stopping your application with a savepoint and redeploying a workable >>> solution for image upgrades? In this way a Job could still be used. >>> >>> >>> @Gyula, how are JMs handled in the operator? Job, Deployment, or >>> something custom? >>> >>> >>> Best, >>> Austin >>> >>> >>> >>> On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: >>> You can use deployments of course , the operator and native k8s integration does exactly that. Even then job updates can be tricky so I believe you are much better off with the operator. Gyula On Sun, 4 Sep 2022 at 11:11, marco andreas wrote: > Hello, > > Thanks for the response, I will take a look at it. > > But if we aren't able to use the flink operator due to technical > constraints is it possible to deploy the JM as deployment without any > consequences that I am not aware of? > > Sincerely, > > Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a > écrit : > >> Hi! >> You should check out the Flink Kubernetes Operator. I think that >> covers all your needs . >> >> >> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >> >> Cheers, >> Gyula >> >> On Sat, 3 Sep 2022 at 13:45, marco andreas >> wrote: >> >>> >>> We are deploying a flink application cluster on k8S. Following the >>> official documentation the JM is deployed As a job resource , however we >>> are deploying a long running flink job that is not supposed to be >>> terminated and also we need to update the image of the flink job. >>> >>> The problem is that the job is an immutable resource, we >>> cant update it. >>> >>> So I'm wondering if it's possible to use a deployment resource for >>> the jobmanager and if there will be any side effects or repercussions. >>> >>> Thanks, >>> >> Gil De Grove
Re: Deploying Jobmanager on k8s as a Deployment
Cool, thanks! How does it clean up the HA data, if the cluster is never able to shut down (due to the k8s Deployment restriction)? Best, Austin On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra wrote: > Hi! > > The operator supports both Flink native and standalone deployment modes > and in both cases the JM is deployed as k8s Deployment. > > During upgrade Flink/operator deletes the deployment after savepoint and > waits for termination before it creates a new one with the updated spec. > > Cheers, > Gyula > > On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hey Marco, >> >> Unfortunately there is no built in k8s API that models an application >> mode JM exactly but Deployments should be fine, in general. As Gyula notes, >> where they can be difficult is during application upgrades as Deployments >> never let their pods exit, even if successful, so there is no way to stop >> the cluster gracefully. >> >> Is stopping your application with a savepoint and redeploying a workable >> solution for image upgrades? In this way a Job could still be used. >> >> >> @Gyula, how are JMs handled in the operator? Job, Deployment, or >> something custom? >> >> >> Best, >> Austin >> >> >> >> On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: >> >>> You can use deployments of course , the operator and native k8s >>> integration does exactly that. >>> >>> Even then job updates can be tricky so I believe you are much better off >>> with the operator. >>> >>> Gyula >>> >>> On Sun, 4 Sep 2022 at 11:11, marco andreas >>> wrote: >>> Hello, Thanks for the response, I will take a look at it. But if we aren't able to use the flink operator due to technical constraints is it possible to deploy the JM as deployment without any consequences that I am not aware of? Sincerely, Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a écrit : > Hi! > You should check out the Flink Kubernetes Operator. I think that > covers all your needs . > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ > > Cheers, > Gyula > > On Sat, 3 Sep 2022 at 13:45, marco andreas > wrote: > >> >> We are deploying a flink application cluster on k8S. Following the >> official documentation the JM is deployed As a job resource , however we >> are deploying a long running flink job that is not supposed to be >> terminated and also we need to update the image of the flink job. >> >> The problem is that the job is an immutable resource, we cant update >> it. >> >> So I'm wondering if it's possible to use a deployment resource for >> the jobmanager and if there will be any side effects or repercussions. >> >> Thanks, >> >
Re: Deploying Jobmanager on k8s as a Deployment
Hi! The operator supports both Flink native and standalone deployment modes and in both cases the JM is deployed as k8s Deployment. During upgrade Flink/operator deletes the deployment after savepoint and waits for termination before it creates a new one with the updated spec. Cheers, Gyula On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards wrote: > Hey Marco, > > Unfortunately there is no built in k8s API that models an application mode > JM exactly but Deployments should be fine, in general. As Gyula notes, > where they can be difficult is during application upgrades as Deployments > never let their pods exit, even if successful, so there is no way to stop > the cluster gracefully. > > Is stopping your application with a savepoint and redeploying a workable > solution for image upgrades? In this way a Job could still be used. > > > @Gyula, how are JMs handled in the operator? Job, Deployment, or something > custom? > > > Best, > Austin > > > > On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: > >> You can use deployments of course , the operator and native k8s >> integration does exactly that. >> >> Even then job updates can be tricky so I believe you are much better off >> with the operator. >> >> Gyula >> >> On Sun, 4 Sep 2022 at 11:11, marco andreas >> wrote: >> >>> Hello, >>> >>> Thanks for the response, I will take a look at it. >>> >>> But if we aren't able to use the flink operator due to technical >>> constraints is it possible to deploy the JM as deployment without any >>> consequences that I am not aware of? >>> >>> Sincerely, >>> >>> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a >>> écrit : >>> Hi! You should check out the Flink Kubernetes Operator. I think that covers all your needs . https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ Cheers, Gyula On Sat, 3 Sep 2022 at 13:45, marco andreas wrote: > > We are deploying a flink application cluster on k8S. Following the > official documentation the JM is deployed As a job resource , however we > are deploying a long running flink job that is not supposed to be > terminated and also we need to update the image of the flink job. > > The problem is that the job is an immutable resource, we cant update > it. > > So I'm wondering if it's possible to use a deployment resource for the > jobmanager and if there will be any side effects or repercussions. > > Thanks, >
Re: Deploying Jobmanager on k8s as a Deployment
Hey Marco, Unfortunately there is no built in k8s API that models an application mode JM exactly but Deployments should be fine, in general. As Gyula notes, where they can be difficult is during application upgrades as Deployments never let their pods exit, even if successful, so there is no way to stop the cluster gracefully. Is stopping your application with a savepoint and redeploying a workable solution for image upgrades? In this way a Job could still be used. @Gyula, how are JMs handled in the operator? Job, Deployment, or something custom? Best, Austin On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra wrote: > You can use deployments of course , the operator and native k8s > integration does exactly that. > > Even then job updates can be tricky so I believe you are much better off > with the operator. > > Gyula > > On Sun, 4 Sep 2022 at 11:11, marco andreas > wrote: > >> Hello, >> >> Thanks for the response, I will take a look at it. >> >> But if we aren't able to use the flink operator due to technical >> constraints is it possible to deploy the JM as deployment without any >> consequences that I am not aware of? >> >> Sincerely, >> >> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a écrit : >> >>> Hi! >>> You should check out the Flink Kubernetes Operator. I think that covers >>> all your needs . >>> >>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >>> >>> Cheers, >>> Gyula >>> >>> On Sat, 3 Sep 2022 at 13:45, marco andreas >>> wrote: >>> We are deploying a flink application cluster on k8S. Following the official documentation the JM is deployed As a job resource , however we are deploying a long running flink job that is not supposed to be terminated and also we need to update the image of the flink job. The problem is that the job is an immutable resource, we cant update it. So I'm wondering if it's possible to use a deployment resource for the jobmanager and if there will be any side effects or repercussions. Thanks, >>>
Re: Deploying Jobmanager on k8s as a Deployment
You can use deployments of course , the operator and native k8s integration does exactly that. Even then job updates can be tricky so I believe you are much better off with the operator. Gyula On Sun, 4 Sep 2022 at 11:11, marco andreas wrote: > Hello, > > Thanks for the response, I will take a look at it. > > But if we aren't able to use the flink operator due to technical > constraints is it possible to deploy the JM as deployment without any > consequences that I am not aware of? > > Sincerely, > > Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a écrit : > >> Hi! >> You should check out the Flink Kubernetes Operator. I think that covers >> all your needs . >> >> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >> >> Cheers, >> Gyula >> >> On Sat, 3 Sep 2022 at 13:45, marco andreas >> wrote: >> >>> >>> We are deploying a flink application cluster on k8S. Following the >>> official documentation the JM is deployed As a job resource , however we >>> are deploying a long running flink job that is not supposed to be >>> terminated and also we need to update the image of the flink job. >>> >>> The problem is that the job is an immutable resource, we cant update it. >>> >>> So I'm wondering if it's possible to use a deployment resource for the >>> jobmanager and if there will be any side effects or repercussions. >>> >>> Thanks, >>> >>
Re: Deploying Jobmanager on k8s as a Deployment
Hello, Thanks for the response, I will take a look at it. But if we aren't able to use the flink operator due to technical constraints is it possible to deploy the JM as deployment without any consequences that I am not aware of? Sincerely, Le sam. 3 sept. 2022 à 23:27, Gyula Fóra a écrit : > Hi! > You should check out the Flink Kubernetes Operator. I think that covers > all your needs . > > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ > > Cheers, > Gyula > > On Sat, 3 Sep 2022 at 13:45, marco andreas > wrote: > >> >> We are deploying a flink application cluster on k8S. Following the >> official documentation the JM is deployed As a job resource , however we >> are deploying a long running flink job that is not supposed to be >> terminated and also we need to update the image of the flink job. >> >> The problem is that the job is an immutable resource, we cant update it. >> >> So I'm wondering if it's possible to use a deployment resource for the >> jobmanager and if there will be any side effects or repercussions. >> >> Thanks, >> >
Re: Deploying Jobmanager on k8s as a Deployment
Hi! You should check out the Flink Kubernetes Operator. I think that covers all your needs . https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ Cheers, Gyula On Sat, 3 Sep 2022 at 13:45, marco andreas wrote: > > We are deploying a flink application cluster on k8S. Following the > official documentation the JM is deployed As a job resource , however we > are deploying a long running flink job that is not supposed to be > terminated and also we need to update the image of the flink job. > > The problem is that the job is an immutable resource, we cant update it. > > So I'm wondering if it's possible to use a deployment resource for the > jobmanager and if there will be any side effects or repercussions. > > Thanks, >
Deploying Jobmanager on k8s as a Deployment
We are deploying a flink application cluster on k8S. Following the official documentation the JM is deployed As a job resource , however we are deploying a long running flink job that is not supposed to be terminated and also we need to update the image of the flink job. The problem is that the job is an immutable resource, we cant update it. So I'm wondering if it's possible to use a deployment resource for the jobmanager and if there will be any side effects or repercussions. Thanks,
