Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang, Please take a look at the PR when you are free. https://github.com/apache/flink/pull/15602<https://urldefense.com/v3/__https:/github.com/apache/flink/pull/15602__;!!GqivPVa7Brio!PvSWfGFchHWHUDIT-tZWHhlT14WLLgt0o8bsKbxezhdomLWIwR-rHuUJl1Y99Po$> Should be a simple change. Thanks! Best, Fuyao From: Fuyao Li Date: Tuesday, April 13, 2021 at 19:10 To: Yang Wang Cc: user Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hello Yang, I also created a PR for this issue. Please take a look. Refer to https://github.com/apache/flink/pull/15602<https://urldefense.com/v3/__https:/github.com/apache/flink/pull/15602__;!!GqivPVa7Brio!PvSWfGFchHWHUDIT-tZWHhlT14WLLgt0o8bsKbxezhdomLWIwR-rHuUJl1Y99Po$> Thanks, Fuyao From: Fuyao Li Date: Tuesday, April 13, 2021 at 18:23 To: Yang Wang Cc: user Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hello Yang, I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/FLINK-22264__;!!GqivPVa7Brio!JAL6kfYxsKxUWjI6RfOyzKgsII1Qvg0NKUSVfiwMVonKnBNQXm7kTHJekM03uUU$> I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue? Thanks. Best, Fuyao From: Yang Wang Date: Tuesday, April 13, 2021 at 03:01 To: Fuyao Li Cc: user , Yan Wang Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode I think it makes sense to have such a simple fix. Could you please create a ticket and attach a PR? Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 下午2:24写道: Hello Yang, It is very kind of you to give such a detailed explanation! Thanks for clarification. For the small document fix I mentioned, what do you think? Best, Fuyao From: Yang Wang mailto:danrtsey...@gmail.com>> Date: Monday, April 12, 2021 at 23:03 To: Fuyao Li mailto:fuyao...@oracle.com>> Cc: user mailto:user@flink.apache.org>>, Yan Wang mailto:y.yan.w.w...@oracle.com>> Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hi Fuyao, Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode. In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11 and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in Kubernetes environment. [1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$> Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 上午8:10写道: Hello Community, Yang, I noticed a conflict in the document for per-job mode support for Kubernetes. In the doc here [1], it mentions in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only. It implies per job mode is supported in Kubernetes. However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes. This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix. I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/nativ
Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang, I also created a PR for this issue. Please take a look. Refer to https://github.com/apache/flink/pull/15602 Thanks, Fuyao From: Fuyao Li Date: Tuesday, April 13, 2021 at 18:23 To: Yang Wang Cc: user Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hello Yang, I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/FLINK-22264__;!!GqivPVa7Brio!JAL6kfYxsKxUWjI6RfOyzKgsII1Qvg0NKUSVfiwMVonKnBNQXm7kTHJekM03uUU$> I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue? Thanks. Best, Fuyao From: Yang Wang Date: Tuesday, April 13, 2021 at 03:01 To: Fuyao Li Cc: user , Yan Wang Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode I think it makes sense to have such a simple fix. Could you please create a ticket and attach a PR? Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 下午2:24写道: Hello Yang, It is very kind of you to give such a detailed explanation! Thanks for clarification. For the small document fix I mentioned, what do you think? Best, Fuyao From: Yang Wang mailto:danrtsey...@gmail.com>> Date: Monday, April 12, 2021 at 23:03 To: Fuyao Li mailto:fuyao...@oracle.com>> Cc: user mailto:user@flink.apache.org>>, Yan Wang mailto:y.yan.w.w...@oracle.com>> Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hi Fuyao, Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode. In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11 and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in Kubernetes environment. [1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$> Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 上午8:10写道: Hello Community, Yang, I noticed a conflict in the document for per-job mode support for Kubernetes. In the doc here [1], it mentions in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only. It implies per job mode is supported in Kubernetes. However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes. This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix. I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$> [4] https://www.youtube.com/watch?v=pdFPr_VOWTU=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$> Best, Fuyao
Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang, I tried to create a ticket https://issues.apache.org/jira/browse/FLINK-22264 I just registered as a user and I can’t find a place to assign the task to myself… Any idea on this jira issue? Thanks. Best, Fuyao From: Yang Wang Date: Tuesday, April 13, 2021 at 03:01 To: Fuyao Li Cc: user , Yan Wang Subject: Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode I think it makes sense to have such a simple fix. Could you please create a ticket and attach a PR? Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 下午2:24写道: Hello Yang, It is very kind of you to give such a detailed explanation! Thanks for clarification. For the small document fix I mentioned, what do you think? Best, Fuyao From: Yang Wang mailto:danrtsey...@gmail.com>> Date: Monday, April 12, 2021 at 23:03 To: Fuyao Li mailto:fuyao...@oracle.com>> Cc: user mailto:user@flink.apache.org>>, Yan Wang mailto:y.yan.w.w...@oracle.com>> Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hi Fuyao, Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode. In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11 and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in Kubernetes environment. [1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$> Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 上午8:10写道: Hello Community, Yang, I noticed a conflict in the document for per-job mode support for Kubernetes. In the doc here [1], it mentions in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only. It implies per job mode is supported in Kubernetes. However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes. This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix. I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$> [4] https://www.youtube.com/watch?v=pdFPr_VOWTU=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$> Best, Fuyao
Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
I think it makes sense to have such a simple fix. Could you please create a ticket and attach a PR? Best, Yang Fuyao Li 于2021年4月13日周二 下午2:24写道: > Hello Yang, > > > > It is very kind of you to give such a detailed explanation! Thanks for > clarification. > > > > For the small document fix I mentioned, what do you think? > > > > Best, > > Fuyao > > > > *From: *Yang Wang > *Date: *Monday, April 12, 2021 at 23:03 > *To: *Fuyao Li > *Cc: *user , Yan Wang > *Subject: *[External] : Re: Conflict in the document - About native > Kubernetes per job mode > > Hi Fuyao, > > > > Currently, Flink only supports perjob mode for Yarn. The standalone job > cluster has been replaced with standalone application mode > > after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration > do not support per-job mode. > > > > In your attached video, it is a PoC implementation for presentation. We > have introduced the Kubernetes application mode in release 1.11 > > and agree to not have a per-job cluster support. The only reason is that > it is not very convenient to ship the job graphs, user jars, artifacts in > > Kubernetes environment. > > > > [1]. > https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode > <https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$> > > > > Best, > > Yang > > > > Fuyao Li 于2021年4月13日周二 上午8:10写道: > > Hello Community, Yang, > > > > I noticed a conflict in the document for per-job mode support for > Kubernetes. > > In the doc here [1], it mentions > > in a Flink Job Cluster, the available cluster manager (like YARN or > Kubernetes) is used to spin up a cluster for each submitted job and this > cluster is available to that job only. > > It implies per job mode is supported in Kubernetes. > > > > However, in the docs [2] and [3], it clearly points out per-job mode is > not supported in Kubernetes. > > > > *This is a conflict statement and is kind of misleading. If needed, I can > create an MR to delete the statement in [1] for Kubernetes.. It is a small > fix. * > > > > I also noticed another thing in the video [4] at 25:08. Yang, you are > executing a command with -e kubernetes-per-job flag. I tried and found such > command is not supported in Flink distribution at all. I noticed the > version you are using is 1.11-snapshot during the demo. Are you modifying > the source code and generated an internal version of Flink….? > > > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster > <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$> > > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode > <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$> > > [3] > https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode > <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$> > > [4] https://www.youtube.com/watch?v=pdFPr_VOWTU=833s > <https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$> > > > > Best, > > Fuyao > >
Re: [External] : Re: Conflict in the document - About native Kubernetes per job mode
Hello Yang, It is very kind of you to give such a detailed explanation! Thanks for clarification. For the small document fix I mentioned, what do you think? Best, Fuyao From: Yang Wang Date: Monday, April 12, 2021 at 23:03 To: Fuyao Li Cc: user , Yan Wang Subject: [External] : Re: Conflict in the document - About native Kubernetes per job mode Hi Fuyao, Currently, Flink only supports perjob mode for Yarn. The standalone job cluster has been replaced with standalone application mode after FLIP-85[1]. Both standalone Flink on K8s and native K8s integration do not support per-job mode. In your attached video, it is a PoC implementation for presentation. We have introduced the Kubernetes application mode in release 1.11 and agree to not have a per-job cluster support. The only reason is that it is not very convenient to ship the job graphs, user jars, artifacts in Kubernetes environment. [1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/FLINK/FLIP-85*Flink*Application*Mode__;Kysr!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_Bg9TMlg$> Best, Yang Fuyao Li mailto:fuyao...@oracle.com>> 于2021年4月13日周二 上午8:10写道: Hello Community, Yang, I noticed a conflict in the document for per-job mode support for Kubernetes. In the doc here [1], it mentions in a Flink Job Cluster, the available cluster manager (like YARN or Kubernetes) is used to spin up a cluster for each submitted job and this cluster is available to that job only. It implies per job mode is supported in Kubernetes. However, in the docs [2] and [3], it clearly points out per-job mode is not supported in Kubernetes. This is a conflict statement and is kind of misleading. If needed, I can create an MR to delete the statement in [1] for Kubernetes.. It is a small fix. I also noticed another thing in the video [4] at 25:08. Yang, you are executing a command with -e kubernetes-per-job flag. I tried and found such command is not supported in Flink distribution at all. I noticed the version you are using is 1.11-snapshot during the demo. Are you modifying the source code and generated an internal version of Flink….? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#flink-job-cluster<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html*flink-job-cluster__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_YCOfjX4$> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_-CMjlT8$> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html#per-job-cluster-mode<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html*per-job-cluster-mode__;Iw!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_yXjZ_Mk$> [4] https://www.youtube.com/watch?v=pdFPr_VOWTU=833s<https://urldefense.com/v3/__https:/www.youtube.com/watch?v=pdFPr_VOWTU=833s__;!!GqivPVa7Brio!JyehsOnEdEuM_rB3oA0QXTln72YUl4ueOuSdQWGAjKmpVFjVnz8jiZb_J8hhHqE$> Best, Fuyao