HI Xintong, Thanks for the feedback and please see responses below -->
On Fri, Jan 28, 2022 at 12:21 AM Xintong Song <tonysong...@gmail.com> wrote: > Thanks Thomas for drafting this FLIP, and everyone for the discussion. > > I also have a few questions and comments. > > ## Job Submission > Deploying a Flink session cluster via kubectl & CR and then submitting jobs > to the cluster via Flink cli / REST is probably the approach that requires > the least effort. However, I'd like to point out 2 weaknesses. > 1. A lot of users use Flink in perjob/application modes. For these users, > having to run the job in two steps (deploy the cluster, and submit the job) > is not that convenient. > 2. One of our motivations is being able to manage Flink applications' > lifecycles with kubectl. Submitting jobs from cli sounds not aligned with > this motivation. > I think it's probably worth it to support submitting jobs via kubectl & CR > in the first version, both together with deploying the cluster like in > perjob/application mode and after deploying the cluster like in session > mode. > The intention is to support application management through operator and CR, which means there won't be any 2 step submission process, which as you allude to would defeat the purpose of this project. The CR example shows the application part. Please note that the bare cluster support is an *additional* feature for scenarios that require external job management. Is there anything on the FLIP page that creates a different impression? > > ## Versioning > Which Flink versions does the operator plan to support? > 1. Native K8s deployment was firstly introduced in Flink 1.10 > 2. Native K8s HA was introduced in Flink 1.12 > 3. The Pod template support was introduced in Flink 1.13 > 4. There was some changes to the Flink docker image entrypoint script in, > IIRC, Flink 1.13 > Great, thanks for providing this. It is important for the compatibility going forward also. We are targeting Flink 1.14.x upwards. Before the operator is ready there will be another Flink release. Let's see if anyone is interested in earlier versions? > > ## Compatibility > What kind of API compatibility we can commit to? It's probably fine to have > alpha / beta version APIs that allow incompatible future changes for the > first version. But eventually we would need to guarantee backwards > compatibility, so that an early version CR can work with a new version > operator. > Another great point and please let me include that on the FLIP page. ;-) I think we should allow incompatible changes for the first one or two versions, similar to how other major features have evolved recently, such as FLIP-27. Would be great to get broader feedback on this one. Cheers, Thomas > > Thank you~ > > Xintong Song > > > > On Fri, Jan 28, 2022 at 1:18 PM Thomas Weise <t...@apache.org> wrote: > > > Thanks for the feedback! > > > > > > > > # 1 Flink Native vs Standalone integration > > > Maybe we should make this more clear in the FLIP but we agreed to do > the > > > first version of the operator based on the native integration. > > > While this clearly does not cover all use-cases and requirements, it > > seems > > > this would lead to a much smaller initial effort and a nicer first > > version. > > > > > > > I'm also leaning towards the native integration, as long as it reduces > the > > MVP effort. Ultimately the operator will need to also support the > > standalone mode. I would like to gain more confidence that native > > integration reduces the effort. While it cuts the effort to handle the TM > > pod creation, some mapping code from the CR to the native integration > > client and config needs to be created. As mentioned in the FLIP, native > > integration requires the Flink job manager to have access to the k8s API > to > > create pods, which in some scenarios may be seen as unfavorable. > > > > > > > # Pod Template > > > > > Is the pod template in CR same with what Flink has already > > > supported[4]? > > > > > Then I am afraid not the arbitrary field(e.g. cpu/memory resources) > > > could > > > > > take effect. > > > > Yes, pod template would look almost identical. There are a few settings > > that the operator will control (and that may need to be blacklisted), but > > in general we would not want to place restrictions. I think a mechanism > > where a pod template is merged from multiple layers would also be > > interesting to make this more flexible. > > > > Cheers, > > Thomas > > >
