Hi folks,
Thanks for the informative discussion.
@Allison @Becket currently the FLIP only focuses on Yarn, but after reading
all your discussions, if I am not mistaken, both Yarn and Kubernetes
clusters should be supported. Does it make sense to update the FLIP
accordingly?
Best regards,
Jing
Hi Weihua,
Just want to clarify. "client.attached.after.submission" is going to be a
pure client side configuration.
On the cluster side, it is only "execution.shutdown-on-attached-exit"
controlling whether the cluster will shutdown or not when an attached
client is disconnected. In order to
Hi, Jiangjie
Thanks for the clarification.
My key point is the meaning of the "submission" in
"client.attached.after.submission".
At first glance, I thought only job submissions were taken into account.
After your clarification, this option also works for cluster submissions.
It's fine for me.
Hi Weihua,
Thanks for the explanation. From the doc, it looks like the current
behaviors of "execution.attached=true" between Yarn and K8S session
cluster are exactly the opposite. For YARN it basically means the cluster
will shutdown if the client disconnects. For K8S, it means the cluster will
Hi, Jiangjie
'execution.attached' can be used to attach an existing cluster and stop it
[1][2],
which is not related to job submission. So does YARN session mode[3].
IMO, this behavior should not be controlled by the new option
'client.attached.after.submission'.
[1]
Hi Weihua,
Just want to clarify a little bit, what is the impact of
`execution.attached` on a cluster startup before a client submits a job to
that cluster? Does this config only become effective after a job submission?
Currently, the cluster behavior has an independent config of
Hi Jiangjie
Sorry for the late reply, I fully agree with the three user sensible
behaviors you described.
I would like to bring up a point.
Currently, 'execution.attached' is not only used for submitting jobs,
But also for starting a new cluster (YARN and Kubernetes). If it's true,
the starting
Hi, Jiangjie
Thanks for your detailed explanation, I got your point. If the
execution.attached is only used for client currently, removing it also make
sense to me.
Best,
Ron
Becket Qin 于2023年8月17日周四 07:37写道：
> Hi Ron,
>
> Isn't the cluster (session or per job) only using the
Hi Ron,
Isn't the cluster (session or per job) only using the execution.attached to
determine whether the client is attached? If so, the client can always
include the information of whether it's an attached client or not in the
JobSubmissoinRequestBody, right? For a shared session cluster, there
Hi, Jiangjie
Sorry for late reply. Thank you for such a detailed response. As you say,
there are three behaviours here for users and I agree with you. The goal of
this FLIP is to clarify the behaviour of the client side, which I also
agree with. However, as weihua said, the config
Hi Ron and Weihua,
Thanks for the feedback.
There seem three user sensible behaviors that we are talking about:
1. The behavior on the client side, i.e. whether blocking until the job
finishes or not.
2. The behavior of the submitted job, whether stop the job execution if the
client is
Hi Allison
Thanks for driving this FLIP. It's a valuable feature for batch jobs.
This helps keep "Drop Per-Job Mode [1]" going.
+1 for this proposal.
However, it seems that the change in this FLIP is not detailed enough.
I have a few questions.
1. The config 'execution.attached' is not only
Hi, Allison
Thanks for driving this proposal, it looks cool for batch jobs under
application mode. But after reading your FLIP document and [1], I have a
question. Why do you want to rename the execution.attached configuration to
client.attached.after.submission and at the same time deprecate
This is definitely a useful feature especially for the flink batch
execution workloads using flow orchestrators like Airflow, Azkaban, Oozie
etc. Thanks for reviving this issue and starting a FLIP.
Regards
Venkata krishnan
On Mon, Aug 7, 2023 at 4:09 PM Allison Chang
wrote:
> Hi all,
>
> I am
Hi all,
I am opening this thread to discuss this proposal to support attached execution
on Flink Application Completion for Batch Jobs. The link to the FLIP proposal
is here:
15 matches
Mail list logo
