[
https://issues.apache.org/jira/browse/FLINK-30669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohemmad Zaid Khan updated FLINK-30669:
---------------------------------------
Description:
User jar has code as -
{code:java}
main() {
init env
pipelines.foreach{
env.fromSource(pipeline.getSource())
.map(pipeline.transform())
.sinkTo(pipeline.getSink())
env.execute(pipeline.getName())
}
}{code}
and below configuration -
{code:java}
execution.runtime-mode: "BATCH"
execution.attached: "true"
$internal.pipeline.job-id: "" {code}
When this single jar executed in Application Mode by using
flink-kubernetes-operator, multiple jobs are submitted sequentially and as per
design only one of the JobStatus is always associated with FlinkDeployment k8s
resource, this job status is periodically updated by operator. To update job
status in k8s resource, it fetches all of the job status from job-manager rest
endpoint and pick the first one and update that one. Problem is, job status
list returned by job-manager rest api is not sorted on time.
!image-2023-01-13-10-04-32-891.png|width=883,height=489!
!image-2023-01-13-09-54-54-280.png|width=353,height=284!
As you can see in above example, job autoscaling-3 is first one in the rest
response and same updated in FlinkDeployment resource, but FlinkDeployment
should have status of job autoscaling-19 because that is the last job finished.
was:
User jar has code as -
{code:java}
main() {
init env
pipelines.foreach{
env.fromSource(pipeline.getSource())
.map(pipeline.transform())
.sinkTo(pipeline.getSink())
env.execute(pipeline.getName())
}
}{code}
and below configuration -
{code:java}
execution.runtime-mode: "BATCH"
execution.attached: "true"
$internal.pipeline.job-id: "" {code}
When this single jar executed in Application Mode by using
flink-kubernetes-operator, multiple jobs are submitted sequentially and as per
design only one of the JobStatus is always associated with FlinkDeployment k8s
resource, this job status is periodically updated by operator. To update job
status in k8s resource, it fetches all of the job status from job-manager rest
endpoint and pick the first one and update that one. Problem is, job status
list returned by job-manager rest api is not sorted on time.
!image-2023-01-13-09-53-18-494.png|width=587,height=326!
!image-2023-01-13-09-54-54-280.png|width=353,height=284!
As you can see in above example, job autoscaling-3 is first one in the rest
response and same updated in FlinkDeployment resource, but FlinkDeployment
should have status of job autoscaling-19 because that is the last job finished.
> Update recent job status in FlinkDeployment resource object.
> ------------------------------------------------------------
>
> Key: FLINK-30669
> URL: https://issues.apache.org/jira/browse/FLINK-30669
> Project: Flink
> Issue Type: Bug
> Reporter: Mohemmad Zaid Khan
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-01-13-09-54-13-457.png,
> image-2023-01-13-09-54-54-280.png, image-2023-01-13-10-04-32-891.png
>
>
> User jar has code as -
> {code:java}
> main() {
> init env
> pipelines.foreach{
> env.fromSource(pipeline.getSource())
> .map(pipeline.transform())
> .sinkTo(pipeline.getSink())
> env.execute(pipeline.getName())
> }
> }{code}
> and below configuration -
> {code:java}
> execution.runtime-mode: "BATCH"
> execution.attached: "true"
> $internal.pipeline.job-id: "" {code}
> When this single jar executed in Application Mode by using
> flink-kubernetes-operator, multiple jobs are submitted sequentially and as
> per design only one of the JobStatus is always associated with
> FlinkDeployment k8s resource, this job status is periodically updated by
> operator. To update job status in k8s resource, it fetches all of the job
> status from job-manager rest endpoint and pick the first one and update that
> one. Problem is, job status list returned by job-manager rest api is not
> sorted on time.
> !image-2023-01-13-10-04-32-891.png|width=883,height=489!
> !image-2023-01-13-09-54-54-280.png|width=353,height=284!
> As you can see in above example, job autoscaling-3 is first one in the rest
> response and same updated in FlinkDeployment resource, but FlinkDeployment
> should have status of job autoscaling-19 because that is the last job
> finished.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
