通过 yarn-cluster 提交多个任务，Flink 页面只显示最后一个提交的任务
大家好，我想请教一个问题哈。 就是我通过 ./flink run -m yarn-cluster -yn 2 -yjm 1024 -ytm 1024 xxx.jar 的方式提交了两个不同的任务程序，在yarn界面里也能看到两个正在运行的 app ，但是点击 对应的 ApplicationMater 跳转到 Flink-Dashboard 页面之后，在 Running job 页面只看到一个运行的最后提交的程序呢？ Task Slot 也只用了 1 个，这是啥情况呢？ 也木有报错。yarn 页面显示两个任务都是 Running 状态，而且先提交的哪个程序任务确实是在运行中的，因为数据库中的数据也是在增加呢。 谢谢大家。 详情图片如下： https://img-blog.csdnimg.cn/20191115175027258.png
Apache Airflow - Question about checkpointing and re-run a job
Hi: I have a Flink job and sometimes I need to cancel and re run it. From what I understand the checkpoints for a job are saved under the job id directory at the checkpoint location. If I run the same job again, it will get a new job id and the checkpoint saved from the previous run job (which is saved under the previous job's id dir) will not be used for this new run. Is that a correct understanding ? If I need to re-run the job from the previous checkpoint - is there any way to do that automatically without using a savepoint ? Also, I believe the internal job restarts do not change the job id so in those cases where the job restarts will pick the state from the saved checkpoint. Is my understanding correct ? Thanks Mans
Re: Re:Apache Flink - Operator name and uuid best practices
Thanks Jiayi for your response. I am thinking on the same lines. Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name. But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check. Mans On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao wrote: Hi Mans! Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1]. > Are there any restrictions on the length of the name and uuid attributes? It’s pretty much the same as you define a string value, so there is no special restrictions on this. > Are there any restrictions on the characters used for name and uuid (blank >spaces, etc) ? I’m not a hundred percent sure about this but I run a testing program and it works fine. > Can the name and uuid be the same ? Yes. But uuids accross operators cannot be same. For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling. Hope this helps. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state Best, Jiayi Liao At 2019-11-16 18:35:38, "M Singh" wrote: Hi: I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators: 1. Are there any restrictions on the length of the name and uuid attributes ?2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?3. Can the name and uuid be the same ? Please let me know if there is any other advice. Thanks Mans
Re:Apache Flink - Operator name and uuid best practices
Hi Mans! Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1]. > Are there any restrictions on the length of the name and uuid attributes? It’s pretty much the same as you define a string value, so there is no special restrictions on this. > Are there any restrictions on the characters used for name and uuid (blank > spaces, etc) ? I’m not a hundred percent sure about this but I run a testing program and it works fine. > Can the name and uuid be the same ? Yes. But uuids accross operators cannot be same. For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling. Hope this helps. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state Best, Jiayi Liao At 2019-11-16 18:35:38, "M Singh" wrote: Hi: I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators: 1. Are there any restrictions on the length of the name and uuid attributes ? 2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ? 3. Can the name and uuid be the same ? Please let me know if there is any other advice. Thanks Mans
unsubscribe
Apache Flink - Operator name and uuid best practices
Hi: I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators: 1. Are there any restrictions on the length of the name and uuid attributes ?2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?3. Can the name and uuid be the same ? Please let me know if there is any other advice. Thanks Mans
Re: how to setup a ha flink cluster on k8s?
Hi Rock, I searched by Google and found a blog[1] talk about how to config JM HA for Flink on k8s. Do not know whether it suitable for you or not. Please feel free to refer to it. Best, Vino [1]: http://shzhangji.com/blog/2019/08/24/deploy-flink-job-cluster-on-kubernetes/ Rock 于2019年11月16日周六 上午11:02写道： > I'm trying to setup a flink cluster on k8s for production use.But the > setup here > https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/kubernetes.html > this > not ha , when job-manager down and rescheduled > > the metadata for running job is lost. > > > > I tried to use ha setup for zk > https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/jobmanager_high_availability.html > on > k8s , but can't get it right. > > > > Stroing job's metadata on k8s using pvc or other external file > system should be very easy.Is there a way to achieve it. >
