Re: Restore from checkpoint
Hi Phil, AFAIK, the error indicated your path was incorrect. your should use '/opt/flink/checkpoints/1875588e19b1d8709ee62be1cdcc' or 'file:///opt/flink/checkpoints/1875588e19b1d8709ee62be1cdcc' instead. Best. Jiadong.Lu On 5/18/24 2:37 AM, Phil Stavridis wrote: Hi, I am trying to test how the checkpoints work for restoring state, but not sure how to run a new instance of a flink job, after I have cancelled it, using the checkpoints which I store in the filesystem of the job manager, e.g. /opt/flink/checkpoints. I have tried passing the checkpoint as an argument in the function and use it while setting the checkpoint but it looks like the way it is done is something like below: docker-compose exec jobmanager flink run -s :/opt/flink/checkpoints/1875588e19b1d8709ee62be1cdcc -py /opt/app/flink_job.py But I am getting error: Caused by: java.io.IOException: Checkpoint/savepoint path ':/opt/flink/checkpoints/1875588e19b1d8709ee62be1cdcc' is not a valid file URI. Either the pointer path is invalid, or the checkpoint was created by a different state backend. What is wrong with the way the job is re-submitted to the cluster? Kind regards Phil
Re: Request to provide example codes on ElasticsearchSinkFunction updaterequest
Hi, Fidea. The ElasticsearchSinkFunction Class has already marked as deprecated[1], please try to use ElasticsearchSink Class. Hope to help you. Best. Jiadong.Lu 1. https://github.com/apache/flink-connector-elasticsearch/blob/153b8fc23e14c03c4bacf2c24fbe0fee286ec6e2/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkFunction.java#L63 2. https://github.com/apache/flink-connector-elasticsearch/blob/153b8fc23e14c03c4bacf2c24fbe0fee286ec6e2/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/connector/elasticsearch/sink/ElasticsearchSink.java#L56C14-L56C31 On 1/29/24 19:15, Fidea Lidea wrote: Hi Team, Could you please share with me a few example codes on how to perform "updaterequest on elasticsearch using apache flink" I.want to use |ElasticsearchSinkFunction| to perform updaterequest. Thanks Nida Shaikh lideafidea...@gmail.com <mailto:lideafidea...@gmail.com>
Re: 404 Jar File Not Found w/ Web Submit Disabled
Hi Patricia Sorry for giving wrong advice. I tried the url path of "/v1/jar/upload" and it did respond the same 404. For now, we may sure that '/jar/upload' cannot work with the configuration of `web.ui.submit=false` . If you really need to disable `web.ui.submit`configuration, maybe you should try @Shammon's solution. By the way, I have been submitting my Flink application by RestClusterClient for a long time. Or you can use the CLI by starting a subprocess. Best Jiadong Lu On 2023/8/17 23:07, jiadong.lu wrote: Hi Patricia Have you tried the url path of '/v1/jars/upload' ? Best Jiadong Lu On 2023/8/16 14:00, patricia lee wrote: Hi, Below are the steps that I take to replicate the issue. I have a requirement to disable both the capability to run and submit jobs in Flink Web UI and Rest Endpoint. I created a docker compose of job manager and task manager. When the property in job manager's config is set to web.submit.enabled: true I can submit a job via rest api as shown below (which is expected) Screenshot 2023-08-16 at 1.05.00 PM.png However, when I disabled the job manager's property web.submit.enabled: false Screenshot 2023-08-16 at 1.55.25 PM.png I can no longer upload a jar file via rest endpoint. According to the Apache Flink Documentation, web.submit.enable=false is only on the front end flag and would not take effect on the rest api. *Reference:* https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/> Screenshot 2023-08-16 at 1.18.25 PM.png On Tue, Aug 15, 2023 at 12:31 AM jiadong.lu <http://jiadong.lu> mailto:archzi...@gmail.com>> wrote: Hi, Patricia I think you should have a look the REST API[1]. > "Even it is disabled sessions clusters still accept jobs through REST requests (Http calls). This flag only guards the feature to upload jobs in the UI" means you cannot upload flink application jar in the flink dashboard UI if the `web.ui.submit=false`, you still can sumbit the flink application jar by the REST API. Best Jiadong Lu. 1. https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/ <https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/> On 2023/8/15 0:14, patricia lee wrote: > Hi, > > Just to add, when I set back to "true" the web.ui submit property, > that is when the rest endpoint /jars/upload worked again. But in the > documentation reference: > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/> > > > Disabling the UI doesnt disable the endpoint. Is this the expected > behavior? > > Regards, > Patricia > > On Mon, Aug 14, 2023, 5:07 PM patricia lee mailto:plee3...@gmail.com>> wrote: > > Hi, > > I disabled the web.ui.submit=false, after that uploading jar files > via rest endpoint is now throwing 404. In the documentation it says: > > "Even it is disabled sessions clusters still accept jobs through > REST requests (Http calls). This flag only guards the feature to > upload jobs in the UI" > > I also set the io.tmp.dirs to my specified directory. > > > But I can no longer upload jar via rest endpoint. > > > Regards, > Patricia >
Re: 404 Jar File Not Found w/ Web Submit Disabled
Hi Patricia Have you tried the url path of '/v1/jars/upload' ? Best Jiadong Lu On 2023/8/16 14:00, patricia lee wrote: Hi, Below are the steps that I take to replicate the issue. I have a requirement to disable both the capability to run and submit jobs in Flink Web UI and Rest Endpoint. I created a docker compose of job manager and task manager. When the property in job manager's config is set to web.submit.enabled: true I can submit a job via rest api as shown below (which is expected) Screenshot 2023-08-16 at 1.05.00 PM.png However, when I disabled the job manager's property web.submit.enabled: false Screenshot 2023-08-16 at 1.55.25 PM.png I can no longer upload a jar file via rest endpoint. According to the Apache Flink Documentation, web.submit.enable=false is only on the front end flag and would not take effect on the rest api. *Reference:* https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/> Screenshot 2023-08-16 at 1.18.25 PM.png On Tue, Aug 15, 2023 at 12:31 AM jiadong.lu <http://jiadong.lu> mailto:archzi...@gmail.com>> wrote: Hi, Patricia I think you should have a look the REST API[1]. > "Even it is disabled sessions clusters still accept jobs through REST requests (Http calls). This flag only guards the feature to upload jobs in the UI" means you cannot upload flink application jar in the flink dashboard UI if the `web.ui.submit=false`, you still can sumbit the flink application jar by the REST API. Best Jiadong Lu. 1. https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/ <https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/> On 2023/8/15 0:14, patricia lee wrote: > Hi, > > Just to add, when I set back to "true" the web.ui submit property, > that is when the rest endpoint /jars/upload worked again. But in the > documentation reference: > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/> > > > Disabling the UI doesnt disable the endpoint. Is this the expected > behavior? > > Regards, > Patricia > > On Mon, Aug 14, 2023, 5:07 PM patricia lee mailto:plee3...@gmail.com>> wrote: > > Hi, > > I disabled the web.ui.submit=false, after that uploading jar files > via rest endpoint is now throwing 404. In the documentation it says: > > "Even it is disabled sessions clusters still accept jobs through > REST requests (Http calls). This flag only guards the feature to > upload jobs in the UI" > > I also set the io.tmp.dirs to my specified directory. > > > But I can no longer upload jar via rest endpoint. > > > Regards, > Patricia >
Re: 404 Jar File Not Found w/ Web Submit Disabled
Hi, Patricia Maybe you should check the code carefully, a reply code of 404 typically indicates that an error occured in client side.Alternatively,could you share a snippet code where the error occured? This would greatly assist us in addressing your issue effectively. Best Jiadong Lu On 2023/8/15 17:34, patricia lee wrote: Hi Jiadong, Yes it is my understanding too, I should be able to upload jars via rest endpoint even the property web.ui.submit=false, but that wasn't the behaviour :( web.ui.submit=false But /jars/upload endpoint throws 404. Is this expected? Thanks On Tue, Aug 15, 2023, 12:31 AM jiadong.lu <http://jiadong.lu> mailto:archzi...@gmail.com>> wrote: Hi, Patricia I think you should have a look the REST API[1]. > "Even it is disabled sessions clusters still accept jobs through REST requests (Http calls). This flag only guards the feature to upload jobs in the UI" means you cannot upload flink application jar in the flink dashboard UI if the `web.ui.submit=false`, you still can sumbit the flink application jar by the REST API. Best Jiadong Lu. 1. https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/ <https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/> On 2023/8/15 0:14, patricia lee wrote: > Hi, > > Just to add, when I set back to "true" the web.ui submit property, > that is when the rest endpoint /jars/upload worked again. But in the > documentation reference: > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/> > > > Disabling the UI doesnt disable the endpoint. Is this the expected > behavior? > > Regards, > Patricia > > On Mon, Aug 14, 2023, 5:07 PM patricia lee mailto:plee3...@gmail.com>> wrote: > > Hi, > > I disabled the web.ui.submit=false, after that uploading jar files > via rest endpoint is now throwing 404. In the documentation it says: > > "Even it is disabled sessions clusters still accept jobs through > REST requests (Http calls). This flag only guards the feature to > upload jobs in the UI" > > I also set the io.tmp.dirs to my specified directory. > > > But I can no longer upload jar via rest endpoint. > > > Regards, > Patricia >
