Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline
Thanks Yang for your help. On Thu, Feb 4, 2021, 8:28 AM Yang Wang wrote: > Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session > ...) for the job management, > it will eventually call the RestClusterClient, which could retrieve the > leader JobManager address from ZK. > > Please ensure that you have specified the HA related config options in CLI > via -D or set them in the flink-conf.yaml. > > Best, > Yang > > sidhant gupta 于2021年2月3日周三 下午10:02写道： > >> Is it possible to use flink CLI instead of flink client for connecting >> zookeeper using network load balancer to retrieve the leader Jobmanager >> address? >> >> On Wed, Feb 3, 2021, 12:42 PM Yang Wang wrote: >> >>> I think the Flink client could make a connection with ZooKeeper via the >>> network load balancer. >>> Flink client is not aware of whether it is a network balancer or >>> multiple ZooKeeper server address. >>> After then Flink client will retrieve the active leader JobManager >>> address via ZooKeeperHAService >>> and submit the job successfully via rest client. >>> >>> Best, >>> Yang >>> >>> >>> sidhant gupta 于2021年2月2日周二 下午11:14写道： >>> >>>> Hi >>>> >>>> I have a flink ECS cluster setup with HA mode using zookeeper where I >>>> have 2 jobmanagers out of which one of will be elected as leader using >>>> zookeeper leader election. I have one application load balancer in front of >>>> the jobmanagers and one network load balancer in front of zookeeper. >>>> >>>> As per [1] >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html> >>>> , >>>> we can provide zookeeper address in the flink cli arguments and it would >>>> upload/ submit the jar to the leader jobmanager. But since I am using >>>> network load balancer in front of zookeeper, I guess it is not able to make >>>> connection with the zookeeper. Please provide suggestions or sample command >>>> for uploading the flink job jar or run the job. >>>> >>>> Is there any way by which we can distinguish between leader and >>>> standby jobmanagers in terms of request or response ? >>>> >>>> Can we use flink cli in jenkins to upload the jar to the flink cluster >>>> and run the jobs? >>>> >>>> >>>> [1] >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html >>>> >>>> Thanks >>>> Sidhant Gupta >>>> >>>
Job submission failure via flink cli
Please help me on this. Thanks Sidhant Gupta flink-conf.yaml Description: application/yaml
Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline
Is it possible to use flink CLI instead of flink client for connecting zookeeper using network load balancer to retrieve the leader Jobmanager address? On Wed, Feb 3, 2021, 12:42 PM Yang Wang wrote: > I think the Flink client could make a connection with ZooKeeper via the > network load balancer. > Flink client is not aware of whether it is a network balancer or multiple > ZooKeeper server address. > After then Flink client will retrieve the active leader JobManager address > via ZooKeeperHAService > and submit the job successfully via rest client. > > Best, > Yang > > > sidhant gupta 于2021年2月2日周二 下午11:14写道： > >> Hi >> >> I have a flink ECS cluster setup with HA mode using zookeeper where I >> have 2 jobmanagers out of which one of will be elected as leader using >> zookeeper leader election. I have one application load balancer in front of >> the jobmanagers and one network load balancer in front of zookeeper. >> >> As per [1] >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html> >> , >> we can provide zookeeper address in the flink cli arguments and it would >> upload/ submit the jar to the leader jobmanager. But since I am using >> network load balancer in front of zookeeper, I guess it is not able to make >> connection with the zookeeper. Please provide suggestions or sample command >> for uploading the flink job jar or run the job. >> >> Is there any way by which we can distinguish between leader and standby >> jobmanagers in terms of request or response ? >> >> Can we use flink cli in jenkins to upload the jar to the flink cluster >> and run the jobs? >> >> >> [1] >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html >> >> Thanks >> Sidhant Gupta >> >
Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline
Hi I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper. As per [1] <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html> , we can provide zookeeper address in the flink cli arguments and it would upload/ submit the jar to the leader jobmanager. But since I am using network load balancer in front of zookeeper, I guess it is not able to make connection with the zookeeper. Please provide suggestions or sample command for uploading the flink job jar or run the job. Is there any way by which we can distinguish between leader and standby jobmanagers in terms of request or response ? Can we use flink cli in jenkins to upload the jar to the flink cluster and run the jobs? [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html Thanks Sidhant Gupta
Re: Uploading job jar via web UI in flink HA mode
Hi Till, Thanks for the clarification and suggestions Regards Sidhant Gupta On Wed, Dec 2, 2020, 10:10 PM Till Rohrmann wrote: > Hi Sidhant, > > Have you seen this discussion [1]? If you want to use S3, then you need to > make sure that you start your Flink processes with the appropriate > FileSystemProvider for S3 [2]. So the problem you are seeing is most likely > caused by the JVM not knowing a S3 file system implementation. > > Be aware that the filesystem used by the FileUploadHandler > is java.nio.file.FileSystem and not > Flink's org.apache.flink.core.fs.FileSystem for which we provide different > FileSystem implementations. > > My best recommendation would be to not use the web submission for Flink > jobs when running Flink behind a load balancer. Instead use the CLI which > will retrieve the leader address from ZooKeeper. > > [1] > https://lists.apache.org/thread.html/1dbaf68bc86ad3a1acc87e4bbc2dd5fa8c7560b3f59eb256f81bc855%40%3Cuser.flink.apache.org%3E > [2] > https://stackoverflow.com/questions/41113119/java-nio-file-implementation-for-aws > > Cheers, > Till > > On Wed, Dec 2, 2020 at 11:31 AM sidhant gupta wrote: > >> Hi All, >> >> I have 2 job managers in flink HA mode cluster setup. I have a load >> balancer forwarding request to both (leader and stand by) the job managers >> in default round-robin fashion. While uploading the job jar the Web UI is >> fluctuating between the leader and standby page. Its difficult to upload >> the job jar and also getting errors after submitting the job as below: >> >> Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: >> Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58 >> c629d5/flink-web-upload/ce291201-31e4-4e55 >> -a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist >> >> >> I tried using *web.upload.dir* property to use a folder location in S3 >> but getting below error: >> >> 2020-12-02 08:26:06,242 WARN >> org.apache.flink.runtime.rest.FileUploadHandler [] - File >> upload failed. >> >> java.lang.IllegalArgumentException: UploadDirectory is not absolute. >> >> at >> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) >> ~[flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.handler.FileUploads.(FileUploads.java:58) >> ~[flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> >> Looks like web.upload.dir only supports local path. >> >> >> Any suggestions on how to upload and submit job jar in flink HA cluster >> setup via web UI and also it from the CLI ? >> >> >> Thanks and regards >> >> Sidhant Gupta >> >> >> >>
Uploading job jar via web UI in flink HA mode
Hi All, I have 2 job managers in flink HA mode cluster setup. I have a load balancer forwarding request to both (leader and stand by) the job managers in default round-robin fashion. While uploading the job jar the Web UI is fluctuating between the leader and standby page. Its difficult to upload the job jar and also getting errors after submitting the job as below: Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58 c629d5/flink-web-upload/ce291201-31e4-4e55 -a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist I tried using *web.upload.dir* property to use a folder location in S3 but getting below error: 2020-12-02 08:26:06,242 WARN org.apache.flink.runtime.rest.FileUploadHandler [] - File upload failed. java.lang.IllegalArgumentException: UploadDirectory is not absolute. at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.runtime.rest.handler.FileUploads.(FileUploads.java:58) ~[flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) [flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) [flink-dist_2.11-1.11.2.jar:1.11.2] at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [flink-dist_2.11-1.11.2.jar:1.11.2] Looks like web.upload.dir only supports local path. Any suggestions on how to upload and submit job jar in flink HA cluster setup via web UI and also it from the CLI ? Thanks and regards Sidhant Gupta
Re: The file STDOUT does not exist on the TaskExecutor
Thanks Yang for providing another alternative solution. On Fri, Oct 9, 2020, 7:49 AM Yang Wang wrote: > I second till's suggestion. Currently in container > environment(docker/K8s), we could not output the > STDOUT/STDERR to console and log > files(taskmanager.out/err) simultaneously. In consideration > of the user experience, we are using "conf/log4j-console.properties" to > only output the STDOUT/STDERR > to console. Then users could use "docker logs " or "kubectl > logs " to view > the logs easily. > > Except for disabling the logging of TaskManagerStdoutFileHandler > in log4j-console.properties, you > could also customize the image entrypoint to redirect the STDOUT/STDERR to > separate files(taskmanager.out/err). > > > Best, > Yang > > Till Rohrmann 于2020年10月8日周四 下午3:30写道： > >> The easiest way to suppress this error would be to disable the logging >> for TaskManagerStdoutFileHandler in your log4j.properties file. >> >> Cheers, >> Till >> >> On Wed, Oct 7, 2020 at 8:48 PM sidhant gupta wrote: >> >>> Hi Till, >>> >>> I understand the errors which appears in my logs are not stopping me >>> from running the job. I am running flink session cluster in ECS and also >>> configured graylog to get the container logs. So getting the docker logs is >>> also not an issue. >>> But is there a way to suppress this error or any work around ? >>> >>> Thanks >>> Sidhant Gupta >>> >>> On Wed, Oct 7, 2020, 9:15 PM Till Rohrmann wrote: >>> >>>> Hi Sidhant, >>>> >>>> when using Flink's Docker image, then the cluster won't create the out >>>> files. Instead the components will directly write to STDOUT which is >>>> captured by Kubernetes and can be viewed using `kubectl logs POD_NAME`. The >>>> error which appears in your logs is not a problem. It is simply the REST >>>> handler which tries to serve the out files. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Wed, Oct 7, 2020 at 5:11 PM 大森林 wrote: >>>> >>>>> what's your running mode? >>>>> if your flink cluster is on yarn mode,then the output you need has no >>>>> relation to $FLINK_HOME/logs/*.out >>>>> >>>>> >>>>> -- 原始邮件 -- >>>>> *发件人:* "sidhant gupta" ; >>>>> *发送时间:* 2020年10月7日(星期三) 晚上11:33 >>>>> *收件人:* "大森林";"user"; >>>>> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >>>>> >>>>> Hi, >>>>> >>>>> I'm running flink cluster in ecs. There is a pipeline which creates >>>>> the job manager and then the task manager using the docker image. >>>>> >>>>> Not sure if we would want to restart the cluster in production. >>>>> >>>>> Is there any way we can make sure the .out files will be created >>>>> without restart ? >>>>> >>>>> I am able to see the logs in the logs tab but not the stdout logs in >>>>> the web ui and getting the below mentioned error after running the job. >>>>> >>>>> Thanks >>>>> Sidhant Gupta >>>>> >>>>> >>>>> On Wed, Oct 7, 2020, 8:00 PM 大森林 wrote: >>>>> >>>>>> it's easy, >>>>>> just restart your flink cluster(standalone mode) >>>>>> >>>>>> if you run flink in yarn mode,then the result will display on >>>>>> $HADOOP/logs/*.out files >>>>>> >>>>>> -- 原始邮件 -- >>>>>> *发件人:* "sidhant gupta" ; >>>>>> *发送时间:* 2020年10月7日(星期三) 晚上9:52 >>>>>> *收件人:* "大森林"; >>>>>> *抄送:* "user"; >>>>>> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >>>>>> >>>>>> ++ user >>>>>> >>>>>> On Wed, Oct 7, 2020, 6:47 PM sidhant gupta >>>>>> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I checked in the $FLINK_HOME/logs. The .out file was not there. Can >>>>>>> you suggest what should be the action item ? >>>>>>> >>>>>>> Thanks >>>>>>>
Re: The file STDOUT does not exist on the TaskExecutor
Hi Till, I understand the errors which appears in my logs are not stopping me from running the job. I am running flink session cluster in ECS and also configured graylog to get the container logs. So getting the docker logs is also not an issue. But is there a way to suppress this error or any work around ? Thanks Sidhant Gupta On Wed, Oct 7, 2020, 9:15 PM Till Rohrmann wrote: > Hi Sidhant, > > when using Flink's Docker image, then the cluster won't create the out > files. Instead the components will directly write to STDOUT which is > captured by Kubernetes and can be viewed using `kubectl logs POD_NAME`. The > error which appears in your logs is not a problem. It is simply the REST > handler which tries to serve the out files. > > Cheers, > Till > > On Wed, Oct 7, 2020 at 5:11 PM 大森林 wrote: > >> what's your running mode? >> if your flink cluster is on yarn mode,then the output you need has no >> relation to $FLINK_HOME/logs/*.out >> >> >> -- 原始邮件 -- >> *发件人:* "sidhant gupta" ; >> *发送时间:* 2020年10月7日(星期三) 晚上11:33 >> *收件人:* "大森林";"user"; >> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >> >> Hi, >> >> I'm running flink cluster in ecs. There is a pipeline which creates the >> job manager and then the task manager using the docker image. >> >> Not sure if we would want to restart the cluster in production. >> >> Is there any way we can make sure the .out files will be created without >> restart ? >> >> I am able to see the logs in the logs tab but not the stdout logs in the >> web ui and getting the below mentioned error after running the job. >> >> Thanks >> Sidhant Gupta >> >> >> On Wed, Oct 7, 2020, 8:00 PM 大森林 wrote: >> >>> it's easy, >>> just restart your flink cluster(standalone mode) >>> >>> if you run flink in yarn mode,then the result will display on >>> $HADOOP/logs/*.out files >>> >>> -- 原始邮件 -- >>> *发件人:* "sidhant gupta" ; >>> *发送时间:* 2020年10月7日(星期三) 晚上9:52 >>> *收件人:* "大森林"; >>> *抄送:* "user"; >>> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >>> >>> ++ user >>> >>> On Wed, Oct 7, 2020, 6:47 PM sidhant gupta wrote: >>> >>>> Hi >>>> >>>> I checked in the $FLINK_HOME/logs. The .out file was not there. Can >>>> you suggest what should be the action item ? >>>> >>>> Thanks >>>> Sidhant Gupta >>>> >>>> >>>> On Wed, Oct 7, 2020, 7:17 AM 大森林 wrote: >>>> >>>>> >>>>> check if the .out file is in $FLINK_HOME/logs please. >>>>> >>>>> -- 原始邮件 -- >>>>> *发件人:* "sidhant gupta" ; >>>>> *发送时间:* 2020年10月7日(星期三) 凌晨1:52 >>>>> *收件人:* "大森林"; >>>>> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >>>>> >>>>> Hi, >>>>> >>>>> I am just running the docker container as it is by adding just the >>>>> conf/flink.yaml . >>>>> I am not sure if the .out file got deleted. Do we need to expose some >>>>> ports ? >>>>> >>>>> Thanks >>>>> Sidhant Gupta >>>>> >>>>> >>>>> >>>>> On Tue, Oct 6, 2020, 8:51 PM 大森林 wrote: >>>>> >>>>>> >>>>>> Hi,I guess you may deleted .out file in $FLINK_HOME/logs. >>>>>> you can just use your default log settings. >>>>>> -- 原始邮件 -- >>>>>> *发件人:* "sidhant gupta" ; >>>>>> *发送时间:* 2020年10月6日(星期二) 晚上10:59 >>>>>> *收件人:* "user"; >>>>>> *主题:* The file STDOUT does not exist on the TaskExecutor >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am running dockerized flink:1.11.0-scala_2.11 container in ecs. I >>>>>> am getting the following error after the job runs: >>>>>> >>>>>> ERROR org.apache.flink.runtime.rest.handler.taskmanager. >>>>>> TaskManagerStdoutFileHandler [] - Unhandled exception. >>>>>> org.apache.flink.util.FlinkException: The file STDOUT does not exist >>>>>> on the TaskExecutor. >>>>>> at org.apache.flink.runtime.taskexecutor.TaskExecutor >>>>>> .lambda$requestFileUploadByFilePath$25(TaskExecutor.java:1742) >>>>>> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >>>>>> at java.util.concurrent.CompletableFuture$AsyncSupply.run( >>>>>> CompletableFuture.java:1604) ~[?:1.8.0_262] >>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>> ThreadPoolExecutor.java:1149) ~[?:1.8.0_262] >>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>> ThreadPoolExecutor.java:624) ~[?:1.8.0_262] >>>>>> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] >>>>>> >>>>>> I guess "file" needs to be added in log4j.properties in the docker >>>>>> container e.g. log4j.rootLogger=INFO, file >>>>>> Are there any other properties which needs to be configured in any of >>>>>> the other property files or any jar needs to be added in the */opt/flink >>>>>> *path ? >>>>>> Thanks >>>>>> Sidhant Gupta >>>>>> >>>>>>
Re: The file STDOUT does not exist on the TaskExecutor
Hi, I'm running flink cluster in ecs. There is a pipeline which creates the job manager and then the task manager using the docker image. Not sure if we would want to restart the cluster in production. Is there any way we can make sure the .out files will be created without restart ? I am able to see the logs in the logs tab but not the stdout logs in the web ui and getting the below mentioned error after running the job. Thanks Sidhant Gupta On Wed, Oct 7, 2020, 8:00 PM 大森林 wrote: > it's easy, > just restart your flink cluster(standalone mode) > > if you run flink in yarn mode,then the result will display on > $HADOOP/logs/*.out files > > -- 原始邮件 ------ > *发件人:* "sidhant gupta" ; > *发送时间:* 2020年10月7日(星期三) 晚上9:52 > *收件人:* "大森林"; > *抄送:* "user"; > *主题:* Re: The file STDOUT does not exist on the TaskExecutor > > ++ user > > On Wed, Oct 7, 2020, 6:47 PM sidhant gupta wrote: > >> Hi >> >> I checked in the $FLINK_HOME/logs. The .out file was not there. Can you >> suggest what should be the action item ? >> >> Thanks >> Sidhant Gupta >> >> >> On Wed, Oct 7, 2020, 7:17 AM 大森林 wrote: >> >>> >>> check if the .out file is in $FLINK_HOME/logs please. >>> >>> -- 原始邮件 -- >>> *发件人:* "sidhant gupta" ; >>> *发送时间:* 2020年10月7日(星期三) 凌晨1:52 >>> *收件人:* "大森林"; >>> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >>> >>> Hi, >>> >>> I am just running the docker container as it is by adding just the >>> conf/flink.yaml . >>> I am not sure if the .out file got deleted. Do we need to expose some >>> ports ? >>> >>> Thanks >>> Sidhant Gupta >>> >>> >>> >>> On Tue, Oct 6, 2020, 8:51 PM 大森林 wrote: >>> >>>> >>>> Hi,I guess you may deleted .out file in $FLINK_HOME/logs. >>>> you can just use your default log settings. >>>> -- 原始邮件 -- >>>> *发件人:* "sidhant gupta" ; >>>> *发送时间:* 2020年10月6日(星期二) 晚上10:59 >>>> *收件人:* "user"; >>>> *主题:* The file STDOUT does not exist on the TaskExecutor >>>> >>>> Hi, >>>> >>>> I am running dockerized flink:1.11.0-scala_2.11 container in ecs. I am >>>> getting the following error after the job runs: >>>> >>>> ERROR org.apache.flink.runtime.rest.handler.taskmanager. >>>> TaskManagerStdoutFileHandler [] - Unhandled exception. >>>> org.apache.flink.util.FlinkException: The file STDOUT does not exist on >>>> the TaskExecutor. >>>> at org.apache.flink.runtime.taskexecutor.TaskExecutor >>>> .lambda$requestFileUploadByFilePath$25(TaskExecutor.java:1742) >>>> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >>>> at java.util.concurrent.CompletableFuture$AsyncSupply.run( >>>> CompletableFuture.java:1604) ~[?:1.8.0_262] >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1149) ~[?:1.8.0_262] >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:624) ~[?:1.8.0_262] >>>> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] >>>> >>>> I guess "file" needs to be added in log4j.properties in the docker >>>> container e.g. log4j.rootLogger=INFO, file >>>> Are there any other properties which needs to be configured in any of >>>> the other property files or any jar needs to be added in the */opt/flink >>>> *path ? >>>> Thanks >>>> Sidhant Gupta >>>> >>>>
Re: The file STDOUT does not exist on the TaskExecutor
++ user On Wed, Oct 7, 2020, 6:47 PM sidhant gupta wrote: > Hi > > I checked in the $FLINK_HOME/logs. The .out file was not there. Can you > suggest what should be the action item ? > > Thanks > Sidhant Gupta > > > On Wed, Oct 7, 2020, 7:17 AM 大森林 wrote: > >> >> check if the .out file is in $FLINK_HOME/logs please. >> >> -- 原始邮件 -- >> *发件人:* "sidhant gupta" ; >> *发送时间:* 2020年10月7日(星期三) 凌晨1:52 >> *收件人:* "大森林"; >> *主题:* Re: The file STDOUT does not exist on the TaskExecutor >> >> Hi, >> >> I am just running the docker container as it is by adding just the >> conf/flink.yaml . >> I am not sure if the .out file got deleted. Do we need to expose some >> ports ? >> >> Thanks >> Sidhant Gupta >> >> >> >> On Tue, Oct 6, 2020, 8:51 PM 大森林 wrote: >> >>> >>> Hi,I guess you may deleted .out file in $FLINK_HOME/logs. >>> you can just use your default log settings. >>> -- 原始邮件 -- >>> *发件人:* "sidhant gupta" ; >>> *发送时间:* 2020年10月6日(星期二) 晚上10:59 >>> *收件人:* "user"; >>> *主题:* The file STDOUT does not exist on the TaskExecutor >>> >>> Hi, >>> >>> I am running dockerized flink:1.11.0-scala_2.11 container in ecs. I am >>> getting the following error after the job runs: >>> >>> ERROR org.apache.flink.runtime.rest.handler.taskmanager. >>> TaskManagerStdoutFileHandler [] - Unhandled exception. >>> org.apache.flink.util.FlinkException: The file STDOUT does not exist on >>> the TaskExecutor. >>> at org.apache.flink.runtime.taskexecutor.TaskExecutor >>> .lambda$requestFileUploadByFilePath$25(TaskExecutor.java:1742) >>> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >>> at java.util.concurrent.CompletableFuture$AsyncSupply.run( >>> CompletableFuture.java:1604) ~[?:1.8.0_262] >>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>> ThreadPoolExecutor.java:1149) ~[?:1.8.0_262] >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>> ThreadPoolExecutor.java:624) ~[?:1.8.0_262] >>> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] >>> >>> I guess "file" needs to be added in log4j.properties in the docker >>> container e.g. log4j.rootLogger=INFO, file >>> Are there any other properties which needs to be configured in any of >>> the other property files or any jar needs to be added in the */opt/flink >>> *path ? >>> Thanks >>> Sidhant Gupta >>> >>>
The file STDOUT does not exist on the TaskExecutor
Hi, I am running dockerized flink:1.11.0-scala_2.11 container in ecs. I am getting the following error after the job runs: ERROR org.apache.flink.runtime.rest.handler.taskmanager. TaskManagerStdoutFileHandler [] - Unhandled exception. org.apache.flink.util.FlinkException: The file STDOUT does not exist on the TaskExecutor. at org.apache.flink.runtime.taskexecutor.TaskExecutor .lambda$requestFileUploadByFilePath$25(TaskExecutor.java:1742) ~[flink-dist_2.11-1.11.0.jar:1.11.0] at java.util.concurrent.CompletableFuture$AsyncSupply.run( CompletableFuture.java:1604) ~[?:1.8.0_262] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor .java:1149) ~[?:1.8.0_262] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:624) ~[?:1.8.0_262] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] I guess "file" needs to be added in log4j.properties in the docker container e.g. log4j.rootLogger=INFO, file Are there any other properties which needs to be configured in any of the other property files or any jar needs to be added in the */opt/flink *path ? Thanks Sidhant Gupta
Re: How jobmanager and task manager communicates with each other ?
Hi Till, Thanks for the reply. (1) If we are not using Flink's HA services then how we can dynamically configure task manager nodes to connect to job manager? Any suggestions or best practices? (2) Which and how flink's HA service can be used for the service discovery of job manager ? Regards Sidhant Gupta On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann wrote: > Hi Sidhant, > > the cluster components use tcp to communicate with each other. If you are > not using Flink's HA services, then the TaskManager nodes need to be > configured with the JobManager's address to connect to them. If you are > using HA services, then the service discovery happens through the HA > services. One requirement for Flink to work is that the different cluster > nodes on which a Flink process is started can communicate with each other. > > Cheers, > Till > > On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta wrote: > >> ++d...@flink.apache.org >> >> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta wrote: >> >> > Hi User >> > >> > How jobmanager and task manager communicates with each other ? How to >> set >> > connection between jobmanager and task manager running in different/same >> > ec2 instance ? Is it http or tcp ? How the service discovery works ? >> > >> > Thanks >> > Sidhant Gupta >> > >> >
Re: How jobmanager and task manager communicates with each other ?
++d...@flink.apache.org On Mon, Aug 24, 2020, 7:31 PM sidhant gupta wrote: > Hi User > > How jobmanager and task manager communicates with each other ? How to set > connection between jobmanager and task manager running in different/same > ec2 instance ? Is it http or tcp ? How the service discovery works ? > > Thanks > Sidhant Gupta >
How jobmanager and task manager communicates with each other ?
Hi User How jobmanager and task manager communicates with each other ? How to set connection between jobmanager and task manager running in different/same ec2 instance ? Is it http or tcp ? How the service discovery works ? Thanks Sidhant Gupta
Re: Flink cluster deployment strategy
Thanks, I will check it out. On Thu, 13 Aug, 2020, 7:55 PM Arvid Heise, wrote: > Hi Sidhant, > > If you are starting fresh with Flink, I strongly recommend to skip ECS and > EMR and directly go to a kubernetes-based solution. Scaling is much easier > on K8s, there will be some kind of autoscaling coming in the next release, > and the best of it all: you even have the option to go to a different cloud > provider if needed. > > The easiest option for you is to use EKS on AWS together with Ververica > community edition [1] or with one of the many kubernetes operators. > > [1] https://www.ververica.com/getting-started > > On Tue, Aug 11, 2020 at 3:23 PM Till Rohrmann > wrote: > >> Hi Sidhant, >> >> see the inline comments for answers >> >> On Tue, Aug 11, 2020 at 3:10 PM sidhant gupta >> wrote: >> >>> Hi Till, >>> >>> Thanks for your response. >>> I have few queries though as mentioned below: >>> (1) Can flink be used in map-reduce fashion with data streaming api ? >>> >> >> What do you understand as map-reduce fashion? You can use Flink's DataSet >> API for processing batch workloads (consisting not only of map and reduce >> operations but also other operations such as groupReduce, flatMap, etc.). >> Flink's DataStream API can be used to process bounded and unbounded >> streaming data. >> >> (2) Does it make sense to use aws EMR if we are not using flink in >>> map-reduce fashion with streaming api ? >>> >> >> I think I don't fully understand what you mean with map-reduce fashion. >> Do you mean multiple stages of map and reduce operations? >> >> >>> (3) Can flink cluster be auto scaled using EMR Managed Scaling when used >>> with yarn as per this link >>> https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/ >>> ? >>> >> >> I am no expert on EMR managed scaling but I believe that it would need >> some custom tooling to scale a Flink job down (by taking a savepoint a >> resuming from it with a lower parallelism) before downsizing the EMR >> cluster. >> >> >>> (4) If we set an explicit max parallelism, and set current parallelism >>> (which might be less than the max parallelism) equal to the maximum number >>> of slots and set slots per task manager while starting the yarn session, >>> then if we increase the task manager as per auto scaling then does the >>> parallelism would increase (till the max parallelism ) and the load would >>> be distributed across the newly spined up task manager ? Refer: >>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/production_ready.html#set-an-explicit-max-parallelism >>> >>> >> >> At the moment, Flink does not support this out of the box but the >> community is working on this feature. >> >>> >>> Regards >>> Sidhant Gupta >>> >>> On Tue, 11 Aug, 2020, 5:19 PM Till Rohrmann, >>> wrote: >>> >>>> Hi Sidhant, >>>> >>>> I am not an expert on AWS services but I believe that EMR might be a >>>> bit easier to start with since AWS EMR comes with Flink support out of the >>>> box [1]. On ECS I believe that you would have to set up the containers >>>> yourself. Another interesting deployment option could be to use Flink's >>>> native Kubernetes integration [2] which would work on AWS EKS. >>>> >>>> [1] >>>> https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-create-cluster.html >>>> [2] >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html >>>> >>>> Cheers, >>>> Till >>>> >>>> On Tue, Aug 11, 2020 at 9:16 AM sidhant gupta >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm kind of new to flink cluster deployment. I wanted to know which >>>>> flink >>>>> cluster deployment and which job mode in aws is better in terms of >>>>> ease of >>>>> deployment, maintenance, HA, cost, etc. As of now I am considering aws >>>>> EMR >>>>> vs ECS (docker containers). We have a usecase of setting up a data >>>>> streaming api which reads records from a Kafka topic, process it and >>>>> then >>>>> write to a another Kafka topic. Please let me know your thoughts on >>>>> this. >>>>> >>>>> Thanks >>>>> Sidhant Gupta >>>>> >>>> > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng >
Re: Flink cluster deployment strategy
Hi Till, Thanks for your response. I have few queries though as mentioned below: (1) Can flink be used in map-reduce fashion with data streaming api ? (2) Does it make sense to use aws EMR if we are not using flink in map-reduce fashion with streaming api ? (3) Can flink cluster be auto scaled using EMR Managed Scaling when used with yarn as per this link https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-managed-scaling-automatically-resize-clusters-to-lower-cost/ ? (4) If we set an explicit max parallelism, and set current parallelism (which might be less than the max parallelism) equal to the maximum number of slots and set slots per task manager while starting the yarn session, then if we increase the task manager as per auto scaling then does the parallelism would increase (till the max parallelism ) and the load would be distributed across the newly spined up task manager ? Refer: https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/production_ready.html#set-an-explicit-max-parallelism Regards Sidhant Gupta On Tue, 11 Aug, 2020, 5:19 PM Till Rohrmann, wrote: > Hi Sidhant, > > I am not an expert on AWS services but I believe that EMR might be a bit > easier to start with since AWS EMR comes with Flink support out of the box > [1]. On ECS I believe that you would have to set up the containers > yourself. Another interesting deployment option could be to use Flink's > native Kubernetes integration [2] which would work on AWS EKS. > > [1] > https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-create-cluster.html > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html > > Cheers, > Till > > On Tue, Aug 11, 2020 at 9:16 AM sidhant gupta wrote: > >> Hi all, >> >> I'm kind of new to flink cluster deployment. I wanted to know which flink >> cluster deployment and which job mode in aws is better in terms of ease of >> deployment, maintenance, HA, cost, etc. As of now I am considering aws EMR >> vs ECS (docker containers). We have a usecase of setting up a data >> streaming api which reads records from a Kafka topic, process it and then >> write to a another Kafka topic. Please let me know your thoughts on this. >> >> Thanks >> Sidhant Gupta >> >
Flink cluster deployment strategy
Hi all, I'm kind of new to flink cluster deployment. I wanted to know which flink cluster deployment and which job mode in aws is better in terms of ease of deployment, maintenance, HA, cost, etc. As of now I am considering aws EMR vs ECS (docker containers). We have a usecase of setting up a data streaming api which reads records from a Kafka topic, process it and then write to a another Kafka topic. Please let me know your thoughts on this. Thanks Sidhant Gupta
