Re: Flink job Deployement problem

:22写道： > Hi, i'm trying to deploy flink job but i have this error. How to solve it > please? > > *Thierry FOKOU *| * IT M.A.Sc <http://M.A.Sc> Student* > > Département de génie logiciel et TI > > École de technologie supérieure | Université du Québec > > 110

Re: Flink job Deployement problem

/java_lambdas/ [2] https://stackoverflow.com/questions/70295970/flink-table-get-type-information De : Fokou Toukam, Thierry Date : jeudi, 6 juin 2024 à 09:04 À : user Objet : Flink job Deployement problem Hi, i'm trying to deploy flink job but i have this error. How to solve it please? [cid:b13357ec

Flink job Deployement problem

Hi, i'm trying to deploy flink job but i have this error. How to solve it please? [cid:b13357ec-00b9-4a15-8d04-1c797a4eced3] Thierry FOKOU | IT M.A.Sc Student Département de génie logiciel et TI École de technologie supérieure | Université du Québec 1100, rue Notre-Dame Ouest Montréal

Re: How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

Hi Sachin, Yes, that's correct. To resume from a savepoint, use the command bin/flink run -s . You can find more details in the Flink documentation on [1]. Additionally, information on how to trigger a savepoint can be found in the section for triggering savepoints [2]. [1]

How to start a flink job on a long running yarn cluster from a checkpoint (with arguments)

Hi, I have a long running yarn cluster and I submit my streaming job using the following command: flink run -m yarn-cluster -yid application_1473169569237_0001 /usr/lib/flink/examples/streaming/WordCount.jar --input file:///input.txt --output file:///output/ Let's say I want to stop this job,

Best Practices? Fault Isolation for Processing Large Number of Same-Shaped Input Kafka Topics in a Big Flink Job

Hi everyone, I'm currently prototyping on a project where we need to process a large number of Kafka input topics (say, a couple of hundred), all of which share the same DataType/Schema. Our objective is to run the same Flink SQL on all of the input topics, but I am concerned about doing this in

Re: Flink job performance

How many taskmanagers and server do you have? Can you also share the task managers page of flink dashboard? On Mon, Apr 15, 2024 at 10:58 AM Oscar Perez via user wrote: > Hi community! > > We have an interesting problem with Flink after increasing parallelism in > a certain way. Here is the

Re: Flink job performance

almost full, could you try allocating more CPUs and see if the > instability persists? > > Best, > Zhanghao Chen > -- > *From:* Oscar Perez > *Sent:* Monday, April 15, 2024 19:24 > *To:* Zhanghao Chen > *Cc:* Oscar Perez via user > *Subje

Re: Flink job performance

if the instability persists? Best, Zhanghao Chen From: Oscar Perez Sent: Monday, April 15, 2024 19:24 To: Zhanghao Chen Cc: Oscar Perez via user Subject: Re: Flink job performance Hei, ok that is weird. Let me resend them. Regards, Oscar On Mon, 15 Apr 2024 at 14:00, Zhanghao

Re: Flink job performance

Hi, there seems to be sth wrong with the two images attached in the latest email. I cannot open them. Best, Zhanghao Chen From: Oscar Perez via user Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user ; pi-team ; Hermes Team Subject: Flink job

Re: Flink job performance

thrown after the change? From: Oscar Perez via user Sent: Monday, April 15, 2024 15:57 To: Oscar Perez via user ; pi-team ; Hermes Team Subject: Flink job performance Hi community! We have an interesting problem with Flink after increasing parallelism

Re: Flink job unable to restore from savepoint

flink version 1.17 Didn't change any job configuration. We are facing this below issue. https://issues.apache.org/jira/browse/FLINK-23886 On Wed, Mar 27, 2024 at 1:39 AM Hangxiang Yu wrote: > Hi, Prashant. > Which Flink version did you use? > And Did you modify your job logic or configurations

Re: Flink job unable to restore from savepoint

Hi Prashant, Compared to the job that generated savepoint, are there any changes in the new job? For example, data fields were added or deleted, or the type serializer was changed? More detailed job manager logs may help. prashant parbhane 于2024年3月27日周三 14:20写道： > > Hello, > > We have been

Re: Flink job unable to restore from savepoint

Hi, Prashant. Which Flink version did you use? And Did you modify your job logic or configurations ? If yes, Could you share changed things ? On Wed, Mar 27, 2024 at 3:37 PM prashant parbhane wrote: > Hello, > > We have been facing this weird issue of not being able to restore from > savepoint,

Flink job unable to restore from savepoint

Hello, We have been facing this weird issue of not being able to restore from savepoint, when we have a significant load on flink jobs. " *org.apache.flink.util.FlinkRuntimeException: Error while deserializing the user key.* *at

Re: 根据flink job web url可以获取到JobGraph信息么？

我在 Yanquan 的回答基础上补充下，通过 /jobs/:jobid/plan 实际上拿到的就是 JSON 表示的 JobGraph 信息（通过 JsonPlanGenerator 这个类生成，包含了绝大部分 jobgraph 里常用的信息），应该能满足你的需要 From: casel.chen Sent: Saturday, March 2, 2024 14:17 To: user-zh@flink.apache.org Subject: 根据flink job web url可以获取到JobGraph信息么

Re: 根据flink job web url可以获取到JobGraph信息么？

https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-plan 通过 /jobs/:jobid/plan 能获得 ExecutionGraph 的信息，不知道能不能包含你需要的信息。 casel.chen 于2024年3月2日周六 14:19写道： > 正在运行的flink作业能够通过其对外暴露的web url获取到JobGraph信息么？

根据flink job web url可以获取到JobGraph信息么？

正在运行的flink作业能够通过其对外暴露的web url获取到JobGraph信息么？

New Relic alerts for Flink job submission failure and recovery

Hi Team, I would like to know the possibilities of configuring the new relic alerts for a Flink job whenever the job is submitted, gets failed and recovers from the failure. In our case, we have configured the Flink environment as a Kubernetes pod running on an EKS cluster and the application

Testing Flink job with bounded input

Hi, I have a question about how to correctly set up a test that will read input from locally provided collection in bounded mode and provide outputs at the end of the computation. My test case looks something like the following: String[] lines = ...; try (StreamExecutionEnvironment env =

RE: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Фролов Cc: user@flink.apache.org Subject: Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS) ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hi Vladimir, As I see, your SQL query is reading data from the Kafka topic and pulls all data to the client

Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

ecord to be inserted to the topic? Although I would not expect those *Node - 1 disconnected* messages in such a scenario. Alexey On Tue, Dec 19, 2023 at 10:40 AM Вова Фролов wrote: > Hello Flink Community, > > I am texting to you with an issue I have encountered while using Apache > Flink versi

Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Hello Flink Community, I am texting to you with an issue I have encountered while using Apache Flink version 1.17.1. In my Flink Job, I am using Kafka version 3.6.0 to ingest data from TPC-DS(current tpcds100 target size tpcds1), and then I am executing SQL queries, specifically, the q77

Re: Query on using two sinks for a Flink job (Flink SQL)

day, December 6, 2023 7:21:50 PM > *收件人:* elakiya udhayanan ; user@flink.apache.org < > user@flink.apache.org> > *主题:* Re: Query on using two sinks for a Flink job (Flink SQL) > > Hi Elakiya, > > You can try executing TableEnvironmentImpl#executeInternal for non-insert &

Re: Query on using two sinks for a Flink job (Flink SQL)

1:50 PM 收件人: elakiya udhayanan ; user@flink.apache.org 主题: Re: Query on using two sinks for a Flink job (Flink SQL) Hi Elakiya, You can try executing TableEnvironmentImpl#executeInternal for non-insert statements, then using StatementSet.addInsertSql to add multiple insertion statetments,

Re: Query on using two sinks for a Flink job (Flink SQL)

s-master/docs/dev/table/sql/insert/ >> >> >> >> -- >> Best！ >> Xuyang >> >> >> At 2023-12-06 17:49:17, "elakiya udhayanan" wrote: >> >> Hi Team, >> I would like to know the possibility of having two sinks in a &g

Re: Query on using two sinks for a Flink job (Flink SQL)

elakiya udhayanan" wrote: > > Hi Team, > I would like to know the possibility of having two sinks in a > single Flink job. In my case I am using the Flink SQL based job where I try > to consume from two different Kafka topics using the create table (as > below) D

Re:Query on using two sinks for a Flink job (Flink SQL)

would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two different Kafka topics using the create table (as below) DDL and then use a join condition to correlate them and at present write it to a

Re: Query on using two sinks for a Flink job (Flink SQL)

Sent: Wednesday, December 6, 2023 17:49 To: user@flink.apache.org Subject: Query on using two sinks for a Flink job (Flink SQL) Hi Team, I would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two

Query on using two sinks for a Flink job (Flink SQL)

Hi Team, I would like to know the possibility of having two sinks in a single Flink job. In my case I am using the Flink SQL based job where I try to consume from two different Kafka topics using the create table (as below) DDL and then use a join condition to correlate them and at present write

Re: Flink Job Failed With Kafka Exception

hile getting metadata but we would like > flink job to continue to consume data from other source topics even if one > of the topic has any issue since failing entire flink application doesn’t > make sense if one if the topic has issue. > > > > Regards, > Madan > > On Nov

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

t; Please let me know if there are any other problems. > > Best, > Yu Chen > > > 2023年11月7日 18:11，arjun s 写道： > > > > Hi Chen, > > I attempted to configure the 'source.path.regex-pattern' property in the > table settings as '^customer.*' to ensure that

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

; Hi Chen, > I attempted to configure the 'source.path.regex-pattern' property in the > table settings as '^customer.*' to ensure that the Flink job only processes > file names starting with "customer" in the specified directory. However, it > appears that this configuratio

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

Hi Chen, I attempted to configure the 'source.path.regex-pattern' property in the table settings as '^customer.*' to ensure that the Flink job only processes file names starting with "customer" in the specified directory. However, it appears that this configuration is not producing th

Re: Flink Job Failed With Kafka Exception

entire flink application is getting failed due to time out while getting metadata but we would like flink job to continue to consume data from other source topics even if one of the topic has any issue since failing entire flink application doesn’t make sense if one if the topic has issue.Regards,Madan

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

-- of `path` >> option >> >> >> Best, >> Yu Chen >> >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ >> >> --

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

ath` option > > > Best, > Yu Chen > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ > > -- > *发件人:* arjun s > *发送时间:* 2023年11月6日 20:50 > *收件人:* user@flink.apache.org > *

回复: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

` option Best, Yu Chen [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ 发件人: arjun s 发送时间: 2023年11月6日 20:50 收件人: user@flink.apache.org 主题: Handling Schema Variability and Applying Regex Patterns in Flink Job

Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

Hi team, I'm currently utilizing the Table API function within my Flink job, with the objective of reading records from CSV files located in a source directory. To obtain the file names, I'm creating a table and specifying the schema using the Table API in Flink. Consequently, when the schema

Re: Flink Job Failed With Kafka Exception

Hi, Madan. This error seems like that there are some problems when the consumer tries to read the topic metadata. If you use the same source for these topics, the kafka connector cannot skip one of them. As you say, you need to modify the connector's default behavior. Maybe you should read the

Re: Flink Job Failed With Kafka Exception

Hi Madan, Do you mean you want to restart only the failed tasks, rather than restarting the entire pipeline region? As far as I know, currently Flink does not support task-level restart, but requires restarting the pipeline region. Best, Junrui Madan D via user 于2023年10月11日周三 12:37写道： > Hello

Flink Job Failed With Kafka Exception

Hello Team, We are running the Flink pipeline by consuming data from multiple topics, but we recently encountered that if there's one topic having issues with participation, etc., the whole Flink pipeline is failing, which is affecting topics. Is there a way we can make Flink Piplein keep

Flink job reading from s3 path

Hi team, I want to implement a flink job to read avro files from s3 path, and output to a kafka topic. Currently, I am using AvroInputFormat like this: AvroInputFormat avroInputFormat = new AvroInputFormat<>(new Path(S3PathString), Session.class); TypeInformation ty

回复： flink-job-history 任务太多页面卡死

-jobmanager-options Best, Shammon FY On Fri, Jul 28, 2023 at 10:17 AM 阿华田 wrote: 目前flink-job-history 已经收录5000+任务，当点击全部任务查看时，job-history就会卡死无法访问，各位大佬有什么好的解决方式？ | | 阿华田 | | a15733178...@163.com | 签名由网易邮箱大师定制

Re: flink-job-history 任务太多页面卡死

数，具体参数可以参考[1] > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#full-jobmanager-options > > Best, > Shammon FY > > On Fri, Jul 28, 2023 at 10:17 AM 阿华田 wrote: > > > 目前flink-job-history > > 已经收录5000+任务，当点击全部任务

Re: flink-job-history 任务太多页面卡死

Hi, 可以通过配置`jobstore.max-capacity`和`jobstore.expiration-time`控制保存的任务数，具体参数可以参考[1] [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#full-jobmanager-options Best, Shammon FY On Fri, Jul 28, 2023 at 10:17 AM 阿华田 wrote: > 目前flink-job-history > 已经收录5000+任务，当点击

flink-job-history 任务太多页面卡死

目前flink-job-history 已经收录5000+任务，当点击全部任务查看时，job-history就会卡死无法访问，各位大佬有什么好的解决方式？ | | 阿华田 | | a15733178...@163.com | 签名由网易邮箱大师定制

Flink job submission to Multi-VM Flink Cluster fails!

Hello, I can launch a Flink cluster (version 1.17.x) on my laptop with 1 Job Manager and 3 Task Managers. The cluster starts, jobs can be submitted correctly on the localhost (my laptop). Next I tried to launch this cluster on 4 VMs - 1 Master

Re: Flink Job Failure for version 1.16

Hi, I may have missed something, So could you share more: I have recently migrated from 1.13.6 to 1.16.1, I can see there is a > performance degradation... Are you referring to a decrease in Checkpoint Performance when you mention performance decline? It just happens when you upgrade from

Re: Flink Job Failure for version 1.16

Hi Everyone, can someone please shade some light when the Checkpoint Coordinator is suspending Error comes and what should I do to avoid this? it is impacting the production pipeline after the version upgrade. It is related to resource crunch in the pipeline? Thank You On Thu, May 11, 2023 at

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

/ops/state/checkpoints/#resuming-from-a-retained-checkpoint Best, Weihua On Thu, May 11, 2023 at 1:43 PM amenreet sodhi wrote: > Hey Hang, > > I am deploying my Flink Job in HA application mode, Whenever I redeploy my > job, or deploy an updated version of the job, it's using the

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

Hey Hang, I am deploying my Flink Job in HA application mode, Whenever I redeploy my job, or deploy an updated version of the job, it's using the same job_id. I haven't configured anywhere to use a fixed job id, I think it's doing it by default. Can you share where I can configure this? I tried

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

Hi Weihua, I am deploying my flink job in HA application mode on a kubernetes cluster. I am using an external nfs mount for storing checkpoints. For some reason, whenever I deploy an updated version of my application, it uses the same job_id for the new job as for the previous job. Thus the flink

Flink Job Failure for version 1.16

I have recently migrated from 1.13.6 to 1.16.1, I can see there is a performance degradation for the Flink pipeline which is using Flink's managed state ListState, MapState, etc. Pipelines are frequently failing with the Exception: 06:59:42.021 [Checkpoint Timer] WARN

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

t;> new job will fail when it tries to generate a new checkpoint. >> I'd like to suggest you to use different JOB_ID for different jobs, or >> set a different checkpoint dir for a new job. >> >> On Tue, May 9, 2023 at 9:38 PM amenreet sodhi >> wrote: >>

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

use different JOB_ID for different jobs, or set > a different checkpoint dir for a new job. > > On Tue, May 9, 2023 at 9:38 PM amenreet sodhi wrote: > >> Hi all, >> >> Is there any way to prevent restart of flink job, or override the >> checkpoint metadata, if for

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

you to use different JOB_ID for different jobs, or set a different checkpoint dir for a new job. On Tue, May 9, 2023 at 9:38 PM amenreet sodhi wrote: > Hi all, > > Is there any way to prevent restart of flink job, or override the > checkpoint metadata, if for some reason there exists

Flink Job Restarts if the metadata already exists for some Checkpoint

Hi all, Is there any way to prevent restart of flink job, or override the checkpoint metadata, if for some reason there exists a checkpoint by same name. I get the following exception and my job restarts, have been trying to find solution for a very long time but havent found anything useful yet

Re: Flink Job across Data Centers

nters? >> >> I am trying to create a Multi site/Geo Replicated Kafka cluster. I want >> that my Flink job to be closely colocated with my Kafka multi site cluster. >> If the Flink job is bound to a single data center, I believe we will >> observe a lot of client latency by

Re: Flink job manager conditional start of flink jobs

. I have to make sure only >> one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running >> flink jobs, later planned to move them to ns2. Now when I start in ns2, I >> can make an api call to ns1 jobmanager about running jobs and if no jobs >> then only I sh

Re: Flink job manager conditional start of flink jobs

Say initially ns1 is running flink > jobs, later planned to move them to ns2. Now when I start in ns2, I can > make an api call to ns1 jobmanager about running jobs and if no jobs then > only I should start in ns2. I can introduce this logic inside the flink job > java main method where my

Re: Flink job manager conditional start of flink jobs

to move them to ns2. Now when I start in ns2, I can make an api call to ns1 jobmanager about running jobs and if no jobs then only I should start in ns2. I can introduce this logic inside the flink job java main method where my total streaming logic present. So if I identify then, how can I stop

Re: Flink job manager conditional start of flink jobs

nks for your email. > I am looking more in terms of running these flinkk jobs in multi names > pace environment and make sure only one namespace flink jobs are running. > So on the Job manager when i try to start a flink job, it has to check if > it's allowed to run in this namespace or not

Re: Flink Job across Data Centers

ss data centers? > > I am trying to create a Multi site/Geo Replicated Kafka cluster. I want > that my Flink job to be closely colocated with my Kafka multi site cluster. > If the Flink job is bound to a single data center, I believe we will > observe a lot of client latency

Flink Job across Data Centers

Hi, Can anyone share any experience on running Flink jobs across data centers? I am trying to create a Multi site/Geo Replicated Kafka cluster. I want that my Flink job to be closely colocated with my Kafka multi site cluster. If the Flink job is bound to a single data center, I believe we

Re: Flink job manager conditional start of flink jobs

Thanks for your email. I am looking more in terms of running these flinkk jobs in multi names pace environment and make sure only one namespace flink jobs are running. So on the Job manager when i try to start a flink job, it has to check if it's allowed to run in this namespace

Re: Flink job manager conditional start of flink jobs

at 10:58 AM naga sudhakar wrote: > Hi Team, > Greetings!! > Just wanted to know when job manager or task manager is being restarted, > is there a way to run the existing flink jobs based on a condition? Same > query when I am starting flink job fresh also. > > Please le

Flink job manager conditional start of flink jobs

Hi Team, Greetings!! Just wanted to know when job manager or task manager is being restarted, is there a way to run the existing flink jobs based on a condition? Same query when I am starting flink job fresh also. Please let me know if any more information is required from my side. Thanks

Re:Re: Re: flink on yarn关于yarn尝试重启flink job问题咨询

quot; 写道： >> >Hi, >> > >> >图片看不到了 >> > >> >按照这个配置，YARN 应该只会拉起 10 次 JobManager。 >> > >> >Best, >> >Weihua >> > >> > >> >On Mon, Mar 13, 2023 at 3:32 PM guanyq wrote: >> > >> >

Re: Re: flink on yarn关于yarn尝试重启flink job问题咨询

> > >On Mon, Mar 13, 2023 at 3:32 PM guanyq wrote: > > > >> flink1.10版本，flink配置如下 > >> yarn.application-attempts = 10 (yarn尝试启动flink job的次数为10) > >> 正常我理解yarn会尝试10次启动flink job，如果起不来应该就会失败，但是在yarn应用页面看到了尝试11次，如下图 > >> 请问appattempt_1678102326043_0006_000409 > >> <http://192.168.63.12:8088/cluster/appattempt/appattempt_1678102326043_0006_000409> > >> 每个序号不是代表一次尝试么 > >> > >

Re:Re: flink on yarn关于yarn尝试重启flink job问题咨询

图片在附件 但是实际却是超过了10次。。 在 2023-03-13 15:39:39，"Weihua Hu" 写道： >Hi, > >图片看不到了 > >按照这个配置，YARN 应该只会拉起 10 次 JobManager。 > >Best, >Weihua > > >On Mon, Mar 13, 2023 at 3:32 PM guanyq wrote: > >> flink1.10版本，flink配置如下 >> yarn.app

Re: flink on yarn关于yarn尝试重启flink job问题咨询

Hi, 图片看不到了 按照这个配置，YARN 应该只会拉起 10 次 JobManager。 Best, Weihua On Mon, Mar 13, 2023 at 3:32 PM guanyq wrote: > flink1.10版本，flink配置如下 > yarn.application-attempts = 10 (yarn尝试启动flink job的次数为10) > 正常我理解yarn会尝试10次启动flink job，如果起不来应该就会失败，但是在yarn应用页面看到了尝试11次

flink on yarn关于yarn尝试重启flink job问题咨询

flink1.10版本，flink配置如下 yarn.application-attempts = 10 (yarn尝试启动flink job的次数为10) 正常我理解yarn会尝试10次启动flink job，如果起不来应该就会失败，但是在yarn应用页面看到了尝试11次，如下图 请问appattempt_1678102326043_0006_000409每个序号不是代表一次尝试么

Re: Flink Job Manager Recovery from EKS Node Terminations

First, JobManager does not store any persistent data to local when the Kubernetes HA + S3 used. It means that you do not need to mount a PV for JobMananger deployment. Secondly, node failures or terminations should not cause the CrashLoopBackOff status. One possible reason I could imagine is a

Flink Job Manager Recovery from EKS Node Terminations

Hi, Have a query on the Job Manager HA for flink 1.15. We currently run a standalone flink cluster with a single JobManager and multiple TaskManagers, deployed on top of a kubernetes cluster (EKS cluster) in application mode (reactive mode). The Task Managers are deployed as a ReplicaSet and

Re: How to get failed streaming Flink job log in Flink Native K8s mode?

I think you might need a sidecar container or daemonset to collect the Flink logs and store into a persistent storage. You could find more information here[1]. [1]. https://www.alibabacloud.com/blog/best-practices-of-kubernetes-log-collection_596356 Best, Yang hjw 于2022年12月22日周四 23:28写道： > On

How to get failed streaming Flink job log in Flink Native K8s mode?

On Flink Native K8s mode, the pod of JM and TM will disappear if the streaming job failed.Are there any ways to get the log of the failed Streaming job? I only think of a solution that is to mount job logs to NFS for persistence through pv-pvc defined in pod-template. ENV: Flink version:1.15.0

Re: Query about flink job manager dashboard

nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-options-for-flink-web-ui >>>> >>>> Best regards, >>>> >>>> Martijn >>>> >>>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>>> wrote: >

Re: Query about flink job manager dashboard

>>> Best regards, >>> >>> Martijn >>> >>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>> wrote: >>> >>>> Hi Team, >>>>> Greetings!!! >>>>> I am a software developer using apache flink an

Re: Query about flink job manager dashboard

ries about flink job manager dashboard. Can you please help with below? 1) is it possible to add login mechanism for the flink job manager dash board and have a role based mechanism for viewing ru

Re: Query about flink job manager dashboard

3qiRrQTWoM_EbECDFr62njjFlxqyUFfLFS6GofOfFjNbK9HcXrwyq0w$> >>> >>> Best regards, >>> >>> Martijn >>> >>> On Thu, Nov 24, 2022 at 3:44 AM naga sudhakar >>> wrote: >>> >>>> Hi Team, >>>>> Greeting

Re: Query about flink job manager dashboard

two queries about flink job manager dashboard. Can you please help with below? 1) is it possible to add login mechanism for the flink job manager dash board and have a role based mechanism for viewing running jobs, cancelling jobs, ad

Re: Query about flink job manager dashboard

akar >> wrote: >> >>> Hi Team, >>>> Greetings!!! >>>> I am a software developer using apache flink and deploying flink jobs >>>> using the same. I have two queries about flink job manager dashboard. Can >>>> you please hel

Re: Query about flink job manager dashboard

;> I am a software developer using apache flink and deploying flink jobs >>> using the same. I have two queries about flink job manager dashboard. Can >>> you please help with below? >>> >>> 1) is it possible to add login mechanism for the flink job manager das

Re: Query about flink job manager dashboard

ying flink jobs >>> using the same. I have two queries about flink job manager dashboard. Can >>> you please help with below? >>> >>> 1) is it possible to add login mechanism for the flink job manager dash >>> board and have a role based mechanism for vi

Re: Query about flink job manager dashboard

, 2022 at 3:44 AM naga sudhakar wrote: > Hi Team, >> Greetings!!! >> I am a software developer using apache flink and deploying flink jobs >> using the same. I have two queries about flink job manager dashboard. Can >> you please help with below? >> >> 1

Query about flink job manager dashboard

> > Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manag

Query about flink job manager dashboard

> Hi Team, > Greetings!!! > I am a software developer using apache flink and deploying flink jobs > using the same. I have two queries about flink job manager dashboard. Can > you please help with below? > > 1) is it possible to add login mechanism for the flink job manager

Re: support escaping `#` in flink job spec in Flink-operator

o flink-core. However, I think we worry that this might be > breaking some applications which rely on the lenient behavior of the > existing parser. > > -Max > > On Tue, Nov 8, 2022 at 12:36 PM liuxiangcao > wrote: > >> Hi Yang, >> >> Do you think flin

Re: support escaping `#` in flink job spec in Flink-operator

`#` in FLINK-15358[1] and Flink > job spec not supporting `#` are caused by some common code? or maybe they > are in different code paths? My first guess was they are in different > code paths. The flink-conf is parsed when starting the flink cluster while > job spec is parsed when star

Re: support escaping `#` in flink job spec in Flink-operator

Hi Yang, Do you think flink-conf not supporting `#` in FLINK-15358[1] and Flink job spec not supporting `#` are caused by some common code? or maybe they are in different code paths? My first guess was they are in different code paths. The flink-conf is parsed when starting the flink cluster

Re: support escaping `#` in flink job spec in Flink-operator

Hi Gyula, Thanks for getting back. Could you share how to submit job to flinkk8operator in json format? We use the java Fabric8 K8 client, which serializes java FlinkDeployment objects to CustomResource YAML (see the code snippet below). Since `#` is considered a special character denoting

Re: support escaping `#` in flink job spec in Flink-operator

This is a known limit of the current Flink options parser. Refer to FLINK-15358[1] for more information. [1]. https://issues.apache.org/jira/browse/FLINK-15358 Best, Yang Gyula Fóra 于2022年11月8日周二 14:41写道： > It is also possible that this is a problem of the Flink native Kubernetes >

Re: support escaping `#` in flink job spec in Flink-operator

It is also possible that this is a problem of the Flink native Kubernetes integration, we have to check where exactly it goes wrong before we try to fix it . We simply set the args into a Flink config and pass it to the native deployment logic in the operator. Gyula On Tue, 8 Nov 2022 at 07:37,

Re: support escaping `#` in flink job spec in Flink-operator

Hi! How do you submit your yaml? It’s possible that this is not operator problem. Did you try submitting the deployment in json format instead? If it still doesn't work please open a JIRA ticket with the details to reproduce and what you have tried :) Cheers Gyula On Tue, 8 Nov 2022 at 04:56,

support escaping `#` in flink job spec in Flink-operator

Hi, We have a job that contains `#` as part of mainArgs and it used to work on Ververica. Now we are switching to our own control plane to deploy to flink-operaotor and the job started to fail due to the main args string getting truncated at `#` character when passed to flink application. I

Re: 请教：关于如何释放 Flink Job 中某个对象持有的资源

在不同的线程或者进程的话，设计上，就不要用同一个EsClientHolder，各个不同阶段各自去new和close对象， > > > Thanks > > > 在 2022-07-12 12:35:31，"Bruce Zu" 写道： > > Flink team好， > > 我有一个很一般的问题，关于如何释放 Flink Job 中某个对象持有的资源。 > > > > 我是 Flink 的新用户。我搜索了很多，

Re:请教：关于如何释放 Flink Job 中某个对象持有的资源

Hi, 如果是访问ES的话，Flink里面自带ES的connector，你可以直接使用，或者参考源码，source和sink接口都有对应的方法 资源是否在一个线程里面，这个取决与你代码逻辑，如果在不同的线程或者进程的话，设计上，就不要用同一个EsClientHolder，各个不同阶段各自去new和close对象， Thanks 在 2022-07-12 12:35:31，"Bruce Zu" 写道： > Flink team好， > 我有一个很一般的问题，关于如何释放 Flink Job 中某个对象持有的资源。 > > 我是 F

请教：关于如何释放 Flink Job 中某个对象持有的资源

Flink team好， 我有一个很一般的问题，关于如何释放 Flink Job 中某个对象持有的资源。 我是 Flink 的新用户。我搜索了很多，但没有找到相关文件。但我确信有一个标准的方法来解决它。 我的Flink 应用程序中需要访问 Elasticsearch 服务器。我们使用从 org.elasticsearch.client.RestHighLevelClient 扩展而来的类 EsClient 来完成查询工作， 一旦不再使用它就需要调用它的`close`方法来释放资源。 所以我需要找到合适的地方来确保资源总是可以被释放，即使在调用的某个地方发生了一些异常 我现在能

Re: Is there an HA solution to run flink job with multiple source

, we would like to process it's data. We use a single flink job, since we find it easier to manage less flink jobs, and that way we also use less resources So far, the idea from Xuyang seems doable to me, I'll explore the idea of subclassing existing Kafka source and making sure that kafka source can

  1   2   3   4   5   6   7   8   9   >