potential software engineering problems

Hi, everyone. I'm a graduate student on software engineering. I focus on Flink a few months. I'm really curious about what software engineering problems (e.g., testing, debugging, etc.) hinder your progress during developing Flink applications. Any response/suggestions I will appreciate. Thank

No activity but checkpoints are failing and backpressure is high

Hi everyone, We have a small QA environment with just one job manager and one task manager. There are several jobs running with parallelism 1. There is a problem with one job. During our regular upgrade process one job wasn't cancelled due to savepoint timeout: Cancelling job

Re: Old job resurrected during HA failover

Hi Elias and Vino, sorry for the late reply. I think your analysis is pretty much to the point. The current implementation does not properly respect the situation with multiple standby JobManagers. In the single JobManager case, a loss of leadership either means that the JobManager has died and,

[ANNOUNCE] Weekly community update #31

Dear community, this is the weekly community update thread #31. Please post any news and updates you want to share with the community to this thread. # Flink 1.5.2 has been released The community released the second bug fix release Flink 1.5.2 [1]. The release contains more than 20 improvements

Re: Old job resurrected during HA failover

Till, Thoughts? On Wed, Aug 1, 2018 at 7:34 PM vino yang wrote: > Your analysis is correct, yes, in theory the old jobgraph should be > deleted, but Flink currently uses the method of locking and asynchronously > deleting Path, so that it can not give you the acknowledgment of deleting, > so

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, -yt,--yarnship Ship files in the specified directory (t for transfer) I guess that you even got a warning in your log files: LOG.warn("Ship directory is not a directory. Ignoring it.”); I’m not sure, but maybe with `-yt` you do

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

> 在 2018年8月3日，19:03，Joshua Fan 写道： > > Hi, > > I'd like to submit a job with dependency jars by flink run, but it failed. > > Here is the script, > > /usr/bin/hadoop/software/flink-1.4.2/bin/flink run \ > -m yarn-cluster -yn 1 -ys 8 -yjm 2148 -ytm 4096 -ynm jarsTest \ > -c StreamExample \

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, Are those paths: file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing) file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar' (missing) accessible from the inside of your container? bin/flink run --help (…) -C,--classpath Adds a URL to each

How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, I'd like to submit a job with dependency jars by flink run, but it failed. Here is the script, /usr/bin/hadoop/software/flink-1.4.2/bin/flink run \ -m yarn-cluster -yn 1 -ys 8 -yjm 2148 -ytm 4096 -ynm jarsTest \ -c StreamExample \ -C file:/home/work/xxx/lib/commons-math3-3.5.jar \ -C

Re: Could not retrieve the redirect address - No REST endpoint has been started

Hi Pedro, it looks as if Flink could not start the WebRuntimeMonitor. Could you maybe share the jobmanager.log of the newly started JobManager which has no WebRuntimeMonitor started with us? Maybe there is a port conflict. Cheers, Till On Thu, Aug 2, 2018 at 11:35 AM vino yang wrote: > Hi

Re: Delay in REST/UI readiness during JM recovery

Hi Joey, your analysis is correct. Currently, the Dispatcher will first try to recover all jobs before it confirms the leadership. 1) The Dispatcher provides much of the relevant information you see in the web-ui. Without a leading Dispatcher, the web-ui cannot show much information. But this

Re: Converting a DataStream into a Table throws error

Apologies should read Vino and Timo Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Converting a DataStream into a Table throws error

Thanks a lot Timo. I will try the changes suggested. Appreciated Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *