Re: improper use of releaseJob() without a matching number of createTmpFiles() calls for jobId

2019-01-22 Thread Chang Liu
of passing it as a method argument. Does it have anything to do with the class Scenario? Many Thanks. Best regards/祝好, Chang Liu 刘畅 > On 22 Jan 2019, at 16:34, Chang Liu wrote: > > Hi Stefan, > > Thanks. I am using 1.6.0. I will upgrade to 1.6.1 and see whether the problem >

Re: improper use of releaseJob() without a matching number of createTmpFiles() calls for jobId

2019-01-22 Thread Chang Liu
Hi Stefan, Thanks. I am using 1.6.0. I will upgrade to 1.6.1 and see whether the problem remains. Best regards/祝好, Chang Liu 刘畅 > On 22 Jan 2019, at 16:10, Stefan Richter wrote: > > Hi, > > Which version of Flink are you using? This issue > https://issues.apache.org

Re: improper use of releaseJob() without a matching number of createTmpFiles() calls for jobId

2019-01-22 Thread Chang Liu
]) extends MapFunction[Indicators, Evaluation] { override def map(inds: Indicators): Evaluation = Evaluation(indicators.id <http://indicator.id/>, evaluateScenarios(indicators, scenarios)) } Best regards/祝好, Chang Liu 刘畅 > On 22 Jan 2019, at 13:33, Chang Liu wrote: > > Ok,

Re: improper use of releaseJob() without a matching number of createTmpFiles() calls for jobId

2019-01-22 Thread Chang Liu
would like to have a configurable scenarios which I can load from config file instead of a static object variable. Any idea why this is happening? I also have other codes where I am also passing arguments and use them as part of my data flow and they are just working fine. Many Thanks. Best regards

improper use of releaseJob() without a matching number of createTmpFiles() calls for jobId

2019-01-22 Thread Chang Liu
r, it will work just fine. But my last operator is justing doing some map operation. I am wondering what could be the cause of this issue? Many Thanks :) Best regards/祝好, Chang Liu 刘畅

Re: question on Flink memory management

2018-12-19 Thread Chang Liu
he jobs running there. If I cancel the job, the memory utilization should be dropped, right? Best regards/祝好, Chang Liu 刘畅 > On 19 Dec 2018, at 13:50, Chang Liu wrote: > > Hi Dawid, > > Thanks for your reply. > > Then, we can just ignore the link and just focus o

Re: question on Flink memory management

2018-12-19 Thread Chang Liu
Hi Dawid, Thanks for your reply. Then, we can just ignore the link and just focus on the questions. Best regards/祝好, Chang Liu 刘畅 > On 19 Dec 2018, at 12:18, Dawid Wysakowicz wrote: > > Hi Chang, > > The link you've posted describes only the Batch execution. It does

question on Flink memory management

2018-12-19 Thread Chang Liu
e same (if there is not off-heap memory used or there is no OOM)? Many Thanks. Best regards/祝好, Chang Liu 刘畅

Re: problem submitting job, it hangs there

2018-12-19 Thread Chang Liu
Many Thanks :) Best regards/祝好, Chang Liu 刘畅 > On 14 Dec 2018, at 11:09, Tzu-Li Chen wrote: > > Hi Chang, > > I think there is a JIRA[1] aimed at harden this case. > > In fact Flink create this directory on started and without other warnings, > we can assume that

Re: problem submitting job, it hangs there

2018-12-14 Thread Chang Liu
, Chang Liu 刘畅 > On 14 Dec 2018, at 10:23, Chang Liu wrote: > > Hi Chesnay, > > What do you mean by "...we can make a small adjustment to the code…"? Do you > mean I, as a flink application developer, can do this in my code, OR, it has > to be a code change in

Re: problem submitting job, it hangs there

2018-12-14 Thread Chang Liu
because I cannot just manually create such directory in Production. Many thanks :) Best regards/祝好, Chang Liu 刘畅 > On 13 Dec 2018, at 14:51, Chesnay Schepler wrote: > > The directory is automatically created when Flink is started; maybe it was > deleted by some cleanup process? &

Re: problem submitting job, it hangs there

2018-12-13 Thread Chang Liu
automatically. Any ideas? Many Thanks. Best regards/祝好, Chang Liu 刘畅 > On 13 Dec 2018, at 12:01, Chang Liu wrote: > > Dear all, > > I am trying to submit a job but it got stuck here: > > ... > 2018-12-13 10:43:11,476 INFO > org.apache.flink.configuration.GlobalConfigu

problem submitting job, it hangs there

2018-12-13 Thread Chang Liu
solving this issue? Many Thanks :) Best regards/祝好, Chang Liu 刘畅

Assign IDs to Operators

2018-11-21 Thread Chang Liu
) .window() .apply() Many Thanks. Best regards/祝好, Chang Liu 刘畅

Re: flink job restarts when flink cluster restarts?

2018-11-08 Thread Chang Liu
Thanks! If I have a cluster more than one node (standalone or YRAN), can I stop and start any single node among them and keep the job running? Best regards/祝好, Chang Liu 刘畅 > On 7 Nov 2018, at 16:17, 秦超峰 <18637156...@163.com> wrote: > > the second > > > >

Re: flink job restarts when flink cluster restarts?

2018-11-08 Thread Chang Liu
Or to say, how can I keep the jobs for system patching, server restart, etc. Is it related to Standalone vs YARN? Or is it related to whether to use Zookeeper? Many thanks! Best regards/祝好, Chang Liu 刘畅 > On 8 Nov 2018, at 13:38, Chang Liu wrote: > > Thanks! > > If I have

flink job restarts when flink cluster restarts?

2018-11-07 Thread Chang Liu
in advance :) Best regards/祝好, Chang Liu 刘畅

Re: Question about Window Tigger

2018-09-25 Thread Chang Liu
this: ExecutionConfig.setAutoWatermarkInterval(…) ? Many thanks :) Best regards/祝好, Chang Liu 刘畅 > On 25 Sep 2018, at 23:41, Till Rohrmann wrote: > > AssignerWithPeriodicWatermarks

Re: Question about Window Tigger

2018-09-25 Thread Chang Liu
Hi Rong, Thanks for your reply. :) Best regards/祝好, Chang Liu 刘畅 > On 19 Sep 2018, at 18:20, Rong Rong wrote: > > Hi Chang, > > There were some previous discussion regarding how to debug watermark and > window triggers[1]. > Basically if there's no data for some part

Question about Window Tigger

2018-09-19 Thread Chang Liu
, which could be coming after a long time. Do you have any idea how can I do this? Many Thanks :) Best regards/祝好, Chang Liu 刘畅

Re: questions about YARN deployment and HDFS integration

2018-09-19 Thread Chang Liu
Thanks for your answers :) Best regards/祝好, Chang Liu 刘畅 > On 17 Sep 2018, at 17:25, Kostas Kloudas wrote: > > Hi Chiang, > > Some of the answers you can find in line: > >> On Sep 17, 2018, at 3:47 PM, Chang Liu > <mailto:fluency...@gmail.com>> wrote:

Re: org.apache.flink.util.FlinkException: Could not cancel job

2018-09-06 Thread Chang Liu
You are correct. Thanks! I misused the job ID. Sorry for bothering you guys. Best regards, Chang from iPhone > On 4 Sep 2018, at 18:06, Chesnay Schepler wrote: > > Please check that the job ID is correct. > >> On 04.09.2018 15:48, Chang Liu wrote: >> Dear All, &g

org.apache.flink.util.FlinkException: Could not cancel job

2018-09-04 Thread Chang Liu
(RestClient.java:225) at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:209) at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ... 5 more Best regards/祝好, Chang Liu 刘畅

Re: How to do test in Flink?

2018-08-24 Thread Chang Liu
No worries, I found it here: org.apache.flink flink-runtime_${scala.binary.version} ${flink.version} test-jar test Best regards/祝好, Chang Liu 刘畅 On Fri, Aug 24, 2018 at 1:16 PM Chang Liu wrote: > Hi Hequn, > > I have added the following dep

Re: How to do test in Flink?

2018-08-24 Thread Chang Liu
/runtime/operators/testutils/MockEnvironmentBuilder Do you know which library contains this class? Thanks :) Best regards/祝好, Chang Liu 刘畅 DevOps Engineer WB TECH / Cyber Crime Prevention Team Mobile: +31(0)687859981 Email: fluency...@gmail.com & chang.l...@ing.nl On Mon, Aug 13, 2018

Re: How to do test in Flink?

2018-08-13 Thread Chang Liu
-java_2.11 flink-connector-kafka-0.11_2.11 Best regards/祝好, Chang Liu 刘畅 > On 13 Aug 2018, at 04:01, Hequn Cheng wrote: > > Hi Chang, > > There are some harness tests which can be used to test your function. It is > also a common way to test function or operator in fl

Re: How to do test in Flink?

2018-08-13 Thread Chang Liu
Hi Dawid, Many Thanks :) Best regards/祝好, Chang Liu 刘畅 > On 13 Aug 2018, at 09:21, Dawid Wysakowicz wrote: > > Hi Chang, > > Just to add to how you could test the function you've posted. The Collector > is an interface so you can just implement a stub that will keep the

Re: How to do test in Flink?

2018-08-13 Thread Chang Liu
-testing <https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/testing.html#integration-testing>) right ? Harness tests provide more easy to use utils to construct source/sink and the execution environment. Best regards/祝好, Chang Liu 刘畅 > On 13 Aug 2018, at 04:01, He

How to do test in Flink?

2018-08-12 Thread Chang Liu
u test this WindowFunction below? The return type is Unit right? We cannot do unit test on like, like how the ReduceFunction was tested in the example link above. Then we only have the option of doing integration testing on it? Your ideas would be very helpful :) Thanks in advance ! Best regards/祝好,

Re: Pass JVM option (-Dconfig.file) per job in standalone mode?

2018-07-31 Thread Chang Liu
ead of val I have to call setup every time I have some other object or vals which initialised depending on the parameters. If the parameters are passed in as program args instead of from JVM option, these object vals cannot be initialised properly. And ideas about it? Many Thanks :) Best reg

Pass JVM option (-Dconfig.file) per job in standalone mode?

2018-07-31 Thread Chang Liu
Dear all, I would like to know is there a way to pass JVM options (for example, -Dconfig.file=application.conf) for each submitted flink job? I am using the Config library from lightbend. ./bin/flink run examples/Example.jar -Dconfig.file=/path/application.conf Best regards/祝好, Chang Liu 刘畅

Re: Questions on Unbounded number of keys

2018-07-30 Thread Chang Liu
aring them carefully). I am thinking about the State objects themselves, which I have no idea what is happening to them and what will happen to them. Many thanks :) Best regards/祝好, Chang Liu 刘畅 > On 26 Jul 2018, at 10:55, Andrey Zagrebin wrote: > > Hi Chang Liu, > > The u

Re: Questions on Unbounded number of keys

2018-07-25 Thread Chang Liu
instances are created, maintained and destroyed. That’s why I would like your help understanding this. Many Thanks. Best regards/祝好, Chang Liu 刘畅 > On 24 Jul 2018, at 14:31, Till Rohrmann wrote: > > Hi Chang Liu, > > if you are dealing with an unlimited number of keys and ke

Questions on Unbounded number of keys

2018-07-24 Thread Chang Liu
you help me understand WHY? Let’s also assume that, we are regularly clearing the KeyedState, so the memory used by the State will not explode. Many Thanks and Looking forward to your reply :) Best regards/祝好, Chang Liu 刘畅