Re: Cannot download Jars from S3 due to resource timestamp changed

Hi, Yan. we have met this problem too when using aliyun-pangu and have commented in FLINK-8801 but no response yet. I think most file systems including s3/s3a/s3n/azure/aliyun-oss etc can encounter this problem since they doesn’t implement FileSystem#setTimes but the PR in FLINK-8801 think they

Partitioning key range

Hi all, I partition DataStream (say dsA) with parallelism 2 and get KeyedStream (say ksA) with parallelism 2. Depending on my keys in dsA, one partition remains empty in ksA. For example when my keys are 10 and 20 in dsA, then both partitions in ksA are full. However, with keys 1000 and 1001, only

Cannot download Jars from S3 due to resource timestamp changed

Hi, I am running issues when trying to move from HDFS to S3 using Flink 1.6. I am getting an exception from Hadoop code: IOException("Resource " + sCopy + " changed on src filesystem (expected " + resource.getTimestamp() + ", was " + sStat.getModificationTime()); Digging into this, I f

Re: Source reinterpretAsKeyedStream

Thanks a lot for the replies. Below I paste my code: DataStreamSource source = env.addSource(new MySource()); KeyedStream keyedStream = DataStreamUtils.reinterpretAsKeyedStream(source, new DummyKeySelector(), TypeInformation.of(Integer.class) ); keyedStream.timeWindow(Tim

Re: [Discuss] Semantics of event time for state TTL

My 2c: Timestamp stored with the state value: Event timestamp Timestamp used to check expiration: Last emitted watermark That follows the event time processing model used elsewhere is Flink. E.g. events are segregated into windows based on their event time, but the windows do not fire until the

print() method does not always print on the taskmanager.out file

Hello, I am studying the parallelism of tasks on DataStream. So, I have configured Flink to execute on my machine (master node) and one virtual machine (worker node). The master has 4 cores (taskmanager.numberOfTaskSlots: 4) and the worker only 2 cores (taskmanager.numberOfTaskSlots: 2). I don't

[Discuss] Semantics of event time for state TTL

Hi All, As you might have already seen there is an effort tracked in FLINK-12005 [1] to support event time scale for state with time-to-live (TTL) [2]. While thinking about design, we realised that there can be multiple options for semantics of this feature, depending on use case. There is also so

Re: FlinkCEP and SQL?

> Should the all sources be combined into one big table before operations with > SQL CEP? Yes, you should combine them into one table/stream. Regards, Dian > 在 2019年4月4日，下午7:11，Esa Heikkinen (TAU) 写道： > > Hi > > Thank you for the information. How this SQL CEP is applicable for situation >

RE: FlinkCEP and SQL?

Hi Thank you for the information. How this SQL CEP is applicable for situation where there are many sources with different type of events ? Should the all sources be combined into one big table before operations with SQL CEP? BR Esa From: jincheng sun Sent: Thursday, April 4, 2019 1:05 PM To:

Re: FlinkCEP and SQL?

Hi BR Esa, CEP is available in Flink SQL, Please the detail here: https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/sql.html#pattern-recognition Best, Jincheng Esa Heikkinen (TAU) 于2019年4月4日周四 下午4:44写道： > Hi > > > > What is the situation of FlinkCEP and SQL? > > > > Is it alr

How to submit Flink program to Yarn without upload the fat jar?

Hi Experts, When submitting a Flink program to Yarn, the app jar( a fat jar about 200M with Flink dependencies ) will be uploaded to Yarn, which will take a lot of time. I check the code in CliFrontend, and found that there is a config item named “yarn.per-job-cluster.include-user-jar”,

FlinkCEP and SQL?

Hi What is the situation of FlinkCEP and SQL? Is it already possible to use SQL in CEP? Is there any example cases where SQL is used in CEP? BR Esa