Re: Flink 1.5 job distribution over cluster nodes

For example state size / logs etc all these are now one one physical node ( or couple rather then spread out ) for that one pipe where we desire to have a large state. We have decreased the slots ( pretty artificial a set up ) to give a node less stress. I wish there was a RFC from the wider

Re: Why data didn't enter the time window in EventTime mode

Hi Soheil, > wait 8 milliseconds (according to my code) to see if any other data with the same key received or not and after 8 millisecond it will be triggered. Yes, but the time is event time, so if there is no data from source the time won't be advanced. There are some reasons why the event

Re: Question about the behavior of TM when it lost the zookeeper client session in HA mode

I just stumbled on this same problem without any associated ZK issues. We had a Kafka broker fail that caused this issue: 2018-07-18 02:48:13,497 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Sink: Produce: (2/4) (7e7d61b286d90c51bbd20a15796633f2) switched from

Re: Flink resource manager unable to connect to mesos after restart

Hi, If you are able to re-produce this reliably, can you post the jobmanager logs? Best, Gary On Wed, Jul 18, 2018 at 10:33 AM, Renjie Liu wrote: > Hi, all: > > I'm testing flink 1.5.0 and find that flink mesos resource manager unable > to connect to mesos after restart. Have you seen this

Re: When a jobmanager fails, it doesn't restart because it tries to restart non existing tasks

Hi Gerard, There is an issue recently fixed for 1.5.2, 1.6.0: https://issues.apache.org/jira/browse/FLINK-9575 It might have caused your problem. Can you please provide log from JobManager/Entry point for further investigation? Cheers, Andrey

Race between window assignment and same window timeout

Hi, It seems like we encounter a race situation between the aggregation thread and the Time Trigger thread. It might not be a bug, but it still seems strange to us, and we would like your help to fix it/work around it please. First, few descriptions about our use case and system: · We

Re: Elasticsearch 6.3.x connector

To Gordon: +1 for your efforts, considering ElasticSearch is a very popular full-text search and analysis engine. If it could be merged into 1.6 version, that would be very good to the community. Thanks, vino. 2018-07-18 17:42 GMT+08:00 Tzu-Li (Gordon) Tai : > Hi Miki, > > The latest stable

Re: Cannot configure akka.ask.timeout

Hello, does anybody have an idea what is going on? I have not yet found a solution. Am I doing something wrong? Or is the 'akka.ask.timeout' parameter not related to the exception stated below? Could somebody please take a look at this? More details can be found in the message prior to this.

Re: FlinkCEP and scientific papers ?

You are right Vino, the initial implementation was based on the above mentioned paper. Cheers, Till On Tue, Jul 17, 2018 at 5:34 PM vino yang wrote: > Hi Esa, > > AFAIK, the earlier Flink CEP refers to the Paper 《Efficient Pattern > Matching over Event Streams》[1]. Flink absorbed two major

Why data didn't enter the time window in EventTime mode

Hi, In a datastream processing problem, the source generated data every 8 millisecond and timestamp is a field of the data. In default Flink time behavior data enter the time window but when I set Flink time to EventTime it will output nothing! Here is the code:

Re: When a jobmanager fails, it doesn't restart because it tries to restart non existing tasks

Hi Gerard, >From you provide information, you mean the path in Zookeeper "/jobgraphs" exists more jobs than you submitted? And can not be restarted because blob files can not be find? Can you provide more details, about the stack trace, log and which version of Flink? Normally, the jobgraph can

Re: Flink CLI properties with HA

Vino, I'm not getting any error but my suspicion was that if I dont specify this `high-availability.storageDir` property in flink CLI side then the CLI will not be able to submit job to flink cluster(HA enabled). But if provide this property in CLI side the job submission will be successful even

Re: When a jobmanager fails, it doesn't restart because it tries to restart non existing tasks

Hi vino, Seems that jobs id stay in /jobgraphs when we cancel them manually. For example, after cancelling the job with id 75e16686cb4fe0d33ead8e29af131d09 the entry is still in zookeeper's path /flink/default/jobgraphs, but the job disappeared from /home/nas/flink/ha/default/blob/. That is the

Flink resource manager unable to connect to mesos after restart

Hi, all: I'm testing flink 1.5.0 and find that flink mesos resource manager unable to connect to mesos after restart. Have you seen this happenen? -- Liu, Renjie Software Engineer, MVAD

Flink 1.5 job distribution over cluster nodes

Hi, We have 4 jobs with parallelism 3 that are running over 3 task manager with 4 slots per each , each task manager runs on a different VM , On Flink 1.3.2 the jobs were evenly distributed per node each job took one task slot of each task manager . After upgrading to flink 1.5 , each job is

Re: Cannot configure akka.ask.timeout

Hi Lukas >From your first two steps' description ("started this in Intellij") and the >exception log, I think you run your program locally within Intellij with >LocalStreamEnvironment. You can view the configuration related code from

Window Stream - Need assistance

Dear Friends, I have 2 streams of the below data types. DataStream> splittedActivationTuple; DataStream> unionReloadsStream; These streams are getting data from Kafka and getting data in different frequencies. "unionReloadsStream" will receive more data than "splittedActivationTuple".

Re: Serialization questions

Hi, Flavio > addDefaultKryoSerializer differs from registerTypeWithKryoSerializer because > addDefaultKryoSerializer use the passed serializer also for subclasses of the > configured class. Am I right? This is not very clear in the method's Javadoc… I think it is not exactly a problem with

Re: Window Stream - Need assistance

Hi Rakkesh, Did you call `execute()`on your `StreamExecutionEnvironment`? Best, Xingcan > On Jul 18, 2018, at 5:12 PM, Titus Rakkesh wrote: > > Dear Friends, > I have 2 streams of the below data types. > > DataStream> splittedActivationTuple; > > DataStream> unionReloadsStream; >

Re: Serialization questions

Hi Minglei, using the registerTypeWithKryoSerializer with the 3 classes works (without disableGenericTypes) but the problem is that I would like to avoid Kryo serialization if this is useful to speedup the job performance, and thus I'd like to be able to run all jobs with disableGenericTypes.

Re: Elasticsearch 6.3.x connector

Hi Miki, The latest stable version of the Elasticsearch connector, as of Flink 1.5.x, is Elasticsearch 5.x. As for Elasticsearch 6.x, there has been some PRs that has been open for a while and have already been discussed quite thoroughly [1] [2]. Till and I have talked about merging these for

Re: Window Stream - Need assistance

Thanks for the reply. I have called "env.execute()". But nothing getting printed. I have a doubt whether "implemented function" is correct with my "requirement". Please assist. On Wed, Jul 18, 2018 at 2:57 PM, Xingcan Cui wrote: > Hi Rakkesh, > > Did you call `execute()`on your

Re: Window Stream - Need assistance

Hi Rakkesh, As Xingcan said, the trigger is required by the window to FIRE, you can use time window (contains a inner trigger) or (ProcessFunction + State + Timer). Thanks, vino. 2018-07-18 21:44 GMT+08:00 Titus Rakkesh : > Thanks Xingcan. I specified as GlobalWindow since I am going to put

Re: Serialization questions

Hi Flavio, According to the current implementation of `disableGenericTypes`, the exception you get should be valid because Kryo still has to be used for `EntitonAtom` which might be classified as generic (non-serialisable by Flink). You cannot specify exceptions for this check at the moment.

Re: Serialization questions

Another way would be also to make `EntitonAtom` extend `org.apache.flink.types.Value` and implement `IOReadableWritable` using custom (Kryo) serialisation. > On 18 Jul 2018, at 18:27, Andrey Zagrebin wrote: > > Hi Flavio, > > According to the current implementation of `disableGenericTypes`,

Re: Race between window assignment and same window timeout

Hi Shay, I would suggest to try Allowed Lateness, like you mention 500 ms: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/operators/windows.html#allowed-lateness

Re: event time and late events - documentation

Hi Tovi, 1. approach 1 : change the watermark let it means the later time, window trigger early. 2. approach 2 : do not change the watermark, but specify the allow lateness for the window[1][2], this way can make window accept late element and re-fire the window after watermark. [1]:

Re: Cannot configure akka.ask.timeout

That stumped us too and I am not sure but could you set up web.timeout higher then t he default 10s. We had issues with timeouts on job submission and were advised to change web.timeout, job submission now being an RPC call.. please do let us know if that helps... On Wed, Jul 18, 2018, 5:11 AM

Re: Cannot configure akka.ask.timeout

Scratch that... that is a different exception On Wed, Jul 18, 2018, 8:40 AM Vishal Santoshi wrote: > That stumped us too and I am not sure but could you set up web.timeout > higher then t he default 10s. We had issues with timeouts on job submission > and were advised to change web.timeout,

Re: Flink 1.5 job distribution over cluster nodes

Answered in another mailing list Hi Shachar, with Flink 1.5 we added resource elasticity. This means that Flink is now able to allocate new containers on a cluster management framework like Yarn or Mesos. Due to these changes (which also apply to the standalone mode), Flink no longer reasons

Re: FlinkKafkaConsumer configuration to consume from Multiple Kafka Topics

Hi Sagar, At the moment number of partitions in Kafka source topics and parallelism of Flink Kafka source operator are completely independent. Flink will internally distribute partitions between a number of source parallel subtasks which you configure. In case of dynamic partition or topic

Re: Window Stream - Need assistance

Hi Rakkesh, The `GlobalWindow` is commonly used for custom window assignment and you should specify a `trigger` for it [1]. If the built-in window (e.g., tumbling window or sliding window) join in DataStream API fails to meet the requirements, you could try the time-windowed join in Table/SQL

Re: Keeping only latest row by key?

Hi James, the easiest solution for this bahavior is to use a user-defined LAST_VALUE aggregate function as discussed here [1]. I hope this helps. Regards, Timo [1]

Re: Keeping only latest row by key?

Hi James, There are over windows in Flink Table API: https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/tableApi.html#over-windows It should be possible to implement this

Re: Flink 1.5 job distribution over cluster nodes

I think there is something to be said about making this distribution more flexible. A stand alone cluster, still the distribution mechanism for many a installations suffers horribly with the above approach. A healthy cluster requires resources wot be used equitably is possible. I have some pipes

Re: Window Stream - Need assistance

Thanks Xingcan. I specified as GlobalWindow since I am going to put all the elements coming with splittedActivationTuple with a 24 hour expiry and then do operations on that when elements coming with stream "unionReloadsStream" (bigger set). On Wed, Jul 18, 2018 at 4:07 PM, Xingcan Cui wrote: >