Re: What happen to state in Flink Task Manager when crash?

With task-local recovery feature [1] flink will try to distribute the state/tasks so that the local snapshot can be reused. Hope that this clears things up. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#task-local-recovery On 12/01/

Re: Flink on Kubernetes (Minikube)

Hi Alexandru, This sounds reasonable that it might be because of this minikube command failed, but I am not a kubernetes expert. I cc Till who knows more on this. Best, Dawid On 19/12/2018 14:16, Alexandru Gutan wrote: > Thanks! > I'm using now the *flink:1.7.0-hadoop24-scala_2.12*

Re: Flink on Kubernetes (Minikube)

Hi, You used a hadoopless docker image, therefore it cannot find hadoop dependencies. It is ok if you don't need to use any, the bolded messages are just INFO, those are not errors. Best, Dawid On 19/12/2018 12:58, Alexandru Gutan wrote: > Dear all, > > I followed the instructions f

Re: question on Flink memory management

Hi Chang, The link you've posted describes only the Batch execution. It does not describe the streaming (which I think is what you use, as only there you have access to flink's state). It is also quite old and therefore I am not 100% sure how up-to-date it is. Best, Dawid On 19/12/2018 12:11

Re: Custom Metrics in Windowed AggregateFunction

(aggregateFunction, windowFunction) and register metrics in the windowFunction? Best, Dawid On 19/12/2018 04:30, Chirag Dewan wrote: > Hi, > > I am writing a Flink job for aggregating events in a window.  > > I am trying to use the /AggregateFunction/ implementation for this. 

Re: Kafka consumer, is there a way to filter out messages using key only?

of KeyedDeserializationSchema#deserialize could be Optional.empty() for invalid keys and Optional.of(deserializedValue) for valid keys. Best, Dawid On 18/12/2018 20:22, Hao Sun wrote: > Hi, I am using 1.7 on K8S. > > I have a huge amount of data in kafka, but I only need a tin

Re: How to migrate Kafka Producer ?

ls of both of the connectors. I pull also Piotr to the thread, maybe he can provide more straightforward workaround. Best, Dawid [1] https://github.com/king/bravo On 18/12/2018 14:33, Edward Rojas wrote: > Hi, > > I'm planning to migrate from kafka connector 0.11 to the new universal kafk

Re: CEP - Support for multiple pattern

s feature. But I think it > may be a very useful feature as it can eliminate the redundant network > transmission compared to multiple operators to support multiple patterns. You > can create an issue and we can discuss further about it on the JIRA page. CC > @Dawid > > Regard

Re: Apache Flink CEP how to detect if event did not occur within x seconds?

.html and http://training.data-artisans.com/exercises/CEP.html Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/libs/cep.html#handling-timed-out-partial-patterns [2] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/process_function.htm

Re: Apache Flink CEP how to detect if event did not occur within x seconds?

Hi Florin, I concur with Dian. If you have any other questions, please do not hesitate to ask. Best, Dawid On 13/12/2018 03:37, fudian.fd wrote: > Hi Florin, > > Are you using processing time or event time? The JIRA FLINK-7384 > allows to emit timed-out patterns without ha

Re: Is there an example of flink cluster "as a job" deployment on k8s ?

Hi Vishal, You might want to have a look at the flink-container/kubernetes module: https://github.com/apache/flink/tree/master/flink-container/kubernetes Best, Dawid On 05/12/2018 22:50, Vishal Santoshi wrote: > signature.asc Description: OpenPGP digital signature

Re: long lived standalone job session cluster in kubernetes

Hi Derek, I am not an expert in kubernetes, so I will cc Till, who should be able to help you more. As for the automation for similar process I would recommend having a look at dA platform[1] which is built on top of kubernetes. Best, Dawid [1] https://data-artisans.com/platform-overview

Re: not able to join data coming from kafka

Hi Rakesh, Could you explain a little bit what is the actual problem? What do you expect as the ouput and what actually happens? It is hard to guess what is the problem you're facing. Best, Dawid On 03/12/2018 12:19, Rakesh Kumar wrote: > > Hello Team, > > > public class Flin

Re: CKAN inputFormat (batch)

Hi Flavio, Thank you for the example. It is definitely gonna be helpful for some people! Best, Dawid On 04/12/2018 09:05, Flavio Pompermaier wrote: > Yesterday it was working...alternatively you can look at  > https://github.com/ckan/ckan-instances/blob/gh-pages/config/instance

Re: Flink CEP support pattern match involving fields of previous events

Hi Florin, This feature is supported with IterativeCondition since 1.3.0. For questions about API and what features are supported in general please always have a look into documentation[1] first. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/libs/cep.html

Re: Changes in Flink 1.6.2

. Best, Dawid On 30/11/2018 15:19, Boris Lublinsky wrote: > Dominik, > Any feedback on this? > > Boris Lublinsky > FDP Architect > boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com> > https://www.lightbend.com/ > >> On Nov 2

Re: FlinkCEP and scientific papers ?

, Dawid On 28/11/2018 10:06, Esa Heikkinen wrote: > > Hi > >   > > What is the situation of SQL/CEP ? > >   > > Flink 1.7 is coming soon, but what about SQL/CEP ? > >   > > Whether it already exists documentation of SQL/CEP ? > > Event better if ther

Re: kafka to hdfs flow avro schema evolution

to write avro records to parquet files. Best, Dawid On 27/11/2018 11:35, CPC wrote: > Hi everybody, > > We are planning to use flink for our kafka to hdfs ingestion. We are > consuming avro messages encoded as json and then writing them to hdfs > as parquet.  But our avro schema i

Re: Exception occurred while processing valve output watermark & NullPointerException

Hi, I think vino is right. It seems that the NullPointerException comes from your condition. Please add handling of the situation when the string that you are comparing is null. Best, Dawid On 21/11/2018 04:32, vino yang wrote: > Hi Steve, > > It seems the NPE caused by the

Re: Could not find previous entry with key.

Hi, Which version of flink do you use? I would recommend upgrading to 1.6.2 for which we reworked internal structures and such problem should not occur. Best, Dawid On 19/11/2018 08:30, Chesnay Schepler wrote: > @klou @dawid What is the most likely cause if this is not caused by an > im

Re: FlinkKafkaProducer and Confluent Schema Registry

Hi Olga, The only thing I can tell is that it definitely won't make it to 1.7 release. The earliest possible is 1.8 then, which is scheduled for the beginning of next year. Best, Dawid On 08/11/2018 00:48, Olga Luganska wrote: > Dawid, > > Is there a projected date to

Re: Flink weird checkpointing behaviour

Hi, I think it is definitely worth checking the alignment time as Yun Tang suggested. There were some changes in the network stack that could influence this behavior between those version. I've also added Stefan as cc, who might have more ideas what would be worth checking. Best, Dawid

Re: "org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files

Hi, Could you post the full stacktrace of the exception? Best, Dawid On 05/11/2018 09:19, wangziyu wrote: > Hi, > I use monitor Restful api ，“/jars/{jars}/run” to test my environment.The > exception is happend. > I did exactly that： > 1.I use “/jars/upload” to upload my

Re: sys.exist(1) led to standalonesession daemon closed

place in the code, which will result in the process executing this code to terminate. In general I would say calling System.exit is rather discouraged. Best, Dawid On 05/11/2018 06:38, Tony Wei wrote: > Hi, > > I used a scala library called scallop[1] to parse my job’s argument

Re: java.io.IOException: NSS is already initialized

Hi Hao, I am not sure, what might be wrong, but I've cc'ed Gary and Kostas who were recently working with S3, maybe they will have some ideas. Best, Dawid On 03/11/2018 03:09, Hao Sun wrote: > Same environment, new error. > I can run the same docker image with my local Mac, but

Re: Question about slot and yarn vcores

per taskamanger if you don't overwrite it with[2]), but each of the jvm will have multiple threads. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/runtime.html#task-slots-and-resources [2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops

Re: Starting a seperate Java process within a Flink cluster

to those user classes. Best, Dawid On 02/11/2018 10:34, Jeff Zhang wrote: > > The error is most likely due to classpath issue. Because classpath is > different when you running flink program in IDE and run it in cluster.  > > And starting another jvm process in SourceFunction doesn't

Re: Job manager UI improvement

, Dawid [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-State-monitoring-td17256.html On 02/11/2018 04:55, Yun Tang wrote: > Hi Michael > > You could view state size metrics in 'Checkpoints' UI tab[1], I think > the state size shown here could meet your n

Re: TaskManagers cannot contact JobManager in Kubernetes when JobManager HA is enabled

Hi John, Glad you resolved the issue. Also thanks for sharing the solution with ML! Best, Dawid On 01/11/2018 16:22, John Stone wrote: > I've managed to resolve the issue. With HA enabled, you will see this > message in the logs: > > 2018-11-01 13:38

Re: Question about serialization and performance

with same logic and events, but without state. Hope those pointers will help improve your job's performance. Best, Dawid On 31/10/2018 23:58, TechnoMage wrote: > In running tests of flink jobs we are seeing some that yield really good > performance (2.5M records in minutes) and

Re: akka timeout exception

Hi, Could you provide us with some more information? Which version of flink are you running? In which cluster setup? When does this exception occur? This exception says that request for status overview (no of taskmanagers, slots info etc.) failed. Best, Dawid On 31/10/2018 20:05, Anil wrote

Re: Flink cluster security conf.: keberos.keytab add to run yarn-cluster

Hi Marke, AFAIK Shuyi is right, there is no such option so far. Maybe you could do though is to extend the "flink" script to substitute those parameters in the file on each run, but I think it is a common practice to run flink jobs on yarn from a single service user. On 31/10/2018 19:52, Shuyi

Re: Flink 1.6, User Interface response time

Hi Oleksandr, Have you checked the jobmanager logs to see if there are any exceptions? What is the response code for request when it doesn't load? Best, Dawid On 31/10/2018 16:49, Oleksandr Nitavskyi wrote: > > Hello! > >   > > We are migrating the the last 1.6.2 version and

Re: Flink CEP Watermark Exception

step by step when the "IllegalStateException: Could not find previous entry with key" happens? The first two issues regarding CEP you've linked concern very old Flink version (1.0.3), CEP library was heavily reworked since then and I would not look for any similarities in those cases. Be

Re: Flink CEP Watermark Exception

e reworked the underlying structure for CEP - SharedBuffer. Best, Dawid On 30/10/2018 20:59, Austin Cawley-Edwards wrote: > Following up, we are using Flink 1.5.0 and Flink-CEP 2.11. > > Thanks, > Austin > > On Tue, Oct 30, 2018 at 3:58 PM Austin Cawley-Edwards > mailto:austin.caw..

Re: FlinkCEP, circular references and checkpointing failures

This is some problem with serializing your events using Kryo. I'm adding Gordon to cc, as he was recently working with serializers. He might give you more insights what is going wrong. Best, Dawid On 25/10/2018 05:41, Shailesh Jain wrote: > Hi Dawid, > > I've upgraded to flink 1.6.1 an

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hi Aaron, Could you share the code of you custom function? I am also adding Aljosha and Kostas to cc, who should be more helpful on that topic. Best, Dawid On 19/10/2018 20:06, Aaron Levin wrote: > Hi, > > I'm writing a custom `SourceFunction` which wraps an u

Re: Manual trigger the window in fold operator or incremental aggregation

Hi Zhen Li, As far as I know that is not possible. For such custom handling I would recommend having a look at ProcessFunction[1], where you have access to timers and state. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/process_function.html

Re: Window State is not being store on check-pointing

stop and then want to start a job with a state from previous run you should use savepoints. For a more thorough explanation of those concepts please have a look here[1] Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html#what-is-a-savepoint-how

Re: Initializing mapstate hangs

itself as a transient field rather than the descriptor. Best, Dawid On 22/10/2018 15:15, Alexander Smirnov wrote: > I think that's because you declared it as transient field. > > Move the declaration inside of "open" function to resolve that > > On Mon, Oct 22, 201

Re: Failed to fetch BLOB - IO Exception

Hi Manjusha, I am not sure what is wrong, but Nico or Till (cc'ed) might be able to help you. Best, Dawid On 23/10/2018 06:58, Manjusha Vuyyuru wrote: > Hello All, > > I have a  job which fails lets say after every 14 days with IO > Exception, failed to fetch blob. > I submitte

Re: how to use the TUMBLE(time_attr, interval) window function in Flink SQL

with implemented watermarks in TableEnvironment. Then you can apply the windowing on a table prepared that way. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/streaming.html#time-attributes On 23/10/2018 06:45, maidangdang44 wrote: > below is one line of

Re: Table API / SQL UIDs not the only one

Hi, Could you rephrase your question? I think some parts of the question are missing. It would be also easier to help you if you could state the final problem a bit more clearly. Best, Dawid On 23/10/2018 04:06, WeiWen Fan wrote: > a job have two sql > source is kafka > sink

Re: FlinkKafkaProducer and Confluent Schema Registry

Hi Olga, There is an open PR[1] that has some in-progress work on corresponding AvroSerializationSchema, you can have a look at it. The bigger issue there is that SerializationSchema does not have access to event's key so using topic pattern might be problematic. Best, Dawid [1] https

Re: [BucketingSink] notify on moving into pending/ final state

Hi Ribat, I haven't checked your PR but we introduced a new connector in flink 1.6 called StreamingFileSink that is supposed to replace BucketingSink long term. I think it might solve a few problems of yours. Have you checked it by chance? Best, Dawid On Thu, 11 Oct 2018, 14:10 Rinat, wrote

Re: getRuntimeContext(): The runtime context has not been initialized.

per each tenant in a single window?     3. I think what you could do is first compute stats for composite key and then aggregate them in subsequent operation(if you need to). This way you could distribute the workload to more parallel instances. Best, Dawid On 11/10/18 11:33, Ahmad Hassan

Re: Apache Flink: Kafka connector in Python streaming API, “Cannot load user class”

Hi Kostas, As far as I know you cannot just use java classes from within python API. I think Python API does not provide wrapper for kafka connector. I am adding Chesnay to cc to correct me if I am wrong. Best, Dawid On 11/10/18 12:18, Kostas Evangelou wrote: > Hey all,  > > Thank yo

Re: Taskmanager times out continuously for registration with Jobmanager

Hi Abdul, I've added Till and Gary to cc, who might be able to help you. Best, Dawid On 11/10/18 03:05, Abdul Qadeer wrote: > > Hi, > > > We are facing an issue in standalone HA mode in Flink 1.4.0 where > Taskmanager restarts and is not able to register with the Jobmanag

Re: 答复: No data issued by flink window after a few hours

, Dawid On 10/10/18 08:44, 潘 功森 wrote: > Hi， > >   > > Cause default state size in one hour is too small，and the max window > size is 24 hours, so I used 500M. > >   > > MemoryStateBackend stateBackend = new > MemoryStateBackend(/MAX_STATE_SIZE/);//500M

Re: cannot resolve constructor when invoke FlinkKafkaProducer011 constructor in scala

Hi, what is the exact error message you are getting? Best, Dawid On 10/10/18 11:51, 远远 wrote: > invoke FlinkKafkaProducer011 constructor in scala: > val producer = new > FlinkKafkaProducer011[PVEvent.Entity](appConf.getPvEventTopic, new > PvEventSerializeSchema, p

Re: Error restoring from savepoint while there's no modification to the job

cannot be changed for a job that shall be restored. If this is not the case, maybe Stefan(cc) have some ideas, what can go wrong. Best, Dawid On 10/10/18 09:23, Averell wrote: > Hi everyone, > > I'm getting the following error when trying to restore from a savepoint. > Here below i

Re: Rowtime for Table from DataStream without explixit fieldnames

Hi Johannes, I am afraid that this is currently not possible and indeed you have to pass all fields again, but Timo cced might want to correct me if I am wrong. Best, Dawid On 04/10/18 15:08, Johannes Schulte wrote: > Hi, > > when converting a DataStream (with Watermarks) to a t

Re: FlinkCEP, circular references and checkpointing failures

previously. Could you check if the problem occurs on vanilla flink as well? Best, Dawid On 27/09/18 08:22, Shailesh Jain wrote: > Hi Dawid, > > Yes, it is version 1.4.2. We are running vanilla flink, but have added > a couple of changes in the CEP operator specifically (top 3 commits &

Re: FlinkCEP, circular references and checkpointing failures

if it is a problem than it is definitely a different one. Last thing I would recommend upgrading to the newest version, as we rewritten the SharedBuffer implementation in 1.6.0. Best, Dawid On 26/09/18 13:50, Shailesh Jain wrote: > Hi, > > I think I've hit this same issue on a 3 node standalon

Re: Null Flink State

ents assigned to that window you can iterate over them with the "itr" in your example. Best, Dawid On 25/09/18 15:07, Taher Koitawala wrote: > Hi All, >            I am trying to access elements stored in the state of the > window. As window, itself is a stateful operator I t

Re: Between Checkpoints in Kafka 11

in case he wants to correct me or add something to the answer. Best, Dawid On 23/09/18 17:48, Harshvardhan Agrawal wrote: > Hi, > > Can someone please help me understand how does the exactly once > semantic work with Kafka 11 in Flink? > > Thanks, > Harsh > > On Tu

Re: Flink TaskManagers do not start until job is submitted in YARN

Hi Suraj, As far as I know this was changed with FLIP-6 to allow dynamic resource allocation. Till, cced might know if there is a switch to restore old behavior or are there plans to support it. Best, Dawid On 24/09/18 12:24, suraj7 wrote: > Hi, > > I am using Amazon EMR to run Flin

Re: error with session window

Hi Yuvraj, It looks as some race condition for me. Would it be ok for you to switch to either Event or Ingestion time[1]? I also cced @Aljosha who might give you a bit more insights Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_time.html#event-time

Re: Running Flink in Google Cloud Platform (GCP) - can Flink be truly elastic?

Maybe Gary or Till cc'ed can share some more details on that topic. Best, Dawid On 21/09/18 17:25, alexander.gard...@rbs.com.INVALID wrote: > Hi > > I'm trying to understand what it means to run a Flink cluster inside the > Google Cloud Platform and whether it can act in an "elas

Re: Migration to Flink 1.6.0, issues with StreamExecutionEnvironment.registerCachedFile

Hi Subramanya, I could reproduce this behavior running a job in YARN cluster. This works in standalone cluster just fine. We've changed a little bit how the cache entries are distributed in 1.6.0. I am investigating this problem right now. Would you like to create a JIRA bug for it? Best, Dawid

Re: [Kerberos] JAAS module content not generated? javax.security.auth.callback.UnsupportedCallbackException: Could not login: the client is being asked for a password, but the Kafka client code does n

FQDN for kafka brokers. Best, Dawid On 13/09/18 16:17, Aljoscha Krettek wrote: > Hi, > > this definitely needs some investigation! Did you try setting the JAAS > settings in the sasl.jaas.config property as described > in  > https://docs.confluent.io/current/kafka/authenticati

Re: Orc Sink Table

Hi Jose, Flink provides only OrcTableSource that is BatchTableSource only as of now. You can read more here[1]. As of now there is no support for orc sink file. The community is working extensively though on extending the set of supported connectors for Table API/SQL, so it might change in the

Re: How to add flink table jar to classpath via bin/flink

Hi Jeff, Right now the suggested procedure is to copy the flink-table dependency from /opt folder to /lib folder. You can read about it here[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/table/#setup On 13/09/18 10:44, Jeff Zhang wrote: > Because flink-table is a

Re: Broadcast managed state

Hi Deepya, It depends how do you want to update this state. If your use-case follows the Broadcast State Pattern[1], that means you perform updates to your state only based on one broadcasted stream (often called control stream) you should check out the linked docs. If you want to perform

Re: Logging metrics from within Elasticsearch ActionRequestFailureHandler

Hi Averell, I am afraid this is currently not possible without modifying ElasticsearchSink. Would you like to open jira ticket for metrics access in the failureHandler? Best, Dawid On 13/09/18 03:50, Averell wrote: > Good day everyone, > > I'm writing to Elasticsearch, and I need

Re: How to get taskmanager hostname and port on runtime

Hi, For the hostname, by default identifier for metrics scoped to taskmanager contain its hostname. You can read more about metrics scope here[1]. What do you mean by port? Which port would you like to report? [1]

Re: Weird behaviour after change sources in a job.

Hi Juan, I think this is somehow expected behaviour. Flink, in order to provide proper processing semantics keeps track of partitions offsets internally, and checkpoints those offsets. FlinkKafkaConsumer supports also new partitions discovery. Having in mind both of those features, if you

Re: Acknowledging Pubsub messages in Flink Runner

Hi Encho, Flink does not provide Pubsub connector. I believe you are using Beam's coder for PubsubIO. If this is correct, I guess you might want to ask this question on Beam's mailing list. Regards, Dawid On 10/09/18 17:24, Encho Mishinev wrote: > Hello, > > I have a simple questi

Re: Unable to start Flink HA cluster with Zookeeper

Hi, It will use HA settings as long as you specify the high-availability: zookeeper. The jobmanager.rpc.adress is used by the jobmanager as a binding address. You can verify it by starting two jobmanagers and then killing the leader. Best, Dawid On Tue, 21 Aug 2018 at 17:46, mozer wrote: > Y

Re: Unable to start Flink HA cluster with Zookeeper

. Regards Dawid On Tue, 21 Aug 2018 at 15:32, mozer wrote: > FQD or full ip; tried all of them, still no changes ... > For ssh connection, I can connect to each machine without passwords. > > > Do you think that the problem can come from : > > *high-availability.storageDir: file:/

Re: How do I investigate checkpoints failures

here [1]). Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/state/checkpointing.html#enabling-and-configuring-checkpointing On 21/08/18 09:10, Alexander Smirnov wrote: > Hello, > > I have a cluster with multiple jobs running on it. One of the

Re: Skip event in case of key extraction exception

Hi, You cannot filter out events in the KeyExtractor. What you can do though is to move the conversion logic to e.g. flatMap function and emit only those events that where successfully converted. Then your KeyExtractor would be a single getter for the UUID. Best, Dawid On 13/08/18 07:45, Jayant

Re: Joining streamed data to reference data

ure. Best, Dawid On Fri, 20 Jul 2018 at 11:59, vino yang wrote: > Hi Porritt, > > Flink does not support streaming and batch join, currently, streaming and > batch job are both independent. > > I guess your use case is streaming and dimension table join? > Unfortunately,

Re: org.apache.flink.streaming.runtime.streamrecord.LatencyMarker cannot be cast to org.apache.flink.streaming.runtime.streamrecord.StreamRecord

Hi Gregory, I think it is some flink bug. Could you file a JIRA for it? Also which version of flink are you using? Best, Dawid On Fri, 20 Jul 2018 at 04:34, vino yang wrote: > Hi Gregory, > > This exception seems a bug, you can create a issues in the JIRA. > > Thanks, vino. >

Re: Flink 1.4.2 -> 1.5.0 upgrade Queryable State debugging

Hi Philip, Could you attach the full stack trace? Are you querying the same job/cluster in both tests? I am also looping in Kostas, who might know more about changes in Queryable state between 1.4.2 and 1.5.0. Best, Dawid On Thu, 19 Jul 2018 at 22:33, Philip Doctor wrote: > Dear Flink Us

Re: Can not get OutPutTag datastream from Windowing function

Hi Soheil, The /getSideOutput/ method is part of /SingleOutputStreamOperator/ which extends /DataStream///. Try using /SingleOutputStreamOperator/ as the type for your res variable. Best, Dawid On 17/07/18 09:36, Soheil Pourbafrani wrote: > Hi, according to the documents I tried to get l

Re: [ANNOUNCE] Apache Flink 1.5.1 released

Good job everyone and Chesnay for being the release manager! On 13/07/18 14:34, Hequn Cheng wrote: > Cool, thanks to Chesnay! > > Best, Hequn > > On Fri, Jul 13, 2018 at 8:25 PM, vino yang > wrote: > > Thanks Chesnay, great job! > > Thanks, > Vino > >

Re: How to create User Notifications/Reminder ?

Hi shyla, It should be doable with CEP. You can create pattern like: Pattern.begin("start").next/followedBy("end").where(...).within(/* two weeks*/) and subscribe for timed out events. You can check very similar example here[1]. Best, Dawid [1] https://github.com/dataArt

Re: CEP: Different consuming strategies within a pattern

ruct match with any subsequent start you can use "followedByAny", but then remember to add the within clause, as otherwise partial matches won't be cleared. Cheers, Dawid On 25/06/18 08:11, Shailesh Jain wrote: > Hi, > > I'm trying to detect a sequence like A followed by B, C,

Re: NPE in flink sql over-window

Hi Yan, I think it is a bug in the ProcTimeBoundedRangeOver. It tries to access a list of elements that was already cleared and does not check against null. Could you please file a JIRA for that? Best, Dawid On 30/05/18 08:27, Yan Zhou [FDS Science] wrote: > > I also get wa

Re: sharebuffer prune code

The logic for SharedBuffer and in result for prunning will be changed in FLINK-9418 [1]. We plan to make it backwards compatible. There is already open PR[2] (in review), you can check if the problem persists. Regards, Dawid [1] https://issues.apache.org/jira/browse/FLINK-9418 [2] https

Re: RocksDBMapState example?

to that key) might be sufficient. You can configure flink further to use RocksDB as the underlying state backend[2] Regards, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state [2] https://ci.apache.org/projects/flink/flink-docs

Re: Dynamic CEP https://issues.apache.org/jira/browse/FLINK-7129?subTaskView=all

; On Thu, Mar 8, 2018 at 10:41 AM, Kostas Kloudas <k.klou...@data-artisans.com> > wrote: > Hi Vishal, > > Dawid (cc’ed) who was working on that stopped because in the past Flink > did not support broadcast state. > > This is now added (in the master) and the implementat

Re: cep code

, but in contrast to OPTIONAL its condition should be taken into account into the combined condition. Hope it helps. Regards, Dawid > On 5 Mar 2018, at 10:42, aitozi <gjying1...@gmail.com> wrote: > > Hi, > > i am reading flink-cep source code based on release-1.3.2 . I c

Re: Retrieving name of last external checkpoint directory

Thx for your suggestions. In the end I’ve integrated altering flink-conf.yaml into job submission, which we do always via some custom ansible scripts. This way each job has its own directory for external checkpoints. Best, Dawid > On 20 Feb 2018, at 17:21, Chesnay Schepler <ches...@apac

Retrieving name of last external checkpoint directory

that we are running. How can we find out the last successful external checkpoint for some particular job? Will be grateful for any pointers. Regards, Dawid signature.asc Description: Message signed with OpenPGP

Re: CEP issue

e ( the cluster ) > is configured to work against the latest and greatest. This does not happen > with smaller range patterns ( 3 out of 5 , 1 of 3 etc) but is always an issue > when it is a larger range ( 20 out of 25 with range of 8 hours ) . Does that > makes sense? > > On Fri, Feb 2,

Re: CEP issue in 1.3.2. Does 1.4 fix this ?

Hi Vishal, I think it might be due to this bug: https://issues.apache.org/jira/browse/FLINK-8226 It was merged for 1.4.1 and 1.5.0. Could you check with this changes applied? Would be really helpful. If the error still persists could you file a jira? Regards Dawid > On 11 Jan 2018, at 19

Re: Flink CEP with event time

or equal. Best Dawid > On 3 Jan 2018, at 17:05, shashank agarwal <shashank...@gmail.com> wrote: > > ssed A with origTimestamp Y. ( signature.asc Description: Message signed with OpenPGP

Re: How to apply patterns from a source onto another datastream?

ant Ameta > > On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz > <wysakowicz.da...@gmail.com> wrote: > It is not possible at this moment. FlinkCEP can handle only one Pattern > applied statically. There is a JIRA ticket for that: > https://issues.apache.org/jira/browse

Re: JVM crash - SIGSEGV in ZIP_GetEntry

; > Hi, > I have seen similar errors when trying to serialize Kryo-typeserializers with > Flink type infos accidentally. > > Maybe that helps :) > > Gyula > > > On Sun, Dec 17, 2017, 15:52 Dawid Wysakowicz <wysakowicz.da...@gmail.com> >

Re: How to apply patterns from a source onto another datastream?

It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 . > On 19 Dec 2017, at 10:10, Jayant Ameta wrote: > > I've a datastream of events, and another

Re: JVM crash - SIGSEGV in ZIP_GetEntry

()V+18 j sun.net.www.protocol.jar.URLJarFile.finalize()V+1 J 10563% C2 java.lang.ref.Finalizer$FinalizerThread.run()V (55 bytes) @ 0x7f9075be90b4 [0x7f9075be8e00+0x2b4] v ~StubRoutines::call_stub > On 17 Dec 2017, at 15:03, Dawid Wysakowicz <wysakowicz.da...@gmail.com> wrote

JVM crash - SIGSEGV in ZIP_GetEntry

Hi, Recently we observe regular taskmanager's JVM crashes just about a minute from the start of our Flink job. We run flink 1.3.2 on YARN (2.6.2.0-205). Java version: JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM

Re: Flink CEP state store

the state by using either within (that will timeout states) or do not use nondeterministic continuity (e.g. followedByAny or looping patterns without until). Hope it helps. Regards Dawid > On 16 Dec 2017, at 08:53, mahesh dhabade <mahesh.dhab...@gmail.com> wrote: > > Hello, >

Re: FlinkCEP behaviour with time constraints not as expected

not returned via sideOutput. > On 8 Nov 2017, at 12:18, Federico D'Ambrosio <federico.dambro...@smartlab.ws> > wrote: > > Thank you very much, Dawid, for your thorough explanation, really useful. I > totally missed the distinction between timed-out events and complete match

Re: FlinkCEP behaviour with time constraints not as expected

urned because within the timeout there was no event with value <100 (in fact there was no event at all to be checked). Hope this "study" helps you understand the behaviour. If you feel I missed something, please provide some example I could reproduce. Regards, Dawid 2017-11-07 1

Re: Flink CEP questions

Hi Basanth, Ad.3 Unfortunately right now, you cannot reset, but there is ongoing work to introduce AfterMatchSkipStrategies(https://issues.apache.org/jira/browse/FLINK-7169?filter=12339990). This will allow the behaviour you described with the SKIP_PAST_LAST strategy. Ad.4 If I understand

Re: Keyed CEP checkpoint fails

to the entryID map. > On 10 Aug 2017, at 14:54, Daiqing Li <lidaiqing1...@gmail.com> wrote: > > Oh sorry, the data in {} is not empty because I hide private information > about my model. Do you have that same conclusion? >> On Aug 10, 2017, at 8:52 AM, Dawid Wysakowicz <

Re: Keyed CEP checkpoint fails

is broken. > On 10 Aug 2017, at 14:29, Daiqing Li <lidaiqing1...@gmail.com> wrote: > > Hi, > > Here is the code. But I am not sure if you can reproduce the problem without > data source. > > Best, > Daiqing > > On Thu, Aug 10, 2017 at 8:15 AM, Dawid Wysakowic

<    1   2   3   4   5   >