Re: Timers and state

Hi Alberto, an ultimate solution for your problem would be a map state with ordered keys (like a TreeMap), but unfortunately, this is still a WIP feature. For now, maybe you could "eagerly remove” the outdated value (with `iterator.remove()`) when iterating the map state in the process

Timers and state

Hello, in a Flink application we have a keyed operator that keeps a map state (MapState). Some of the elements in the state need a timeout so we use a Timer. When the timer is called the state is scoped to the key as expected but we would like to 'pass' (or have available elsewhere) to

Using time window with SQL nested query

I tried to use write a nested query with HOP window in a streaming environment. Table `source` consists of 3 column, a, b, and timestamp. SELECT a FROM (SELECT a, COUNT(*) FROM source GROUP BY HOP(timestamp, ..., ...), a, b) GROUP BY HOP(timestamp, ..., ...), a HAVING ... And flink throws an

cep code

Hi, i am reading flink-cep source code based on release-1.3.2 . I cant understand here , can anyone help me on this in NFACompiler? private List, String>> getCurrentNotCondition() { List, String>> notConditions = new

Re: cep code

Hi, It is a very low level detail of the CEP library, that should be transparent for the end-user. However, just to clarify it a bit, it is expected. The reason behind this function is to create an optional path that bypasses all optional states. NOT_FOLLOW is treated as part of optional

Simple CEP examples with full Maven project

Hi I have tried to learn CEP, but from some reasons it seems to be little difficult. It looks very complex. Are there exist some simple (Scala) examples about CEP with full Maven projects ? I have only found TaxiRide of Dataartisan example [1]. For example what variables, classes and

Re: CsvTableSource Types.TIMESTAMP

Hi, SQL_TIMESTAMP is the same. A couple of months ago it was decided to rename this property such that it can be used for timestamps with timezone support in the future. Regards, Tiom Am 3/5/18 um 2:10 PM schrieb Esa Heikkinen: I have tried to following example to work, but no succeed

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi, quick question: what is your exact checkpointing configuration? In particular, what is your value for the maximum parallel checkpoints and the minimum time interval to wait between two checkpoints? Best, Stefan > Am 05.03.2018 um 06:34 schrieb Tony Wei : > > Hi

CsvTableSource Types.TIMESTAMP

I have tried to following example to work, but no succeed yet. https://flink.apache.org/news/2017/03/29/table-sql-api-update.html Error .. value TIMESTAMP is not a member of object org.apache.glink.table.api.Types What would be the problem ? What the imports should I use ? Or should I use

Re: Using time window with SQL nested query

Hi Bill, you can use HOP_ROWTIME()/HOP_PROCTIME() to propagate the time attribute to the outer query. See also [1] for an example. Regards, Timo [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/sql.html#selecting-group-window-start-and-end-timestamps Am 3/5/18

Re: Table API Compilation Error in Flink

Hi Nagananda, could you show us your entire pom.xml? From what I see it seems that you are using the wrong StreamTableEnvironment. First you need to decide if you want to program in Scala or Java. Depending on that you can add the dependencies as descriped in [1]. There are two

Re: Rocksdb in production

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink. Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct? Jayant Ameta On Mon, Mar 5, 2018 at 9:51 PM, Fabian

Re: Timers and state

Hi Alberto, You can also add another MapState. The key is a timestamps and the value is the key that you want to discard. When onTimer() is called, you look up the key in the MapState and and remove it from the original MapState. Best, Fabian 2018-03-05 0:48 GMT-08:00

Re: Rocksdb in production

Hi, RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink. As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself. Besides that, RocksDB is

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi, thanks for all the info. I had a look into the problem and opened https://issues.apache.org/jira/browse/FLINK-8871 to fix this. From your stack trace, you can see many checkpointing threads are running on your TM for checkpoints that have

Kafka offset auto-commit stops after timeout

We have noticed that the Kafka offset auto-commit functionality seems to stop working after it encounters a timeout. It appears in the logs like this: 2018-03-04 07:02:54,779 INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the coordinator kafka06:9092 (id:

Re: Rocksdb in production

Yes, that is correct. 2018-03-05 8:57 GMT-08:00 Jayant Ameta : > Oh! Somehow I missed while reading the documentation that RocksDB is > embedded in Flink. > > Also, irrespective of state backend being filesystem or rocksdb, I'll have > to setup a shared filesystem (HDFS,

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Stefan, Here is my checkpointing configuration. Checkpointing Mode Exactly Once Interval 20m 0s Timeout 10m 0s Minimum Pause Between Checkpoints 0ms Maximum Concurrent Checkpoints 1 Persist Checkpoints Externally Enabled (delete on cancellation) Best Regards, Tony Wei 2018-03-05 21:30

Re: Delta iteration not spilling to disk

Hi Joshua, I am running into a similar problem. Can you explain your solution a bit more ? A code snippet will help. Thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: cep code

Then what is STOP state in NFA, i haven't seen this state in event pattern match paper ? Does each Not pattern will be transformed to stop state? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Stefan, I see. That explains why the loading of machines grew up. However, I think it is not the root cause that led to these consecutive checkpoint timeout. As I said in my first mail, the checkpointing progress usually took 1.5 mins to upload states, and this operator and kafka consumer are

Re: Using time window with SQL nested query

Thanks. It works. I missed it while reading the document. Timo Walther 于 2018年3月5日周一 下午9:20写道： > Hi Bill, > > you can use HOP_ROWTIME()/HOP_PROCTIME() to propagate the time attribute > to the outer query. See also [1] for an example. > > Regards, > Timo > > [1] >

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Sihua, Thanks for your suggestion. "incremental checkpoint" is what I will try out next and I know it will give a better performance. However, it might not solve this issue completely, because as I said, the average end to end latency of checkpointing is less than 1.5 mins currently, and it is

Re: Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Sihua, You are right. The incremental checkpoint might release machine from high cpu loading and make the bad machines recover quickly, but I was wondering why the first checkpoint failed by timeout. You can see when the bad machine recovered, the cpu loading for each checkpoint is not so

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Tony, About to your question: average end to end latency of checkpoint is less than 1.5 mins, doesn't means that checkpoint won't timeout. indeed, it determined byt the max end to end latency (the slowest one), a checkpoint truly completed only after all task's checkpoint have completed.

Visual CEP pattern & State transformation

Hi, When i read about the CEP code, i am feeling that it is a bit difficult to understand the whole transformation, I hava a idea that we can have a draw of the pattern and State transformation like the DAG plan picture in web UI. May be it can be the "ignore, take, processd" like this

Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

Sent to the wrong mailing list. Forward it to the correct one. -- Forwarded message -- From: Tony Wei Date: 2018-03-06 14:43 GMT+08:00 Subject: Re: checkpoint stuck with rocksdb statebackend and s3 filesystem To: 周思华 , Stefan Richter