How & Where does flink stores data for aggregations.

2017-11-23 Thread Shivam Sharma
Hi All, I have a small question regarding where does Flink stores data for doing window aggregations. Lets say I am running following query on Flink table: SELECT name, count(*) FROM testTable GROUP BY TUMBLE(rowtime, INTERVAL '1' MINUTE), name So, If I understand above query properly so it must

Re: FlinkKafkaConsumer010 does not start from the next record on startup from offsets in Kafka

2017-11-23 Thread r. r.
Gordon, thanks for clarifying this! For my experimental project I decided to disable checkpointing and use kafkaConsumer.setStartFromGroupOffsets() (explicitly, although docs state it is the default). I verified with kafka-consumer-offset-checker.sh that, after the job fails and is restarted, i

Re: Bad entry in block exception with RocksDB

2017-11-23 Thread Kostas Kloudas
Hi Kien, Could you share some more information about your job? What operators are you using, the format of your elements, etc? Thanks, Kostas > On Nov 23, 2017, at 2:23 AM, Kien Truong wrote: > > Hi, > > We are seeing this exception in one of our job, whenever a check point or > save point i

Re: Problem with SQL-API and nested objects in case class

2017-11-23 Thread Lothium
Hey Timo, when I try to access the array in the case class via the SQL syntax, I get back an error that the syntax is invalid. Here is an example of the case class structure: case class Envelope(name: String, entity: Product) case class Product(name: String, items: List[Item]) case class Item(attr

Re: Flink stress testing and metrics

2017-11-23 Thread Timo Walther
Yes, I agree that this looks like a bug. You can open an issue about that. Maybe with a small reproduceble example to give others the chance to fix it. Am 11/22/17 um 10:18 PM schrieb Ladhari Sadok: Normally it should return 0ms in case of no latency not NaN, and my real data size is 1kb, but

Re: How to write dataset as parquet format

2017-11-23 Thread ebru
Flavio and Fabian thanks for your quick answers, it was very helpful. -Ebru > On 22 Nov 2017, at 20:47, Flavio Pompermaier wrote: > > I usually refer to this: > > https://github.com/FelixNeutatz/parquet-flinktacular > > > On 22 Nov 2017 1