Re: Re: Use Flink to process request with list of queries and aggregate

2021-01-15 Thread Arvid Heise
Hi Li, what you can do is to add the #queries when splitting: user1 - [query1, query2, query3] -> [user1, query1, 3] [user1, query2, 3] [user1, query2, 3] -> Then while collecting the results, you just compare the current number of records in the window and emit if it reaches the expected number.

Re: Re: Use Flink to process request with list of queries and aggregate

2021-01-11 Thread Yun Gao
Hi Li, From my view I think it would not be eaily use a countWindow if you have different number of records for each key (namely user in this case). I think you may need to user the low level KeyedProcessFunction [1] to keep some state by yourself. For example, each request might also

Re: Use Flink to process request with list of queries and aggregate

2021-01-10 Thread Li Wang
Can I get any suggestion? Thanks a lot. - Li -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Use Flink to process request with list of queries and aggregate

2021-01-08 Thread Li Wang
Hi Team, I have a Flink application reading from Kafka. Each payload is a request sent by a user containing a list of queries. What I would like to do is use Flink to process the queries parallelly and aggregate results and send back to the user. For example, let's say we have two messages in