Re: Parallelism vs task manager allocation

2016-09-21 Thread Greg Hogan
Is the query stream also a Flink job? Is this use case not supported by keeping state within a single Flink job? https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html FLINK-3779 recently added "queryable state" to allow external processes access to operator state. https://issue

Re: Parallelism vs task manager allocation

2016-09-20 Thread pushpendra.jaiswal
Hi Greg My sink is a hashmap(acting as db store), I am reading from this hashmap from other query stream. I need one instance / slot per task manager of sink, so that everyone refers the same instance/slot. So is there any way so that I can restrict 1 slot per task manager. Thanks Pushpendra Jais

Re: Parallelism vs task manager allocation

2016-09-20 Thread Greg Hogan
Hi Pushpendra, This is the expected system behavior. Slots local to the same TaskManager can transfer buffers in memory. Are you able to also run the Sink with a parallelism of 4? Greg On Tue, Sep 20, 2016 at 6:16 AM, Pushpendra Jaiswal < pushpendra.jaiswa...@gmail.com> wrote: > Hi > I have lau