Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

Not to derail this thread onto another topic but the problem with using a static instance is that there's no way to shut it down when the job stops. So if, for example, it starts threads, I don't think those threads will stop when the job stops. I'm not very well versed in how various Java 8

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

Hi Shannon, That's right. This FLIP aims to boost TPS of the task workers with async i/o operation. As what Stephan has mentioned, by placing static attribute to shared resources(like event pool, connection), it is possible to share those resources among different slots in the same JVM. I will

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

@Shannon: One could have a "static" broker to share the same netty across slots in the same JVM. Implicitly, Flink does the same with broadcast variables. On Wed, Sep 21, 2016 at 5:04 PM, Shannon Carey wrote: > David, > > I just wanted to say "thanks" for making this

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

David, I just wanted to say "thanks" for making this proposal! I'm also interested in performing nonblocking I/O (multiplexing threads/reactive programming) within Flink operators so that we can, for example, communicate with external web services with Netty/RxNetty without blocking an entire

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

Hi David, thanks for the FLIP! It looks pretty good. A few questions / suggestions: - It would be good to make the number of concurrent AsyncFunction calls configurable. Maybe overload the unorderedWait and orderedWait methods with an additional int parameter? - Do you plan to also add a

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

Hi Henry, Here is the FLIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65870673 Thanks, David 2016-09-15 12:26 GMT+08:00 Henry Saputra : > HI David, > > Thanks so much for the interest to contribute to Apache Flink. > > To help review and DISCUSS

Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK

HI David, Thanks so much for the interest to contribute to Apache Flink. To help review and DISCUSS for new feature in Flink, please do submit FLIP [1] proposal. It will help the PMCs managing the new feature proposals and keep resources with ASF realm. Thanks, Henry [1]

[DISCUSS] Proposal for Asynchronous I/O in FLINK

Hi ALL, Recently, we have designed and implemented a proposal in FLINK to support Asynchronous Operation while streaming. The main feature of this proposal is to introduce async i/o operation in FLINK to boost the TPS of streaming job without delaying the checkpoint, and provide an easy way for

[DISCUSS] Proposal for Asynchronous I/O in FLINK

Hi ALL, Recently, we have designed and implemented a new proposal in FLINK to support Asynchronous Operation while streaming. The main feature of this proposal is to introduce async i/o operation in FLINK to boost the TPS of streaming job without delaying the checkpoint, and provide an easy way