Re: [Discuss] Design of the python execution operator

Serialization and resource manager are not involved in CONTAINER_LOCAL. However, CONTAINER_LOCAL is probably not what you want, since it would force all partitions to run in the same JVM. Thomas On Fri, Dec 22, 2017 at 1:19 PM, Ananth G wrote: > I guess my comment

Re: [Discuss] Design of the python execution operator

I guess my comment below regarding overhead of serialisation in container local is wrong ? Nevertheless having a local thread implementation gives some benefits . For example I am using to whether sleep if there is no request in the queue or spin checking for request presence in the request

Re: [Discuss] Design of the python execution operator

Thanks for the comments Thomas and Pramod. Apologies for the delayed response on this thread. > I believe the thread implementation still adds some value over a container > local approach. It is more of a “thread local” equivalent which is more > efficient as opposed to a container local

Re: [Discuss] Design of the python execution operator

On Wed, Dec 20, 2017 at 3:34 PM, Thomas Weise wrote: > It is exciting to see this move forward, the ability to use Python opens > many new possibilities. > > Regarding use of worker threads, this is a pattern that we are using > elsewhere (for example in the Kafka input

Re: [Discuss] Design of the python execution operator

It is exciting to see this move forward, the ability to use Python opens many new possibilities. Regarding use of worker threads, this is a pattern that we are using elsewhere (for example in the Kafka input operator). When the operator performs blocking operations and consumes little memory

Re: [Discuss] Design of the python execution operator

Hi Ananth, >From your explanation, it looks like the threads overall allow you to achieve two things. Have some sort of overall timeout if by which a tuple doesn't finish processing then it is flagged as such. Second, it doesn't block processing of subsequent tuples and you can still process them

Re: [Discuss] Design of the python execution operator

Hello Pramod, Thanks for the comments. I adjusted the title of the JIRA. Here is what I was thinking for the worker pool implementation. - The main reason ( which I forgot to mention in the design points below ) is that the java embedded engine allows only the thread that created the instance

Re: [Discuss] Design of the python execution operator

Hi Anath, Sounds interesting and looks like you have put quite a bit of work on it. Might I suggest changing the title of 2260 to better fit your proposal and implementation, mainly so that there is differentiation from 2261. I wanted to discuss the proposal to use multiple threads in an