Enumerable is flawed because it uses the Volcano execution model, which requires blocking threads.
Interpreter might be a better execution model. Take a look at org.apache.calcite.interpreter.Node. The ‘run()’ method returns void but we could make it return boolean, where ‘false’ means ‘I’m not yet done, call me in a while’. Then you could have many more queries running than you have threads. Julian > On Mar 4, 2020, at 9:48 AM, Scott Reynolds <[email protected]> wrote: > > List, > > Our team's current application runs a thread pool for handling queries. > Each time a query comes it it is placed in the run queue and executed > outside of the HTTP RPC thread. This allows us to handle N concurrent > queries where N is the size of the thread pool. I would like to change this > to a N:M system allowing our service to handle more concurrent queries then > the thread pool size. Our TableScan uses Netty RPCs and Queues to fetch > results from the remote database giving use the opportunity to park the > current query while awaiting RPC responses. > > Has anyone been able to achieve this? Is the best way to own Convention? > > Our current service doesn't use JDBC and therefore have direct access to > the compiled Enumerable. One solution I haven't been able to get working is > using InterruptedException (or something similar -- Kafka created a > WakeUpException), which would allow a sub-enumerable the ability to signal > above that it needs to save it's state and yield. > > Is this something that could be added to EnumerableDefaults? If not would > it be a InterruptableEnumerable convention instead?
