On 02/09/14 11:18, Claude Warren wrote:
I have had several cases where I have unions of service calls. since the
service calls take time and since the operation is not sensitive to the
order of the items being unioned I was thinking of implementing a union
operation for service endpoints such that both endpoints would be opened
and the results fed into the result stream as they are returned. This
wiould involve a polling mechanism on the connections to see if there was a
result available. First available data element is added to the result and
the polling continues.
pseudo code something like
create a list of connections.
loop:
poll each connection
if connection finished close
if connection has data add to result
if there are connections still open go to loop;
Using multiple threads would be less implementation work, then have a
query iterator that picks items from a BlockingQueue.
Otherwise, you have hidden some magic in there which might result in
significant reengineering of result format parsers to cope with partial
bytes for a row. This is the usual difficulty for singe-threaded
multi-input processing in a partial-pull style.
I think that this would improve performance when used with multiple service
calls. It may also improve performance across slower storeage devices.
I don't follow this bit. The storage device is a long way from the
query engine and the union block is doing more than a single storage access.
Now, if you mean multi-threaded query execution on one machine, yes,
there can be gains. If done too simply with queues and locks the
overheads and contention problems of multi-user access may make it
negligible or even negative gain. It needs an experiment!
Andy
Thoughts?
Claude
