Sorabh Hamirwasia created DRILL-6498:
----------------------------------------
Summary: Support for EMIT outcome in ExternalSortBatch
Key: DRILL-6498
URL: https://issues.apache.org/jira/browse/DRILL-6498
Project: Apache Drill
Issue Type: Task
Components: Execution - Relational Operators
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.14.0
With Lateral and Unnest if Sort is present in the sub-query, then it needs to
handle the EMIT outcome correctly. This means when a EMIT is received then
perform the Sort operation on the records buffered so far and produce output
with it. After EMIT Sort should refresh it's state and again work on next
batches of incoming record unless an EMIT is seen again.
For first cut Sort will not support spilling in the subquery between Lateral
and Unnest since spilling is very unlikely. The worst case that can happen is
that Lateral will get a batch with only 1 row of data because of repeated type
column data size being too big. In that case Unnest will produce 1 output batch
only and Sort or other blocking operators anyways needs enough memory to at
least hold 1 incoming batch. So in ideal cases spilling should not happen. But
if there is a operator between Sort and Unnest which increases the data size
then Sort might be in a situation to spill but thats not a common case for now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
