Kenneth Knowles created BEAM-4719:
-------------------------------------
Summary: Enhanced LIMIT support
Key: BEAM-4719
URL: https://issues.apache.org/jira/browse/BEAM-4719
Project: Beam
Issue Type: New Feature
Components: dsl-sql
Reporter: Kenneth Knowles
Currently, Beam SQL supports LIMIT in two ways:
1. Within a query, the results are subject to LIMIT. This works.
2. The shell knows to cancel a pipeline when the limit is reached, even if
there is unfinished unbounded data.
The canceling of a pipeline works via a basic pattern match against the query
execution plan, checking a few child nodes of the BeamEnumerableConverter for a
BeamSortRel without a collation. If it can figure out what the limit is for the
outermost query, then it will cancel the pipeline.
A more robust approach might be to use traits (or some other thorough analysis)
to see if there is a known size for the outermost query. This would, for
example, be unaffected by any number of layer of non-size-changing
transformations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
