Todd Lipcon has posted comments on this change. ( )

Change subject: WIP KUDU-16 pt 3: add per-scan-token limits

Patch Set 1:

(1 comment)
Commit Message:
PS1, Line 19: sure this is actually a useful API or the one that we want to 
yea, Im not sure this API provides much value beyond the caller just specifying 
the limit when they hydrate the token before they call Open() on the scan, 
which has the same net effect.

In general I feel like the APIs we provide for scan tokens should be things 
that offer some optimization opportunity at that "planning" stage. For example, 
it's beneficial to express predicates because it may reduce the number of 
generated tokens based on partition pruning, but for something like limit, 
especially given the odd semantics that you described, it's less obvious. A 
further reason it might be better for the user to handle the limits is that, at 
least in the case of Impala, a single backend is responsible for processing 
several tokens (in parallel, even), and so they'll still need to do their own 
limit tracking there to early-stop scanning after the limit is exhausted.

Maybe worth chatting with the Impala folks to see whether they think they would 
prefer to use an API like this to attach the limit to the scan tokens or 
whether or would be more useful for them to manage the limit themselves?

To view, visit
To unsubscribe, visit

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I655f07f10a9a99e9766402729361de97e929136a
Gerrit-Change-Number: 9923
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <>
Gerrit-Reviewer: Andrew Wong <>
Gerrit-Reviewer: Dan Burkert <>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <>
Gerrit-Comment-Date: Wed, 04 Apr 2018 19:23:10 +0000
Gerrit-HasComments: Yes

Reply via email to