Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9923 )
Change subject: WIP KUDU-16 pt 3: add per-scan-token limits ...................................................................... Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9923/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9923/1//COMMIT_MSG@19 PS1, Line 19: sure this is actually a useful API or the one that we want to expose. > We chatted about this on Slack yesterday, but my thoughts are that we shoul I agree with both of your points; if a token is a logical concept that represents some optimization on a query plan, limiting probably doesn't make much sense (unless we're doing something fancy like sampling and joining, but AFAIK we're [Impala/Spark] not). That said, as an optimization for how Impala or some other parallel-scanning service might handles limits, it still might be a convenient API to have. I took a look at Spark and it seems like limiting is handled at a different layer of abstraction than the one at which KuduRDD sits (something like https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/sql/execution/Limit.html). I'll have to dig a bit more, still not sure how KuduRDD fits in. As for Impala, yeah it'd be helpful to get some feedback from the Impala devs. I pinged Thomas since he worked on runtime filters and runtime limits seems in a similar ballpark. In the meantime, I've filed IMPALA-6821 and pointed it at this CR. -- To view, visit http://gerrit.cloudera.org:8080/9923 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I655f07f10a9a99e9766402729361de97e929136a Gerrit-Change-Number: 9923 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Sat, 07 Apr 2018 06:58:30 +0000 Gerrit-HasComments: Yes
