kkhatua edited a comment on issue #1608: DRILL-6960: AutoLimit the size of ResultSet for a WebUI (or REST) client URL: https://github.com/apache/drill/pull/1608#issuecomment-461693969 @vvysotskyi (cc @ihuzenko ) I was discussing with Sorabh the issue of whether I can fetch only an initial small part of a resultset for a query and terminate it without consuming the remaining rows, and he pointed out that cancellation is the only way to do this, which is not a very graceful way of executing the query to fetch a limited set of rows. My current problem is that a massive join query was able to create the small dataset ready for the WebServer when I tried you approach of not building the resultset beyond what the limit was. However, I am still forced to iterate through the remaining incoming batches till the query completes. That means, while I have the benefit of not overload the WebServer (original intent of DRILL-6050 and DRILL-6477), the Foreman is still forced to run through all the remaining batches only to discard them. As a result, the Calcite approach suggested by Igor seems to be the best approach, because it also ensures that the Foreman does not process records that will not be sent to the Web/REST client. I'm in the process of doing some final testing with the last commit on the Calcite-based solution, which are mostly UX related. Should I submit the commit ID to this PR or open a new PR for the review?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
