[
https://issues.apache.org/jira/browse/DRILL-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150837#comment-17150837
]
ASF GitHub Bot commented on DRILL-7763:
---------------------------------------
vvysotskyi commented on pull request #2092:
URL: https://github.com/apache/drill/pull/2092#issuecomment-653427039
@cgivre, how it would work for the case when there was created multiple
fragments with their own scan? From the code, it looks like every fragment
would read the same number of rows specified in the limit. Also, will the limit
operator be preserved in the plan if the scan supports limit pushdown?
Metastore also provides capabilities for pushing the limit, but it works
slightly differently - it prunes files and leaves only minimum files number
with specific row count. Would these two features coexist and work correctly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Limit Pushdown to File Based Storage Plugins
> ------------------------------------------------
>
> Key: DRILL-7763
> URL: https://issues.apache.org/jira/browse/DRILL-7763
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.17.0
> Reporter: Charles Givre
> Assignee: Charles Givre
> Priority: Major
> Fix For: 1.18.0
>
>
> As currently implemented, when querying a file, Drill will read the entire
> file even if a limit is specified in the query. This PR does a few things:
> # Refactors the EasyGroupScan, EasySubScan, and EasyFormatConfig to allow
> the option of pushing down limits.
> # Applies this to all the EVF based format plugins which are: LogRegex,
> PCAP, SPSS, Esri, Excel and Text (CSV).
> Due to JSON's fluid schema, it would be unwise to adopt the limit pushdown as
> it could result in very inconsistent schemata.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
