alamb commented on PR #21828:
URL: https://github.com/apache/datafusion/pull/21828#issuecomment-4347147466
BTW here is the explain plan
```sql
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion/benchmarks/data$
~/Downloads/datafusion-cli-feat_offset-pushdown
DataFusion CLI v53.1.0
> explain select * from 'hits.parquet' OFFSET 99000000 LIMIT 5;
+---------------+-------------------------------+
| plan_type | plan |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
| | │ GlobalLimitExec │ |
| | │ -------------------- │ |
| | │ limit: 5 │ |
| | │ skip: 99000000 │ |
| | └─────────────┬─────────────┘ |
| | ┌─────────────┴─────────────┐ |
| | │ CoalescePartitionsExec │ |
| | │ -------------------- │ |
| | │ limit: 99000005 │ |
| | └─────────────┬─────────────┘ |
| | ┌─────────────┴─────────────┐ |
| | │ DataSourceExec │ |
| | │ -------------------- │ |
| | │ files: 16 │ |
| | │ format: parquet │ |
| | └───────────────────────────┘ |
| | |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.017 seconds.
> explain format indent select * from 'hits.parquet' OFFSET 99000000 LIMIT 5;
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Limit: skip=99000000, fetch=5
|
| | TableScan: hits.parquet projection=[WatchID, JavaEnable,
Title, GoodEvent, EventTime, EventDate, CounterID, ClientIP, RegionID, UserID,
CounterClass, OS, UserAgent, URL, Referer, IsRefresh, RefererCategoryID,
RefererRegionID, URLCategoryID, URLRegionID, ResolutionWidth, ResolutionHeight,
ResolutionDepth, FlashMajor, FlashMinor, FlashMinor2, NetMajor, NetMinor,
UserAgentMajor, UserAgentMinor, CookieEnable, JavascriptEnable, IsMobile,
MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID,
SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth,
WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1,
SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset,
CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID,
IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor,
LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName,
OpenerName, HistoryLeng
th, BrowserLanguage, BrowserCountry, SocialNetwork, SocialAction, HTTPError,
SendTiming, DNSTiming, ConnectTiming, ResponseStartTiming, ResponseEndTiming,
FetchTiming, SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID,
ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID,
OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent,
UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], fetch=99000005
|
| physical_plan | GlobalLimitExec: skip=99000000, fetch=5
|
| | CoalescePartitionsExec: fetch=99000005
|
| | DataSourceExec: file_groups={16 groups:
[[Users/andrewlamb/Software/datafusion/benchmarks/data/hits.parquet:0..923748528],
[Users/andrewlamb/Software/datafusion/benchmarks/data/hits.parquet:923748528..1847497056],
[Users/andrewlamb/Software/datafusion/benchmarks/data/hits.parquet:1847497056..2771245584],
[Users/andrewlamb/Software/datafusion/benchmarks/data/hits.parquet:2771245584..3694994112],
[Users/andrewlamb/Software/datafusion/benchmarks/data/hits.parquet:3694994112..4618742640],
...]}, projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime,
EventDate, CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent,
URL, Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID,
URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor,
FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor,
CookieEnable, JavascriptEnable, IsMobile, MobilePhone, MobilePhoneModel,
Params, IPNetworkID, TraficSourc
eID, SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical,
WindowClientWidth, WindowClientHeight, ClientTimeZone, ClientEventTime,
SilverlightVersion1, SilverlightVersion2, SilverlightVersion3,
SilverlightVersion4, PageCharset, CodeVersion, IsLink, IsDownload, IsNotBounce,
FUniqID, OriginalURL, HID, IsOldCounter, IsEvent, IsParameter, DontCountHits,
WithHash, HitColor, LocalEventTime, Age, Sex, Income, Interests, Robotness,
RemoteIP, WindowName, OpenerName, HistoryLength, BrowserLanguage,
BrowserCountry, SocialNetwork, SocialAction, HTTPError, SendTiming, DNSTiming,
ConnectTiming, ResponseStartTiming, ResponseEndTiming, FetchTiming,
SocialSourceNetworkID, SocialSourcePage, ParamPrice, ParamOrderID,
ParamCurrency, ParamCurrencyID, OpenstatServiceName, OpenstatCampaignID,
OpenstatAdID, OpenstatSourceID, UTMSource, UTMMedium, UTMCampaign, UTMContent,
UTMTerm, FromTag, HasGCLID, RefererHash, URLHash, CLID], limit=99000005,
file_type=parquet |
| |
|
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.009 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
