jychen7 commented on issue #5404: URL: https://github.com/apache/arrow-datafusion/issues/5404#issuecomment-1445277435
**Additional context**
Datafusion Plan
```
> explain ANALYZE SELECT * FROM hits WHERE "URL" LIKE '%google%' limit 1;
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[WatchID@0 as WatchID,
JavaEnable@1 as JavaEnable, Title@2 as Title, GoodEvent@3 as GoodEvent,
EventTime@4 as EventTime, EventDate@5 as EventDate, CounterID@6 as CounterID,
ClientIP@7 as ClientIP, RegionID@8 as RegionID, UserID@9 as UserID,
CounterClass@10 as CounterClass, OS@11 as OS, UserAgent@12 as UserAgent, URL@13
as URL, Referer@14 as Referer, IsRefresh@15 as IsRefresh, RefererCategoryID@16
as RefererCategoryID, RefererRegionID@17 as RefererRegionID, URLCategoryID@18
as URLCategoryID, URLRegionID@19 as URLRegionID, ResolutionWidth@20 as
ResolutionWidth, ResolutionHeight@21 as ResolutionHeight, ResolutionDepth@22 as
ResolutionDepth, FlashMajor@23 as FlashMajor, FlashMinor@24 as FlashMinor,
FlashMinor2@25 as FlashMinor2, NetMajor@26 as NetMajor, NetMinor@27 as
NetMinor, UserAgentMajor@28 as UserAgentMajor, UserAgentMinor@29 as
UserAgentMinor, CookieEnable@30 as CookieEnable, JavascriptEnable@31 as
JavascriptEnable, IsMobile@32 as IsMo
bile, MobilePhone@33 as MobilePhone, MobilePhoneModel@34 as MobilePhoneModel,
Params@35 as Params, IPNetworkID@36 as IPNetworkID, TraficSourceID@37 as
TraficSourceID, SearchEngineID@38 as SearchEngineID, SearchPhrase@39 as
SearchPhrase, AdvEngineID@40 as AdvEngineID, IsArtifical@41 as IsArtifical,
WindowClientWidth@42 as WindowClientWidth, WindowClientHeight@43 as
WindowClientHeight, ClientTimeZone@44 as ClientTimeZone, ClientEventTime@45 as
ClientEventTime, SilverlightVersion1@46 as SilverlightVersion1,
SilverlightVersion2@47 as SilverlightVersion2, SilverlightVersion3@48 as
SilverlightVersion3, SilverlightVersion4@49 as SilverlightVersion4,
PageCharset@50 as PageCharset, CodeVersion@51 as CodeVersion, IsLink@52 as
IsLink, IsDownload@53 as IsDownload, IsNotBounce@54 as IsNotBounce, FUniqID@55
as FUniqID, OriginalURL@56 as OriginalURL, HID@57 as HID, IsOldCounter@58 as
IsOldCounter, IsEvent@59 as IsEvent, IsParameter@60 as IsParameter,
DontCountHits@61 as DontCountHits, WithHash@62
as WithHash, HitColor@63 as HitColor, LocalEventTime@64 as LocalEventTime,
Age@65 as Age, Sex@66 as Sex, Income@67 as Income, Interests@68 as Interests,
Robotness@69 as Robotness, RemoteIP@70 as RemoteIP, WindowName@71 as
WindowName, OpenerName@72 as OpenerName, HistoryLength@73 as HistoryLength,
BrowserLanguage@74 as BrowserLanguage, BrowserCountry@75 as BrowserCountry,
SocialNetwork@76 as SocialNetwork, SocialAction@77 as SocialAction,
HTTPError@78 as HTTPError, SendTiming@79 as SendTiming, DNSTiming@80 as
DNSTiming, ConnectTiming@81 as ConnectTiming, ResponseStartTiming@82 as
ResponseStartTiming, ResponseEndTiming@83 as ResponseEndTiming, FetchTiming@84
as FetchTiming, SocialSourceNetworkID@85 as SocialSourceNetworkID,
SocialSourcePage@86 as SocialSourcePage, ParamPrice@87 as ParamPrice,
ParamOrderID@88 as ParamOrderID, ParamCurrency@89 as ParamCurrency,
ParamCurrencyID@90 as ParamCurrencyID, OpenstatServiceName@91 as
OpenstatServiceName, OpenstatCampaignID@92 as OpenstatCampaign
ID, OpenstatAdID@93 as OpenstatAdID, OpenstatSourceID@94 as OpenstatSourceID,
UTMSource@95 as UTMSource, UTMMedium@96 as UTMMedium, UTMCampaign@97 as
UTMCampaign, UTMContent@98 as UTMContent, UTMTerm@99 as UTMTerm, FromTag@100 as
FromTag, HasGCLID@101 as HasGCLID, RefererHash@102 as RefererHash, URLHash@103
as URLHash, CLID@104 as CLID], metrics=[output_rows=1,
elapsed_compute=21.342µs, spill_count=0, spilled_bytes=0, mem_used=0] |
| | GlobalLimitExec: skip=0, fetch=1,
metrics=[output_rows=1, elapsed_compute=55.749µs, spill_count=0,
spilled_bytes=0, mem_used=0]
|
| | CoalescePartitionsExec, metrics=[output_rows=410,
elapsed_compute=19.372µs, spill_count=0, spilled_bytes=0, mem_used=0]
|
| | CoalesceBatchesExec: target_batch_size=8192,
metrics=[output_rows=410, elapsed_compute=25.377822ms, spill_count=0,
spilled_bytes=0, mem_used=0]
|
| | FilterExec: URL@13 LIKE %google%,
metrics=[output_rows=10767, elapsed_compute=12.106422195s, spill_count=0,
spilled_bytes=0, mem_used=0]
|
| | ParquetExec: limit=None, partitions={12
groups: [[hits.parquet:0..1231664704], [hits.parquet:1231664704..2463329408],
[hits.parquet:2463329408..3694994112], [hits.parquet:3694994112..4926658816],
[hits.parquet:4926658816..6158323520], [hits.parquet:6158323520..7389988224],
[hits.parquet:7389988224..8621652928], [hits.parquet:8621652928..9853317632],
[hits.parquet:9853317632..11084982336],
[hits.parquet:11084982336..12316647040],
[hits.parquet:12316647040..13548311744],
[hits.parquet:13548311744..14779976446]]}, predicate=URL LIKE Utf8("%google%"),
projection=[WatchID, JavaEnable, Title, GoodEvent, EventTime, EventDate,
CounterID, ClientIP, RegionID, UserID, CounterClass, OS, UserAgent, URL,
Referer, IsRefresh, RefererCategoryID, RefererRegionID, URLCategoryID,
URLRegionID, ResolutionWidth, ResolutionHeight, ResolutionDepth, FlashMajor,
FlashMinor, FlashMinor2, NetMajor, NetMinor, UserAgentMajor, UserAgentMinor,
CookieEnable, JavascriptEnable, IsMobi
le, MobilePhone, MobilePhoneModel, Params, IPNetworkID, TraficSourceID,
SearchEngineID, SearchPhrase, AdvEngineID, IsArtifical, WindowClientWidth,
WindowClientHeight, ClientTimeZone, ClientEventTime, SilverlightVersion1,
SilverlightVersion2, SilverlightVersion3, SilverlightVersion4, PageCharset,
CodeVersion, IsLink, IsDownload, IsNotBounce, FUniqID, OriginalURL, HID,
IsOldCounter, IsEvent, IsParameter, DontCountHits, WithHash, HitColor,
LocalEventTime, Age, Sex, Income, Interests, Robotness, RemoteIP, WindowName,
OpenerName, HistoryLength, BrowserLanguage, BrowserCountry, SocialNetwork,
SocialAction, HTTPError, SendTiming, DNSTiming, ConnectTiming,
ResponseStartTiming, ResponseEndTiming, FetchTiming, SocialSourceNetworkID,
SocialSourcePage, ParamPrice, ParamOrderID, ParamCurrency, ParamCurrencyID,
OpenstatServiceName, OpenstatCampaignID, OpenstatAdID, OpenstatSourceID,
UTMSource, UTMMedium, UTMCampaign, UTMContent, UTMTerm, FromTag, HasGCLID,
RefererHash, URLHash, CLID], metrics=[ou
tput_rows=73363236, elapsed_compute=12ns, spill_count=0, spilled_bytes=0,
mem_used=0, predicate_evaluation_errors=0, pushdown_rows_filtered=0,
row_groups_pruned=0, bytes_scanned=11496382994, page_index_rows_filtered=0,
num_predicate_creation_errors=0, time_elapsed_scanning_total=151.775198747s,
pushdown_eval_time=24ns, time_elapsed_processing=133.355923578s,
time_elapsed_opening=1.135712999s, page_index_eval_time=24ns,
time_elapsed_scanning_until_data=2.005873968s]
|
| |
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set. Query took 12.817 seconds.
```
DuckDB Plan
```
> explain ANALYZE SELECT * FROM read_parquet('hits.parquet') WHERE URL LIKE
'%google%' LIMIT 1;
┌─────────────────────────────┐
│┌───────────────────────────┐│
│└───────────────────────────┘│
└─────────────────────────────┘
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Query Profiling Information ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
explain ANALYZE SELECT * FROM read_parquet('hits.parquet') WHERE URL LIKE
'%google%' LIMIT 1;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Total Time: 0.475s ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌───────────────────────────┐
│ EXPLAIN_ANALYZE │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 0 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ LIMIT │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 1 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PROJECTION │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ WatchID │
│ JavaEnable │
│ Title │
│ GoodEvent │
│ EventTime │
│ EventDate │
│ CounterID │
│ ClientIP │
│ RegionID │
│ UserID │
│ CounterClass │
│ OS │
│ UserAgent │
│ URL │
│ Referer │
│ IsRefresh │
│ RefererCategoryID │
│ RefererRegionID │
│ URLCategoryID │
│ URLRegionID │
│ ResolutionWidth │
│ ResolutionHeight │
│ ResolutionDepth │
│ FlashMajor │
│ FlashMinor │
│ FlashMinor2 │
│ NetMajor │
│ NetMinor │
│ UserAgentMajor │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ FILTER │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ contains(URL, 'google') │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 29 │
│ (0.07s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PARQUET_SCAN │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ URL │
│ WatchID │
│ JavaEnable │
│ Title │
│ GoodEvent │
│ EventTime │
│ EventDate │
│ CounterID │
│ ClientIP │
│ RegionID │
│ UserID │
│ CounterClass │
│ OS │
│ UserAgent │
│ Referer │
│ IsRefresh │
│ RefererCategoryID │
│ RefererRegionID │
│ URLCategoryID │
│ URLRegionID │
│ ResolutionWidth │
│ ResolutionHeight │
│ ResolutionDepth │
│ FlashMajor │
│ FlashMinor │
│ FlashMinor2 │
│ NetMajor │
│ NetMinor │
│ UserAgentMajor │
└───────────────────────────┘
Run Time (s): real 0.590 user 1.888174 sys 0.381627
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
