Re: [I] Support limit pruning [datafusion]

Fri, 21 Nov 2025 20:23:57 -0800 


2010YOUY01 commented on issue #18860:
URL: https://github.com/apache/datafusion/issues/18860#issuecomment-3565575261

   > > the parquet reader will stop (and not fetch any more row groups) once 
the limit is hit.
   > 
   > There is a picture from snowflake, we can set the partition in the pic as 
row group.
   > 
   > The filter is `WHERE species LIKE'Alpine%' AND s >= 50`
   > 
   > <img alt="Image" width="469" height="291" 
src="https://private-user-images.githubusercontent.com/41979257/517446334-0fc77177-eda4-467a-835b-c6952f4077d9.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjM3ODUyMDEsIm5iZiI6MTc2Mzc4NDkwMSwicGF0aCI6Ii80MTk3OTI1Ny81MTc0NDYzMzQtMGZjNzcxNzctZWRhNC00NjdhLTgzNWItYzY5NTJmNDA3N2Q5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTExMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMTIyVDA0MTUwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkyNWEzNzZmZjA1OWY1YjU4ZjQ5ZTAzOWIyMWU5MjEwYWVmMDljZWE4ZjkzZDFmMjBkMzUwOTNmNmY0OTFkOTcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.VRtEH4clPjunmyMgAcnwAC-CFMQO4_-AT5f66rx0rvE";>
   > rg1 -> pruned, rg2, rg4 -> partially matched, rg3 fully matched.
   > 
   > We need to read rg2, rg3, then get the satisfied rows from rg3. But if we 
have limit pruning and a fully matched rg definition, we can directly fetch 
limit rows from rg3.
   
   I also want to double check I get it correctly, did you mean:
   
   Now we have a query
   ```sql
   SELECT * 
   FROM t 
   WHERE species LIKE'Alpine%' AND s >= 50
   LIMIT 3
   ```
   And without this limit pruning feature, the plan might look like
   ```
   -- Executing with `target_partition=3`, and we already got partition 1 
skipped because it's stat shows 'not matched'
   DataSourceExec (partition 2)
   DataSourceExec (partition 3)
   DataSourceExec (partition 4)
   ```
   From the stat, we can ensure partition 3 will output 3 rows satisfying the 
`LIMIT 3`, so we can safely skip reading partition 2 and 4.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to