bchalk101 opened a new issue, #13745:
URL: https://github.com/apache/datafusion/issues/13745
### Describe the bug
I am trying to load a parquet dataset, using both a limit and filter. When
combining this with the `pushdown_filters` config, no data is found.
If I either remove the `limit` or remove `pushdown_filters` than it works.
### To Reproduce
The following code reproduces the issues.
The dataset here is a path in S3, which contains 120 parquets, and each
parquet has about 7000-8000 rows.
```
let object_store = Arc::new(aws_s3);
let mut config = SessionConfig::new();
config.options_mut().execution.parquet.pushdown_filters = true;
let state = SessionStateBuilder::new().with_config(config).build();
let ctx = SessionContext::from(state);
ctx.register_object_store(object_store_url.as_ref(),
object_store.clone());
let mut parquet_options = ParquetReadOptions::new();
parquet_options = parquet_options.parquet_pruning(true);
let mut df = ctx
.read_parquet(path, parquet_options.clone())
.await
.unwrap();
df = df
.filter(col("clip_name").eq(lit(
"GAS2_Mz01_240225_062204_100706_s001_p_Front-1_s60_0029",
)))
.unwrap();
df = df.limit(0, Some(1)).unwrap();
let batch = df.collect().await.unwrap();
```
### Expected behavior
Expected to apply both limit and filter.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
