progval commented on code in PR #12585:
URL: https://github.com/apache/datafusion/pull/12585#discussion_r1773675575
##########
datafusion/core/src/datasource/physical_plan/parquet/opener.rs:
##########
@@ -187,15 +190,18 @@ impl FileOpener for ParquetOpener {
}
// If there is a predicate that can be evaluated against the
metadata
if let Some(predicate) = predicate.as_ref() {
+ let mut timer = file_metrics.statistics_eval_time.timer();
row_groups.prune_by_statistics(
&file_schema,
builder.parquet_schema(),
rg_metadata,
predicate,
&file_metrics,
);
+ timer.stop();
if enable_bloom_filter && !row_groups.is_empty() {
+ let mut timer =
file_metrics.bloom_filter_eval_time.timer();
Review Comment:
Good point I did the latter, because it's probably closer to what users
expect.
##########
datafusion/core/src/datasource/physical_plan/parquet/opener.rs:
##########
@@ -187,15 +190,18 @@ impl FileOpener for ParquetOpener {
}
// If there is a predicate that can be evaluated against the
metadata
if let Some(predicate) = predicate.as_ref() {
+ let mut timer = file_metrics.statistics_eval_time.timer();
row_groups.prune_by_statistics(
&file_schema,
builder.parquet_schema(),
rg_metadata,
predicate,
&file_metrics,
);
+ timer.stop();
Review Comment:
Doesn't matter anymore, as I moved both calls to the callees.
##########
datafusion/core/src/datasource/physical_plan/parquet/opener.rs:
##########
@@ -187,15 +190,18 @@ impl FileOpener for ParquetOpener {
}
// If there is a predicate that can be evaluated against the
metadata
if let Some(predicate) = predicate.as_ref() {
+ let mut timer = file_metrics.statistics_eval_time.timer();
row_groups.prune_by_statistics(
&file_schema,
builder.parquet_schema(),
rg_metadata,
predicate,
&file_metrics,
);
+ timer.stop();
if enable_bloom_filter && !row_groups.is_empty() {
+ let mut timer =
file_metrics.bloom_filter_eval_time.timer();
Review Comment:
Good point. I did the latter, because it's probably closer to what users
expect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
