BlakeOrth commented on PR #17266:
URL: https://github.com/apache/datafusion/pull/17266#issuecomment-3215858079
@alamb
> In my mind I was going to have time to work on this myself, but I fear
that is not likely to be the case for a while (and I will be out for the next
week or so on vacation, though I will be reviewing PRs as much as possible).
>
> If you are willing to help push this forward that would be most appreciated
I'm happy to keep driving this effort as long as it sounds like it's moving
in the right direction. My interpretation of your above comments is that we
currently are moving along at least mostly the right path, so I should be able
to generally make progress. (And do your best to enjoy your vacation! The code
will be here when you return)
I just pushed some changes the implement the summary output. I get the
feeling we're about to learn quite a lot...
```sql
DataFusion CLI v49.0.1
> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION
's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
0 row(s) fetched.
Elapsed 2.587 seconds.
List Summary:
count: 1
Get Summary:
count: 288
duration min: 0.058361s
duration max: 0.374491s
duration avg: 0.122724s
size min: 8
size max: 44247
size avg: 18870
size sum: 5434702
List Summary:
count: 1
> select count(*) from
's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' where
vendor_id='CMT';
+-----------+
| count(*) |
+-----------+
| 505603754 |
+-----------+
1 row(s) fetched.
Elapsed 56.057 seconds.
Get Summary:
count: 1126
duration min: 0.062342s
duration max: 1.831455s
duration avg: 0.397414s
size min: 47
size max: 112
size avg: 69
size sum: 78422
List Summary:
count: 4
```
(caveat: this is a debug build, not `release-nonlto`, but interesting data
nonetheless)
> I think it would be ok to always use the "Instrumented Object REgistry"
and then only pass back an instrumented object store if the profiling was
enabled
I think this is an interesting thought and I'll take some time to explore
this as a solution. Are you thinking the `InstrumentedObjectStoreRegistry`
would just pass back the provided `inner` object store when profiling is
disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
