BlakeOrth commented on PR #18103: URL: https://github.com/apache/datafusion/pull/18103#issuecomment-3412526966
> I tested this locally, and I don't see any LIST appearing in the output 🤔 > > I think you have to also instrument `list_wit_delimiter` 🤔 > > ```sql > andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ cargo run -p datafusion-cli > Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.18s > Running `target/debug/datafusion-cli` > DataFusion CLI v50.2.0 > > \object_store_profiling trace > ObjectStore Profile mode set to Trace > > select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet'; > +----------+ > | count(*) | > +----------+ > | 1000000 | > +----------+ > 1 row(s) fetched. > Elapsed 0.454 seconds. > > Object Store Profiling > Instrumented Object Store: instrument_mode: Trace, inner: HttpStore > 2025-10-16T19:01:11.225234+00:00 operation=Get duration=0.036593s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet > 2025-10-16T19:01:11.262391+00:00 operation=Get duration=0.036872s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet > > Summaries: > Get > count: 2 > duration min: 0.036593s > duration max: 0.036872s > duration avg: 0.036733s > size min: 8 B > size max: 34322 B > size avg: 17165 B > size sum: 34330 B > ``` > > EDIT: I do see it when I use `CREATE EXTERNAL TABLE`: > > ```sql > Object Store Profiling > Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data) > 2025-10-16T19:03:44.140517+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet > 2025-10-16T19:03:44.268832+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet > ``` So I think this is more or less expected. `https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet` is a single file behind an `https` endpoint, and as such there's no need (and in some sense no functional mechanism) for `ObjectStore`to list that `http` endpoint. I think we'd expect list to apply to buckets (like s3 in my example and your edit) and local directories. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
