BlakeOrth commented on issue #18119: URL: https://github.com/apache/datafusion/issues/18119#issuecomment-3416433123
@alamb I believe you may have been "bit" by the excellent performance improvements from @nuno-faria and the metadata cache here :) Disabling the cache shows some expected activity from the underlying store. ```sql DataFusion CLI v50.2.0 > set datafusion.runtime.metadata_cache_limit = '0M'; 0 row(s) fetched. Elapsed 0.002 seconds. > COPY 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' to 'hits_1_index.parquet'; +---------+ | count | +---------+ | 1000000 | +---------+ 1 row(s) fetched. Elapsed 21.988 seconds. > \object_store_profiling trace ObjectStore Profile mode set to Trace > SELECT COUNT(*) from 'hits_1_index.parquet' where "SearchPhrase" <> ''; +----------+ | count(*) | +----------+ | 131559 | +----------+ 1 row(s) fetched. Elapsed 0.258 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Trace, inner: LocalFileSystem(file:///) 2025-10-17T17:17:17.229500671+00:00 operation=Get duration=0.000036s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.229594018+00:00 operation=Get duration=0.000029s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.230720213+00:00 operation=Get duration=0.000086s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.243778013+00:00 operation=Get duration=0.000034s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.243842805+00:00 operation=Get duration=0.000033s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.244849498+00:00 operation=Get duration=0.000065s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260849153+00:00 operation=Get duration=0.000216s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260922554+00:00 operation=Get duration=0.000324s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260922546+00:00 operation=Get duration=0.000463s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261163562+00:00 operation=Get duration=0.000222s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260947121+00:00 operation=Get duration=0.000603s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260984618+00:00 operation=Get duration=0.000626s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261123655+00:00 operation=Get duration=0.000592s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261312958+00:00 operation=Get duration=0.000460s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261123654+00:00 operation=Get duration=0.000767s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261218152+00:00 operation=Get duration=0.000694s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261232355+00:00 operation=Get duration=0.000704s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261141902+00:00 operation=Get duration=0.000885s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261244353+00:00 operation=Get duration=0.000801s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261606333+00:00 operation=Get duration=0.000443s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.260927601+00:00 operation=Get duration=0.001221s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261939845+00:00 operation=Get duration=0.000253s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262110141+00:00 operation=Get duration=0.000181s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262063427+00:00 operation=Get duration=0.000348s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262150416+00:00 operation=Get duration=0.000400s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262176237+00:00 operation=Get duration=0.000465s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261241804+00:00 operation=Get duration=0.001466s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261100559+00:00 operation=Get duration=0.001607s size=8 range: bytes=110630760-110630767 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.261834926+00:00 operation=Get duration=0.000896s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262171990+00:00 operation=Get duration=0.000868s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262873290+00:00 operation=Get duration=0.000171s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262064519+00:00 operation=Get duration=0.001049s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262930937+00:00 operation=Get duration=0.001465s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262932113+00:00 operation=Get duration=0.001481s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.263070727+00:00 operation=Get duration=0.001354s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262737063+00:00 operation=Get duration=0.001896s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.262455380+00:00 operation=Get duration=0.002193s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264338510+00:00 operation=Get duration=0.000327s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264595981+00:00 operation=Get duration=0.000294s size=23460 range: bytes=110607300-110630759 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264616932+00:00 operation=Get duration=0.000289s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264661781+00:00 operation=Get duration=0.000313s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264533660+00:00 operation=Get duration=0.000461s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264834065+00:00 operation=Get duration=0.000221s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264890179+00:00 operation=Get duration=0.000361s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264930130+00:00 operation=Get duration=0.001093s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.264998856+00:00 operation=Get duration=0.001042s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.265190045+00:00 operation=Get duration=0.000878s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.266018787+00:00 operation=Get duration=0.000118s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.266208265+00:00 operation=Get duration=0.000146s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.266456771+00:00 operation=Get duration=0.000107s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.266611206+00:00 operation=Get duration=0.000120s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.266788020+00:00 operation=Get duration=0.000101s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.267139786+00:00 operation=Get duration=0.001899s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.267952933+00:00 operation=Get duration=0.001088s size=179295 range: bytes=110428005-110607299 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet 2025-10-17T17:17:17.286706015+00:00 operation=Get duration=0.000176s size=2306987 range: bytes=66885459-69192445 path=home/blake/open_source_src/datafusion-BlakeOrth/hits_1_index.parquet Summaries: Get count: 55 duration min: 0.000029s duration max: 0.002193s duration avg: 0.000626s size min: 8 B size max: 2306987 B size avg: 108304 B size sum: 5956721 B > ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
