BlakeOrth commented on issue #17207: URL: https://github.com/apache/datafusion/issues/17207#issuecomment-3208159926
@alamb I believe I, more or less, have what you've suggested there. My approach is currently subtly different, but I think the general idea is the same. I will push a draft PR so you can see the concrete implementation, as that may help show the issue a bit better, or give you the opportunity to explain something that I may be missing and/or over-complicating. I've also realized there's another subtle complication here. Producing output from individual object store requests is only part of the story, and the summary statistics I initially recommended are potentially misleading. The issue is object store requests are (rightfully) executed in parallel in order to limit the overall latency penalty they apply to any individual operation. In the in-progress output below, the `total_duration` (sum of all individual requests) for creating the table is reported as `37.010551s` but the real elapsed time is `2.281s`. (note: the issue I noted above for listing operations can also be seen here, clearly listing from remote storage is not actually completing in single microsecond durations) ```sql DataFusion CLI v49.0.1 > CREATE EXTERNAL TABLE nyc_taxi_rides STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/'; 0 row(s) fetched. Elapsed 2.281 seconds. 2025-08-20T21:20:24.580384173+00:00 operation=List duration=0.000013s path=nyc_taxi_rides/data/tripdata_parquet Total duration: 0.000013 2025-08-20T21:20:24.918107150+00:00 operation=List duration=0.000006s path=nyc_taxi_rides/data/tripdata_parquet 2025-08-20T21:20:25.172969667+00:00 operation=Get duration=0.260397s path=nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet 2025-08-20T21:20:25.167966077+00:00 operation=Get duration=0.268494s path=nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet 2025-08-20T21:20:25.170653460+00:00 operation=Get duration=0.276504s path=nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet 2025-08-20T21:20:25.167315490+00:00 operation=Get duration=0.282914s path=nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet 2025-08-20T21:20:25.166233527+00:00 operation=Get duration=0.288857s path=nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet 2025-08-20T21:20:25.165742384+00:00 operation=Get duration=0.290442s path=nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet 2025-08-20T21:20:25.171304487+00:00 operation=Get duration=0.285823s path=nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet 2025-08-20T21:20:25.171645253+00:00 operation=Get duration=0.289283s path=nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet 2025-08-20T21:20:25.169966122+00:00 operation=Get duration=0.291278s path=nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet 2025-08-20T21:20:25.165205488+00:00 operation=Get duration=0.305158s path=nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet 2025-08-20T21:20:25.165995504+00:00 operation=Get duration=0.307101s path=nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet 2025-08-20T21:20:25.166756565+00:00 operation=Get duration=0.309858s path=nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet 2025-08-20T21:20:25.164821169+00:00 operation=Get duration=0.312160s path=nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet 2025-08-20T21:20:25.165482889+00:00 operation=Get duration=0.312901s path=nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet 2025-08-20T21:20:25.170308450+00:00 operation=Get duration=0.313416s path=nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet 2025-08-20T21:20:25.169638732+00:00 operation=Get duration=0.315370s path=nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet 2025-08-20T21:20:25.167648050+00:00 operation=Get duration=0.321611s path=nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet 2025-08-20T21:20:25.166476219+00:00 operation=Get duration=0.325231s path=nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet 2025-08-20T21:20:25.168305592+00:00 operation=Get duration=0.324585s path=nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet 2025-08-20T21:20:25.173300552+00:00 operation=Get duration=0.320743s path=nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet 2025-08-20T21:20:25.433458511+00:00 operation=Get duration=0.064714s path=nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet 2025-08-20T21:20:25.174622822+00:00 operation=Get duration=0.324518s path=nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet 2025-08-20T21:20:25.173626668+00:00 operation=Get duration=0.328044s path=nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet 2025-08-20T21:20:25.172631825+00:00 operation=Get duration=0.334044s path=nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet 2025-08-20T21:20:25.168975608+00:00 operation=Get duration=0.338430s path=nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet 2025-08-20T21:20:25.450262945+00:00 operation=Get duration=0.060749s path=nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet 2025-08-20T21:20:25.456217057+00:00 operation=Get duration=0.059382s path=nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet 2025-08-20T21:20:25.173956464+00:00 operation=Get duration=0.345046s path=nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet 2025-08-20T21:20:25.436496530+00:00 operation=Get duration=0.083541s path=nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet 2025-08-20T21:20:25.455118283+00:00 operation=Get duration=0.066443s path=nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet 2025-08-20T21:20:25.169306279+00:00 operation=Get duration=0.352345s path=nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet 2025-08-20T21:20:25.460957238+00:00 operation=Get duration=0.062608s path=nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet 2025-08-20T21:20:25.172298112+00:00 operation=Get duration=0.352401s path=nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet 2025-08-20T21:20:25.461270454+00:00 operation=Get duration=0.066084s path=nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet 2025-08-20T21:20:25.470389541+00:00 operation=Get duration=0.062182s path=nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet 2025-08-20T21:20:25.170979431+00:00 operation=Get duration=0.364597s path=nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet 2025-08-20T21:20:25.477006932+00:00 operation=Get duration=0.068757s path=nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet 2025-08-20T21:20:25.171975127+00:00 operation=Get duration=0.373856s path=nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet 2025-08-20T21:20:25.473122698+00:00 operation=Get duration=0.076832s path=nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet 2025-08-20T21:20:25.476640870+00:00 operation=Get duration=0.073384s path=nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet 2025-08-20T21:20:25.483751428+00:00 operation=Get duration=0.067850s path=nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet 2025-08-20T21:20:25.489287595+00:00 operation=Get duration=0.068233s path=nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet 2025-08-20T21:20:25.494070223+00:00 operation=Get duration=0.067057s path=nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet 2025-08-20T21:20:25.167088028+00:00 operation=Get duration=0.394123s path=nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet 2025-08-20T21:20:25.478409402+00:00 operation=Get duration=0.085458s path=nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet 2025-08-20T21:20:25.499166919+00:00 operation=Get duration=0.064774s path=nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet 2025-08-20T21:20:25.492916725+00:00 operation=Get duration=0.074454s path=nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet 2025-08-20T21:20:25.501705023+00:00 operation=Get duration=0.066853s path=nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet 2025-08-20T21:20:25.485036287+00:00 operation=Get duration=0.087981s path=nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet 2025-08-20T21:20:25.174291616+00:00 operation=Get duration=0.404580s path=nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet 2025-08-20T21:20:25.447185097+00:00 operation=Get duration=0.136710s path=nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet 2025-08-20T21:20:25.519079992+00:00 operation=Get duration=0.068945s path=nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet 2025-08-20T21:20:25.457154942+00:00 operation=Get duration=0.133995s path=nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet 2025-08-20T21:20:25.491734372+00:00 operation=Get duration=0.101180s path=nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet 2025-08-20T21:20:25.521675965+00:00 operation=Get duration=0.077484s path=nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet 2025-08-20T21:20:25.535604283+00:00 operation=Get duration=0.073358s path=nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet 2025-08-20T21:20:25.506701151+00:00 operation=Get duration=0.104436s path=nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet 2025-08-20T21:20:25.549676455+00:00 operation=Get duration=0.063781s path=nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet 2025-08-20T21:20:25.545854542+00:00 operation=Get duration=0.072466s path=nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet 2025-08-20T21:20:25.545456359+00:00 operation=Get duration=0.072967s path=nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet 2025-08-20T21:20:25.557265446+00:00 operation=Get duration=0.061247s path=nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet 2025-08-20T21:20:25.563564905+00:00 operation=Get duration=0.065479s path=nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet 2025-08-20T21:20:25.567128036+00:00 operation=Get duration=0.070290s path=nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet 2025-08-20T21:20:25.561234403+00:00 operation=Get duration=0.078902s path=nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet 2025-08-20T21:20:25.524723630+00:00 operation=Get duration=0.115541s path=nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet 2025-08-20T21:20:25.576673265+00:00 operation=Get duration=0.071180s path=nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet 2025-08-20T21:20:25.572769892+00:00 operation=Get duration=0.075184s path=nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet 2025-08-20T21:20:25.581458767+00:00 operation=Get duration=0.071764s path=nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet 2025-08-20T21:20:25.578901959+00:00 operation=Get duration=0.077023s path=nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet 2025-08-20T21:20:25.596792767+00:00 operation=Get duration=0.062112s path=nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet 2025-08-20T21:20:25.507432225+00:00 operation=Get duration=0.157101s path=nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet 2025-08-20T21:20:25.606462029+00:00 operation=Get duration=0.058189s path=nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet 2025-08-20T21:20:25.601228269+00:00 operation=Get duration=0.068654s path=nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet 2025-08-20T21:20:25.598868918+00:00 operation=Get duration=0.074760s path=nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet 2025-08-20T21:20:25.610842423+00:00 operation=Get duration=0.065517s path=nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet 2025-08-20T21:20:25.608669694+00:00 operation=Get duration=0.070496s path=nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet 2025-08-20T21:20:25.613169978+00:00 operation=Get duration=0.070279s path=nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet 2025-08-20T21:20:25.615461756+00:00 operation=Get duration=0.070634s path=nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet 2025-08-20T21:20:25.621057378+00:00 operation=Get duration=0.071968s path=nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet 2025-08-20T21:20:25.168634929+00:00 operation=Get duration=0.528040s path=nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet 2025-08-20T21:20:25.570553207+00:00 operation=Get duration=0.132171s path=nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet 2025-08-20T21:20:25.636623888+00:00 operation=Get duration=0.072859s path=nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet 2025-08-20T21:20:25.642385489+00:00 operation=Get duration=0.067181s path=nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet 2025-08-20T21:20:25.633591382+00:00 operation=Get duration=0.076071s path=nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet 2025-08-20T21:20:25.644710789+00:00 operation=Get duration=0.065039s path=nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet 2025-08-20T21:20:25.628741058+00:00 operation=Get duration=0.090550s path=nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet 2025-08-20T21:20:25.658544407+00:00 operation=Get duration=0.068048s path=nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet 2025-08-20T21:20:25.666884828+00:00 operation=Get duration=0.063899s path=nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet 2025-08-20T21:20:25.661329853+00:00 operation=Get duration=0.086199s path=nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet 2025-08-20T21:20:25.647520004+00:00 operation=Get duration=0.104351s path=nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet 2025-08-20T21:20:25.688065562+00:00 operation=Get duration=0.068063s path=nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet 2025-08-20T21:20:25.692727972+00:00 operation=Get duration=0.067918s path=nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet 2025-08-20T21:20:25.704834458+00:00 operation=Get duration=0.064188s path=nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet 2025-08-20T21:20:25.709192074+00:00 operation=Get duration=0.080505s path=nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet 2025-08-20T21:20:25.696710971+00:00 operation=Get duration=0.094618s path=nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet 2025-08-20T21:20:25.655963457+00:00 operation=Get duration=0.159518s path=nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet 2025-08-20T21:20:25.618540972+00:00 operation=Get duration=0.209641s path=nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet 2025-08-20T21:20:25.670630844+00:00 operation=Get duration=0.168034s path=nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet 2025-08-20T21:20:25.669921633+00:00 operation=Get duration=0.187359s path=nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet 2025-08-20T21:20:25.719561868+00:00 operation=Get duration=0.156295s path=nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet 2025-08-20T21:20:25.670400759+00:00 operation=Get duration=0.206251s path=nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet 2025-08-20T21:20:25.815515036+00:00 operation=Get duration=0.068054s path=nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet 2025-08-20T21:20:25.719327042+00:00 operation=Get duration=0.171263s path=nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet 2025-08-20T21:20:25.828212765+00:00 operation=Get duration=0.066983s path=nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet 2025-08-20T21:20:25.670167131+00:00 operation=Get duration=0.228254s path=nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet 2025-08-20T21:20:25.836047403+00:00 operation=Get duration=0.063971s path=nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet 2025-08-20T21:20:25.759522223+00:00 operation=Get duration=0.151869s path=nyc_taxi_rides/data/tripdata_parquet/data-201205.parquet 2025-08-20T21:20:25.839014091+00:00 operation=Get duration=0.078779s path=nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet 2025-08-20T21:20:25.857309890+00:00 operation=Get duration=0.071435s path=nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet 2025-08-20T21:20:25.760383966+00:00 operation=Get duration=0.180220s path=nyc_taxi_rides/data/tripdata_parquet/data-201209.parquet 2025-08-20T21:20:25.875927967+00:00 operation=Get duration=0.070672s path=nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet 2025-08-20T21:20:25.760186326+00:00 operation=Get duration=0.191529s path=nyc_taxi_rides/data/tripdata_parquet/data-201208.parquet 2025-08-20T21:20:25.898453308+00:00 operation=Get duration=0.059756s path=nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet 2025-08-20T21:20:25.898079235+00:00 operation=Get duration=0.066267s path=nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet 2025-08-20T21:20:25.759791972+00:00 operation=Get duration=0.208448s path=nyc_taxi_rides/data/tripdata_parquet/data-201206.parquet 2025-08-20T21:20:25.876685439+00:00 operation=Get duration=0.099136s path=nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet 2025-08-20T21:20:25.911433587+00:00 operation=Get duration=0.067630s path=nyc_taxi_rides/data/tripdata_parquet/data-201205.parquet 2025-08-20T21:20:25.759994684+00:00 operation=Get duration=0.227309s path=nyc_taxi_rides/data/tripdata_parquet/data-201207.parquet 2025-08-20T21:20:25.920486366+00:00 operation=Get duration=0.069928s path=nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet 2025-08-20T21:20:25.931925458+00:00 operation=Get duration=0.066873s path=nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet 2025-08-20T21:20:25.934196067+00:00 operation=Get duration=0.068808s path=nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet 2025-08-20T21:20:25.890638617+00:00 operation=Get duration=0.122213s path=nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet 2025-08-20T21:20:25.940632784+00:00 operation=Get duration=0.074947s path=nyc_taxi_rides/data/tripdata_parquet/data-201209.parquet 2025-08-20T21:20:25.951742870+00:00 operation=Get duration=0.068883s path=nyc_taxi_rides/data/tripdata_parquet/data-201208.parquet 2025-08-20T21:20:25.962336069+00:00 operation=Get duration=0.064992s path=nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet 2025-08-20T21:20:25.905106619+00:00 operation=Get duration=0.129363s path=nyc_taxi_rides/data/tripdata_parquet/data-201306.parquet 2025-08-20T21:20:25.905957504+00:00 operation=Get duration=0.129897s path=nyc_taxi_rides/data/tripdata_parquet/data-201310.parquet 2025-08-20T21:20:25.903996433+00:00 operation=Get duration=0.140445s path=nyc_taxi_rides/data/tripdata_parquet/data-201301.parquet 2025-08-20T21:20:25.987327618+00:00 operation=Get duration=0.065464s path=nyc_taxi_rides/data/tripdata_parquet/data-201207.parquet 2025-08-20T21:20:25.906172917+00:00 operation=Get duration=0.146737s path=nyc_taxi_rides/data/tripdata_parquet/data-201311.parquet 2025-08-20T21:20:25.905746056+00:00 operation=Get duration=0.154410s path=nyc_taxi_rides/data/tripdata_parquet/data-201309.parquet 2025-08-20T21:20:25.996517166+00:00 operation=Get duration=0.068749s path=nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet 2025-08-20T21:20:25.906604224+00:00 operation=Get duration=0.159214s path=nyc_taxi_rides/data/tripdata_parquet/data-201401.parquet 2025-08-20T21:20:25.904905324+00:00 operation=Get duration=0.164659s path=nyc_taxi_rides/data/tripdata_parquet/data-201305.parquet 2025-08-20T21:20:25.968267244+00:00 operation=Get duration=0.101613s path=nyc_taxi_rides/data/tripdata_parquet/data-201206.parquet 2025-08-20T21:20:25.904235435+00:00 operation=Get duration=0.165724s path=nyc_taxi_rides/data/tripdata_parquet/data-201302.parquet 2025-08-20T21:20:25.907034651+00:00 operation=Get duration=0.167907s path=nyc_taxi_rides/data/tripdata_parquet/data-201403.parquet 2025-08-20T21:20:25.907233846+00:00 operation=Get duration=0.171073s path=nyc_taxi_rides/data/tripdata_parquet/data-201404.parquet 2025-08-20T21:20:25.904695536+00:00 operation=Get duration=0.182500s path=nyc_taxi_rides/data/tripdata_parquet/data-201304.parquet 2025-08-20T21:20:26.027043505+00:00 operation=Get duration=0.066848s path=nyc_taxi_rides/data/tripdata_parquet/data-201205.parquet 2025-08-20T21:20:26.020169666+00:00 operation=Get duration=0.074572s path=nyc_taxi_rides/data/tripdata_parquet/data-201209.parquet 2025-08-20T21:20:26.022851454+00:00 operation=Get duration=0.071954s path=nyc_taxi_rides/data/tripdata_parquet/data-201208.parquet 2025-08-20T21:20:25.903512374+00:00 operation=Get duration=0.195566s path=nyc_taxi_rides/data/tripdata_parquet/data-201211.parquet 2025-08-20T21:20:26.034503528+00:00 operation=Get duration=0.068446s path=nyc_taxi_rides/data/tripdata_parquet/data-201306.parquet 2025-08-20T21:20:25.905312663+00:00 operation=Get duration=0.205541s path=nyc_taxi_rides/data/tripdata_parquet/data-201307.parquet 2025-08-20T21:20:25.904448592+00:00 operation=Get duration=0.206783s path=nyc_taxi_rides/data/tripdata_parquet/data-201303.parquet 2025-08-20T21:20:26.044496302+00:00 operation=Get duration=0.072565s path=nyc_taxi_rides/data/tripdata_parquet/data-201301.parquet 2025-08-20T21:20:26.057017143+00:00 operation=Get duration=0.062931s path=nyc_taxi_rides/data/tripdata_parquet/data-201207.parquet 2025-08-20T21:20:26.035880382+00:00 operation=Get duration=0.086006s path=nyc_taxi_rides/data/tripdata_parquet/data-201310.parquet 2025-08-20T21:20:25.906388744+00:00 operation=Get duration=0.220260s path=nyc_taxi_rides/data/tripdata_parquet/data-201312.parquet 2025-08-20T21:20:26.060184008+00:00 operation=Get duration=0.068049s path=nyc_taxi_rides/data/tripdata_parquet/data-201309.parquet 2025-08-20T21:20:25.978428933+00:00 operation=Get duration=0.152305s path=nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet 2025-08-20T21:20:25.906816904+00:00 operation=Get duration=0.223996s path=nyc_taxi_rides/data/tripdata_parquet/data-201402.parquet 2025-08-20T21:20:26.053731717+00:00 operation=Get duration=0.077353s path=nyc_taxi_rides/data/tripdata_parquet/data-201311.parquet 2025-08-20T21:20:26.069594762+00:00 operation=Get duration=0.066262s path=nyc_taxi_rides/data/tripdata_parquet/data-201305.parquet 2025-08-20T21:20:26.069982730+00:00 operation=Get duration=0.068650s path=nyc_taxi_rides/data/tripdata_parquet/data-201302.parquet 2025-08-20T21:20:25.903761039+00:00 operation=Get duration=0.234958s path=nyc_taxi_rides/data/tripdata_parquet/data-201212.parquet 2025-08-20T21:20:26.059512229+00:00 operation=Get duration=0.085881s path=nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet 2025-08-20T21:20:26.078333750+00:00 operation=Get duration=0.074823s path=nyc_taxi_rides/data/tripdata_parquet/data-201404.parquet 2025-08-20T21:20:26.087225701+00:00 operation=Get duration=0.076338s path=nyc_taxi_rides/data/tripdata_parquet/data-201304.parquet 2025-08-20T21:20:26.099105939+00:00 operation=Get duration=0.067905s path=nyc_taxi_rides/data/tripdata_parquet/data-201211.parquet 2025-08-20T21:20:26.065844626+00:00 operation=Get duration=0.101251s path=nyc_taxi_rides/data/tripdata_parquet/data-201401.parquet 2025-08-20T21:20:26.110888577+00:00 operation=Get duration=0.062792s path=nyc_taxi_rides/data/tripdata_parquet/data-201307.parquet 2025-08-20T21:20:25.903201706+00:00 operation=Get duration=0.273774s path=nyc_taxi_rides/data/tripdata_parquet/data-201210.parquet 2025-08-20T21:20:26.126308045+00:00 operation=Get duration=0.060952s path=nyc_taxi_rides/data/tripdata_parquet/data-201310.parquet 2025-08-20T21:20:26.074969005+00:00 operation=Get duration=0.113668s path=nyc_taxi_rides/data/tripdata_parquet/data-201403.parquet 2025-08-20T21:20:26.111265770+00:00 operation=Get duration=0.079879s path=nyc_taxi_rides/data/tripdata_parquet/data-201303.parquet 2025-08-20T21:20:26.110492164+00:00 operation=Get duration=0.083188s path=nyc_taxi_rides/data/tripdata_parquet/data-201306.parquet 2025-08-20T21:20:26.116796884+00:00 operation=Get duration=0.077006s path=nyc_taxi_rides/data/tripdata_parquet/data-201206.parquet 2025-08-20T21:20:25.967982800+00:00 operation=Get duration=0.225915s path=nyc_taxi_rides/data/tripdata_parquet/data-201405.parquet 2025-08-20T21:20:26.126674011+00:00 operation=Get duration=0.072289s path=nyc_taxi_rides/data/tripdata_parquet/data-201312.parquet 2025-08-20T21:20:26.138354048+00:00 operation=Get duration=0.064087s path=nyc_taxi_rides/data/tripdata_parquet/data-201311.parquet 2025-08-20T21:20:26.138745981+00:00 operation=Get duration=0.066209s path=nyc_taxi_rides/data/tripdata_parquet/data-201212.parquet 2025-08-20T21:20:25.905530804+00:00 operation=Get duration=0.299508s path=nyc_taxi_rides/data/tripdata_parquet/data-201308.parquet 2025-08-20T21:20:26.130837327+00:00 operation=Get duration=0.074490s path=nyc_taxi_rides/data/tripdata_parquet/data-201402.parquet 2025-08-20T21:20:26.133116231+00:00 operation=Get duration=0.075043s path=nyc_taxi_rides/data/tripdata_parquet/data-201309.parquet 2025-08-20T21:20:26.140989674+00:00 operation=Get duration=0.071959s path=nyc_taxi_rides/data/tripdata_parquet/data-201302.parquet 2025-08-20T21:20:26.156961114+00:00 operation=Get duration=0.065397s path=nyc_taxi_rides/data/tripdata_parquet/data-201404.parquet 2025-08-20T21:20:26.031072418+00:00 operation=Get duration=0.191377s path=nyc_taxi_rides/data/tripdata_parquet/data-201408.parquet 2025-08-20T21:20:26.006408108+00:00 operation=Get duration=0.228261s path=nyc_taxi_rides/data/tripdata_parquet/data-201406.parquet 2025-08-20T21:20:26.175673702+00:00 operation=Get duration=0.067001s path=nyc_taxi_rides/data/tripdata_parquet/data-201307.parquet 2025-08-20T21:20:26.171269419+00:00 operation=Get duration=0.076037s path=nyc_taxi_rides/data/tripdata_parquet/data-201211.parquet 2025-08-20T21:20:26.177004333+00:00 operation=Get duration=0.070394s path=nyc_taxi_rides/data/tripdata_parquet/data-201210.parquet 2025-08-20T21:20:26.173305795+00:00 operation=Get duration=0.076543s path=nyc_taxi_rides/data/tripdata_parquet/data-201401.parquet 2025-08-20T21:20:26.006612588+00:00 operation=Get duration=0.247997s path=nyc_taxi_rides/data/tripdata_parquet/data-201407.parquet 2025-08-20T21:20:26.182229674+00:00 operation=Get duration=0.072977s path=nyc_taxi_rides/data/tripdata_parquet/data-201305.parquet 2025-08-20T21:20:26.202166550+00:00 operation=Get duration=0.070412s path=nyc_taxi_rides/data/tripdata_parquet/data-201312.parquet 2025-08-20T21:20:26.210202228+00:00 operation=Get duration=0.062455s path=nyc_taxi_rides/data/tripdata_parquet/data-201402.parquet 2025-08-20T21:20:26.205064716+00:00 operation=Get duration=0.072242s path=nyc_taxi_rides/data/tripdata_parquet/data-201308.parquet 2025-08-20T21:20:26.207611718+00:00 operation=Get duration=0.071888s path=nyc_taxi_rides/data/tripdata_parquet/data-201212.parquet 2025-08-20T21:20:26.193937752+00:00 operation=Get duration=0.085690s path=nyc_taxi_rides/data/tripdata_parquet/data-201405.parquet 2025-08-20T21:20:26.212658826+00:00 operation=Get duration=0.067089s path=nyc_taxi_rides/data/tripdata_parquet/data-201304.parquet 2025-08-20T21:20:26.222647238+00:00 operation=Get duration=0.063873s path=nyc_taxi_rides/data/tripdata_parquet/data-201408.parquet 2025-08-20T21:20:26.135871204+00:00 operation=Get duration=0.153840s path=nyc_taxi_rides/data/tripdata_parquet/data-201409.parquet 2025-08-20T21:20:26.136062969+00:00 operation=Get duration=0.153950s path=nyc_taxi_rides/data/tripdata_parquet/data-201410.parquet 2025-08-20T21:20:26.234372112+00:00 operation=Get duration=0.060520s path=nyc_taxi_rides/data/tripdata_parquet/data-201403.parquet 2025-08-20T21:20:26.239882514+00:00 operation=Get duration=0.065360s path=nyc_taxi_rides/data/tripdata_parquet/data-201303.parquet 2025-08-20T21:20:26.150538763+00:00 operation=Get duration=0.154787s path=nyc_taxi_rides/data/tripdata_parquet/data-201501.parquet 2025-08-20T21:20:26.169103288+00:00 operation=Get duration=0.139901s path=nyc_taxi_rides/data/tripdata_parquet/data-201301.parquet 2025-08-20T21:20:26.252047649+00:00 operation=Get duration=0.062229s path=nyc_taxi_rides/data/tripdata_parquet/data-201210.parquet 2025-08-20T21:20:26.254642112+00:00 operation=Get duration=0.073736s path=nyc_taxi_rides/data/tripdata_parquet/data-201407.parquet 2025-08-20T21:20:26.199590508+00:00 operation=Get duration=0.130384s path=nyc_taxi_rides/data/tripdata_parquet/data-201505.parquet 2025-08-20T21:20:26.150345802+00:00 operation=Get duration=0.188090s path=nyc_taxi_rides/data/tripdata_parquet/data-201412.parquet 2025-08-20T21:20:26.234698134+00:00 operation=Get duration=0.115458s path=nyc_taxi_rides/data/tripdata_parquet/data-201406.parquet 2025-08-20T21:20:26.290041256+00:00 operation=Get duration=0.068535s path=nyc_taxi_rides/data/tripdata_parquet/data-201410.parquet 2025-08-20T21:20:26.286198365+00:00 operation=Get duration=0.072521s path=nyc_taxi_rides/data/tripdata_parquet/data-201308.parquet 2025-08-20T21:20:26.289362340+00:00 operation=Get duration=0.070093s path=nyc_taxi_rides/data/tripdata_parquet/data-201408.parquet 2025-08-20T21:20:26.199189023+00:00 operation=Get duration=0.168427s path=nyc_taxi_rides/data/tripdata_parquet/data-201503.parquet 2025-08-20T21:20:26.150146480+00:00 operation=Get duration=0.225432s path=nyc_taxi_rides/data/tripdata_parquet/data-201411.parquet 2025-08-20T21:20:26.305351490+00:00 operation=Get duration=0.070347s path=nyc_taxi_rides/data/tripdata_parquet/data-201501.parquet 2025-08-20T21:20:26.198988172+00:00 operation=Get duration=0.176793s path=nyc_taxi_rides/data/tripdata_parquet/data-201502.parquet 2025-08-20T21:20:26.289741964+00:00 operation=Get duration=0.088652s path=nyc_taxi_rides/data/tripdata_parquet/data-201409.parquet 2025-08-20T21:20:26.322092062+00:00 operation=Get duration=0.067403s path=nyc_taxi_rides/data/tripdata_parquet/data-201405.parquet 2025-08-20T21:20:26.330005048+00:00 operation=Get duration=0.064982s path=nyc_taxi_rides/data/tripdata_parquet/data-201505.parquet 2025-08-20T21:20:26.338466358+00:00 operation=Get duration=0.060029s path=nyc_taxi_rides/data/tripdata_parquet/data-201412.parquet 2025-08-20T21:20:26.332018364+00:00 operation=Get duration=0.070365s path=nyc_taxi_rides/data/tripdata_parquet/data-201407.parquet 2025-08-20T21:20:26.199380499+00:00 operation=Get duration=0.213543s path=nyc_taxi_rides/data/tripdata_parquet/data-201504.parquet 2025-08-20T21:20:26.354169366+00:00 operation=Get duration=0.065521s path=nyc_taxi_rides/data/tripdata_parquet/data-201406.parquet 2025-08-20T21:20:26.376026309+00:00 operation=Get duration=0.061102s path=nyc_taxi_rides/data/tripdata_parquet/data-201411.parquet 2025-08-20T21:20:26.378113580+00:00 operation=Get duration=0.063482s path=nyc_taxi_rides/data/tripdata_parquet/data-201501.parquet 2025-08-20T21:20:26.367647578+00:00 operation=Get duration=0.078123s path=nyc_taxi_rides/data/tripdata_parquet/data-201503.parquet 2025-08-20T21:20:26.375807007+00:00 operation=Get duration=0.072382s path=nyc_taxi_rides/data/tripdata_parquet/data-201502.parquet 2025-08-20T21:20:26.380906400+00:00 operation=Get duration=0.069522s path=nyc_taxi_rides/data/tripdata_parquet/data-201409.parquet 2025-08-20T21:20:26.363518921+00:00 operation=Get duration=0.090815s path=nyc_taxi_rides/data/tripdata_parquet/data-201410.parquet 2025-08-20T21:20:26.318394816+00:00 operation=Get duration=0.143049s path=nyc_taxi_rides/data/tripdata_parquet/data-201508.parquet 2025-08-20T21:20:26.398215901+00:00 operation=Get duration=0.067203s path=nyc_taxi_rides/data/tripdata_parquet/data-201505.parquet 2025-08-20T21:20:26.319248399+00:00 operation=Get duration=0.150150s path=nyc_taxi_rides/data/tripdata_parquet/data-201512.parquet 2025-08-20T21:20:26.401417858+00:00 operation=Get duration=0.068286s path=nyc_taxi_rides/data/tripdata_parquet/data-201412.parquet 2025-08-20T21:20:26.319904612+00:00 operation=Get duration=0.152276s path=nyc_taxi_rides/data/tripdata_parquet/data-201603.parquet 2025-08-20T21:20:26.412954128+00:00 operation=Get duration=0.062466s path=nyc_taxi_rides/data/tripdata_parquet/data-201504.parquet 2025-08-20T21:20:26.319699816+00:00 operation=Get duration=0.170309s path=nyc_taxi_rides/data/tripdata_parquet/data-201602.parquet 2025-08-20T21:20:26.319035381+00:00 operation=Get duration=0.179149s path=nyc_taxi_rides/data/tripdata_parquet/data-201511.parquet 2025-08-20T21:20:26.318596709+00:00 operation=Get duration=0.188141s path=nyc_taxi_rides/data/tripdata_parquet/data-201509.parquet 2025-08-20T21:20:26.440219988+00:00 operation=Get duration=0.069692s path=nyc_taxi_rides/data/tripdata_parquet/data-201411.parquet 2025-08-20T21:20:26.318192952+00:00 operation=Get duration=0.193458s path=nyc_taxi_rides/data/tripdata_parquet/data-201507.parquet 2025-08-20T21:20:26.319454611+00:00 operation=Get duration=0.192505s path=nyc_taxi_rides/data/tripdata_parquet/data-201601.parquet 2025-08-20T21:20:26.447920073+00:00 operation=Get duration=0.066637s path=nyc_taxi_rides/data/tripdata_parquet/data-201503.parquet 2025-08-20T21:20:26.450159425+00:00 operation=Get duration=0.068485s path=nyc_taxi_rides/data/tripdata_parquet/data-201502.parquet 2025-08-20T21:20:26.365891758+00:00 operation=Get duration=0.155038s path=nyc_taxi_rides/data/tripdata_parquet/data-201605.parquet 2025-08-20T21:20:26.317981933+00:00 operation=Get duration=0.204229s path=nyc_taxi_rides/data/tripdata_parquet/data-201506.parquet 2025-08-20T21:20:26.461475770+00:00 operation=Get duration=0.072875s path=nyc_taxi_rides/data/tripdata_parquet/data-201508.parquet 2025-08-20T21:20:26.318826455+00:00 operation=Get duration=0.222435s path=nyc_taxi_rides/data/tripdata_parquet/data-201510.parquet 2025-08-20T21:20:26.366102448+00:00 operation=Get duration=0.177727s path=nyc_taxi_rides/data/tripdata_parquet/data-201606.parquet 2025-08-20T21:20:26.469426474+00:00 operation=Get duration=0.074703s path=nyc_taxi_rides/data/tripdata_parquet/data-201512.parquet 2025-08-20T21:20:26.366307273+00:00 operation=Get duration=0.177894s path=nyc_taxi_rides/data/tripdata_parquet/data-201607.parquet 2025-08-20T21:20:26.478874849+00:00 operation=Get duration=0.069262s path=nyc_taxi_rides/data/tripdata_parquet/data-201504.parquet 2025-08-20T21:20:26.366927111+00:00 operation=Get duration=0.182364s path=nyc_taxi_rides/data/tripdata_parquet/data-201610.parquet 2025-08-20T21:20:26.367130133+00:00 operation=Get duration=0.184652s path=nyc_taxi_rides/data/tripdata_parquet/data-201611.parquet 2025-08-20T21:20:26.366516591+00:00 operation=Get duration=0.185707s path=nyc_taxi_rides/data/tripdata_parquet/data-201608.parquet 2025-08-20T21:20:26.367333544+00:00 operation=Get duration=0.190307s path=nyc_taxi_rides/data/tripdata_parquet/data-201612.parquet 2025-08-20T21:20:26.472207026+00:00 operation=Get duration=0.087004s path=nyc_taxi_rides/data/tripdata_parquet/data-201603.parquet 2025-08-20T21:20:26.498215540+00:00 operation=Get duration=0.064440s path=nyc_taxi_rides/data/tripdata_parquet/data-201511.parquet 2025-08-20T21:20:26.490039602+00:00 operation=Get duration=0.082418s path=nyc_taxi_rides/data/tripdata_parquet/data-201602.parquet 2025-08-20T21:20:26.522237747+00:00 operation=Get duration=0.063539s path=nyc_taxi_rides/data/tripdata_parquet/data-201506.parquet 2025-08-20T21:20:26.511988250+00:00 operation=Get duration=0.076112s path=nyc_taxi_rides/data/tripdata_parquet/data-201601.parquet 2025-08-20T21:20:26.520958135+00:00 operation=Get duration=0.071346s path=nyc_taxi_rides/data/tripdata_parquet/data-201605.parquet 2025-08-20T21:20:26.511680838+00:00 operation=Get duration=0.091637s path=nyc_taxi_rides/data/tripdata_parquet/data-201507.parquet 2025-08-20T21:20:26.547834993+00:00 operation=Get duration=0.058344s path=nyc_taxi_rides/data/tripdata_parquet/data-201512.parquet 2025-08-20T21:20:26.538694325+00:00 operation=Get duration=0.071311s path=nyc_taxi_rides/data/tripdata_parquet/data-201508.parquet 2025-08-20T21:20:26.543863146+00:00 operation=Get duration=0.066248s path=nyc_taxi_rides/data/tripdata_parquet/data-201606.parquet 2025-08-20T21:20:26.541289507+00:00 operation=Get duration=0.068916s path=nyc_taxi_rides/data/tripdata_parquet/data-201510.parquet 2025-08-20T21:20:26.549326865+00:00 operation=Get duration=0.062900s path=nyc_taxi_rides/data/tripdata_parquet/data-201610.parquet 2025-08-20T21:20:26.506773684+00:00 operation=Get duration=0.114425s path=nyc_taxi_rides/data/tripdata_parquet/data-201509.parquet 2025-08-20T21:20:26.544223643+00:00 operation=Get duration=0.078399s path=nyc_taxi_rides/data/tripdata_parquet/data-201607.parquet 2025-08-20T21:20:26.557676114+00:00 operation=Get duration=0.069342s path=nyc_taxi_rides/data/tripdata_parquet/data-201612.parquet 2025-08-20T21:20:26.551810476+00:00 operation=Get duration=0.075291s path=nyc_taxi_rides/data/tripdata_parquet/data-201611.parquet 2025-08-20T21:20:26.552249803+00:00 operation=Get duration=0.076787s path=nyc_taxi_rides/data/tripdata_parquet/data-201608.parquet 2025-08-20T21:20:26.562376281+00:00 operation=Get duration=0.068401s path=nyc_taxi_rides/data/tripdata_parquet/data-201603.parquet 2025-08-20T21:20:26.564530607+00:00 operation=Get duration=0.070076s path=nyc_taxi_rides/data/tripdata_parquet/data-201511.parquet 2025-08-20T21:20:26.574675505+00:00 operation=Get duration=0.062372s path=nyc_taxi_rides/data/tripdata_parquet/data-201602.parquet 2025-08-20T21:20:26.365688742+00:00 operation=Get duration=0.271426s path=nyc_taxi_rides/data/tripdata_parquet/data-201604.parquet 2025-08-20T21:20:26.366721145+00:00 operation=Get duration=0.276411s path=nyc_taxi_rides/data/tripdata_parquet/data-201609.parquet 2025-08-20T21:20:26.587780315+00:00 operation=Get duration=0.065229s path=nyc_taxi_rides/data/tripdata_parquet/data-201506.parquet 2025-08-20T21:20:26.591805296+00:00 operation=Get duration=0.069428s path=nyc_taxi_rides/data/tripdata_parquet/data-201601.parquet 2025-08-20T21:20:26.595042962+00:00 operation=Get duration=0.071232s path=nyc_taxi_rides/data/tripdata_parquet/data-201605.parquet 2025-08-20T21:20:26.605005007+00:00 operation=Get duration=0.066641s path=nyc_taxi_rides/data/tripdata_parquet/data-201507.parquet 2025-08-20T21:20:26.617536069+00:00 operation=Get duration=0.069896s path=nyc_taxi_rides/data/tripdata_parquet/data-201610.parquet 2025-08-20T21:20:26.615811584+00:00 operation=Get duration=0.073930s path=nyc_taxi_rides/data/tripdata_parquet/data-201510.parquet 2025-08-20T21:20:26.626278172+00:00 operation=Get duration=0.070453s path=nyc_taxi_rides/data/tripdata_parquet/data-201607.parquet 2025-08-20T21:20:26.630506963+00:00 operation=Get duration=0.070813s path=nyc_taxi_rides/data/tripdata_parquet/data-201612.parquet 2025-08-20T21:20:26.628765074+00:00 operation=Get duration=0.074402s path=nyc_taxi_rides/data/tripdata_parquet/data-201611.parquet 2025-08-20T21:20:26.634335888+00:00 operation=Get duration=0.070587s path=nyc_taxi_rides/data/tripdata_parquet/data-201608.parquet 2025-08-20T21:20:26.624615676+00:00 operation=Get duration=0.082005s path=nyc_taxi_rides/data/tripdata_parquet/data-201509.parquet 2025-08-20T21:20:26.637142663+00:00 operation=Get duration=0.073370s path=nyc_taxi_rides/data/tripdata_parquet/data-201604.parquet 2025-08-20T21:20:26.613816785+00:00 operation=Get duration=0.098655s path=nyc_taxi_rides/data/tripdata_parquet/data-201606.parquet 2025-08-20T21:20:26.643166056+00:00 operation=Get duration=0.080656s path=nyc_taxi_rides/data/tripdata_parquet/data-201609.parquet 2025-08-20T21:20:26.727081939+00:00 operation=Get duration=0.066146s path=nyc_taxi_rides/data/tripdata_parquet/data-201609.parquet 2025-08-20T21:20:26.714227976+00:00 operation=Get duration=0.099141s path=nyc_taxi_rides/data/tripdata_parquet/data-201604.parquet Total duration: 37.010551 > select count(*) from nyc_taxi_rides; +------------+ | count(*) | +------------+ | 1310903963 | +------------+ 1 row(s) fetched. Elapsed 0.328 seconds. Total duration: 0.000000 2025-08-20T21:21:08.406923199+00:00 operation=List duration=0.000007s path=nyc_taxi_rides/data/tripdata_parquet Total duration: 0.000007 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
