BlakeOrth opened a new pull request, #18139:
URL: https://github.com/apache/datafusion/pull/18139
## Which issue does this PR close?
This does not fully close, but is an incremental building block component
for:
- https://github.com/apache/datafusion/issues/17207
The full context of how this code is likely to progress can be seen in the
POC for this effort:
- https://github.com/apache/datafusion/pull/17266
## Rationale for this change
Further fills out the missing methods that have yet to be instrumented in
the instrumented object store.
## What changes are included in this PR?
- Adds instrumentation around put_opts
- Adds instrumentation around put_multipart
- Adds tests for newly instrumented methods
## Are these changes tested?
Yes. Unit tests have been added for the new methods
Example output:
```sql
DataFusion CLI v50.2.0
> CREATE EXTERNAL TABLE
test(a bigint, b bigint)
STORED AS parquet LOCATION '../../test_table/';
0 row(s) fetched.
Elapsed 0.003 seconds.
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> INSERT INTO test values (1, 2), (3, 4);
+-------+
| count |
+-------+
| 2 |
+-------+
1 row(s) fetched.
Elapsed 0.007 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner:
LocalFileSystem(file:///)
2025-10-17T19:02:15.440246215+00:00 operation=List
path=home/blake/open_source_src/datafusion-BlakeOrth/test_table
2025-10-17T19:02:15.444096012+00:00 operation=Put duration=0.000249s
size=815
path=home/blake/open_source_src/datafusion-BlakeOrth/test_table/a9pjKBxSOtXZobJO_0.parquet
Summaries:
List
count: 1
Put
count: 1
duration min: 0.000249s
duration max: 0.000249s
duration avg: 0.000249s
size min: 815 B
size max: 815 B
size avg: 815 B
size sum: 815 B
>
```
(note: I have no idea how to exercise/show a multi-part put operation, or if
DataFusion even utilizes multipart puts for large files)
## Are there any user-facing changes?
No-ish
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
