Tpt commented on code in PR #24924:
URL: https://github.com/apache/datafusion/pull/24924#discussion_r4049848228
##########
datafusion/datasource/src/write/orchestration.rs:
##########
@@ -111,9 +112,11 @@ pub(crate) async fn serialize_rb_stream_to_object_store(
});
let mut row_count = 0;
+ let mut serialized_bytes = 0;
while let Some(task) = rx.recv().await {
match task.join().await {
Ok(Ok((cnt, bytes))) => {
+ serialized_bytes += bytes.len();
Review Comment:
Thank you! I indeed missed that!
> Could we account for bytes at the compressed writer/object-store boundary
Indeed. I have updated `ObjectWriterBuilder` to count the number of bytes
flushed to the underlying writer. It is done with a new intermediate
`AsyncWrite` implementation that sits between object store and the compression
implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
