JeonDaehong commented on code in PR #3433:
URL: https://github.com/apache/iceberg-rust/pull/3433#discussion_r4259898371
##########
crates/iceberg/src/arrow/delete_file_loader.rs:
##########
@@ -66,13 +66,45 @@ impl BasicDeleteFileLoader {
pub(crate) async fn parquet_to_batch_stream(
&self,
data_file_path: &str,
- file_size_in_bytes: u64,
+ file_size_in_bytes: Option<u64>,
key_metadata: Option<&[u8]>,
) -> Result<ArrowRecordBatchStream> {
/*
Essentially a super-cut-down ArrowReader. We can't use ArrowReader
directly
as that introduces a circular dependency.
*/
+ // `None` means the task did not carry a size (see
`FileScanTaskDeleteFile`).
+ // Resolve it when the file is first loaded.
+ let file_size_in_bytes = match file_size_in_bytes {
+ Some(file_size_in_bytes) => file_size_in_bytes,
+ None => {
+ self.file_io
+ .new_input(data_file_path)?
+ .metadata()
+ .await
+ .map_err(|error| {
+ // Keep the storage error's kind and retryability.
+ Error::new(
+ error.kind(),
+ format!("Failed to stat delete file
'{data_file_path}'"),
+ )
+ .with_retryable(error.retryable())
Review Comment:
Reproduced on `main` at b154603a7 with a known size as well: the first load
fails, a second load on the same loader after the file appears doesn't complete
within 3s, and a new loader reads it fine. Filed #3434 for the positional half.
I'm happy to take the fix there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
