gaborkaszab commented on code in PR #18486:
URL: https://github.com/apache/iceberg/pull/18486#discussion_r4323360651
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java:
##########
@@ -148,12 +148,8 @@ public boolean next() throws IOException {
}
}
} catch (IOException | RuntimeException e) {
- if (currentTask != null && !currentTask.isDataTask()) {
- String filePaths =
- referencedFiles(currentTask)
- .map(ContentFile::location)
- .collect(Collectors.joining(", "));
- LOG.error("Error reading file(s): {}", filePaths, e);
+ if (currentTask instanceof ContentScanTask<?> contentTask &&
!currentTask.isDataTask()) {
+ LOG.error("Error reading file: {}", contentTask.file().location(), e);
Review Comment:
1. Logging delete files
I'm afraid we don't have a unified interface the get the delete files across
the implementations: We have a `ContentScanTask` here that doesn't give an
interface for deletes, and I don't think we want to cast to specific types here
just for logging.
I agree that we loose some of the previous functionality here, but I believe
that since this is logging, we won't sacrifice any actual functionality by
simply logging the data file's location.
2. referencedFiles()
One of the main goals of the decryptor call is that once that's wired in, we
don;t have to change all the readers to add column files to the referenced
files list, we could do that internally in the decryptor making it available to
all readers out of the box. I agree that the changeling reader and position
delete reader are special in this case constructing their own referenced files
list. I made some adjustments to the decryptor to do that internally, moving
code out from these readers. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
