HyukjinKwon commented on code in PR #57814:
URL: https://github.com/apache/spark/pull/57814#discussion_r3732396350
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala:
##########
@@ -281,6 +281,10 @@ class FileScanRDD(
// Throw FileNotFoundException even if `ignoreCorruptFiles`
is true
case e: FileNotFoundException if !ignoreMissingFiles =>
throw e
case e @ (_ : AccessControlException | _ :
BlockMissingException) => throw e
+ // A resource-safety limit is not a corrupt file: skipping
the rest of the archive
+ // would return partial data with no signal that the limit
stopped the read.
+ case e: SparkRuntimeException
+ if e.getCondition == "MAX_ARCHIVE_DEPTH_EXCEEDED" =>
throw e
Review Comment:
This carve-out keeps the depth guard authoritative on the scan path, but
schema inference is a separate path that still swallows it.
`CSVDataSource.inferWithArchives`, `JsonDataSource`/`XmlDataSource`
`skipInputOnError`, and `AvroUtils.firstArchiveEntrySchema` each catch
`RuntimeException if ignoreCorruptFiles`, and `SparkRuntimeException` is a
`RuntimeException` -- so with `ignoreCorruptFiles=true`, inferring a schema
over a too-deeply-nested archive logs it as corrupt and infers from partial
data, the exact silent-partial-data outcome this case prevents for scans. The
`the depth limit still fails the read under ignoreCorruptFiles` test only
exercises `read(...).collect()`, so the inference gap is untested.
Rather than repeat this `case` at all five sites, make the guard
authoritative where every site already funnels: add it to
`DataSourceUtils.shouldIgnoreCorruptFileException` so a `SparkThrowable` whose
condition is `MAX_ARCHIVE_DEPTH_EXCEEDED` returns false. Every inference path
and this `FileScanRDD` clause consult it, so this explicit `case` then becomes
redundant and a future archive reader can't forget the carve-out. Please add an
inference-path test alongside the scan-path one.
##########
.isaac/config.json:
##########
@@ -0,0 +1,3 @@
+{
+ "sync_reminder_last_shown": "2026-08-06"
Review Comment:
This file and `build/sbt-launch-1.12.13.jar.part` (a 0-byte partial-download
artifact) look accidentally committed -- both are unrelated to the PR and
neither is gitignored, so they'd ship to master. Please `git rm` both from the
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
