andygrove opened a new issue, #7213:
URL: https://github.com/apache/datafusion-comet/issues/7213
### Describe the bug
When a native Parquet scan fails to read a file, the
`FAILED_READ_FILE.NO_HINT` error does not name the file the way Spark's does.
- **Corrupt file** (random bytes in place of one part file): Spark's message
names the bad file. Comet's message reads `Encountered error while reading file
.`, with an empty path. The native error does carry the file name in its inner
message (`Failed to fetch metadata for file .../part-00001-....parquet: ...
Corrupt footer`), but `params.filePath` is `""`. In the branch-1.2 audit, a
test with several files per task got the opposite problem: the path listed
every file of the task's partition. That comes from the per-partition path
fallback (`perPartitionPaths`).
- **Duplicate field in case-insensitive mode** (seen in the audit, not
re-run here): Spark wraps `Found duplicate field(s) "a": [a, A] in
case-insensitive mode` in `FAILED_READ_FILE.NO_HINT` with the path. Comet
raises the bare `SparkRuntimeException`. In field-id mode Comet wraps it, but
with an empty path.
The empty path for `PARQUET_COLUMN_DATA_TYPE_MISMATCH` is already tracked in
#4316.
The 1.1.0 release behaves the same way by code. The error conversion and the
per-partition path fallback are unchanged, and #7094's metadata prefetch does
not change the outcome.
### Steps to reproduce
Spark 4.1, default configs:
```scala
import org.apache.hadoop.fs.{FileSystem, Path}
spark.range(0, 100, 1, 4).write.parquet("/tmp/badread")
val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
val victim = fs.listStatus(new Path("/tmp/badread")).map(_.getPath)
.filter(_.getName.endsWith(".parquet")).sortBy(_.getName).apply(1)
val bytes = new Array[Byte](fs.getFileStatus(victim).getLen.toInt)
new scala.util.Random(1).nextBytes(bytes)
val os = fs.create(victim, true); os.write(bytes); os.close()
fs.delete(new Path(victim.getParent, "." + victim.getName + ".crc"), false)
spark.read.parquet("/tmp/badread").count()
```
- Spark: `[FAILED_READ_FILE.NO_HINT] Encountered error while reading file
file:///tmp/badread/part-00001-....snappy.parquet.`
- Comet: `[FAILED_READ_FILE.NO_HINT] Encountered error while reading file .`
### Expected behavior
The error names the file that failed and uses the same error class as Spark,
so users can find the bad file and tests that parse the path work.
### Additional context
Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression.
Related: #4529 (closed) and #4316.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
