cloud-fan commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r3928083269
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -752,15 +757,30 @@ case class FileSourceScanExec(
lazy val inputRDD: RDD[InternalRow] = {
val options = relation.options +
(FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString)
- val readFile: (PartitionedFile) => Iterator[InternalRow] =
- relation.fileFormat.buildReaderWithPartitionValues(
- sparkSession = relation.sparkSession,
- dataSchema = relation.dataSchema,
- partitionSchema = relation.partitionSchema,
- requiredSchema = requiredSchema,
- filters = pushedDownFilters,
- options = options,
- hadoopConf = getHadoopConf(relation.sparkSession, relation.options))
+ val hadoopConf = getHadoopConf(relation.sparkSession, relation.options)
+ val readFile: (PartitionedFile) => Iterator[InternalRow] =
relation.fileFormat match {
+ case format: OrcFileFormat
Review Comment:
**Blocking (P1):** This exact-class branch restores the existing subclass
override, but it still
loses the analyzed mode when that override delegates to `super`: the public
`OrcFileFormat.buildReaderWithPartitionValues` overload always forwards
`charVarcharStandardSemantics = false`. Under standard semantics, a
delegating subclass can
therefore read physical `"abcdef"` as `VARCHAR(4)`, receive native ORC
truncation to `"abcd"`,
and bypass Spark's `EXCEED_LIMIT_LENGTH` check.
Please define a typed mode in Catalyst:
```scala
sealed trait CharVarcharScanMode
object CharVarcharScanMode {
case object PreserveNative extends CharVarcharScanMode
case object SparkStandard extends CharVarcharScanMode
}
```
Relations and scans should carry `Option[CharVarcharScanMode]`: `None` is
not applicable,
`Some(PreserveNative)` preserves native ORC constraints, and
`Some(SparkStandard)` requests
physical STRING so Spark sees and validates the original value.
Then remove the format match here by adding a mode-aware overload to
`FileFormat.buildReaderWithPartitionValues`. Its default implementation
should:
1. clone the per-call Hadoop configuration;
2. overwrite an engine-private entry with the explicit mode; and
3. invoke the existing seven-argument method virtually.
`FileSourceScanExec` calls the new overload whenever the relation has a
bound mode, regardless of
the concrete file format. The existing public ORC method reads the
compatibility entry and passes
the mode to a separate non-virtual reader builder. Existing subclasses
therefore keep their old
override, and a call to `super` retains the bound mode. The Hadoop entry is
only a bridge across
the legacy signature; the authoritative state remains the typed plan field
and overload parameter.
Please extend the subclass regression with an over-length value:
`SparkStandard` must raise
`EXCEED_LIMIT_LENGTH`, while `PreserveNative` retains the intended native
truncation. Cover
both row and vectorized readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
