jubins commented on code in PR #56581:
URL: https://github.com/apache/spark/pull/56581#discussion_r3457053036
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala:
##########
@@ -3588,6 +3593,79 @@ abstract class CSVSuite
matchPVals = true)
}
+ test("SPARK-57515: non-multiLine CSV read with header exceeding maxColumns
surfaces " +
+ "MALFORMED_CSV_RECORD") {
+ // inferFromDataset called csvParser.parseLine(header) directly without
the AIOOBE guard
Review Comment:
thanks @MaxGekk you're right, without an explicit schema, `DataFrameReader`
calls `inferFromDataset` eagerly and the error surfaces there, never reaching
`checkHeaderColumnNames`. Added two new explicit-schema variants (one for
file-based non-multiLine, one for `Dataset[String]`) that skip inference and
exercise the `CSVHeaderChecker` guards directly. Also updated the comments on
the original two tests to clarify they validate the `inferFromDataset` path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
