Anika Kelhanka created SPARK-59629:
--------------------------------------
Summary: CSV Reader, mapping columnNameOfCorruptRecord does not
honor spark.sql.caseSensitive=false
Key: SPARK-59629
URL: https://issues.apache.org/jira/browse/SPARK-59629
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.1.2, 3.5.3
Reporter: Anika Kelhanka
The Spark CSV reader resolves \{{columnNameOfCorruptRecord}} against the
user-supplied schema using exact string equality, and
\{{spark.sql.caseSensitive}}
is not honoured. When the schema declares the corrupt-record field in a
different case to the option value, the reader does not fail — it silently
returns \{{NULL}} in that column and discards the malformed record.
XML was made to honour \{{spark.sql.caseSensitive}} in 4.0.0 (SPARK-45844), and
SPARK-46954 extracted the lookup into
{\{StructType.getFieldIndexCaseInsensitive}}. The CSV corrupt-record path still
uses the case-sensitive \{{getFieldIndex}}.
Within the CSV reader, header-to-schema matching already honours
{\{spark.sql.caseSensitive}} (SPARK-23786); the corrupt-record name does not.
h2. Reproducer
{code:scala}
spark.conf.set("spark.sql.caseSensitive", "false") // the default
// id,name
// 1,Alice
// invalid,Bob
val schema = new StructType()
.add("id", IntegerType)
.add("name", StringType)
.add("_CORRUPT_RECORD", StringType) // option default is "_corrupt_record"
spark.read.option("header", "true").option("mode", "PERMISSIVE")
.schema(schema).csv(path).show(false)
{code}
*Expected* — and what you get if the field is spelled \{{_corrupt_record}}:
{noformat}
|1 |Alice|NULL |
|NULL|Bob |invalid,Bob|
{noformat}
*Actual*:
{noformat}
|1 |Alice|NULL|
|NULL|Bob |NULL|
{noformat}
The bad row is still detected, but \{{invalid,Bob}} is discarded. The only
signal
is a WARN from \{{CSVHeaderChecker}} about the column count. Reproduced on 3.5.3
and 4.1.2.
h2. Suggested fix
Match \{{columnNameOfCorruptRecord}} case-insensitively. Or, if case-sensitive
matching must be preserved for any reason, then
{\{verifyColumnNameOfCorruptRecord}} should throw an \{{AnalysisException}}
when a
field matches case-insensitively but not exactly. Failing at analysis time is
strictly better than silently returning NULL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
