[
https://issues.apache.org/jira/browse/SPARK-59629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anika Kelhanka updated SPARK-59629:
-----------------------------------
Description:
The Spark CSV reader resolves `{{{}columnNameOfCorruptRecord`{}}} against the
user-supplied schema using exact string equality, and
`{{{}spark.sql.caseSensitive`{}}}
is not honoured. When the schema declares the corrupt-record field in a
different case to the option value, the reader does not fail — it silently
returns {{NULL}} in that column and discards the malformed record.
XML was made to honour {{spark.sql.caseSensitive}} in 4.0.0 (SPARK-45844), and
SPARK-46954 extracted the lookup into
{{{}StructType.getFieldIndexCaseInsensitive{}}}. The CSV corrupt-record path
still
uses the case-sensitive {{{}getFieldIndex{}}}.
Within the CSV reader, header-to-schema matching already honours
{{spark.sql.caseSensitive}} (SPARK-23786); the corrupt-record name does not.
h2. Reproducer
{code:scala}
spark.conf.set("spark.sql.caseSensitive", "false") // the default
// id,name
// 1,Alice
// invalid,Bob
val schema = new StructType()
.add("id", IntegerType)
.add("name", StringType)
.add("_CORRUPT_RECORD", StringType) // option default is "_corrupt_record"
spark.read.option("header", "true").option("mode", "PERMISSIVE")
.schema(schema).csv(path).show(false)
{code}
*Expected* — and what you get if the field is spelled {{{}_corrupt_record{}}}:
{noformat}
|1 |Alice|NULL |
|NULL|Bob |invalid,Bob|
{noformat}
{*}Actual{*}:
{noformat}
|1 |Alice|NULL|
|NULL|Bob |NULL|
{noformat}
The bad row is still detected, but {{invalid,Bob}} is discarded. The only signal
is a WARN from {{CSVHeaderChecker}} about the column count. Reproduced on 3.5.3
and 4.1.2.
h2. Suggested fix
Match {{columnNameOfCorruptRecord}} case-insensitively. Or, if case-sensitive
matching must be preserved for any reason, then
{{verifyColumnNameOfCorruptRecord}} should throw an {{AnalysisException}} when a
field matches case-insensitively but not exactly. Failing at analysis time is
strictly better than silently returning NULL.
was:
The Spark CSV reader resolves \{{columnNameOfCorruptRecord}} against the
user-supplied schema using exact string equality, and
\{{spark.sql.caseSensitive}}
is not honoured. When the schema declares the corrupt-record field in a
different case to the option value, the reader does not fail — it silently
returns \{{NULL}} in that column and discards the malformed record.
XML was made to honour \{{spark.sql.caseSensitive}} in 4.0.0 (SPARK-45844), and
SPARK-46954 extracted the lookup into
{\{StructType.getFieldIndexCaseInsensitive}}. The CSV corrupt-record path still
uses the case-sensitive \{{getFieldIndex}}.
Within the CSV reader, header-to-schema matching already honours
{\{spark.sql.caseSensitive}} (SPARK-23786); the corrupt-record name does not.
h2. Reproducer
{code:scala}
spark.conf.set("spark.sql.caseSensitive", "false") // the default
// id,name
// 1,Alice
// invalid,Bob
val schema = new StructType()
.add("id", IntegerType)
.add("name", StringType)
.add("_CORRUPT_RECORD", StringType) // option default is "_corrupt_record"
spark.read.option("header", "true").option("mode", "PERMISSIVE")
.schema(schema).csv(path).show(false)
{code}
*Expected* — and what you get if the field is spelled \{{_corrupt_record}}:
{noformat}
|1 |Alice|NULL |
|NULL|Bob |invalid,Bob|
{noformat}
*Actual*:
{noformat}
|1 |Alice|NULL|
|NULL|Bob |NULL|
{noformat}
The bad row is still detected, but \{{invalid,Bob}} is discarded. The only
signal
is a WARN from \{{CSVHeaderChecker}} about the column count. Reproduced on 3.5.3
and 4.1.2.
h2. Suggested fix
Match \{{columnNameOfCorruptRecord}} case-insensitively. Or, if case-sensitive
matching must be preserved for any reason, then
{\{verifyColumnNameOfCorruptRecord}} should throw an \{{AnalysisException}}
when a
field matches case-insensitively but not exactly. Failing at analysis time is
strictly better than silently returning NULL.
> CSV Reader, mapping columnNameOfCorruptRecord does not honor
> spark.sql.caseSensitive=false
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59629
> URL: https://issues.apache.org/jira/browse/SPARK-59629
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 3.5.3, 4.1.2
> Reporter: Anika Kelhanka
> Priority: Major
> Labels: correctness
>
> The Spark CSV reader resolves `{{{}columnNameOfCorruptRecord`{}}} against the
> user-supplied schema using exact string equality, and
> `{{{}spark.sql.caseSensitive`{}}}
> is not honoured. When the schema declares the corrupt-record field in a
> different case to the option value, the reader does not fail — it silently
> returns {{NULL}} in that column and discards the malformed record.
> XML was made to honour {{spark.sql.caseSensitive}} in 4.0.0 (SPARK-45844), and
> SPARK-46954 extracted the lookup into
> {{{}StructType.getFieldIndexCaseInsensitive{}}}. The CSV corrupt-record path
> still
> uses the case-sensitive {{{}getFieldIndex{}}}.
> Within the CSV reader, header-to-schema matching already honours
> {{spark.sql.caseSensitive}} (SPARK-23786); the corrupt-record name does not.
> h2. Reproducer
> {code:scala}
> spark.conf.set("spark.sql.caseSensitive", "false") // the default
> // id,name
> // 1,Alice
> // invalid,Bob
> val schema = new StructType()
> .add("id", IntegerType)
> .add("name", StringType)
> .add("_CORRUPT_RECORD", StringType) // option default is "_corrupt_record"
> spark.read.option("header", "true").option("mode", "PERMISSIVE")
> .schema(schema).csv(path).show(false)
> {code}
> *Expected* — and what you get if the field is spelled {{{}_corrupt_record{}}}:
> {noformat}
> |1 |Alice|NULL |
> |NULL|Bob |invalid,Bob|
> {noformat}
> {*}Actual{*}:
> {noformat}
> |1 |Alice|NULL|
> |NULL|Bob |NULL|
> {noformat}
> The bad row is still detected, but {{invalid,Bob}} is discarded. The only
> signal
> is a WARN from {{CSVHeaderChecker}} about the column count. Reproduced on
> 3.5.3
> and 4.1.2.
> h2. Suggested fix
> Match {{columnNameOfCorruptRecord}} case-insensitively. Or, if case-sensitive
> matching must be preserved for any reason, then
> {{verifyColumnNameOfCorruptRecord}} should throw an {{AnalysisException}}
> when a
> field matches case-insensitively but not exactly. Failing at analysis time is
> strictly better than silently returning NULL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
