Subhramit Basu Bhowmick created SPARK-58946:
-----------------------------------------------
Summary: CSV extension option validation uses `&&` instead of
`||`, accepting invalid extensions
Key: SPARK-58946
URL: https://issues.apache.org/jira/browse/SPARK-58946
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Subhramit Basu Bhowmick
{{CSVOptions.scala:127}} validates the {{extension}} write option with:
{code:scala}
if (ext.size != 3 && !ext.forall(_.isLetter))
{code}
The error message states the extension is limited to exactly three letters,
which requires rejecting anything that is not both three characters and all
letters. The {{&&}} rejects only values that fail *both* clauses, so invalid
extensions are accepted:
{noformat}
"abcd" four letters accepted
"data" four letters accepted
"ab1" three chars, not all letters accepted
"a" one letter accepted
"a/b" three chars, path separator accepted
"12" rejected (fails both clauses)
{noformat}
The value flows into the output filename at {{CSVWrite.scala:61}} and
{{CSVFileFormat.scala:91}}, so {{"a/b"}} places a path separator inside a
filename component.
The fix is:
{code:scala}
if (ext.size != 3 || !ext.forall(_.isLetter))
{code}
which matches the documented contract. Only {{"tsv"}} is used in existing tests
and remains valid.
h3. Dependency
Depends on SPARK-58945. Until that lands, this error path raises
{{INTERNAL_ERROR}} rather than {{INVALID_PARAMETER_VALUE.EXTENSION}}, because
{{invalidFileExtensionError}} passes a {{fileExtension}} key that the template
does not declare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
