[
https://issues.apache.org/jira/browse/SPARK-58946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58946:
--------------------------------------------
Description:
[{{CSVOptions.scala:127}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala#L127]
validates the {{extension}} write option with:
{code:scala}
if (ext.size != 3 && !ext.forall(_.isLetter))
{code}
The error message states the extension is limited to exactly three letters,
which requires rejecting anything that is not both three characters and all
letters. The {{&&}} rejects only values that fail *both* clauses, so invalid
extensions are accepted:
{noformat}
"abcd" four letters accepted
"data" four letters accepted
"ab1" three chars, not all letters accepted
"a" one letter accepted
"a/b" three chars, path separator accepted
"12" rejected (fails both clauses)
{noformat}
The value flows into the output filename at {{CSVWrite.scala:61}} and
{{CSVFileFormat.scala:91}}, so {{"a/b"}} places a path separator inside a
filename component.
The fix is:
{code:scala}
if (ext.size != 3 || !ext.forall(_.isLetter))
{code}
which matches the documented contract. Only {{"tsv"}} is used in existing tests
and remains valid.
h3. Dependency
Depends on SPARK-58945. Until that lands, this error path raises
{{INTERNAL_ERROR}} rather than {{INVALID_PARAMETER_VALUE.EXTENSION}}, because
{{invalidFileExtensionError}} passes a {{fileExtension}} key that the template
does not declare.
was:
{{CSVOptions.scala:127}} validates the {{extension}} write option with:
{code:scala}
if (ext.size != 3 && !ext.forall(_.isLetter))
{code}
The error message states the extension is limited to exactly three letters,
which requires rejecting anything that is not both three characters and all
letters. The {{&&}} rejects only values that fail *both* clauses, so invalid
extensions are accepted:
{noformat}
"abcd" four letters accepted
"data" four letters accepted
"ab1" three chars, not all letters accepted
"a" one letter accepted
"a/b" three chars, path separator accepted
"12" rejected (fails both clauses)
{noformat}
The value flows into the output filename at {{CSVWrite.scala:61}} and
{{CSVFileFormat.scala:91}}, so {{"a/b"}} places a path separator inside a
filename component.
The fix is:
{code:scala}
if (ext.size != 3 || !ext.forall(_.isLetter))
{code}
which matches the documented contract. Only {{"tsv"}} is used in existing tests
and remains valid.
h3. Dependency
Depends on SPARK-58945. Until that lands, this error path raises
{{INTERNAL_ERROR}} rather than {{INVALID_PARAMETER_VALUE.EXTENSION}}, because
{{invalidFileExtensionError}} passes a {{fileExtension}} key that the template
does not declare.
> CSV extension option validation uses `&&` instead of `||`, accepting invalid
> extensions
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-58946
> URL: https://issues.apache.org/jira/browse/SPARK-58946
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
>
> [{{CSVOptions.scala:127}}|https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala#L127]
> validates the {{extension}} write option with:
> {code:scala}
> if (ext.size != 3 && !ext.forall(_.isLetter))
> {code}
> The error message states the extension is limited to exactly three letters,
> which requires rejecting anything that is not both three characters and all
> letters. The {{&&}} rejects only values that fail *both* clauses, so invalid
> extensions are accepted:
> {noformat}
> "abcd" four letters accepted
> "data" four letters accepted
> "ab1" three chars, not all letters accepted
> "a" one letter accepted
> "a/b" three chars, path separator accepted
> "12" rejected (fails both clauses)
> {noformat}
> The value flows into the output filename at {{CSVWrite.scala:61}} and
> {{CSVFileFormat.scala:91}}, so {{"a/b"}} places a path separator inside a
> filename component.
> The fix is:
> {code:scala}
> if (ext.size != 3 || !ext.forall(_.isLetter))
> {code}
> which matches the documented contract. Only {{"tsv"}} is used in existing
> tests and remains valid.
> h3. Dependency
> Depends on SPARK-58945. Until that lands, this error path raises
> {{INTERNAL_ERROR}} rather than {{INVALID_PARAMETER_VALUE.EXTENSION}}, because
> {{invalidFileExtensionError}} passes a {{fileExtension}} key that the
> template does not declare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
