Joorgem commented on PR #57658:
URL: https://github.com/apache/spark/pull/57658#issuecomment-5180571469
Scope note, so this is not a silent expansion: 4a1ad5e adds a second
correction to the same bullet, on @szehon-ho's suggestion from #57608.
Alongside the "drops corrupt records" claim this PR already fixes, the
bullet also says:
> A record with less/more tokens than schema is not a corrupted record to
CSV.
That is false. `UnivocityParser.convert`:
```scala
var badRecordException: Option[Throwable] = if (tokens.length !=
parsedSchema.length) {
// If the number of tokens doesn't match the schema, we should treat it as
a malformed record.
// However, we still have chance to parse some of the tokens. It continues
to parses the
// tokens normally and sets null when `ArrayIndexOutOfBoundsException`
occurs for missing
// tokens.
```
The record is flagged malformed *and* parsing continues with `null` for the
tokens it does not have. So the sentence's second half was accurate and its
first half was not, which is likely why it lasted: what a user observes matches
the accurate half.
It landed here rather than on #57608 because this PR already edits this
bullet, and #57608 would otherwise have shipped a table cell contradicting
itself two lines apart. Both changes are now one concern — what `PERMISSIVE`
actually does with a corrupt record — in one bullet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
