Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21380#discussion_r189791852
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -196,7 +198,7 @@ class UnivocityParser(
}
def getPartialResult(): Option[InternalRow] = {
try {
- Some(convert(checkedTokens))
+ convert(checkedTokens).headOption
--- End diff --
Yes, it is. I changed returned type of the _convert()_ method from
InternalRow to Seq[InternalRow] to catch the cases when uniVocity parser
returns nulls (comments and empty lines). As a consequence of that, I have to
change this function too because it returns Option which is required by the
BadRecordException exception. It is safe because Seq can be either empty or
contain only one element. And I though it is better to modify body of
_getPartialResult()_ than places where BadRecordException is handled.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
