Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18865#discussion_r137741783
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
---
@@ -113,6 +113,18 @@ class JsonFileFormat extends TextBasedFileFormat with
DataSourceRegister {
}
}
+ if (requiredSchema.length == 1 &&
+ requiredSchema.head.name == parsedOptions.columnNameOfCorruptRecord)
{
+ throw new AnalysisException(
+ s"'${parsedOptions.columnNameOfCorruptRecord}' cannot be selected
alone without other " +
+ "data columns, because its content is completely derived from the
data columns parsed.\n" +
+ "If you want to select corrupt records only, cache or save the
Dataset " +
--- End diff --
Add a `\n` here too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
