Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18865#discussion_r137742562
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
---
@@ -113,6 +113,18 @@ class JsonFileFormat extends TextBasedFileFormat with
DataSourceRegister {
}
}
+ if (requiredSchema.length == 1 &&
+ requiredSchema.head.name == parsedOptions.columnNameOfCorruptRecord)
{
+ throw new AnalysisException(
+ s"'${parsedOptions.columnNameOfCorruptRecord}' cannot be selected
alone without other " +
+ "data columns, because its content is completely derived from the
data columns parsed.\n" +
--- End diff --
Add a more sentence like `Even your queries looks not only select this
column, if after column pruning it isn't involving paring any data fields,
e.g., filtering on the column followed by a counting, it can produce incorrect
results and so disallowed.`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
