Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8515#discussion_r38252484
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/CatalystRowConverter.scala
---
@@ -196,6 +196,13 @@ private[parquet] class CatalystRowConverter(
}
}
+ if (paddedParquetFields.length != catalystType.length) {
+ throw new IllegalStateException(
+ "A Parquet file's schema has different number of fields with the
table schema. " +
+ "Please enable schema merging by setting \"mergeSchema\" to true
when load " +
+ "a Parquet dataset or set spark.sql.parquet.mergeSchema to true
in SQLConf.")
+ }
--- End diff --
@liancheng @marmbrus I add the check to make sure the number of fields of a
parquet file's struct have the same number of the corresponding struct in the
table schema. If this check fails, we will ask users to enable mergeSchema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
