Serge Rielau created SPARK-47467: ------------------------------------ Summary: Error message regressed when creating hive table with illegal column name Key: SPARK-47467 URL: https://issues.apache.org/jira/browse/SPARK-47467 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 4.0.0 Reporter: Serge Rielau
The following statement used to result in: CREATE TABLE test5(`北京, ` INT) USING HIVE; [[INVALID_HIVE_COLUMN_NAME|https://docs.databricks.com/error-messages/error-classes.html#invalid_hive_column_name]] Cannot create the table `hive_metastore`.`srielau`.`test5` having the column `北京, ` whose name contains invalid characters ',' in Hive metastore. SQLSTATE: 42K05 Now it results in: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 2 elements while columns.types has 1 elements!) at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$2(HiveExternalCatalog.scala:168) at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:115) at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:153) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:405) at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:391) at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34) at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:152) at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:312) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.$anonfun$createTable$1(ExternalCatalogWithListener.scala:122) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.catalyst.MetricKeyUtils$.measure(MetricKey.scala:661) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.$anonfun$profile$1(ExternalCatalogWithListener.scala:54) at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94) at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.profile(ExternalCatalogWithListener.scala:53) This may be related to: https://github.com/apache/spark/pull/45180 -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org