Rohit Saxena created HIVE-24017:
-----------------------------------
Summary: "CREATE VIEW" command failing with AlreadyExistsException
Key: HIVE-24017
URL: https://issues.apache.org/jira/browse/HIVE-24017
Project: Hive
Issue Type: Bug
Components: HiveServer2, Spark
Affects Versions: 2.1.1
Reporter: Rohit Saxena
We are using our custom code to fire 50 concurrent hive view creation queries.
The view name is randomly generated for each query and is always unique. We get
this stacktrace which we do not get for low concurrency like 20. This is very
puzzling, we tried following scenarios to avoid the error:
# using "DROP VIEW IF EXISTS" then "CREATE VIEW IF NOT EXISTS"
# using "DROP VIEW IF EXISTS" then "CREATE VIEW"
# using "CREATE OR REPLACE VIEW"
We still get the "AlreadyExistsException". Seems like a concurrency/threading
issue from either Hive or Spark. We are running our queries in SparkSQL which
is internally invoking the Hive libraries. These are one of the query sequence
from our custom code :
// [4] -> [USE default]
// [5] -> [DROP VIEW IF EXISTS
`default`.`w2314287698276073922_generatedsource_46_view_102__m_sparkengine_alltx_allsrc`]
// [6] -> [CREATE VIEW IF NOT EXISTS
`default`.`w2314287698276073922_generatedsource_46_view_102__m_sparkengine_alltx_allsrc`
(`a0`, `a1`, `a2`, `a3`) AS SELECT CAST(CAST(5 * CAST(alias.c_custkey AS
DECIMAL(18, 0)) AS DECIMAL(28, 0)) AS DECIMAL(18, 0)) as a0, alias.c_name as
a1, alias.c_address as a2, alias.c_nationkey as a3 FROM
default.dst_sanity_test_customer_hive alias WHERE (CAST(CAST(5 *
CAST(alias.c_custkey AS DECIMAL(18, 0)) AS DECIMAL(28, 0)) AS DECIMAL(18, 0)) %
3) = 0]
This is the stacktrace we are getting:-
_[com.informatica.sdk.dtm.ExecutionException: [SPARK_1003] Spark task
[InfaSpark0] failed with the following error: [User class threw exception:
java.lang.reflect.InvocationTargetException_
_at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
_at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
_at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
_at java.lang.reflect.Method.invoke(Method.java:498)_
_at com.informatica.compiler.InfaSparkMain$.main(InfaSparkMain.scala:124)_
_at com.informatica.compiler.InfaSparkMain.main(InfaSparkMain.scala)_
_at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
_at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
_at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
_at java.lang.reflect.Method.invoke(Method.java:498)_
_at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)_
_Caused by: org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException:
AlreadyExistsException(message:Table
w2314287698276073922_generatedsource_46_view_102__m_sparkengine_alltx_allsrc
already exists);_
_at
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)_
_at
org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:238)_
_at
org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:102)_
_at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:319)_
_at
org.apache.spark.sql.execution.command.CreateViewCommand.run(views.scala:175)_
_at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)_
_at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)_
_at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)_
_at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:196)_
_at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:196)_
_at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3384)_
_at
org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)_
_at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)_
_at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)_
_at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3383)_
_at org.apache.spark.sql.Dataset.<init>(Dataset.scala:196)_
_at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:80)_
_at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)_
_at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)_
_at com.informatica.exec.InfaSpark0$.main(InfaSpark0.scala:55)_
_at com.informatica.exec.InfaSpark0.main(InfaSpark0.scala)_
_... 11 more_
_Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
AlreadyExistsException(message:Table
w2314287698276073922_generatedsource_46_view_102__m_sparkengine_alltx_allsrc
already exists)_
_at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:864)_
_at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:872)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createTable$1.apply$mcV$sp(HiveClientImpl.scala:489)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createTable$1.apply(HiveClientImpl.scala:487)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$createTable$1.apply(HiveClientImpl.scala:487)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:221)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:220)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:266)_
_at
org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:487)_
_at
org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createTable$1.apply$mcV$sp(HiveExternalCatalog.scala:309)_
_at
org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createTable$1.apply(HiveExternalCatalog.scala:238)_
_at
org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$createTable$1.apply(HiveExternalCatalog.scala:238)_
_at
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)_
_... 31 more_
_Caused by: AlreadyExistsException(message:Table
w2314287698276073922_generatedsource_46_view_102__m_sparkengine_alltx_allsrc
already exists)_
_at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41631)_
_at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41617)_
_at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:41543)_
_at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)_
_at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1187)_
_at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1173)_
_at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2549)_
_at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)_
_at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:814)_
_at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:800)_
_at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
_at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
_at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
_at java.lang.reflect.Method.invoke(Method.java:498)_
_at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)_
_at com.sun.proxy.$Proxy43.createTable(Unknown Source)_
_at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
_at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
_at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
_at java.lang.reflect.Method.invoke(Method.java:498)_
_at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2477)_
_at com.sun.proxy.$Proxy43.createTable(Unknown Source)_
_at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:857)_
_... 44 more_
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
