[jira] [Created] (HIVE-19874) JSONMessageFactory::getTObj is not thread safe
Dan Burkert created HIVE-19874:
--
Summary: JSONMessageFactory::getTObj is not thread safe
Key: HIVE-19874
URL: https://issues.apache.org/jira/browse/HIVE-19874
Project: Hive
Issue Type: Bug
Components: Standalone Metastore
Affects Versions: 2.3.2
Reporter: Dan Burkert
The static {{JSONMessageFactory::getTObj}} internally uses the non-threadsafe
static field {{thriftDeSerializer}}, which causes concurrent calls to fail with
non-intuitive exceptions. For instance, here's the output from concurrent
{{ALTER TABLE RENAME}} operations on different tables:
{code:java}
2018-06-12T15:34:44,466 ERROR [pool-8-thread-2] metastore.HiveAlterHandler:
Failed to alter table default.table_b_0
2018-06-12T15:34:44,466 ERROR [pool-8-thread-3] metastore.HiveAlterHandler:
Failed to alter table default.table_a_0
2018-06-12T15:34:44,476 ERROR [pool-8-thread-2] metastore.RetryingHMSHandler:
MetaException(message:java.lang.IllegalStateException: Event not set up
correctly)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4063)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:4020)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy25.alter_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11631)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_table_with_environment_context.getResult(ThriftHiveMetastore.java:11615)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:103)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Event not set up correctly
at
org.apache.hadoop.hive.metastore.messaging.AlterTableMessage.checkValid(AlterTableMessage.java:49)
at
org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.(JSONAlterTableMessage.java:57)
at
org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.buildAlterTableMessage(JSONMessageFactory.java:115)
at
org.apache.hive.hcatalog.listener.DbNotificationListener.onAlterTable(DbNotificationListener.java:187)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$8.notify(MetaStoreListenerNotifier.java:107)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)
at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:317)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:4049)
... 16 more
Caused by: org.apache.thrift.protocol.TProtocolException: Unexpected character:{
at
org.apache.thrift.protocol.TJSONProtocol.readJSONSyntaxChar(TJSONProtocol.java:337)
at
org.apache.thrift.protocol.TJSONProtocol.readJSONInteger(TJSONProtocol.java:719)
at
org.apache.thrift.protocol.TJSONProtocol.readFieldBegin(TJSONProtocol.java:858)
at
org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1580)
at
org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.read(Table.java:1573)
at org.apache.hadoop.hive.metastore.api.Table.read(Table.java:1407)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:98)
at
org.apache.hadoop.hive.metastore.messaging.json.JSONMessageFactory.getTObj(JSONMessageFactory.java:270)
at
org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage.getTableObjAfter(JSONAlterTableMessage.java:97)
at
[jira] [Created] (HIVE-18852) Misleading error message in alter table validation
Dan Burkert created HIVE-18852:
--
Summary: Misleading error message in alter table validation
Key: HIVE-18852
URL: https://issues.apache.org/jira/browse/HIVE-18852
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 2.4.0
Reporter: Dan Burkert
The metastore's validation error message when attempting to rename a table to a
non-existent database is wrong. For instance, attempting to alter table
'db.table' to 'non_existent_database.table' results in the Thrift error:
{{TException - service has thrown: InvalidOperationException(message=Unable to
change partition or table. Database db does not exist Check metastore logs for
detailed stack.non_existent_database)}}
I believe the offending line of code is
[here|https://github.com/apache/hive/blob/branch-2/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java?utf8=%E2%9C%93#L331-L333],
notice that {{dbname}} is used in the message, not {{newDbName}}. I don't
know if switching that would cause the case of a non-existing {{dbname}} case
to regress, though.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-17747) HMS DropTableMessage should include the full table object
Dan Burkert created HIVE-17747:
--
Summary: HMS DropTableMessage should include the full table object
Key: HIVE-17747
URL: https://issues.apache.org/jira/browse/HIVE-17747
Project: Hive
Issue Type: Improvement
Components: HCatalog, Metastore
Affects Versions: 2.3.0
Reporter: Dan Burkert
Assignee: Dan Burkert
I have a notification log follower use-case which requires accessing the
parameters of dropped tables, so it would be useful if the {{DROP_TABLE}}
events in the notification log included the full table object, as the create
and alter events do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
Dan Burkert created HIVE-17008:
--
Summary: HiveMetastore.drop_database can return NPE if database
does not exist
Key: HIVE-17008
URL: https://issues.apache.org/jira/browse/HIVE-17008
Project: Hive
Issue Type: Bug
Components: HBase Metastore
Reporter: Dan Burkert
When dropping a non-existent database, the HMS will still fire registered
{{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-16993) ThriftHiveMetastore.create_database can fail if the locationUri is not set
Dan Burkert created HIVE-16993:
--
Summary: ThriftHiveMetastore.create_database can fail if the
locationUri is not set
Key: HIVE-16993
URL: https://issues.apache.org/jira/browse/HIVE-16993
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Dan Burkert
Calling
[{{ThriftHiveMetastore.create_database}}|https://github.com/apache/hive/blob/3fa48346d509813977cd3c7622d581c0ccd51e99/metastore/if/hive_metastore.thrift#L1078]
with a database with an unset {{locationUri}} field through the C++
implementation fails with:
{code}
MetaException(message=java.lang.IllegalArgumentException: Can not create a Path
from an empty string)
{code}
The
[{{locationUri}}|https://github.com/apache/hive/blob/3fa48346d509813977cd3c7622d581c0ccd51e99/metastore/if/hive_metastore.thrift#L270]
Thrift field is 'default requiredness (implicit)', and Thrift [does not
specify|https://thrift.apache.org/docs/idl#default-requiredness-implicit]
whether unset default requiredness fields are encoded. Empirically, the Java
generated code [does not write the
{{locationUri}}|https://github.com/apache/hive/blob/3fa48346d509813977cd3c7622d581c0ccd51e99/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java#L938-L942]
when the field is unset, while the C++ generated code
[does|https://github.com/apache/hive/blob/3fa48346d509813977cd3c7622d581c0ccd51e99/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp#L3888-L3890].
The MetaStore treats the field as optional, and [fills in a default
value|https://github.com/apache/hive/blob/3fa48346d509813977cd3c7622d581c0ccd51e99/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L867-L871]
if the field is unset.
The end result is that when the C++ implementation sends a {{Database}} without
the field set, it actually writes an empty string, and the MetaStore treats it
as a set field (non-null), and then calls a {{Path}} API which rejects the
empty string. The fix is simple: make the {{locationUri}} field optional in
metastore.thrift.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
