[
https://issues.apache.org/jira/browse/HIVE-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593695#comment-15593695
]
Eric Lin edited comment on HIVE-14482 at 10/21/16 2:12 AM:
-----------------------------------------------------------
The error seems to be unrelated (please see the attached screenshot) and there
is no existing test coverage for HMS audit logs, also it is pretty hard to test
it.
Please advise if the patch can be accepted.
Thanks
was (Author: ericlin):
The error seems to be unrelated and there is no existing test coverage for HMS
audit logs, also it is pretty hard to test it.
Please advise if the patch can be accepted.
Thanks
> Add and drop table partition is not audit logged in HMS
> -------------------------------------------------------
>
> Key: HIVE-14482
> URL: https://issues.apache.org/jira/browse/HIVE-14482
> Project: Hive
> Issue Type: Improvement
> Affects Versions: 2.1.0
> Reporter: Eric Lin
> Assignee: Eric Lin
> Priority: Minor
> Attachments: HIVE-14482.2.patch, HIVE-14482.patch, Screen Shot
> 2016-10-21 at 1.06.33 pm.png
>
>
> When running:
> {code}
> ALTER TABLE test DROP PARTITION (b=140);
> {code}
> I only see the following in the HMS log:
> {code}
> 2016-08-08 23:12:34,081 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: <PERFLOG method=get_table
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,082 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:
> [pool-4-thread-2]: 2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,082 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
> ugi=hive ip=xx.xx.xxx.xxx cmd=source:xx.xx.xxx.xxx get_table : db=default
> tbl=test
> 2016-08-08 23:12:34,094 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: </PERFLOG method=get_table start=1470723154081
> end=1470723154094 duration=13
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2
> retryCount=0 error=false>
> 2016-08-08 23:12:34,095 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: <PERFLOG method=get_partitions_by_expr
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,095 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:
> [pool-4-thread-2]: 2: source:xx.xx.xxx.xxx get_partitions_by_expr :
> db=default tbl=test
> 2016-08-08 23:12:34,096 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
> ugi=hive ip=xx.xx.xxx.xxx cmd=source:xx.xx.xxx.xxx get_partitions_by_expr
> : db=default tbl=test
> 2016-08-08 23:12:34,112 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: </PERFLOG method=get_partitions_by_expr
> start=1470723154095 end=1470723154112 duration=17
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2
> retryCount=0 error=false>
> 2016-08-08 23:12:34,172 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: <PERFLOG method=get_table
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,173 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:
> [pool-4-thread-2]: 2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,173 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
> ugi=hive ip=xx.xx.xxx.xxx cmd=source:xx.xx.xxx.xxx get_table : db=default
> tbl=test
> 2016-08-08 23:12:34,186 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: </PERFLOG method=get_table start=1470723154172
> end=1470723154186 duration=14
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2
> retryCount=0 error=false>
> 2016-08-08 23:12:34,186 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: <PERFLOG method=get_table
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,187 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:
> [pool-4-thread-2]: 2: source:xx.xx.xxx.xxx get_table : db=default tbl=test
> 2016-08-08 23:12:34,187 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
> ugi=hive ip=xx.xx.xxx.xxx cmd=source:xx.xx.xxx.xxx get_table : db=default
> tbl=test
> 2016-08-08 23:12:34,199 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: </PERFLOG method=get_table start=1470723154186
> end=1470723154199 duration=13
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2
> retryCount=0 error=false>
> 2016-08-08 23:12:34,203 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: <PERFLOG method=drop_partitions_req
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
> 2016-08-08 23:12:34,215 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-2]: JDO filter pushdown cannot be used: Filtering is supported
> only on partition keys of type string
> 2016-08-08 23:12:34,226 ERROR org.apache.hadoop.hdfs.KeyProviderCache:
> [pool-4-thread-2]: Could not find uri with key
> [dfs.encryption.key.provider.uri] to create a keyProvider !!
> 2016-08-08 23:12:34,239 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:
> [pool-4-thread-2]: dropPartition() will move partition-directories to
> trash-directory.
> 2016-08-08 23:12:34,239 INFO hive.metastore.hivemetastoressimpl:
> [pool-4-thread-2]: deleting
> hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO org.apache.hadoop.fs.TrashPolicyDefault:
> [pool-4-thread-2]: Moved:
> 'hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140' to trash at:
> hdfs://<host-name>:8020/user/hive/.Trash/Current/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO hive.metastore.hivemetastoressimpl:
> [pool-4-thread-2]: Moved to trash:
> hdfs://<host-name>:8020/user/hive/warehouse/default/test/b=140
> 2016-08-08 23:12:34,247 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
> [pool-4-thread-2]: </PERFLOG method=drop_partitions_req start=1470723154203
> end=1470723154247 duration=44
> from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=2
> retryCount=0 error=false>
> {code}
> There is no entry in the "HiveMetaStore.audit" to show that partition b=140
> was dropped.
> When we add a new partition, we can see the following:
> {code}
> 2016-08-08 23:04:48,534 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-4-thread-2]:
> ugi=hive ip=xx.xx.xxx.xxx cmd=source:xx.xx.xxx.xxx append_partition :
> db=default tbl=test[130]
> {code}
> Ideally we should see the similar message when dropping partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
