[
https://issues.apache.org/jira/browse/HIVE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945892#comment-14945892
]
Siddharth Seth commented on HIVE-11786:
---------------------------------------
This patch seems to have caused a pretty big performance regression on the
client.
With the patch
{code}
2015-10-06T17:25:01,133 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:25:21,237 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444166701133 end=1444166721237 duration=20104
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:25:21,248 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:25:41,074 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444166721248 end=1444166741074 duration=19826
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:25:41,170 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:26:59,585 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444166741170 end=1444166819585 duration=78415
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:27:01,439 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:27:01,662 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444166821439 end=1444166821662 duration=223
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
{code}
Without the patch.
{code}
2015-10-06T17:04:51,205 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:04:51,440 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444165491205 end=1444165491439 duration=234
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:04:51,450 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:04:51,655 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444165491450 end=1444165491655 duration=205
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:04:51,751 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:04:52,355 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444165491751 end=1444165492355 duration=604
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
2015-10-06T17:04:54,224 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=get_aggr_stats_for
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
2015-10-06T17:04:54,624 INFO [main([])]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(157)) - </PERFLOG method=get_aggr_stats_for
start=1444165494224 end=1444165494624 duration=400
from=org.apache.hadoop.hive.metastore.RetryingHMSHandler threadId=0
retryCount=0 error=false>
{code}
This is running against a MySQL database with a client-embedded metastore.
Are there specific log lines which would help to debug this ? I have logs with
debug logging for org.apache.hadoop.hive.metastore
> Deprecate the use of redundant column in colunm stats related tables
> --------------------------------------------------------------------
>
> Key: HIVE-11786
> URL: https://issues.apache.org/jira/browse/HIVE-11786
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Chaoyu Tang
> Assignee: Chaoyu Tang
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-11786.1.patch, HIVE-11786.1.patch,
> HIVE-11786.2.patch, HIVE-11786.patch
>
>
> The stats tables such as TAB_COL_STATS, PART_COL_STATS have redundant columns
> such as DB_NAME, TABLE_NAME, PARTITION_NAME since these tables already have
> foreign key like TBL_ID, or PART_ID referencing to TBLS or PARTITIONS.
> These redundant columns violate database normalization rules and cause a lot
> of inconvenience (sometimes difficult) in column stats related feature
> implementation. For example, when renaming a table, we have to update
> TABLE_NAME column in these tables as well which is unnecessary.
> This JIRA is first to deprecate the use of these columns at HMS code level. A
> followed JIRA is to be opened to focus on DB schema change and upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
