[
https://issues.apache.org/jira/browse/HIVE-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448301#comment-16448301
]
Adam Szita commented on HIVE-18705:
-----------------------------------
Uploaded [^HIVE-18705.4.patch] with my tweaks on this one.
The original implementation does the following when dropping DB cascade:
- get list of table names
- loop:
-- get Table object (by its name) from HMS
-- drop the table in HMS
-- execute drop hook
- drop DB in HMS
This results in 1+100*2+1=202 HMS calls for just removing a DB with 100
tables...
My approach to speed this up was:
- get list of table names
- if list size > {{MetastoreConf#ConfVars.BATCH_RETRIEVE_MAX}} (aka we don't
want too many Table objects in our limited memory space)
-- do the same like before but use batched Table retrieval using TableIterable
-- e.g. if BATCH_RETRIEVE_MAX=50, it will only send one API call to HMS, to
get 50 Table objects
-- drop table calls in loop as usual and run hooks
- else we query all Table objects in one batch
-- send only one call, a dropDB to HMS
-- loop on Tables to execute hook
In the first scenario we reduced the original API call count from 202 to
1+1+50+1+50+1=104.
In the second scenario to 1+1=2...
Then I started profiling the calls on a cluster and saw that although the
second scenario does save a lot of time (about 4-5X times faster), the first
scenario makes no difference from the original one.
After some debugging and more profiling I found the root cause:
HMS's dropTable takes about 120 msec, (even if perflogger says it finished in
20ms), while getTable takes just 10ms.
So reducing the number of getTable calls makes little difference while we have
a heavy dropTable method.
The reason why dropTable takes a lot of time is the clean-up part of the call
[here|https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/TUGIBasedProcessor.java#L132].
Drop table call comes in, it is executed and perflogger logs ~20 msec time for
the operation, result is then written back to the client - BUT:
Closing down the FD opened by the call takes at least 100 msec. This is
because Hadoop adds this time when the hdfs client closes connection to the
namenode
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1323]
This latter finding is what makes the current implementation of dropDB in HMS
client so bad: it makes the server side open the same HDFS connection for each
table in the DB and then of course close it on every occasion....
After all of this I started having doubts if the first (batched Table
retrieval) scenario even makes sense to be introduced when we don't see
relevant speed-up. Then I've seen this: [In DDLSemanticAnalyzer Hive already
fetches ALL the Table objects for a given
DB.|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java#L1393]
(I believe this is for ACID reasons) So after all, all Table objects have
already fit in memory when the client's dropDB method is invoked - I might
started this being too careful with memory?
+So here's a question+: should I get rid of the batched scenario as all the
tables are queried and are accessible at a time already, and there's little
reason for me to query them in batches later (for memory reasons) instead of
all of them at once. This way I could have the non-batched (send one dropDB
only) scenario only which doesn't suffer from all the slowing effects I
described above, and is generally 4-5 times faster than the current
implementation.
My gut feeling is to go with only the dropDB scenario and forget about batching.
What are your thoughts?
> Improve HiveMetaStoreClient.dropDatabase
> ----------------------------------------
>
> Key: HIVE-18705
> URL: https://issues.apache.org/jira/browse/HIVE-18705
> Project: Hive
> Issue Type: Improvement
> Reporter: Adam Szita
> Assignee: Adam Szita
> Priority: Major
> Attachments: HIVE-18705.0.patch, HIVE-18705.1.patch,
> HIVE-18705.2.patch, HIVE-18705.4.patch
>
>
> {{HiveMetaStoreClient.dropDatabase}} has a strange implementation to ensure
> dealing with client side hooks (for non-native tables e.g. HBase). Currently
> it starts by retrieving all the tables from HMS, and then sends {{dropTable}}
> calls to HMS table-by-table. At the end a {{dropDatabase}} just to be sure :)
> I believe this could be refactored so that it speeds up the dropDB in
> situations where the average table count per DB is very high.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
