Alexander Kolbasov created SENTRY-1687:
------------------------------------------
Summary: FullUpdateInitializer can be more efficient
Key: SENTRY-1687
URL: https://issues.apache.org/jira/browse/SENTRY-1687
Project: Sentry
Issue Type: Improvement
Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov
Priority: Minor
The FullUpdateInitializer follows the {{MetastoreCacheInitializer}}. It reads a
bunch of information from HMS and uses Thrift structures to pass around, but in
the end it just constructs {{Map<String, Set<String>>}}. It uses concurrent
fetches from HMS, but synchronizes a lot on common data structures to update
them.
I think that we can refactor all this code to make it faster and consume less
memory. The idea is the following:
Use background threads to collect Thrift results from HMS calls (database,
table and partition data). Then we can use a single thread to construct the
resulting update and return it without using intermediate Thrift methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
