[
https://issues.apache.org/jira/browse/SENTRY-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289783#comment-16289783
]
Liam Sargent commented on SENTRY-1753:
--------------------------------------
[~akolb] After a little research, it appears the Impala team uses the following
pattern to solve this problem, as shown in [the source for their
MetaStoreClientPool:|https://github.com/apache/impala/blob/2e63752858d71cc745534367a686980e060a8180/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java#L41]
1.) Wait for inital configurable delay before first attempt. (default 0) (L43)
2.) Attempt to connect to HMS with configurable retry time using
RetryingHMSClient in a while loop, until maximum time is reached (L81)
3.) Fail out if no valid HMS client returned.
I propose we use the same pattern within SentryHMSClient (or HMSFollower) using
exponential backoff retries in order to reduce log noise.
This would work as follows:
1.) Wait for initial configurable delay before first attempt
2.) Attempt to connect to HMS,
* if fail due to no service, wait configurable initial delay amount and try
again.
* If fail again, exponentially increase (usually power of 2) the wait time, and
try again.
3.) Fail out if maximum connection attempt time reached.
Say we set the exponential backoff to 2 seconds.
This means the 1st connection attempt would happen immediately (initial_delay=0)
2nd try: 2 seconds later than 1st
3rd try: 4 seconds later than 2nd
4th try: 8 seconds later than 3rd attempt.
5th try: 16 seconds, etc
continue until HMSFOLLOWER_CONNECTION_TIMEOUT is reached.
On the other hand, just suggesting sensible default startup and retry delays
without really touching the code would mitigate the log noise issue, but has
the tradeoff of potentially slowing down system startup for no reason, and
still bombarding logs with errors if not configured properly.
> Make HMSFollower initial delay configurable
> -------------------------------------------
>
> Key: SENTRY-1753
> URL: https://issues.apache.org/jira/browse/SENTRY-1753
> Project: Sentry
> Issue Type: Task
> Components: Sentry
> Affects Versions: 2.0.0
> Reporter: Na Li
> Assignee: Liam Sargent
> Priority: Minor
>
> HMSFollower initial delay affects how fast it can get notification from Hive
> meta store. We are changing it from 60 seconds to 0 second. It may cause some
> issue. It is better to make this value configurable, so we can adjust it
> based on testing result.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
