[
https://issues.apache.org/jira/browse/IMPALA-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-4953.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
Impala 3.0
> Prevent large statestore updates from head-of-line blocking subsequent
> updates to different topics
> --------------------------------------------------------------------------------------------------
>
> Key: IMPALA-4953
> URL: https://issues.apache.org/jira/browse/IMPALA-4953
> Project: IMPALA
> Issue Type: Improvement
> Components: Distributed Exec
> Affects Versions: Impala 2.9.0
> Reporter: Henry Robinson
> Assignee: Tim Armstrong
> Priority: Major
> Fix For: Impala 3.0, Impala 2.12.0
>
>
> The statestore currently tries to send all its topic updates to a particular
> host at the same time. If a topic update is very large (e.g. large metadata
> change) that can delay the receipt of subsequent topic updates (like
> membership or load data).
> The obvious solution is to send topic updates independently, at different
> frequencies. But the statestore->subscriber protocol is based on the idea
> that there's only one update in flight at once, and changing that might be
> more complex than necessary.
> A simpler approach is to have per-topic queues in the subscriber, and to only
> move the topic update payload onto the queue in the {{UpdateState()}}
> handler. That means that topic updates can happen frequently (the statestore
> won't send data update for a topic with no changes, of course). If the
> subscriber can't keep up with the rate of change, there's already a mechanism
> for dealing with that by setting the {{skipped}} parameter in the RPC
> response, which tells the statestore to retry the update later. That can be
> extended to be per-topic, so the subscriber can keep up-to-date with
> lightweight topics.
> We can always set a topic update frequency to be a multiple of the 'base'
> frequency, but we also want a mechanism to prevent large updates from
> blocking small ones, and that can be addressed better by processing topic
> updates asynchronously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
