Tim Armstrong resolved IMPALA-4953.
       Resolution: Fixed
    Fix Version/s: Impala 2.12.0
                   Impala 3.0

> Prevent large statestore updates from head-of-line blocking subsequent 
> updates to different topics
> --------------------------------------------------------------------------------------------------
>                 Key: IMPALA-4953
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4953
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Distributed Exec
>    Affects Versions: Impala 2.9.0
>            Reporter: Henry Robinson
>            Assignee: Tim Armstrong
>            Priority: Major
>             Fix For: Impala 3.0, Impala 2.12.0
> The statestore currently tries to send all its topic updates to a particular 
> host at the same time. If a topic update is very large (e.g. large metadata 
> change) that can delay the receipt of subsequent topic updates (like 
> membership or load data). 
> The obvious solution is to send topic updates independently, at different 
> frequencies. But the statestore->subscriber protocol is based on the idea 
> that there's only one update in flight at once, and changing that might be 
> more complex than necessary. 
> A simpler approach is to have per-topic queues in the subscriber, and to only 
> move the topic update payload onto the queue in the {{UpdateState()}} 
> handler. That means that topic updates can happen frequently (the statestore 
> won't send data update for a topic with no changes, of course). If the 
> subscriber can't keep up with the rate of change, there's already a mechanism 
> for dealing with that by setting the {{skipped}} parameter in the RPC 
> response, which tells the statestore to retry the update later. That can be 
> extended to be per-topic, so the subscriber can keep up-to-date with 
> lightweight topics.
> We can always set a topic update frequency to be a multiple of the 'base' 
> frequency, but we also want a mechanism to prevent large updates from 
> blocking small ones, and that can be addressed better by processing topic 
> updates asynchronously.

This message was sent by Atlassian JIRA

Reply via email to