[
https://issues.apache.org/jira/browse/ACCUMULO-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347595#comment-15347595
]
Shawn Walker edited comment on ACCUMULO-4353 at 6/24/16 3:04 AM:
-----------------------------------------------------------------
I did have rolling restarts as a primary motivation for doing this work, though
a few other scenarios did come to mind as potential applications:
* tserver loses lock (possibly due to load), dies, and is restarted quickly via
some external infrastructure, e.g. Puppet
* temporary network connectivity loss
I was thinking a `table.suspend.duration` on the order of 2-3 minutes might
make sense for general purposes in a large cluster. Long enough to catch most
truly transient problems, sufficiently short that many applications wouldn't be
unduly impacted. Particularly seeing as any application already has to deal
with a ~30 second wait before the master really notices a tablet server gone
anyways. After all, Accumulo is ultimately a consistent+partition tolerant
database, not an available+partition tolerant database. If availability is a
user's top priority, other databases (e.g. Apache Cassandra) offer tradeoffs in
that direction.
I hadn't seen ACCUMULO-1454, I'll take a closer look in the morning. One
concern that I had with some rolling-restart ideas was a matter of ops
complexity. In my (admittedly limited) experience, orchestrating a rolling
restart that needs to do much more than "kill daemon, restart daemon" over a
large cluster can be a huge headache.
was (Author: shawnwalker):
I did have rolling restarts as a primary motivation for doing this work, though
a few other scenarios did come to mind as potential applications:
* tserver loses lock (possibly due to load), dies, and is restarted quickly via
some external infrastructure, e.g. Puppet
* temporary network connectivity loss
I was thinking a `table.suspend.duration` on the order of 2-3 minutes might
make sense for general purposes in a large cluster. Long enough to catch most
truly transient problems, sufficiently short that many applications wouldn't be
unduly impacted. Particularly seeing as any application already has to deal
with a ~30 second wait before the master really notices a tablet server gone
anyways.
I hadn't seen ACCUMULO-1454, I'll take a closer look in the morning. One
concern that I had with some rolling-restart ideas was a matter of ops
complexity. In my (admittedly limited) experience, orchestrating a rolling
restart that needs to do much more than "kill daemon, restart daemon" over a
large cluster can be a huge headache.
> Stabilize tablet assignment during transient failure
> ----------------------------------------------------
>
> Key: ACCUMULO-4353
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4353
> Project: Accumulo
> Issue Type: Improvement
> Reporter: Shawn Walker
> Assignee: Shawn Walker
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When a tablet server dies, Accumulo attempts to reassign the tablets it was
> hosting as quickly as possible to maintain availability. If multiple tablet
> servers die in quick succession, such as from a rolling restart of the
> Accumulo cluster or a network partition, this behavior can cause a storm of
> reassignment and rebalancing, placing significant load on the master.
> To avert such load, Accumulo should be capable of maintaining a steady tablet
> assignment state in the face of transient tablet server loss. Instead of
> reassigning tablets as quickly as possible, Accumulo should be await the
> return of a temporarily downed tablet server (for some configurable duration)
> before assigning its tablets to other tablet servers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
