> On Aug. 19, 2014, 6:31 p.m., Josh Elser wrote:
> > One big design concern I have is what gains the final solution would 
> > actually have over what is currently possible with Accumulo as it stands.
> > 
> > Right now, you can force tablets to migrate by stopping a tserver. This 
> > goes back through the balancer, so you have a bit of churn in however many 
> > "rounds" the Balancer takes to choose where those tablets should go, and 
> > then for the master to process the necessary assignments for each tserver. 
> > How I'm seeing it described is that the only piece of the puzzle that we're 
> > making better is removing the migration components in favor of letting the 
> > user control this directly. How much does a "smart" Balancer implementation 
> > close the gap between the user providing migrations in regards to 
> > performance? Also, how does removing the Balancer from the equation change 
> > the wall time to get a tablet assigned (is it significant)?
> > 
> > We have to also understand that while we can decompose the problem into 
> > some simple primitives, I believe this approach is still a rather difficult 
> > distributed state problem that I'm worried is being over-architected. My 
> > $0.02.
> 
> Josh Elser wrote:
>     For context, I was reading about HBase's support on the subject and found 
> http://hbase.apache.org/book/node.management.html. Their general approach is 
> to provide a graceful shutdown for regionservers. This is still subject to 
> problems in mass amounts of servers being stopped at one time. To alleviate 
> some of this pain, they use ZK to store what servers are currently in a 
> "draining state" to avoid new assignments to those nodes -- "[...] 
> decommissioning mulitple nodes may be non-optimal because regions that are 
> being drained from one region server may be moved to other regionservers that 
> are also draining. Marking RegionServers to be in the draining state prevents 
> this from happening",
> 
> kturner wrote:
>     An alternative to this design, is one that Mike mentioned on the issue.   
> Temporarily replace the balancer.  I am thinking that providing these 
> primitves for manipulating tablets will allow an administrator to quickly 
> script a one off solution to a problem, in addition to solving the rolling 
> restart problem.  You do not get this quick flexibility with writing a new 
> balancer.
>     
>     Killing tablet servers is a solution.  I think it would be nice to have a 
> solution that avoids log recovery, minimizes down time of individual tablets, 
> preserves locality, and is easy to use.  It does not have to be this 
> solution.  W/o additional scripts, the primary use case in 1454 would not be 
> easy to use.   A balancer alone would not be enough to achieve the goal of 
> migrating tablets between old and new tservers on the same node.  However a 
> balancer + tservers states like you mentioned from HBAse may provide enough.  
> Should probably try to explore the balancer option a bit more.
> 
> kturner wrote:
>     One other thing I was thinking about was that you can not make 
> assumptions about the environment.  Users may not use the Accumulo scripts to 
> start and stop tservers.
> 
> Josh Elser wrote:
>     I think there would be merit in enumerating what would be needed by a 
> custom Balancer. Is it really something that would need to be written on a 
> per-instance basis, or is there something we could provide that would be more 
> conducive to "heavy" tserver churn.
>     
>     I would definitely not advocate killing tservers. A graceful shutdown 
> would be much more desirable. We get a little bit of help here by the 
> client-side scan retries for not having to quiesce all reads to a tablet, but 
> that could still introduce more latency for a query (e.g. lots of filtering 
> over a large row).
>     
>     As mentioned about concerns with the final two-tservers-per-node 
> approach, I'm not entirely convinced that "sibling" tservers is worth the 
> complexity. We really don't have that much locality in how we use HDFS now. 
> Is trying to keep all of the tablets assigned on the same node going to make 
> things much more efficient over assigning them to nodes elsewhere? I don't 
> even have a good grasp for what these perf numbers would be at a high level.
> 
> kturner wrote:
>     Eric looked into locality once when running continuous ingest and found 
> that ~50% of tablets had local data.    This matches expectations as the 
> default balancer will try to migrate one child after a split.
>     
>     The sibling tserver concept may be too complex to implement.  Sigh, but 
> its so cool :)
> 
> Josh Elser wrote:
>     Clarification on what I meant by locality: we don't consider HDFS block 
> locations when we chose where Tablets get assigned, AFAIK. Yes, we'll have 
> locality when we're slamming Accumulo with ingest, but once we start 
> agitating at any reasonable rate, that's going to be lost.
>     
>     Requiring sibling tservers also implies that you have ample extra 
> resources on a node which is absolutely not going to be the case for most 
> systems. It would be nice, but it sounds to me like a one-off from what would 
> be the norm. :)

On the mailing list Adam thought locality reached 90% in long running CI test.  
Need to ask Eric what he saw.  Seems plausable that as time since split 
increases that locality would increase in a stable system.


- kturner


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24855/#review51006
-----------------------------------------------------------


On Aug. 19, 2014, 5:50 p.m., kturner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24855/
> -----------------------------------------------------------
> 
> (Updated Aug. 19, 2014, 5:50 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-1454
>     https://issues.apache.org/jira/browse/ACCUMULO-1454
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> Positing ACCUMULO-1454 design doc for review
> 
> 
> Diffs
> -----
> 
>   docs/src/main/asciidoc/design/ACCUMULO-1454-proposal-01.adoc PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24855/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> kturner
> 
>

Reply via email to