[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15764837#comment-15764837
]
Joep Rottinghuis commented on HBASE-17018:
------------------------------------------
Thanks for the feedback Sangjin.
(1) The code to transition states is indeed not complete yet. I agree that the
ExceptionListener and information we get back from the BufferedMutator will be
indicative for error conditions. I was thinking what upon the first successful
submission when in bad state, we'd immediately transition to the transitioning
state, but we might want to hedge there before we go back to a good state to
avoid flapping.
(2) You're right that the two-tiered timeout is more complex than a single one.
I was planning to add a jmx bean to be able to poke the coordinator to go to a
"BAD" state in advance of HBase maintenance and/or to force it to spool for
other reasons. I was also thinking of making the timeout dynamic (perhaps
through a bean, perhaps through other mechanisms). If the timeout can reduce
we'd need some reasonable response time for the new timeout to take affect. The
final reason why I chose the double timeout is to allow an exception handler to
call back into the coordinator on a different thread. We could make an
assumption that the exception handler will be called only when the last
submission also times out, but I think that might be making too strong of an
assumption of the current implementation of the BufferedMutatorImpl. That said,
I'll have to work out a little more of the details of state transitioning to
know for sure if two tiers is indeed needed.
(3) I agree that the timeout of 0 is somewhat explicit. The design was aimed to
separate concerns of the coordinator and the processor (for design and testing
purposes). Similarly I didn't want to bleed the state to the flusher, but
simply have it ask whether it should flush or not. If the state itself is
exposed, then the logic will be spread between the coordinator, the processor
and the flusher. One could argue that this is already the case, so I'll have to
re-examine this. When state transitioning evolves, it might be needed to give
up on the strict separation.
(4) Indeed if I were to wrap the put calls, it would alleviate the out of order
submissions between puts and flushes. It would mean I have to change the
flushCount in the SpoolingBufferedMutatorSubmission from a final field set in
the constructor to a volatile (or otherwise synchnonized) setter. Were you
thinking I'd add a new interface around the BlockingQueue, or simply have a
local method and hope the current code and later maintainers keep the
discipline to always call a local method to enqueue? Alternatively the entire
construction of the submission could be pushed into a separate class that keeps
the flushCount and does object creation and queue submission under a single
lock (perhaps simply a synchronized method). I can see if the coordinator could
do this work so we can avoid yet another class.
One open item with respect to timeouts is that there currently isn't a way to
make the spoolingBufferedMutator have a different timeout than the wrapped
BufferedMutatorImpl. There is only one params passed in. This is a challenge in
terms of coordinating the timeouts descibed above, but also in terms of
closing. If the flush in close times out, we might not close. If the close
times out, we may not have time to shut the queues down. We may have to divvy
up the timeout between the various needed operations. TDB how that will work.
> Spooling BufferedMutator
> ------------------------
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
> Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018.master.004.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
> https://reviews.apache.org/r/54882/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
