[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746798#comment-15746798
]
stack commented on HBASE-17018:
-------------------------------
Nice writeup/design. Looks good. Like the tracking of 'batches'. There seems to
be a bunch of moving parts but all seem necessary.
On "...wrapped in a SpoolingBufferedMutatorSubmission which comes in three
SpoolingBufferedMutatorSubmissionType...", if you were looking for a simpler
approach, you might just add markers to the Mutation Map of Attributes...
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/OperationWithAttributes.html#setAttribute-java.lang.String-byte:A-
On the patch:
This will not be committed to hbase? It'll be part of timeline v2? (Or
looking at the patch, there is some change needed in core ... in
BufferedMutatorParams... these we can take).
The bulk of the change in BufferedMutatorParams is unrelated. You want to do a
patch w/ just the changes to hbase core removing the non-changes: i.e. in
BufferedMutatorParams only change should be the clone method addition, not the
reformatting of javadoc.
Is FilesystemMutationSpooler still TODO? Is it needed? There doesn't seem to be
much filesystem-ey about FilesystemMutationSpooler, at least just yet.
If SpoolingBufferedMutatorCoordinator Interface a bit over the top? Is there
ever going to be another type of cooridinator implementation?
nit: Could change this
145 public boolean shouldSpool(SpoolingBufferedMutatorSubmission
submission) {
146 // TODO: should we keep track of the last successful flush that
went through
147 // and perhaps still drop some items from outbound, even if we just
went
148 // into a bad state?
149 if (state == State.GOOD) {
150 return false;
151 } else {
152 // Spool for both BAD and TRANSITIONING states.
153 return true;
154 }
155 }
to....
return state != State.GOOD;
Otherwise skimmed the rest.. Where are the tests?
Looks good [~jrottinghuis]
> Spooling BufferedMutator
> ------------------------
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
> Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
