[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963342#comment-15963342 ] Andrew Purtell commented on HBASE-17018: [~enis] [~jrottinghuis] Should we move this forward? What's the right move? I think this could be useful, FWIW > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, HBASE-17018.master.005.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ > For design of SpoolingBufferedMutatorImpl see > https://docs.google.com/document/d/1GTSk1Hd887gGJduUr8ZJ2m-VKrIXDUv9K3dr4u2YGls/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768703#comment-15768703 ] Enis Soztutar commented on HBASE-17018: --- Thanks for entertaining my suggestion. bq. On our case we launch ~1K containers per second. If we write 100 metrics each, the total volume written into HBase is considerable HBase in the end will end up writing these events to it's own WAL's on HDFS. So in terms of scalability, you should be able to achieve HBase throughput much much easier, since HBase is doing a lot more work (RPC, sorting data, flushing to disk, compaction, etc). bq. For large deployments that means that there could be hundreds of parallel writers. That should be fine for HDFS, as long as you have 1 writer per application, rather than 1 writer per task. bq. It would essentially double the hdfs requirement for the storage I was thinking that you would delete the records once the reader has persisted them to HBase. If the application writer is dead, some other application writer eventually finishes the persisting to HBase (because WALs are already there in HDFS). For example, HBase keeps rolling the WAL to a new file every ~100MB's. Then the whole file is deleted once we determine that it is not needed anymore. bq. Would the reader still query hbase only and return no data if hbase is missing the data? I think that is determined by the requirements for ATS. You have to determine the "commit point" and the read point semantics. For example, you can have it so that commit point is the HDFS write. Once it is complete, you ACK the write which means HBase write will be "eventual consitent" with the benefit of not depending on HBase availability. Or you can make it so that, the commit point is HDFS write + wait for Hbase write for 30seconds. In this case, you wait for HBase for 30 sec, but still ACK the write once it hits HDFS after timeout. It also depends on whether you need read-after-write semantics or not. If so, maybe you do a in-memory cache for stuff waiting to be written to HBase. Not sure on ATS requirements. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ > For design of SpoolingBufferedMutatorImpl see > https://docs.google.com/document/d/1GTSk1Hd887gGJduUr8ZJ2m-VKrIXDUv9K3dr4u2YGls/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768468#comment-15768468 ] Sangjin Lee commented on HBASE-17018: - Your suggestion is interesting [~enis]. Thanks for the idea. In addition to what Joep mentioned above, I do worry about the capacity requirement a dual-writing system would have. It would essentially double the hdfs requirement for the storage, and at large scale it would add up to a meaningful amount. Also, how would a reader work in the case where the data made it into hdfs but not into hbase (e.g. hbase cluster was down for a while for an upgrade)? Would the reader still query hbase only and return no data if hbase is missing the data? If we want to address that situation, we're putting back the unspooling (migrating missing data from the backup location to hbase). I'm just trying to round out the idea... Thanks! > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ > For design of SpoolingBufferedMutatorImpl see > https://docs.google.com/document/d/1GTSk1Hd887gGJduUr8ZJ2m-VKrIXDUv9K3dr4u2YGls/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765627#comment-15765627 ] Joep Rottinghuis commented on HBASE-17018: -- [~enis] thanks for the comments. Interesting food for thought. I should have made that more clear in the requirements doc, but for ATS we actually have many writers and a shared bank of readers. The Yarn Resource Manager does have its own writer, but there will be a writer per active application in the cluster. In the current version these are spawned in the NodeManager as an auxiliary service, but we intend to make the per application writer a specially managed standalone container. For large deployments that means that there could be hundreds of parallel writers. On our case we launch ~1K containers per second. If we write 100 metrics each, the total volume written into HBase is considerable, and that isn't counting the longer running applications that want to send their data between once per minute and once per 10 minutes (depending on what the particular HBase cluster can handle). On top of that we will probably have one shared HBase cluster (per datacenter) hosting data for multiple Yarn clusters. The simplicity of always dual writing does sound appealing. The dependency on yet another service would be hard to sell. The hard dependency on HBase being up is exactly what we're trying to tackle in this jira. We'll discuss with the other ATS devs to see if adding a BK type of solution is tenable. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765286#comment-15765286 ] Joep Rottinghuis commented on HBASE-17018: -- [~devaraj] thanks for your feedback. Indeed I'm well aware that this feature may not make it into HBase if the consensus is indeed that this doesn't fit the intended core use-cases. I'm trying to keep the code separate so that it is relatively easy to pull out wholesale. For modifications of existing code I'll file separate sub-tasks. Do you mind if I develop the code a little further here under the watchful eye of HBase gurus? Also, would you have any suggestions how I should go about tackling sub-task HBASE-17327: "Allow for lazy connection / BufferedMutator creation"? I'm happy to do the work, but would like an opinion on the desired approach from HBase folks such as yourself. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765091#comment-15765091 ] Enis Soztutar commented on HBASE-17018: --- I've read through the requirements a bit, and left with feeling that you need persistent queue, a WAL (Kafka / Bookeeper) in front of HBase, rather than spooling for a much simpler design. The semantics would be that the writes will not fail as long as the kafka / BK is available. A single consumer would consume the WAL and propagate the writes via BufferedMutator. Of course having this q would mean that writes are doubled, but I think the failure conditions and simplicity makes it so much simpler to deal with to justify the design. Other concern is to have to depend on yet another system for ATS deployment. Since for ATS, there is single writer and single consumer, how about doing a WAL on top of HDFS, for effectively implementing BK? In that you would not need to depend on another system, and affectively achieve high availability. Writes will succeed as long as HDFS/local fs is up and running. You can do what HBase/Kafka does in terms of periodically rolling the WAL, and keeping track of where the readers are (the reader is the BuffereMutator that replicates to HBase), and deleting the WAL files once they are not needed anymore. This way, recovery is a very simple task as well (just restart). You probably do not even need to wait for the HDFS write to finish before sending the Put to HBase. You can affectively have an in-memory buffer for the recent WAL appends and have the consumer switch to that, if it is sufficiently up-to-date. Anyway a suggestion to consider. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765011#comment-15765011
]
Sangjin Lee commented on HBASE-17018:
-
{quote}
Indeed if I were to wrap the put calls, it would alleviate the out of order
submissions between puts and flushes. It would mean I have to change the
flushCount in the SpoolingBufferedMutatorSubmission from a final field set in
the constructor to a volatile (or otherwise synchnonized) setter. Were you
thinking I'd add a new interface around the BlockingQueue, or simply have a
local method and hope the current code and later maintainers keep the
discipline to always call a local method to enqueue? Alternatively the entire
construction of the submission could be pushed into a separate class that keeps
the flushCount and does object creation and queue submission under a single
lock (perhaps simply a synchronized method). I can see if the coordinator could
do this work so we can avoid yet another class.
{quote}
I suppose there are a couple of options each of which has pros and cons. First
was extending {{LinkedBlockingQueue}} within this class to add the logic of
handling the flush count strongly. Something like:
{code}
final BlockingQueue inbound =
new LinkedBlockingQueue<>() {
public synchronized void put(SpoolingBufferedMutatorSubmission s)
throws InterruptedException {
// handle and set the flush count on the submission based on the type
super.put(s);
}
};
{code}
That way, any new code within this class that would put items in the inbound
queue would benefit from the same treatment. One drawback is that {{put()}} is
not the only way to add items to the queue (there is {{add()}}), and to be
complete we'd need to either override {{add()}} too or ensure no one calls it
(by overriding it to throw an exception or something). This overridden class
can be totally private to the SBMI.
I also thought of simply doing this as a private method within SBMI so we don't
need to play with extending the LinkedBlockingQueue. But the downside of that
is we need to ensure all new code that puts items in that inbound queue goes
through that method.
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018.master.004.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
> https://reviews.apache.org/r/54882/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764837#comment-15764837 ] Joep Rottinghuis commented on HBASE-17018: -- Thanks for the feedback Sangjin. (1) The code to transition states is indeed not complete yet. I agree that the ExceptionListener and information we get back from the BufferedMutator will be indicative for error conditions. I was thinking what upon the first successful submission when in bad state, we'd immediately transition to the transitioning state, but we might want to hedge there before we go back to a good state to avoid flapping. (2) You're right that the two-tiered timeout is more complex than a single one. I was planning to add a jmx bean to be able to poke the coordinator to go to a "BAD" state in advance of HBase maintenance and/or to force it to spool for other reasons. I was also thinking of making the timeout dynamic (perhaps through a bean, perhaps through other mechanisms). If the timeout can reduce we'd need some reasonable response time for the new timeout to take affect. The final reason why I chose the double timeout is to allow an exception handler to call back into the coordinator on a different thread. We could make an assumption that the exception handler will be called only when the last submission also times out, but I think that might be making too strong of an assumption of the current implementation of the BufferedMutatorImpl. That said, I'll have to work out a little more of the details of state transitioning to know for sure if two tiers is indeed needed. (3) I agree that the timeout of 0 is somewhat explicit. The design was aimed to separate concerns of the coordinator and the processor (for design and testing purposes). Similarly I didn't want to bleed the state to the flusher, but simply have it ask whether it should flush or not. If the state itself is exposed, then the logic will be spread between the coordinator, the processor and the flusher. One could argue that this is already the case, so I'll have to re-examine this. When state transitioning evolves, it might be needed to give up on the strict separation. (4) Indeed if I were to wrap the put calls, it would alleviate the out of order submissions between puts and flushes. It would mean I have to change the flushCount in the SpoolingBufferedMutatorSubmission from a final field set in the constructor to a volatile (or otherwise synchnonized) setter. Were you thinking I'd add a new interface around the BlockingQueue, or simply have a local method and hope the current code and later maintainers keep the discipline to always call a local method to enqueue? Alternatively the entire construction of the submission could be pushed into a separate class that keeps the flushCount and does object creation and queue submission under a single lock (perhaps simply a synchronized method). I can see if the coordinator could do this work so we can avoid yet another class. One open item with respect to timeouts is that there currently isn't a way to make the spoolingBufferedMutator have a different timeout than the wrapped BufferedMutatorImpl. There is only one params passed in. This is a challenge in terms of coordinating the timeouts descibed above, but also in terms of closing. If the flush in close times out, we might not close. If the close times out, we may not have time to shut the queues down. We may have to divvy up the timeout between the various needed operations. TDB how that will work. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762788#comment-15762788
]
Sangjin Lee commented on HBASE-17018:
-
I am supportive of the design (obviously) as I consulted with Joep and provided
feedback. I like the overall approach and the proposed WIP patch.
I went over the patch at a high level and I do have several comments and
questions.
(1)
What's not entirely clear to me from the patch is exactly how the state
transition will occur, especially how it will transition out of BAD into
TRANSITIONING or GOOD. Maybe that is still TODO? Did I read that right? We
talked offline about attempting flushing periodically if it is in a bad state
to probe the state of the HBase cluster. One idea we could use is to use the
ExceptionListener/RetriesExhaustedWithDetailsException/mayHaveClusterIssues.
Also, we want to think about under what condition we transition from GOOD to
BAD. I still think the exception listener has a lot of value as it can tell us
(more) about the cluster status. We should see if we can utilize the exception
listener in determining whether to transition. We should also tune the timeout
value to the same timeout value we're willing to wait for these operations so
that in most cases we would detect the timeout in the form of an exception
coming from the underlying {{BufferedMutatorImpl}} with the proper exception
listener.
(2)
In {{SpoolingBufferedMutatorProcessor.call()}}, I'm not quite sure how useful
the two-tiered timeout setup is. It appears that the overall timeout is
something like 100 seconds, but we divide it with finer-grained 1-second timed
waits and keep looping until we exhaust the overall timeout. Is it truly
necessary? Do we gain value by having the two-tiered mechanism? Since this is
all done in the same single thread, it's not like the thread will do anything
other than looping right back onto {{future.get()}} on the same submission.
IMO, this seems to be adding more complexity than is needed with not much
payoff. Why not simply have a single timeout with the desired overall timeout?
That would make this much simpler without losing any flexibility.
(3)
Somewhat related to the above, I'd like to see the submission state used more
explicitly in that method. Currently a bad state and a subsequent behavior
difference are implied by {{timeout == 0}}. Instead, it might be great if we
explicitly use the state from the coordinator to do different things (e.g.
{{coordinator.getState()}} instead of {{coordinator.getTimeout()}}). We can add
things like the timeout value into the state enum so the state becomes more
useful.
(4)
Regarding the out-of-order mutate calls with respect to flushes, one idea might
be to extend the {{BlockingQueue}} (specifically override {{put()}}) so that
each {{put()}} call can handle the flush count synchronously and internally as
part of the call. Then we may be able to eliminate the need for handling
out-of-order mutate calls and thus simplify further.
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762388#comment-15762388 ] Devaraj Das commented on HBASE-17018: - Folks, let's consider putting the Spool* implementations outside HBase. The refactor of the Buffered* classes is fine IMO but I'd hesitate to open this feature up for general consumption yet. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15757654#comment-15757654 ] Joep Rottinghuis commented on HBASE-17018: -- Thanks for the feedback [~tedyu] I'll go through and apply your suggestions. I do plan to resolve all the TODOs as well before I declare this patch as ready. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756773#comment-15756773
]
Ted Yu commented on HBASE-17018:
For SpoolingBufferedMutatorImpl :
{code}
this.maxKeyValueSize =
params.getMaxKeyValueSize() != BufferedMutatorParams.UNSET
? params.getMaxKeyValueSize() : Integer.MAX_VALUE;
{code}
We already have hbase.client.keyvalue.maxsize
Should use the config value in place of Integer.MAX_VALUE
{code}
if ((currentBufferSize.get() > writeBufferSize)
&& (previousBufferSize < writeBufferSize)) {
{code}
Should the first line above be:
{code}
if ((currentBufferSize.get() >= writeBufferSize)
{code}
For close(), why is the call to flush(true) outside of the try block ?
{code}
if (!processES.awaitTermination(operationTimeout, TimeUnit.MILLISECONDS))
{
{code}
Should we obtain timestamp (now) at the beginning of the method and wait for
what remains w.r.t. operationTimeout ?
{code}
} catch (ExecutionException ee) {
LOG.error("ExecutionException while waiting for shutdown.", ee);
{code}
Should some IOException be thrown in case of ExecutionException ?
{code}
} catch (ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
{code}
Convert the above to LOG.error().
Both setRpcTimeout() and setOperationTimeout() call
wrapped.setOperationTimeout(timeout), do we need both methods ?
Please put patch on review board - it is getting bigger.
Thanks
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755171#comment-15755171 ] stack commented on HBASE-17018: --- Skimmed. Looking clean. Tests look great. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748827#comment-15748827 ] Joep Rottinghuis commented on HBASE-17018: -- If I can make it so that the SpoolingBufferedMutator doesn't even connect to an actual BufferedMutatorImpl, this setup could help in some tests where you want to be able to send mutations but just have them appear in a file. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747493#comment-15747493 ] stack commented on HBASE-17018: --- HBASE-17313 and HBASE-17277 were pushed. Hope I think I got it right. ExceptionListener seems like good idea -- especially if it will help this project. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747443#comment-15747443 ] Joep Rottinghuis commented on HBASE-17018: -- Filed HBASE-17313 with patch. If HBASE-17313 goes in first, then HBASE-17277 should be updated to add the new field in the clone method (and in the unit test comparison), or visa versa. bq. Is there anything point in AP that could be exposed that might help simplify the implementation at all? Good question. Not quite sure. I was thinking of adding two more things: a) an exception listener that can be used to capture exceptions from the BufferedMutatorImpl and pass them to the coordinator. I'll have to see if it is clear for the submission to catch these and shove all of this info into the Future to pass back, or if I want to have this all appear asynchronously in the coordinator. I think the former might be cleaner. b) If the outbound queue reaches a certain size it should trigger a flush. As [~sjlee0] pointed out, the current design would allow for the outbound queue to grow very large if the user keeps sending mutations without calling flush. The BufferdMutatorImpl could have flushed, but we don't know that. I think the cleanest solution would be to call flush, but that cannot be a blocking call from the processor, otherwise we'll have a deadlock on our hands. I probably have to make flush on the SpoolingBufferedMutatorImpl have a boolean argument to block or not block. Perhaps this is a bit where modifying the AP or the BMI to indicate that a size-based flush happened might be a good thing. On the other hand, to treat the BMI as a total black-box has a certain elegance... > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747317#comment-15747317 ] stack commented on HBASE-17018: --- bq, I don't want those attributes to be transmitted to HBase, that would be a waste. The bit where you'd have to peel-apart-a-submission to mark all Mutations kills my suggestion I'd say. bq. Is that a question or statement? If the HBase community is interested to have this part of HBase, that would be great and I'll continue the code in place. If not, then I'll move this to yarn. Question. IMO, it is an exotic feature. Lets see if anyone else interested. If not, probably better have it live w/ TimelineV2. bq. Reformatting javadoc wasn't intended. I'll remove that. In fact, I'll file a separate sub-jira to just add the clone method to the BufferedMutatorParams to separate out that concern. Good. Lets get that in. Patch looks great so far. Is there anything point in AP that could be exposed that might help simplify the implementation at all? Your approach of treating BMI as black box is probably best way to go... justifies all the rigging you have about... rigging that you'd probably end up building anyway because AP signaling would likely be mixed, not-clean. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746879#comment-15746879 ] Joep Rottinghuis commented on HBASE-17018: -- Interesting idea about using attributes on the mutation itself. That wouldn't mess with the way the BufferedMutatorImpl would deal with them? I don't want those attributes to be transmitted to HBase, that would be a waste. I'll read up more, but upon first inspection I should be able to stash flushCount, submit- and completion times in a byte[]. Putting a flushLatch there would be harder. I'd have to think if that can be stashed and communicated in a different way. The other impact is that I'd have to peel apart a list of mutations and set attributes on each. Right now a submission maintains a List of Mutations so that they can be added to BufferedMutatorImpl in batch. bq. This will not be committed to hbase? It'll be part of timeline v2? Is that a question or statement? ;) If the HBase community is interested to have this part of HBase, that would be great and I'll continue the code in place. If not, then I'll move this to yarn. bq. The bulk of the change in BufferedMutatorParams is unrelated. You want to do a patch w/ just the changes to hbase core removing the non-changes: i.e. in BufferedMutatorParams only change should be the clone method addition, not the reformatting of javadoc. Reformatting javadoc wasn't intended. I'll remove that. In fact, I'll file a separate sub-jira to just add the clone method to the BufferedMutatorParams to separate out that concern. bq. Is FilesystemMutationSpooler still TODO? Is it needed? There doesn't seem to be much filesystem-ey about FilesystemMutationSpooler, at least just yet. Indeed this is still completely an empty template. Actual implementation still open. I didn't want to go too far with implementation, just sketch out enough to make the design clear to get feedback on that. I started with the diagram and a description, but as I was thinking through more details the design had to be tweaked. I figured POC code would do best job in describing the design. With respect to public boolean shouldSpool() indeed, the code right now it a bit more verbose than needed. I'll collapse to the simple format if indeed I don't need to keep track of the max successful flushCount. I need to add actual tests before I can get those details added. bq. If SpoolingBufferedMutatorCoordinator Interface a bit over the top? Is there ever going to be another type of cooridinator implementation? Yes indeed and no probably not. I started with this thinking I needed to make it pluggable for testing. But you're right that no interface is needed there, I can simply use inheritance and still control tests. bq. Otherwise skimmed the rest.. Where are the tests? Tests are still missing indeed. Just a design sketch at the moment. If the approach seems sane, I'll add unit tests. Thanks for the feedback [~stack] > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746798#comment-15746798
]
stack commented on HBASE-17018:
---
Nice writeup/design. Looks good. Like the tracking of 'batches'. There seems to
be a bunch of moving parts but all seem necessary.
On "...wrapped in a SpoolingBufferedMutatorSubmission which comes in three
SpoolingBufferedMutatorSubmissionType...", if you were looking for a simpler
approach, you might just add markers to the Mutation Map of Attributes...
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/OperationWithAttributes.html#setAttribute-java.lang.String-byte:A-
On the patch:
This will not be committed to hbase? It'll be part of timeline v2? (Or
looking at the patch, there is some change needed in core ... in
BufferedMutatorParams... these we can take).
The bulk of the change in BufferedMutatorParams is unrelated. You want to do a
patch w/ just the changes to hbase core removing the non-changes: i.e. in
BufferedMutatorParams only change should be the clone method addition, not the
reformatting of javadoc.
Is FilesystemMutationSpooler still TODO? Is it needed? There doesn't seem to be
much filesystem-ey about FilesystemMutationSpooler, at least just yet.
If SpoolingBufferedMutatorCoordinator Interface a bit over the top? Is there
ever going to be another type of cooridinator implementation?
nit: Could change this
145 public boolean shouldSpool(SpoolingBufferedMutatorSubmission
submission) {
146 // TODO: should we keep track of the last successful flush that
went through
147 // and perhaps still drop some items from outbound, even if we just
went
148 // into a bad state?
149 if (state == State.GOOD) {
150 return false;
151 } else {
152 // Spool for both BAD and TRANSITIONING states.
153 return true;
154 }
155 }
to
return state != State.GOOD;
Otherwise skimmed the rest.. Where are the tests?
Looks good [~jrottinghuis]
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730113#comment-15730113 ] Vrushali C commented on HBASE-17018: bq. You are setting timestamp on the Puts. You are not relying on HBase for this? We use our generated timestamps for certain cells. Especially for those cell which the coprocessor needs to look at. The cell timestamps for these cells are our unique timestamp (generated at https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimestampGenerator.java#L65) which is left shifted by 1million and lower digits of the app id added to them (at https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TimestampGenerator.java#L90). This is required to avoid cell over-writes for the case when say two or more applications have to write the same column happen to write at the same instant in time. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730082#comment-15730082 ] stack commented on HBASE-17018: --- Looking at attached PDF (in future, attach google doc with comments permissions for those w/ URL... then we can write comment on the doc itself): You are setting timestamp on the Puts. You are not relying on HBase for this? How long would you allow spooling to go on? What you thinking? Agree focus should be on spooling to distributed storage. If wrote using hbase WAL, could use WALPlayer to replay. Just a suggestion. Might be too heavyweight for what you need. FYI, lots of support in hbase for serializing pojos as protobufs and back again. Section 'Replaying spooled puts' especially #2 is awkward. Needs working through. Connection needs to be able to ride over vagaries such as the coming and going of cluster (Application servers can't be expected to check/reset Connections): i.e. most scenarios in this doc are general hbase connection issues that we should have test coverage for and deal with; if missing from general client, its a bug. Client is not currently interruptible which is wrong.I like where you finger ClusterStatusListener as a font for connection state signal. TODO. See subtask on this issue for cut at making it so can supply own BM. AP is convoluted. Better would be an Interface on AP that exposes API you'd need to run your spooling BM. Its YAI (yet-another-interface) but AP is inscrutable and subject to change; not to be depended on. Thanks for posting the doc. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638727#comment-15638727 ] Joep Rottinghuis commented on HBASE-17018: -- Thanks for the comments. My thought around using MR were because of easy of implementation and stemmed from my use case where Yarn is present and therefore MR trivially available. It is a fair point that as a standalone feature in HBase this doesn't have to be true. Using MR isn't a requirement, but was merely a (naive) suggestion. I don't think that atomicity is a requirement, nor are we asking for "guarantees". If you want to be guaranteed to write something to HBase you probably shouldn't use a BufferedMutator in the first place. Please see attached PDF where I try to sketch out our use case and what behavior we're hoping to see. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637715#comment-15637715 ] Mikhail Antonov commented on HBASE-17018: - At a high level idea of having BufferedMutator or similar client API manage separate persistent storage with atomicity / replay guarantees sounds somewhat weird to me. Is that the problem to be solved outside of HBase? Or should it be bulk ingest or some sort as mentioned above? > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635287#comment-15635287 ] Anoop Sam John commented on HBASE-17018: If an MR way is what we take, then better it can be an ImportTSV kind of job which splits data into HFiles and do bulk load? > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635154#comment-15635154 ] Phil Yang commented on HBASE-17018: --- An interesting and useful feature :) Cassandra has a feature/api called "atomic batches". If the client get response from server, we can say all mutations have be executed. If the request timeout, server can guarantee "all or nothing" -- all will be executed eventually or nothing will be executed. I think we can provide a similar feature to HBase users as a Table's API. And BufferedMutator can have an optional logic to switch to this API when the initial request failed. For implementation, I think we'd better not use MR because users may not have MR for a HBase cluster, or at least RS should be able to replay mutations. If we save mutations to HDFS, the client will be like a RS that writes WAL to HDFS, right? We can use a logic just like distribute log replay to read log entries and use client api to write them to all region servers. Thanks. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634964#comment-15634964 ] Joep Rottinghuis commented on HBASE-17018: -- [~sduskis] I've made you a watcher of this jira, since we discussed this at the last HBase meetup in NYC. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: YARN-4061 HBase requirements for fault tolerant > writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
