Dylan Hutchison wrote:
Hi Josh, responses follow:
Using a BatchWriter implies that you'll need some sort of resource
management - both ensuring that the BatchWriter is close()'ed whenever a
compaction/procedure ends and handling rejected mutations. Have you put any
thought into how you would address these?
Yes, some thoughts:
- Accumulo guarantees that iterators at major compaction time are never
killed until tablet compaction completes, barring tablet server failure or
exception.
- We can close the BatchWriter in the finalize() method of the
RemoteWriteIterator. This is already done in my code for the
RemoteSourceIterator, though I don't think it matters for a Scanner. Not
ideal but maybe good enough.
Remember that finalize() is not guaranteed to be called by the JVM. You
are correct that you don't need to close() a Scanner (it's just good
practice).
- Actually, if we do all computation for a tablet from the seek() method
of the branching iterator, then we can create, use and close the
BatchWriter in a single method invocation per tablet. This way, we may
enclose all BatchWriter use in a try-finally loop, closing it in the
finally.
- This thread is relevant
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%[email protected]%3E>.
Note that our scenario is different in that we're operating from a major
compaction.
- ACCUMULO-1280<https://issues.apache.org/jira/browse/ACCUMULO-1280>
proposes changing SKVI API to add a close() method, such that iterators
know when they are about to be closed and why they are about to be closed.
The iterators may then clean themselves up.
One solution I did not include this at first because its complexity may not
be necessary: *checkpoint*-and-restart-style. When an iterator dies,
Accumulo will recreate it, re-init() it and then seek() it to an
appropriate range. This range should start *right after the last entry
returned* by the iterator. If we structure the computation in such a way
that it can restart in the middle, then we can employ the following
strategy:
- Suppose the initial computation range from seek() is [a,f].
- The stored procedure table runs as normal. Eventually its
RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
RemoteSourceIterator passes c back to the BranchIterator which returns from
its seek() and prepares a top entry with row c and a top value of ""
(hasTop() is true). The computation continues when next() is called on the
BranchIterator
- Suppose Accumulo kills the iterator instead of calling next().
Accumulo re-init()s the BranchIterator which sets up the custom computation
stack again. Accumulo then seek()s with range (c,f].
- The custom computation stack may now continue with the corresponding
entries for range (c,f].
Notice that all we need is a correspondence between seek ranges and entries
computed. I still believe the above scheme is a bit too much complexity.
Suppose in the worst case we need to restart an entire tablet's worth of
computation. Okay, we have some redundant computation but *preserve
correctness*, because the result table's VersioningIterator will handle
multiple (identical) entries, in most cases.
using a wrench as a hammer -- iterators are great for performing some
passing computation, but not really for doing some arbitrary read/writes.
It gets back to how Accumulo/HBase comparisons where people try to compare
Iterators and Coprocessors. They can sometimes do the same thing, but
they're definitely different features.
Yes-- we are talking about different kinds of computation. That's why I
distinguish between the types of iterators in the design doc.
Traditional iterators are wrenches-- passing computation done at scan or
compaction time, not too heavy, done in a sorted, streaming manner, limited
to within a single table. They come in three flavors: one-time scan
iterators, one-time compaction iterators, and "permanent table property"
iterators set on the table at scan or compaction scope. Do correct me if I
oversimplify or if I missed a large use case.
The iterators I propose in the design doc are hammers-- heavy, one-time,
multi-table computation, that may or may not return results in sorted
order. That's why I use the phrase "stored procedure."
On *HBase Coprocessors*-- I looked up what they are here
<https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
seems our "stored procedure" idea bears many similarities to HBase
Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
the Accumulo Fluo subproject). Maybe we can take some architecture lessons
from HBase. Anyone have experience they would like to share?
IIRC, HBase coprocessors aren't 100% as described in the Percolator
paper. I think Fluo tried to model more closely what was described, but
I don't remember anymore. Keith and Mike would know much better (since
they're building it :D).
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser<[email protected]> wrote:
Thanks for taking the time to write this up, Dylan.
I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
implies that you'll need some sort of resource management - both ensuring
that the BatchWriter is close()'ed whenever a compaction/procedure ends and
handling rejected mutations. Have you put any thought into how you would
address these?
I'm not familiar enough with the internals anymore, but I remember that I
had some pains trying to write to another table during compactions when I
was working on replication. I think as long as it's not triggered off of
the metadata table, it wouldn't have any deadlock issues.
Architecturally, it's a little worrisome, because it feels a bit like
using a wrench as a hammer -- iterators are great for performing some
passing computation, but not really for doing some arbitrary read/writes.
It gets back to how Accumulo/HBase comparisons where people try to compare
Iterators and Coprocessors. They can sometimes do the same thing, but
they're definitely different features.
Anyways, I need to stew on it some more and give it a few more reads.
Thanks again for sharing!
Dylan Hutchison wrote:
Hello all,
As promised
<https://mail-archives.apache.org/mod_mbox/accumulo-user/
201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
here is a design doc open for comments on implementing server-side
computation in Accumulo.
https://github.com/Accla/accumulo_stored_procedure_design
Would love to hear your opinion, especially if the proposed design
pattern matches one of /your use cases/.
Regards,
Dylan Hutchison
