On 6/27/14, 12:32 PM, Joe Stein wrote:
Hey Josh, the user manual is really great and helpful as a starting point!!!
Great, glad to hear. Please feel free to let me know areas that are
unclear or need more substance.
Have you thought of / considered what an active/active solution might look
like?
A little bit. There are two concerns here that will help you decide if
active-active replication will work out. The first is the eventual
consistency. If both "sides" are accepting writes, you'll get into a
situation where you might get different results from the instances.
The second is updates to both systems for the same key at the same time.
The timestamp will be preserved across two systems and should provide
decent ordering, but, if you have two updates at the same time, it's
going to appear as non-deterministic to you as the user. This isn't too
bad if you can pin updates to certain entities to a single instance.
There are other ways you could address it.
Combiners configured on replicating tables can also be problematic if
they are not idempotent:
http://people.apache.org/~elserj/accumulo_user_manual.html#_table_configured_iterators
One thought I had (perhaps naive don't know though) is having a table for
each data center. and then some iterators to materialize that on query and
compacted perhaps somewhere/how.
This way each data center can operate active / active and folks can get
queries from both data center tables (from a data perspective so DC 1
replicates the DC1 table to DC2 and DC2 replicates the DC2 table to DC1).
Having certain users query certain DCs certainly makes sense. I'm not
sure I understand your comment about materializing queries across DCs(?)
though.
As far as testing goes with resources for more testing can you be more
specific? Are you talking about 3 servers, 30 servers, 300 servers?
I've done most of my testing on the <5 servers scale with minimal
injected fault (kill -9'ing things). I've been considering taking a
weekend and some EC2 $$ to test on the 10-100 node scale. I would
ecstatic to do greater than 100 node testing.
I have at least two tests that could be run, both of which would also be
good candidates for fault injection.
<employer-hat>
<client-feather>
We are currently using HDP 2.1 , fwiw
http://hortonworks.com/customer/bloomberg/ so that is great news, what is
the scheduled release date?
</client-feather>
<employer-hat>
I'll reply to you off-list with some more info.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Thu, Jun 26, 2014 at 10:48 PM, Josh Elser <[email protected]> wrote:
Hi Joe,
I'm the guy to ask if you'd like more information about the replication
feature. You already found the parent ticket, so that has a bunch of
technical "what's been done".
At a high level, replication was implemented as a framework in Accumulo to
copy data that was written to a table to another "location". The provided
initial implementation is to replicate the data as-is to another Accumulo
table (usually some other Accumulo instance). You'll also find a new page
in the monitor some basic administration tools in the code via Instance#
replicationOperations.
I've published a recent version of the user manual[1] which goes into some
more detail on the feature, as well as how to configure it.
You can also check the replication component on JIRA [2] to see what I
have lined up. Automatically replicating bulk-loaded files will be a bit of
work. There are some other minor things that could be improved. We can
delve into the more technical implementation difficulties if you'd like.
I've written a basic test to evaluate equivalence by generating a Merkle
tree for two tables. This has been promising so far, but it currently is
living in my Github[3]. I need to figure out where/how best to include it
in Apache.
Finally, having resources to do a larger-scale test would be great, and
testing failure conditions over multiple nodes is probably the biggest area
that needs to be tested more. I can simulate this on a small scale, but I
don't have the resources to do an appropriate larger test with injected
failure.
If you have something specific you'd like to help out with, I'd be happy
to work with you.
<employer-hat>This feature will also be included in the next version of
Accumulo shipped in HDP</employer-hat>
- Josh
[1] http://people.apache.org/~elserj/accumulo_user_manual.
html#_replication
[2] https://issues.apache.org/jira/issues/?jql=project%20%
3D%20ACCUMULO%20AND%20component%20%3D%20replication%20AND%
20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20key%
20DESC
[3] https://github.com/joshelser/merkle
On 6/26/14, 8:50 PM, Joe Stein wrote:
Hi, I was hoping to get some more info around the 1.7 release and what are
the to-be-dos and plans around it?
Is there any help that is needed from a contribution perspective in
anyways? Testing? Documentation? Pending coding or such?
We are going to be rolling trunk into two of our lab environments
specifically for https://issues.apache.org/jira/browse/ACCUMULO-378 as it
is a requirement for one of my projects at Bloomberg for Accumulo to have
data center replication before we go live. This works is going to be
over
the next month(s) with lots of cycles dedicated to Accumulo 1.7 in the
next
few sprints.
Also, I wanted to reach out if folks are looking for full time, contract
or
even side work with Accumulo. We have projects right now going on and are
looking for more hands on keyboards.
Anyways, thanks for all the great work!!!! I am looking forward to more
continued success with the system, more integrations and to be able to
become more active in the community.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
