-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21752/
-----------------------------------------------------------
Review request for accumulo.
Bugs: ACCUMULO-378
https://issues.apache.org/jira/browse/ACCUMULO-378
Repository: accumulo
Description
-------
Initial implementation that covers (high-level) replication of "live" data
using WAls (no support for bulk-imports yet), master->slave and master->master
configurations, replication implementations to another Accumulo instance, and
multiple replication "assignment" implementations (in what order do we
replicate WALs, replicating WALs concurrently, etc). Normal code path (no
tables configured for replication) should not have been altered. Some effort
was put into code cleanup (compiler warnings, unused imports, formatting
issues), but I'd ask that reviewers focus on the substance rather than the
nitty-gritty at the moment. I think this is at a fairly stable point right now,
so I wanted to make it available to those who plan to look at it.
"Big" things that aren't yet addressed: insight via monitor, bulk-import
replication, authentication/authorization to the remote Accumulo cluster and
encryption of data to remote Accumulo cluster. All of these (maybe not
bulk-import replication) should be done before this gets merged in upstream.
The ITs are a great place to start looking to see how things work.
Diffs
-----
core/pom.xml 2c3f648
core/src/main/java/org/apache/accumulo/core/Constants.java 7d602bb
core/src/main/java/org/apache/accumulo/core/client/Connector.java 4a2acff
core/src/main/java/org/apache/accumulo/core/client/admin/ReplicationOperations.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java
0df35f6
core/src/main/java/org/apache/accumulo/core/client/impl/MasterClient.java
1fa8f12
core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationClient.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/mock/MockAccumulo.java
2c26ecc
core/src/main/java/org/apache/accumulo/core/client/mock/MockConnector.java
996198c
core/src/main/java/org/apache/accumulo/core/client/mock/MockTable.java
cb50761
core/src/main/java/org/apache/accumulo/core/client/mock/MockTableOperations.java
a44a027
core/src/main/java/org/apache/accumulo/core/client/replication/PeerExistsException.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/replication/PeerNotFoundException.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/replication/ReplicaSystem.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/replication/ReplicaSystemFactory.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/replication/ReplicationTable.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33
core/src/main/java/org/apache/accumulo/core/data/ConditionalMutation.java
e43968c
core/src/main/java/org/apache/accumulo/core/data/Mutation.java 42fa143
core/src/main/java/org/apache/accumulo/core/data/thrift/MultiScanResult.java
f0b8a87
core/src/main/java/org/apache/accumulo/core/data/thrift/ScanResult.java
6472587
core/src/main/java/org/apache/accumulo/core/data/thrift/TConditionalMutation.java
3f9b3f7
core/src/main/java/org/apache/accumulo/core/data/thrift/TMutation.java
f698892
core/src/main/java/org/apache/accumulo/core/data/thrift/UpdateErrors.java
59ce5cb
core/src/main/java/org/apache/accumulo/core/iterators/Combiner.java ceb4411
core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java
06eae23
core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/AccumuloReplicationReplayer.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/PrintReplicationRecords.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/RemoteReplicationErrorCode.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/ReplicationConfigurationUtil.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/ReplicationCoordinatorErrorCode.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/ReplicationSchema.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/ReplicationTarget.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/StatusUtil.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/proto/Replication.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/KeyValues.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/RemoteCoordinationException.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/RemoteReplicationException.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/Replication.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/ReplicationCoordinator.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/ReplicationServicer.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/replication/thrift/WalEdits.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/schema/Section.java PRE-CREATION
core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ReplicationFailedException.java
PRE-CREATION
core/src/main/protobuf/replication.proto PRE-CREATION
core/src/main/scripts/generate-protobuf.sh PRE-CREATION
core/src/main/scripts/generate-thrift.sh 5a5d69f
core/src/main/thrift/data.thrift ae6f439
core/src/main/thrift/replication.thrift PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/ReplicationConfigurationUtilTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/ReplicationOperationsImplTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/ReplicationSchemaTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/ReplicationTargetTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/StatusUtilTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/replication/proto/StatusTest.java
PRE-CREATION
docs/src/main/resources/design/ACCUMULO-378-design.mdtext PRE-CREATION
docs/src/main/resources/state/replicationstatus.gv PRE-CREATION
docs/src/main/resources/state/replicationstatus.png PRE-CREATION
minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java
3258991
server/base/pom.xml f698621
server/base/src/main/java/org/apache/accumulo/server/conf/TableConfigurationCopy.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/constraints/MetadataConstraints.java
ecb0db8
server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
436667c
server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java
10cdce6
server/base/src/main/java/org/apache/accumulo/server/replication/AbstractWorkAssigner.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/replication/ReplicationTable.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/replication/StatusCombiner.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/replication/WorkAssigner.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java
e936b97
server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
341f39a
server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/util/SystemPropUtil.java
1d93f90
server/base/src/main/java/org/apache/accumulo/server/zookeeper/DistributedWorkQueue.java
4738c2b
server/base/src/test/java/org/apache/accumulo/server/master/balancer/TableLoadBalancerTest.java
82e5885
server/base/src/test/java/org/apache/accumulo/server/replication/ReplicationTableTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/replication/StatusCombinerTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
PRE-CREATION
server/gc/pom.xml 5c501de
server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java
ae850af
server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionAlgorithm.java
40fb847
server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionEnvironment.java
3e36617
server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java
7ebfff0
server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
PRE-CREATION
server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
f90b965
server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectionTest.java
537fc46
server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
PRE-CREATION
server/gc/src/test/resources/log4j.properties e74337f
server/master/pom.xml fd53a8c
server/master/src/main/java/org/apache/accumulo/master/Master.java 526ca8a
server/master/src/main/java/org/apache/accumulo/master/replication/DistributedWorkQueueWorkAssigner.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/FinishedWorkUpdater.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/MasterReplicationCoordinator.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/RemoveCompleteReplicationRecords.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/ReplicationDriver.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/SequentialWorkAssigner.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/StatusMaker.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/WorkDriver.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/replication/WorkMaker.java
PRE-CREATION
server/master/src/main/java/org/apache/accumulo/master/tableOps/CreateTable.java
5b64053
server/master/src/test/java/org/apache/accumulo/master/replication/AbstractWorkAssignerTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/DistributedWorkQueueWorkAssignerTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/FinishedWorkUpdaterTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/MasterReplicationCoordinatorTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/RemoveCompleteReplicationRecordsTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/SequentialWorkAssignerTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/StatusMakerTest.java
PRE-CREATION
server/master/src/test/java/org/apache/accumulo/master/replication/WorkMakerTest.java
PRE-CREATION
server/master/src/test/resources/log4j.properties e74337f
server/tserver/pom.xml 100bc2d
server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java fadf4ed
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
144d59b
server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java
871f4ae
server/tserver/src/main/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystem.java
PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/replication/BatchWriterReplicationReplayer.java
PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationProcessor.java
PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationServicerHandler.java
PRE-CREATION
server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationWorker.java
PRE-CREATION
server/tserver/src/test/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystemTest.java
PRE-CREATION
server/tserver/src/test/java/org/apache/accumulo/tserver/replication/ReplicationProcessorTest.java
PRE-CREATION
server/tserver/src/test/resources/log4j.properties e74337f
test/src/main/java/org/apache/accumulo/test/replication/MockReplicaSystem.java
PRE-CREATION
test/src/main/java/org/apache/accumulo/test/replication/ReplicationTablesPrinterThread.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/CyclicReplicationIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationDeadlockTest.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationFilesClosedAfterUnusedTest.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationPortAdvertisementIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationSequentialIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationSourceOnlyIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationTablesMacTest.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationTest.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationWithGCIT.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/ReplicationWithMakerTest.java
PRE-CREATION
test/src/test/java/org/apache/accumulo/test/replication/StatusCombinerMacTest.java
PRE-CREATION
test/src/test/resources/log4j.properties 3fa1983
Diff: https://reviews.apache.org/r/21752/diff/
Testing
-------
All existing unit and integration tests pass. Expected as normal code path
hasn't changed.
Loads of new tests included, many ITs which ensure that replication
"bookkeeping" is cleaned up when appropriate, all data is replicated to a peer
(and not over-replicated or re-replicated).
Thanks,
Josh Elser
