Murtadha Hubail has submitted this change and it was merged. Change subject: Asterix NCs Failback Support ......................................................................
Asterix NCs Failback Support - Allow Failed NCs to failback and takeover their partitions. - New cluster API servlet for cluster state description. - Remove nodeId from txn logs except remote FLUSH_LOG. - Add partition id in UPDATE and Entity_COMMIT logs. - Adapt remote recovery to new logs format. - Refactor RecoveryManager and split Analysis and Redo phase. - Spill remote recover logs to temporary file. - Replicate files to remote replicas for partitions of interest only. - Introduce NC active/inactive partitions concept. - Test case for failback. Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24 Reviewed-on: https://asterix-gerrit.ics.uci.edu/613 Tested-by: Jenkins <[email protected]> Reviewed-by: abdullah alamoudi <[email protected]> --- M .gitignore M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntime.java M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitRuntimeFactory.java M asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/UpsertCommitRuntime.java M asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ReplaceSinkOpWithCommitOpRule.java M asterix-app/src/main/java/org/apache/asterix/api/common/AsterixAppRuntimeContext.java A asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ClusterAPIServlet.java M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/ClusterLifecycleListener.java M asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplicationEntryPoint.java M asterix-app/src/main/java/org/apache/asterix/messaging/CCMessageBroker.java M asterix-app/src/main/java/org/apache/asterix/messaging/NCMessageBroker.java M asterix-common/src/main/java/org/apache/asterix/common/api/IAsterixAppRuntimeContext.java M asterix-common/src/main/java/org/apache/asterix/common/api/IClusterManagementWork.java M asterix-common/src/main/java/org/apache/asterix/common/api/IDatasetLifecycleManager.java M asterix-common/src/main/java/org/apache/asterix/common/config/AsterixReplicationProperties.java M asterix-common/src/main/java/org/apache/asterix/common/context/DatasetLifecycleManager.java M asterix-common/src/main/java/org/apache/asterix/common/dataflow/AsterixLSMInsertDeleteOperatorNodePushable.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/AbstractFailbackPlanMessage.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackRequestMessage.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/CompleteFailbackResponseMessage.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackRequestMessage.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/PreparePartitionsFailbackResponseMessage.java A asterix-common/src/main/java/org/apache/asterix/common/messaging/ReplicaEventMessage.java M asterix-common/src/main/java/org/apache/asterix/common/messaging/TakeoverPartitionsRequestMessage.java M asterix-common/src/main/java/org/apache/asterix/common/messaging/api/IApplicationMessage.java M asterix-common/src/main/java/org/apache/asterix/common/replication/IRemoteRecoveryManager.java M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java M asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicationManager.java A asterix-common/src/main/java/org/apache/asterix/common/replication/NodeFailbackPlan.java M asterix-common/src/main/java/org/apache/asterix/common/replication/ReplicaEvent.java M asterix-common/src/main/java/org/apache/asterix/common/transactions/ILogRecord.java M asterix-common/src/main/java/org/apache/asterix/common/transactions/IRecoveryManager.java M asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java A asterix-common/src/main/java/org/apache/asterix/common/utils/ServletUtil.java M asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java M asterix-common/src/main/java/org/apache/asterix/common/utils/TransactionUtil.java M asterix-common/src/test/java/org/apache/asterix/test/aql/TestExecutor.java M asterix-installer/src/test/java/org/apache/asterix/installer/test/ReplicationIT.java A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.1.ddl.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.10.cstate.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.11.query.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.2.update.aql C asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.3.vscript.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.4.sleep.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.5.cstate.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.6.query.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.7.update.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.8.vmgx.aql A asterix-installer/src/test/resources/integrationts/replication/queries/failback/node_failback/node_failback.9.sleep.aql M asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.2.update.aql R asterix-installer/src/test/resources/integrationts/replication/queries/failover/bulkload/bulkload.4.vscript.aql M asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.2.update.aql R asterix-installer/src/test/resources/integrationts/replication/queries/failover/mem_component_recovery/mem_component_recovery.4.vscript.aql R asterix-installer/src/test/resources/integrationts/replication/queries/failover/metadata_node/metadata_node.3.vscript.aql A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.10.adm A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.cluster_state.5.adm A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.11.adm A asterix-installer/src/test/resources/integrationts/replication/results/failback/node_failback/node_failback.query.6.adm M asterix-installer/src/test/resources/integrationts/replication/testsuite.xml M asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java M asterix-om/src/main/java/org/apache/asterix/om/util/AsterixClusterProperties.java M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicaFilesRequest.java M asterix-replication/src/main/java/org/apache/asterix/replication/functions/ReplicationProtocol.java D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicaEventNotifier.java M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java D asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationLifecycleListener.java M asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java M asterix-replication/src/main/java/org/apache/asterix/replication/recovery/RemoteRecoveryManager.java M asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java M asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java M asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AsterixLSMPrimaryUpsertOperatorNodePushable.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/AbstractIndexModificationOperationCallback.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallback.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/PrimaryIndexModificationOperationCallbackFactory.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallback.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/SecondaryIndexModificationOperationCallbackFactory.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetIndexModificationOperationCallback.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetPrimaryIndexModificationOperationCallbackFactory.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/TempDatasetSecondaryIndexModificationOperationCallbackFactory.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallbackFactory.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogManagerWithReplication.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/LogReader.java A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/logging/RemoteLogReader.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/RecoveryManager.java A asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/TxnId.java M asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/transaction/TransactionContext.java 90 files changed, 3,401 insertions(+), 2,229 deletions(-) Approvals: abdullah alamoudi: Looks good to me, approved Jenkins: Verified -- To view, visit https://asterix-gerrit.ics.uci.edu/613 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id17819542d6b9c4e32647e64737c4a467b630f24 Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Ian Maxon <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Till Westmann <[email protected]> Gerrit-Reviewer: abdullah alamoudi <[email protected]>
