[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/20151 ) Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. Patch Set 16: Code-Review-2 Make it not push to remote repo. -- To view, visit http://gerrit.cloudera.org:8080/20151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ec587821f41eef75078152d7912c11456906ecf Gerrit-Change-Number: 20151 Gerrit-PatchSet: 16 Gerrit-Owner: Yuqi Du Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Mon, 31 Jul 2023 07:49:17 + Gerrit-HasComments: No
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/20151 ) Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. Patch Set 16: > Patch Set 14: > > I'm not sure it make sense to pile like patches like this if you want them to > be reviewed in eventually pushed into the repo. > > It's hard to review and revv these. > > Please prefer to maintain a stack of patches, each being logically separated > and doing a well-defined part of the overall task. > > Also, have you addressed the feedback on the design document for this feature? > > Thanks! This patch is for passed all tests and then I'll split this patch into several individual patches. The document: https://docs.google.com/document/d/1ihqPFO1vulpYDYYKcHmhes0LCOXKQtdAf5ub0y2LiaM/edit -- To view, visit http://gerrit.cloudera.org:8080/20151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ec587821f41eef75078152d7912c11456906ecf Gerrit-Change-Number: 20151 Gerrit-PatchSet: 16 Gerrit-Owner: Yuqi Du Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Fri, 28 Jul 2023 09:19:22 + Gerrit-HasComments: No
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#16). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#15). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20151 ) Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. Patch Set 14: I'm not sure it make sense to pile like patches like this if you want them to be reviewed in eventually pushed into the repo. It's hard to review and revv these. Please prefer to maintain a stack of patches, each being logically separated and doing a well-defined part of the overall task. Also, have you addressed the feedback on the design document for this feature? Thanks! -- To view, visit http://gerrit.cloudera.org:8080/20151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ec587821f41eef75078152d7912c11456906ecf Gerrit-Change-Number: 20151 Gerrit-PatchSet: 14 Gerrit-Owner: Yuqi Du Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Thu, 20 Jul 2023 04:22:33 + Gerrit-HasComments: No
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#14). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#13). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#12). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#11). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#10). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#9). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#8). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Yuqi Du has posted comments on this change. ( http://gerrit.cloudera.org:8080/20151 ) Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/20151/4/src/kudu/rebalance/cluster_status.h File src/kudu/rebalance/cluster_status.h: http://gerrit.cloudera.org:8080/#/c/20151/4/src/kudu/rebalance/cluster_status.h@76 PS4, Line 76: std::optional opid_index, > warning: pass by value and use std::move [modernize-pass-by-value] Done http://gerrit.cloudera.org:8080/#/c/20151/4/src/kudu/rebalance/cluster_status.h@83 PS4, Line 83: opid_index(std::move(opid_index)), > warning: parameter 'opid_index' is passed by value and only copied once; co Done http://gerrit.cloudera.org:8080/#/c/20151/4/src/kudu/tablet/tablet_bootstrap.cc File src/kudu/tablet/tablet_bootstrap.cc: http://gerrit.cloudera.org:8080/#/c/20151/4/src/kudu/tablet/tablet_bootstrap.cc@1404 PS4, Line 1404: *all_skipped &= action != NEEDS_REPLAY; > warning: The left operand of '!=' is a garbage value [clang-analyzer-core.U Done -- To view, visit http://gerrit.cloudera.org:8080/20151 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ec587821f41eef75078152d7912c11456906ecf Gerrit-Change-Number: 20151 Gerrit-PatchSet: 7 Gerrit-Owner: Yuqi Du Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Thu, 06 Jul 2023 03:59:40 + Gerrit-HasComments: Yes
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#7). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#6). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Tidy Bot, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#5). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake M java/config/spotbugs/excludeFilter.xml A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#4). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_bootstrap.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#3). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_bootstrap.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20151 to look at the new patch set (#2). Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_bootstrap.cc M
[kudu-CR] WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches
Yuqi Du has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20151 Change subject: WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches .. WIP [duplication] KUDU-3290 Unite all patches as one to prepare these patches This patch is for duplication, it includes all about duplication. The purpose of this patch is do some preparing works for duplication. Some refactors need to do such as tests. This patch is a little too many codes, so I plan to split this patch into 10 patches. They are: 1. Adding 'Duplicator' which will write records to kafka, adding extra 2 thirdparty libraries: cppkafka, librdkafka. librdkfka supports c language apis to operate kafka, and cppkafka is a C++ wrapper for librdkakfa, its an api sugger. 2. Add a new raft role DUPLICATOR and raft processing 3. recovering duplication when a leader switches 4. creating/altering in server-side for duplication 5. cpp client apis and CLI tool 6. java client apis 7. suppports kerberos 8. alter a duplication 9. registe kafka uri and validate and unregiste During this progressing, plans maybe adjust according to advices. Change-Id: I2ec587821f41eef75078152d7912c11456906ecf --- M CMakeLists.txt A cmake_modules/FindCppKafka.cmake A cmake_modules/FindRdKafka.cmake A java/kudu-client/src/main/java/org/apache/kudu/DuplicationDownstreamType.java M java/kudu-client/src/main/java/org/apache/kudu/client/AlterTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/CreateTableOptions.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTablesResponse.java M java/kudu-client/src/main/java/org/apache/kudu/client/ListTabletsResponse.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-proto/build.gradle M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/replica_controller-internal.cc M src/kudu/client/replica_controller-internal.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.h M src/kudu/common/common.proto M src/kudu/common/types-test.cc M src/kudu/common/types.cc M src/kudu/common/types.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_meta.cc M src/kudu/consensus/consensus_meta.h M src/kudu/consensus/consensus_meta_manager.cc M src/kudu/consensus/consensus_meta_manager.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h A src/kudu/duplicator/CMakeLists.txt A src/kudu/duplicator/connector.cc A src/kudu/duplicator/connector.h A src/kudu/duplicator/connector_manager.h A src/kudu/duplicator/duplication_replay-test.cc A src/kudu/duplicator/duplication_replay.cc A src/kudu/duplicator/duplication_replay.h A src/kudu/duplicator/duplicator.cc A src/kudu/duplicator/duplicator.h A src/kudu/duplicator/kafka/kafka.proto A src/kudu/duplicator/kafka/kafka_connector-test.cc A src/kudu/duplicator/kafka/kafka_connector.cc A src/kudu/duplicator/kafka/kafka_connector.h A src/kudu/duplicator/log_segment_reader.cc A src/kudu/duplicator/log_segment_reader.h M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/duplication_with_fuzzy_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kafka-itest.cc A src/kudu/integration-tests/duplication_with_kerberos_kafka-itest.cc M src/kudu/integration-tests/fuzz-itest.cc A src/kudu/integration-tests/single_broker_kafka.h M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto M src/kudu/master/master_service.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/ts_manager.h M src/kudu/rebalance/cluster_status.h A src/kudu/scripts/kafka-simple-control.sh M src/kudu/subprocess/subprocess_server-test.cc M src/kudu/tablet/CMakeLists.txt M src/kudu/tablet/diskrowset-test.cc A src/kudu/tablet/ops/duplication_op.cc A src/kudu/tablet/ops/duplication_op.h M src/kudu/tablet/ops/op.h M src/kudu/tablet/ops/op_driver.cc M src/kudu/tablet/ops/op_tracker-test.cc M src/kudu/tablet/ops/op_tracker.cc M src/kudu/tablet/ops/write_op.cc M src/kudu/tablet/ops/write_op.h M src/kudu/tablet/tablet-harness.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M
