Caideyipi commented on code in PR #15275:
URL: https://github.com/apache/iotdb/pull/15275#discussion_r2044343928
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/pipe/runtime/PipeMetaSyncProcedure.java:
##########
@@ -100,8 +108,49 @@ public boolean
executeFromValidateTask(ConfigNodeProcedureEnv env) {
@Override
public void executeFromCalculateInfoForTask(ConfigNodeProcedureEnv env) {
LOGGER.info("PipeMetaSyncProcedure: executeFromCalculateInfoForTask");
-
- // Do nothing
+ // Re-balance the external source tasks here in case of any changes in the
dataRegion
+ pipeTaskInfo
+ .get()
+ .getPipeMetaList()
+ .forEach(
+ pipeMeta -> {
+ if (pipeMeta.getStaticMeta().isSourceExternal()) {
+ final ExternalLoadBalancer loadBalancer =
+ new ExternalLoadBalancer(
+ pipeMeta
+ .getStaticMeta()
+ .getExtractorParameters()
+ .getStringOrDefault(
+ Arrays.asList(
+
PipeExtractorConstant.EXTERNAL_EXTRACTOR_BALANCE_STRATEGY_KEY,
+
PipeExtractorConstant.EXTERNAL_SOURCE_BALANCE_STRATEGY_KEY),
+ PipeExtractorConstant
+
.EXTERNAL_EXTRACTOR_BALANCE_PROPORTION_STRATEGY));
+
+ final int parallelism =
+ pipeMeta
+ .getStaticMeta()
+ .getExtractorParameters()
+ .getIntOrDefault(
+ Arrays.asList(
+ EXTERNAL_EXTRACTOR_PARALLELISM_KEY,
+ EXTERNAL_SOURCE_PARALLELISM_KEY),
+ EXTERNAL_EXTRACTOR_PARALLELISM_DEFAULT_VALUE);
+ loadBalancer
+ .balance(
+ parallelism,
+ pipeMeta.getStaticMeta(),
+ ConfigNode.getInstance().getConfigManager())
+ .forEach(
+ (taskIndex, newLeader) -> {
+ pipeMeta
+ .getRuntimeMeta()
+ .getConsensusGroupId2TaskMetaMap()
+ .get(taskIndex)
Review Comment:
Well, the nodes may be added, and if the extractor is a singleton per node,
and the node count < parallelism previously.....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
