XNX02 commented on code in PR #15275:
URL: https://github.com/apache/iotdb/pull/15275#discussion_r2044225386
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/pipe/runtime/PipeMetaSyncProcedure.java:
##########
@@ -100,8 +108,49 @@ public boolean
executeFromValidateTask(ConfigNodeProcedureEnv env) {
@Override
public void executeFromCalculateInfoForTask(ConfigNodeProcedureEnv env) {
LOGGER.info("PipeMetaSyncProcedure: executeFromCalculateInfoForTask");
-
- // Do nothing
+ // Re-balance the external source tasks here in case of any changes in the
dataRegion
+ pipeTaskInfo
+ .get()
+ .getPipeMetaList()
+ .forEach(
+ pipeMeta -> {
+ if (pipeMeta.getStaticMeta().isSourceExternal()) {
+ final ExternalLoadBalancer loadBalancer =
+ new ExternalLoadBalancer(
+ pipeMeta
+ .getStaticMeta()
+ .getExtractorParameters()
+ .getStringOrDefault(
+ Arrays.asList(
+
PipeExtractorConstant.EXTERNAL_EXTRACTOR_BALANCE_STRATEGY_KEY,
+
PipeExtractorConstant.EXTERNAL_SOURCE_BALANCE_STRATEGY_KEY),
+ PipeExtractorConstant
+
.EXTERNAL_EXTRACTOR_BALANCE_PROPORTION_STRATEGY));
+
+ final int parallelism =
+ pipeMeta
+ .getStaticMeta()
+ .getExtractorParameters()
+ .getIntOrDefault(
+ Arrays.asList(
+ EXTERNAL_EXTRACTOR_PARALLELISM_KEY,
+ EXTERNAL_SOURCE_PARALLELISM_KEY),
+ EXTERNAL_EXTRACTOR_PARALLELISM_DEFAULT_VALUE);
+ loadBalancer
+ .balance(
+ parallelism,
+ pipeMeta.getStaticMeta(),
+ ConfigNode.getInstance().getConfigManager())
+ .forEach(
+ (taskIndex, newLeader) -> {
+ pipeMeta
+ .getRuntimeMeta()
+ .getConsensusGroupId2TaskMetaMap()
+ .get(taskIndex)
Review Comment:
I'm not sure weather this would happen since the parallelism of each pipe is
not allowed to change for now. Added the logic in case something happens..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
