Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
sonarcloud[bot] commented on PR #12331: URL: https://github.com/apache/iotdb/pull/12331#issuecomment-2054312410 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12331) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12331=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12331) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
lancelly commented on code in PR #12331:
URL: https://github.com/apache/iotdb/pull/12331#discussion_r1565071729
##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlannerTest.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.planner;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.MeasurementPath;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.PlanFragmentId;
+import org.apache.iotdb.db.queryengine.common.QueryId;
+import org.apache.iotdb.db.queryengine.execution.driver.DriverContext;
+import
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext;
+import
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceStateMachine;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import
org.apache.iotdb.db.queryengine.execution.operator.source.SeriesScanOperator;
+import org.apache.iotdb.db.queryengine.plan.analyze.TypeProvider;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.DeviceViewNode;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.join.FullOuterTimeJoinNode;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.parameter.SeriesScanOptions;
+import org.apache.iotdb.db.queryengine.plan.statement.component.Ordering;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+
+import com.google.common.collect.Sets;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+
+import static
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext.createFragmentInstanceContext;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.createLocalExecutionPlanContext;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.initDeviceViewNodeWithSeriesScanAsChildren;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.initFullOuterTimeJoinNode;
+
+public class LocalExecutionPlannerTest {
+
+ private static final long ESTIMATED_FI_NUM = 8;
+
+ private static final long QUERY_THREAD_COUNT = 8;
+
+ private static final SeriesScanOperator MOCK_SERIES_SCAN =
mockSeriesScanOperator();
+
+ /**
+ * This test will test dop = 2. Expected result is five pipelines with
dependency:
+ *
+ * The first is: DeviceView - [ExchangeOperator, ExchangeOperator,
ExchangeOperator,
+ * ExchangeOperator];
+ *
+ * The second is: ExchangeOperator - SeriesScan0.
+ *
+ * The third is: ExchangeOperator - SeriesScan1, which has dependency
second pipeline.
+ *
+ * The forth is: ExchangeOperator - SeriesScan2, which has dependency
third pipeline.
+ *
+ * The fifth is: ExchangeOperator - SeriesScan3, which has dependency
forth pipeline.
+ */
+ @Test
+ public void testCheckMemoryWithDeviceView() {
+try {
+ TypeProvider typeProvider = new TypeProvider();
+ DeviceViewNode deviceViewNode =
initDeviceViewNodeWithSeriesScanAsChildren(typeProvider, 4);
+ LocalExecutionPlanContext context =
createLocalExecutionPlanContext(typeProvider);
+ context.setDegreeOfParallelism(2);
+ Operator root = deviceViewNode.accept(new OperatorTreeGenerator(),
context);
+
+ // all the four children pipelines are ScanOperator
+ context
+ .getPipelineDriverFactories()
+ .forEach(
+ pipelineDriverFactory ->
+ Assert.assertEquals(
+ MOCK_SERIES_SCAN.calculateMaxPeekMemory(),
+ pipelineDriverFactory.getEstimatedMemorySize()));
+
+ context.addPipelineDriverFactory(
+ root, context.getDriverContext(), root.calculateMaxPeekMemory());
+
+ long expected =
+ 4 * MOCK_SERIES_SCAN.calculateRetainedSizeAfterCallingNext()
+
Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
SzyWilliam commented on code in PR #12331:
URL: https://github.com/apache/iotdb/pull/12331#discussion_r1564514434
##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlannerTest.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.planner;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.MeasurementPath;
+import org.apache.iotdb.db.queryengine.common.FragmentInstanceId;
+import org.apache.iotdb.db.queryengine.common.PlanFragmentId;
+import org.apache.iotdb.db.queryengine.common.QueryId;
+import org.apache.iotdb.db.queryengine.execution.driver.DriverContext;
+import
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext;
+import
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceStateMachine;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import
org.apache.iotdb.db.queryengine.execution.operator.source.SeriesScanOperator;
+import org.apache.iotdb.db.queryengine.plan.analyze.TypeProvider;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.DeviceViewNode;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.join.FullOuterTimeJoinNode;
+import
org.apache.iotdb.db.queryengine.plan.planner.plan.parameter.SeriesScanOptions;
+import org.apache.iotdb.db.queryengine.plan.statement.component.Ordering;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+
+import com.google.common.collect.Sets;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+
+import static
org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext.createFragmentInstanceContext;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.createLocalExecutionPlanContext;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.initDeviceViewNodeWithSeriesScanAsChildren;
+import static
org.apache.iotdb.db.queryengine.plan.planner.FEPlanUtil.initFullOuterTimeJoinNode;
+
+public class LocalExecutionPlannerTest {
+
+ private static final long ESTIMATED_FI_NUM = 8;
+
+ private static final long QUERY_THREAD_COUNT = 8;
+
+ private static final SeriesScanOperator MOCK_SERIES_SCAN =
mockSeriesScanOperator();
+
+ /**
+ * This test will test dop = 2. Expected result is five pipelines with
dependency:
+ *
+ * The first is: DeviceView - [ExchangeOperator, ExchangeOperator,
ExchangeOperator,
+ * ExchangeOperator];
+ *
+ * The second is: ExchangeOperator - SeriesScan0.
+ *
+ * The third is: ExchangeOperator - SeriesScan1, which has dependency
second pipeline.
+ *
+ * The forth is: ExchangeOperator - SeriesScan2, which has dependency
third pipeline.
+ *
+ * The fifth is: ExchangeOperator - SeriesScan3, which has dependency
forth pipeline.
+ */
+ @Test
+ public void testCheckMemoryWithDeviceView() {
+try {
+ TypeProvider typeProvider = new TypeProvider();
+ DeviceViewNode deviceViewNode =
initDeviceViewNodeWithSeriesScanAsChildren(typeProvider, 4);
+ LocalExecutionPlanContext context =
createLocalExecutionPlanContext(typeProvider);
+ context.setDegreeOfParallelism(2);
+ Operator root = deviceViewNode.accept(new OperatorTreeGenerator(),
context);
+
+ // all the four children pipelines are ScanOperator
+ context
+ .getPipelineDriverFactories()
+ .forEach(
+ pipelineDriverFactory ->
+ Assert.assertEquals(
+ MOCK_SERIES_SCAN.calculateMaxPeekMemory(),
+ pipelineDriverFactory.getEstimatedMemorySize()));
+
+ context.addPipelineDriverFactory(
+ root, context.getDriverContext(), root.calculateMaxPeekMemory());
+
+ long expected =
+ 4 * MOCK_SERIES_SCAN.calculateRetainedSizeAfterCallingNext()
+
Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
codecov[bot] commented on PR #12331: URL: https://github.com/apache/iotdb/pull/12331#issuecomment-2053619947 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12331?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `86.20690%` with `4 lines` in your changes are missing coverage. Please review. > Project coverage is 46.50%. Comparing base [(`199342d`)](https://app.codecov.io/gh/apache/iotdb/commit/199342da8df903ffa7ba821fd94fa4e17bbac11a?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`3381afd`)](https://app.codecov.io/gh/apache/iotdb/pull/12331?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). | [Files](https://app.codecov.io/gh/apache/iotdb/pull/12331?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...ueryengine/plan/planner/LocalExecutionPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/12331?src=pr=tree=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2FLocalExecutionPlanner.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL0xvY2FsRXhlY3V0aW9uUGxhbm5lci5qYXZh) | 82.60% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/12331?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12331 +/- ## + Coverage 46.49% 46.50% +0.01% Complexity 69 69 Files 3368 3368 Lines207377 207397 +20 Branches 2495524955 + Hits 9641096445 +35 + Misses 110967 110952 -15 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12331?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
sonarcloud[bot] commented on PR #12331: URL: https://github.com/apache/iotdb/pull/12331#issuecomment-2053609338 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12331) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12331=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12331) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
lancelly opened a new pull request, #12331: URL: https://github.com/apache/iotdb/pull/12331 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Take all Drivers of one FragmentInstance into account when estimating memory usage of FI [iotdb]
lancelly closed pull request #10922: Take all Drivers of one FragmentInstance into account when estimating memory usage of FI URL: https://github.com/apache/iotdb/pull/10922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
