JackieTien97 commented on code in PR #19086:
URL: https://github.com/apache/iotdb/pull/19086#discussion_r4462121249


##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -53,6 +55,11 @@
 @Category({TableLocalStandaloneIT.class, TableClusterIT.class})
 public class IoTDBMultiTAGsWithAttributesTableIT {
   private static final String DATABASE_NAME = "db";
+  // Default heap of the test DataNodes (see DataNodeWrapper#initVMConfig).
+  private static final int DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB = 256;
+  // fourTableJoinTest needs about 3.5 MB of operator memory per DataRegion. A 
256 MB heap has
+  // about 16 MB of it, enough for about 4 regions; 128 MB of heap per region 
leaves a 2x margin.
+  private static final int DATANODE_HEAP_SIZE_PER_DATA_REGION_IN_MB = 128;

Review Comment:
   The per-region model doesn't hold beyond a few regions. 
`students`/`teachers`/`courses`/`grades` have 5/3/5/5 devices, all in one time 
partition, so the number of scan FragmentInstances of `fourTableJoinTest` is 
bounded by those 18 devices, however many regions exist. The heap has no bound, 
though. On 64 cores it is 32 × 128 MB = 4 GB per DataNode, or 12 GB of `-Xmx` 
for 1C3D, where each DataNode only runs about a third of the scans anyway. If 
this approach stays, please cap it, for example at 1024 MB, which is what CI 
uses.



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -53,6 +55,11 @@
 @Category({TableLocalStandaloneIT.class, TableClusterIT.class})
 public class IoTDBMultiTAGsWithAttributesTableIT {
   private static final String DATABASE_NAME = "db";
+  // Default heap of the test DataNodes (see DataNodeWrapper#initVMConfig).
+  private static final int DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB = 256;

Review Comment:
   This duplicates the literal and the system-property lookup in 
`DataNodeWrapper#initVMConfig` 
([L178](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/integration-test/src/main/java/org/apache/iotdb/it/env/cluster/node/DataNodeWrapper.java#L178)).
 `MppJVMConfig.override()` replaces any positive value and doesn't re-check 
`init <= max` 
([L89-L97](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/integration-test/src/main/java/org/apache/iotdb/it/env/cluster/config/MppJVMConfig.java#L89-L97)).
 So if the framework default ever changes (for example, aligned with CI's 1024 
MB, or with a larger init heap), this test would quietly lower the heap on 
small machines, or produce `-Xms > -Xmx` and the DataNodes would fail to start. 
If the heap override stays, it would be safer to override only when the 
computed value is larger than the effective default, or to get the effective 
default from the framework instead of copying it.



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -185,6 +192,22 @@ public class IoTDBMultiTAGsWithAttributesTableIT {
   @BeforeClass
   public static void setUp() throws Exception {
     
EnvFactory.getEnv().getConfig().getDataNodeCommonConfig().setSortBufferSize(1024
 * 1024L);
+    // The number of DataRegionGroups follows the CPU cores (0.5 per core), 
and with it the number
+    // of FragmentInstances of fourTableJoinTest, each reserving up to the 1 
MB sort buffer. Scale
+    // the DataNode heap, and so its operator memory, with the regions. On a 
4-core machine
+    // (2 regions) this keeps the default heap.
+    final int expectedDataRegionsPerDataNode =
+        Math.max(1, Runtime.getRuntime().availableProcessors() / 2);
+    EnvFactory.getEnv()
+        .getConfig()
+        .getDataNodeJVMConfig()
+        .setMaxHeapSize(
+            Math.max(
+                EnvUtils.getIntFromSysVar(
+                    ClusterConstant.DATANODE_MAX_HEAP_SIZE,
+                    DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB,
+                    0),
+                expectedDataRegionsPerDataNode * 
DATANODE_HEAP_SIZE_PER_DATA_REGION_IN_MB));
     EnvFactory.getEnv().initClusterEnvironment();

Review Comment:
   Pre-existing, and not caused by this PR, but since this `setUp` is being 
touched anyway: the `setMaxTsBlockLineNumber(2).setMaxNumberOfPointsInPage(5)` 
call just below runs after `initClusterEnvironment()`. The node configuration 
is only written when the nodes are created, so these two settings never reach 
the running cluster, and the test has always run with the defaults. Moving them 
before `initClusterEnvironment()` would apply them, but it changes the 
effective configuration (and possibly the memory numbers above), so it may be 
better as a separate change.



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -185,6 +192,22 @@ public class IoTDBMultiTAGsWithAttributesTableIT {
   @BeforeClass
   public static void setUp() throws Exception {
     
EnvFactory.getEnv().getConfig().getDataNodeCommonConfig().setSortBufferSize(1024
 * 1024L);
+    // The number of DataRegionGroups follows the CPU cores (0.5 per core), 
and with it the number
+    // of FragmentInstances of fourTableJoinTest, each reserving up to the 1 
MB sort buffer. Scale
+    // the DataNode heap, and so its operator memory, with the regions. On a 
4-core machine
+    // (2 regions) this keeps the default heap.
+    final int expectedDataRegionsPerDataNode =
+        Math.max(1, Runtime.getRuntime().availableProcessors() / 2);

Review Comment:
   Minor: this re-derives the ConfigNode's formula, and it already differs from 
it in three ways:
   - The ConfigNode rounds up (`Math.ceil(...)`, [ClusterSchemaManager 
L595-L598](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/schema/ClusterSchemaManager.java#L595-L598)).
   - It uses the cores that each DataNode JVM reports ([DataNode 
L1027](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNode.java#L1027)),
 not the test JVM's.
   - It honors `data_region_per_data_node` overrides, for example from 
`-DClusterConfigurations`.
   
   On odd core counts, this underestimates by one region and relies on the 2x 
margin. That's another reason to pin the region count instead (see the review 
summary).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to