JackieTien97 commented on code in PR #19086:
URL: https://github.com/apache/iotdb/pull/19086#discussion_r4462121249
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -53,6 +55,11 @@
@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
public class IoTDBMultiTAGsWithAttributesTableIT {
private static final String DATABASE_NAME = "db";
+ // Default heap of the test DataNodes (see DataNodeWrapper#initVMConfig).
+ private static final int DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB = 256;
+ // fourTableJoinTest needs about 3.5 MB of operator memory per DataRegion. A
256 MB heap has
+ // about 16 MB of it, enough for about 4 regions; 128 MB of heap per region
leaves a 2x margin.
+ private static final int DATANODE_HEAP_SIZE_PER_DATA_REGION_IN_MB = 128;
Review Comment:
The per-region model doesn't hold beyond a few regions.
`students`/`teachers`/`courses`/`grades` have 5/3/5/5 devices, all in one time
partition, so the number of scan FragmentInstances of `fourTableJoinTest` is
bounded by those 18 devices, however many regions exist. The heap has no bound,
though. On 64 cores it is 32 × 128 MB = 4 GB per DataNode, or 12 GB of `-Xmx`
for 1C3D, where each DataNode only runs about a third of the scans anyway. If
this approach stays, please cap it, for example at 1024 MB, which is what CI
uses.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -53,6 +55,11 @@
@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
public class IoTDBMultiTAGsWithAttributesTableIT {
private static final String DATABASE_NAME = "db";
+ // Default heap of the test DataNodes (see DataNodeWrapper#initVMConfig).
+ private static final int DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB = 256;
Review Comment:
This duplicates the literal and the system-property lookup in
`DataNodeWrapper#initVMConfig`
([L178](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/integration-test/src/main/java/org/apache/iotdb/it/env/cluster/node/DataNodeWrapper.java#L178)).
`MppJVMConfig.override()` replaces any positive value and doesn't re-check
`init <= max`
([L89-L97](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/integration-test/src/main/java/org/apache/iotdb/it/env/cluster/config/MppJVMConfig.java#L89-L97)).
So if the framework default ever changes (for example, aligned with CI's 1024
MB, or with a larger init heap), this test would quietly lower the heap on
small machines, or produce `-Xms > -Xmx` and the DataNodes would fail to start.
If the heap override stays, it would be safer to override only when the
computed value is larger than the effective default, or to get the effective
default from the framework instead of copying it.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -185,6 +192,22 @@ public class IoTDBMultiTAGsWithAttributesTableIT {
@BeforeClass
public static void setUp() throws Exception {
EnvFactory.getEnv().getConfig().getDataNodeCommonConfig().setSortBufferSize(1024
* 1024L);
+ // The number of DataRegionGroups follows the CPU cores (0.5 per core),
and with it the number
+ // of FragmentInstances of fourTableJoinTest, each reserving up to the 1
MB sort buffer. Scale
+ // the DataNode heap, and so its operator memory, with the regions. On a
4-core machine
+ // (2 regions) this keeps the default heap.
+ final int expectedDataRegionsPerDataNode =
+ Math.max(1, Runtime.getRuntime().availableProcessors() / 2);
+ EnvFactory.getEnv()
+ .getConfig()
+ .getDataNodeJVMConfig()
+ .setMaxHeapSize(
+ Math.max(
+ EnvUtils.getIntFromSysVar(
+ ClusterConstant.DATANODE_MAX_HEAP_SIZE,
+ DEFAULT_DATANODE_MAX_HEAP_SIZE_IN_MB,
+ 0),
+ expectedDataRegionsPerDataNode *
DATANODE_HEAP_SIZE_PER_DATA_REGION_IN_MB));
EnvFactory.getEnv().initClusterEnvironment();
Review Comment:
Pre-existing, and not caused by this PR, but since this `setUp` is being
touched anyway: the `setMaxTsBlockLineNumber(2).setMaxNumberOfPointsInPage(5)`
call just below runs after `initClusterEnvironment()`. The node configuration
is only written when the nodes are created, so these two settings never reach
the running cluster, and the test has always run with the defaults. Moving them
before `initClusterEnvironment()` would apply them, but it changes the
effective configuration (and possibly the memory numbers above), so it may be
better as a separate change.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java:
##########
@@ -185,6 +192,22 @@ public class IoTDBMultiTAGsWithAttributesTableIT {
@BeforeClass
public static void setUp() throws Exception {
EnvFactory.getEnv().getConfig().getDataNodeCommonConfig().setSortBufferSize(1024
* 1024L);
+ // The number of DataRegionGroups follows the CPU cores (0.5 per core),
and with it the number
+ // of FragmentInstances of fourTableJoinTest, each reserving up to the 1
MB sort buffer. Scale
+ // the DataNode heap, and so its operator memory, with the regions. On a
4-core machine
+ // (2 regions) this keeps the default heap.
+ final int expectedDataRegionsPerDataNode =
+ Math.max(1, Runtime.getRuntime().availableProcessors() / 2);
Review Comment:
Minor: this re-derives the ConfigNode's formula, and it already differs from
it in three ways:
- The ConfigNode rounds up (`Math.ceil(...)`, [ClusterSchemaManager
L595-L598](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/schema/ClusterSchemaManager.java#L595-L598)).
- It uses the cores that each DataNode JVM reports ([DataNode
L1027](https://github.com/apache/iotdb/blob/88f85321744697d747a6c6ed8ec41eeb6aa6b978/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNode.java#L1027)),
not the test JVM's.
- It honors `data_region_per_data_node` overrides, for example from
`-DClusterConfigurations`.
On odd core counts, this underestimates by one region and relies on the 2x
margin. That's another reason to pin the region count instead (see the review
summary).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]