d-wang-commit opened a new pull request, #18692:
URL: https://github.com/apache/iotdb/pull/18692

   ## Description
   
   ### Policy use cases
   
   The configuration template describes the intended scenarios as follows 
(quoted verbatim).
   
   SchemaRegionGroups:
   
   ```properties
   # 1. CUSTOM: For known workloads that need manual resource allocation; 
creates the configured target
   #    number of SchemaRegionGroups per Database when schema partitions are 
first allocated.
   # 2. AUTO: For most routine workloads (roughly 80% as an approximate 
planning guideline); gradually
   #    meets the configured minimum and expands with slot occupancy up to the 
resource-based maximum,
   #    balancing parallelism and RegionGroup management overhead.
   # 3. PROACTIVE: Complements AUTO where it does not expand eagerly enough, 
such as very few devices with
   #    many measurements and a high load; creates groups earlier for more 
parallelism at the cost of
   #    additional RegionGroup management overhead.
   ```
   
   DataRegionGroups:
   
   ```properties
   # 1. CUSTOM: For known workloads that need manual resource allocation; 
creates the configured target
   #    number of DataRegionGroups per Database when data partitions are first 
allocated.
   # 2. AUTO: For most routine workloads (roughly 80% as an approximate 
planning guideline); gradually
   #    meets the configured minimum and expands with slot occupancy up to the 
resource-based maximum,
   #    balancing parallelism and RegionGroup management overhead.
   # 3. PROACTIVE: Complements AUTO where it does not expand eagerly enough, 
such as very few devices with
   #    many measurements and a high load; creates groups earlier for more 
parallelism at the cost of
   #    additional RegionGroup management overhead.
   ```
   
   ### Proactive RegionGroup allocation
   
   Sparse databases can keep using only a few RegionGroups under AUTO even when 
resources are available. Add PROACTIVE for SchemaRegionGroups and 
DataRegionGroups, and make it the default in ConfigNodeConfig and the 
configuration template.
   
   PROACTIVE counts distinct persisted and pending series slots separately for 
schema and data. It grows toward one group per active slot, respects the 
resource-derived maximum, and follows AUTO's incremental growth toward the 
configured minimum. New time partitions do not inflate the distinct-slot count. 
Existing groups are retained; if all groups of the requested type are disabled, 
one replacement can be allocated within the maximum.
   
   For data allocation and recovery, calculate the retained-slot limit from 
active assignments, with a minimum of one, instead of the total configured slot 
count. With 1,000 slots and six active devices, switching from AUTO with two 
groups to PROACTIVE can grow to six groups: each old group retains one slot and 
four slots use the four new groups for new partitions. Historical partition 
mappings remain unchanged.
   
   ### Policy hot reload
   
   Recalculate resource-derived maxima on the ConfigNode leader when either 
extension policy changes, as well as when per-node quotas change. Previously, 
leaving CUSTOM without changing quotas could retain its manual cap and prevent 
expected growth.
   
   Cover CUSTOM below and above the PROACTIVE target, PROACTIVE to CUSTOM, 
CUSTOM to AUTO, and independent schema/data switching. Switching to CUSTOM 
preserves existing groups and uses the stored maximum; it does not promise to 
redistribute sparse slots across every group.
   
   ### Tests
   
   Added UTs:
   - PartitionManagerProactiveExtensionTest: distinct slots, minimum growth, 
batching, caps, unavailable groups, policy switches, and failed-allocation 
retries.
   - DataPartitionPolicyTableProactiveTest: sparse assignment, use of new 
groups, recovery, and AUTO/CUSTOM compatibility.
   - RegionGroupExtensionPolicyConfigTest: parsing, invalid values, file 
loading, and independent hot reload.
   
   Existing ClusterSchemaManagerTest and DataPartitionPolicyTableTest provide 
resource-limit and allocation regression coverage.
   
   Added ITs:
   - IoTDBProactiveRegionGroupDistributionIT: AUTO/PROACTIVE comparisons on 
1C3D, 1C6D, and 1C9D; replication, SQL reads/writes, hot reload, new time 
partitions, and ConfigNode restart. Six targeted hot-reload cases cover 
AUTO/CUSTOM/PROACTIVE transitions and preserved historical mappings.
   - IoTDBProactiveRegionGroupExtensionIT: five topology/consensus combinations 
covering partition counts, shared slots, caps, ConfigNode recovery, and mapping 
retention.
   
   The recorded local targeted run completed with 65 UT cases (0 failures, 0 
errors, 3 skipped) and all six hot-reload IT cases passing. The topology matrix 
and existing IoTDBPartitionInheritStrategyIT were not part of that run.
   
   ### Side effects and risks
   
   Changing the default from AUTO to PROACTIVE can create more RegionGroups 
earlier, increasing consensus, memory, and storage overhead within resource 
limits. Explicit AUTO and CUSTOM settings remain available. Policy changes can 
affect placement of newly allocated partitions, but do not migrate historical 
data or shrink existing groups. Allocation may still fail when insufficient 
DataNodes are available.
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are 
corner cases and error 
       conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, 
inheritance, composition, 
       design patterns)
    - Method organization and design (how the logic is split between methods, 
parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of 
emitted metrics)
   -->
   
   <!-- It's good to describe an alternative design (or mention an alternative 
name) for every design 
   (or naming) decision point and compare the alternatives with the designs 
that you've implemented 
   (or the names you've chosen) to highlight the advantages of the chosen 
designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in 
this PR elsewhere 
   (e. g. a "Proposal" issue, any other issue, or a thread in the development 
mailing list), 
   link to that discussion from this PR description and explain what have 
changed in your final design 
   compared to your original proposal or the consensus version in the end of 
the discussion. 
   If something hasn't changed since the original discussion, you can omit a 
detailed discussion of 
   those aspects of the design here, perhaps apart from brief mentioning for 
the sake of readability 
   of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small 
changes. -->
   
   <hr>
   
   This PR has:
   - [x] added documentation for new or modified features or behaviors.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious 
     for an unfamiliar reader.
   - [x] added integration tests.
   - [x] been tested in a test IoTDB cluster.
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not 
all of these items 
   apply to every PR. Remove the items which are not done or not relevant to 
the PR. None of the items 
   from the checklist above are strictly necessary, but it would be very 
helpful if you at least 
   self-review the PR. -->
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - RegionGroupExtensionPolicy and ConfigNodeConfig: PROACTIVE policy and 
defaults.
   - PartitionManager, PartitionInfo, and DatabasePartitionTable: distinct-slot 
counting and bounded proactive allocation.
   - DataPartitionPolicyTable: active-slot-based retention during balancing and 
recovery.
   - ConfigManager: refresh resource-derived maxima after policy hot reload.
   - New proactive/configuration UTs and proactive distribution/extension ITs: 
allocation and switching coverage.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to