alpass163gmail opened a new pull request, #18127:
URL: https://github.com/apache/iotdb/pull/18127
This PR introduces a lease-based self-fencing framework to enable high
availability for table metadata operations. It prevents DataNodes from
serving stale schema during network partitions.
Problem
Currently, in a cluster deployment (e.g., 1 ConfigNode, 3 DataNodes),
metadata operations lack true high availability. If a single DataNode (DN)
crashes or experiences a network partition, execution of DDL procedures
(such as Create/Alter/Drop Table, View management, TTL adjustments, and
Drop Database) will directly fail and trigger a rollback.
Solution
Lease Mechanism
- MetadataLeaseManager (DataNode): Tracks lease via ConfigNode heartbeats.
Uses monotonic clock. Self-fences (clears cache, blocks reads/writes) if no
heartbeat received within metadata_lease_fence_ms (T_fence).
- MetadataLeaseFencedException: Thrown when operations are blocked on a
fenced DataNode.
Broadcast Coordination
- DataNodeContactTracker (ConfigNode): Records time of last successful
heartbeat response per DataNode. Separately maintained from load-balancing
samples to ensure correctness.
- MetadataBroadcastVerdict: Pure decision logic — PROCEED if all unacked
DataNodes have been silent for T_proceed = T_fence + margin, WAIT
otherwise, FAIL when retry budget exhausted.
- ClusterCachePropagator: Broadcasts cache invalidations with retry loop,
waiting up to T_proceed for unresponsive DataNodes to prove self-fenced.
Schema Change Integration
- Procedures now propagate metadata invalidations via
ClusterCachePropagator before proceeding.
- Pre-deletion marker (PreDeleteTsTable) added for safe table state
transitions.
- Rollback mechanism (RollbackPreDeleteTablePlan) for failed schema
changes.
Configuration
- New config: metadata_lease_fence_ms (default in
iotdb-system.properties.template).
Testing
- Unit tests for MetadataLeaseManager, DataNodeContactTracker,
MetadataBroadcastVerdict, ClusterCachePropagator
- Lease integration tests for DataNodeTableCache, PartitionCache,
ClusterAuthorityFetcher
- New HA IT: IoTDBTableDDLHAIT
Key Components Added
- MetadataLeaseManager (DataNode): Tracks the lease and performs
self-fencing upon expiration.
- DataNodeContactTracker (ConfigNode): Tracks the last successful heartbeat
timestamp for each DataNode.
- ClusterCachePropagator (ConfigNode): Broadcasts cache invalidations with
a fencing-aware retry mechanism.
- MetadataBroadcastVerdict (ConfigNode): Decides when unacknowledged
DataNodes are logically safe to skip.
- MetadataLeaseFencedException (node-commons): The explicit exception
thrown on a fenced DataNode.
- PreDeleteTsTable (node-commons): Represents the marker table state to
ensure safe schema deletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
