yuqi1129 opened a new pull request, #13619: URL: https://github.com/apache/gravitino/pull/13619
### What changes were proposed in this pull request? - Schedule each change-log poll cycle individually. When a cycle reads a full batch, the next cycle starts immediately; an empty or partial batch, or a failed cycle, waits `pollIntervalSecs` as before. - Each cycle still reads at most one batch, so the work per cycle stays bounded and `close()` stops the chain between batches. The scheduler drops a pending delayed poll on shutdown, so `close()` neither waits for it nor runs it against a closing store. - Add `gravitino.entityChangeLog.pollBatchSize` (default `2000`, unchanged) and document propagation capacity and the backlog signal in `batch-size-records`. ### Why are the changes needed? The poller read at most 2,000 records and then always slept for the full interval, so the default configuration could not consume more than about 667 records/s, even with a backlog. Under sustained multi-node writes, the backlog kept growing and remote caches stayed stale. Fix: #13581 ### Does this PR introduce _any_ user-facing change? - New server property `gravitino.entityChangeLog.pollBatchSize`. - A server that is behind now drains the change log continuously instead of at a fixed `pollBatchSize / pollIntervalSecs` rate. Idle and failing pollers still poll once per interval. ### How was this patch tested? - `TestEntityChangeLogPoller`: a full batch reports a backlog and a partial or empty batch does not; with a mocked scheduler, a full batch schedules the next cycle with no delay, while a partial batch and a failed poll wait the interval, and nothing is scheduled after `close()`. The scheduling test fails if every cycle waits the full interval. - `TestEntityChangeLogPoller` with the real scheduler: `close()` returns immediately and the pending poll never runs; without the shutdown policy, `close()` blocks for 5 s. - `TestEntityCacheCrossNodeInvalidation` (H2): five distinct cached tables altered on node A with a batch size of two; one poll invalidates only the first batch, and draining until a partial batch invalidates all five on node B. - `TestEntityChangeLogPoller`, `TestEntityCacheCrossNodeInvalidation`, `TestEntityCacheChangeLogListener`, `TestEntityChangeLogCleaner`, `TestEntityChangeLogService`, and `TestRelationalEntityStore` pass with H2; `:core:spotlessCheck`, `:core:javadoc`, and `git diff --check` pass. The full `:core:test` suite was not run locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
