vigneshio opened a new issue, #4922: URL: https://github.com/apache/polaris/issues/4922
## Problem Polaris writes asynchronous task metadata (such as file cleanup tasks after a `DROP TABLE` operation) to the database as `TASK` entities. These tasks are scheduled to run asynchronously. However, if a node crashes, the JVM restarts, or a transient error occurs during execution, these tasks become orphaned and remain stuck in the database indefinitely. Although the metastore layer implements CAS-based task leasing (`PolarisMetaStoreManager#loadTasks`), there is currently no background thread or daemon in Polaris that polls this endpoint to recover and execute orphaned tasks. ## Proposal Introduce a distributed background task sweeper daemon (`PolarisTaskSweeper`) to periodically lease, run, and clean up stuck tasks across all active realms. ### Design Highlights * **CAS Leasing:** Periodically calls `loadTasks` which uses internal CAS updates (`LAST_ATTEMPT_START_TIME`, `ATTEMPT_COUNT`) to lease tasks. Multiple Polaris nodes can safely sweep concurrently without double-executing tasks. * **Async Processing:** Claims are processed asynchronously on the shared `task-executor` thread pool to avoid blocking the scheduler thread during long cloud storage cleanups. * **Observability:** Fires standard `BEFORE_ATTEMPT_TASK` and `AFTER_ATTEMPT_TASK` events so sweeps are visible to audit loggers and OpenTelemetry listeners. * **Thundering Herd Mitigation:** Applies a random jitter (up to 20% of the interval) before runs so multi-node deployments do not hit the database simultaneously. * **Configuration:** Exposes `polaris.tasks.sweeper.enabled` (default: `true`) and `polaris.tasks.sweeper.interval` (default: `1m`) configuration properties. I have a fully tested local implementation of this daemon (including unit tests and documentation updates) ready on a branch and can submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
