danielhumanmod opened a new pull request, #1523:
URL: https://github.com/apache/polaris/pull/1523
Fix #774
# Context
Polaris uses async tasks to perform operations such as table and manifest
file cleanup. These tasks are executed asynchronously in a separate thread
within the same JVM, and retries are handled inline within the task execution.
However, this mechanism does not guarantee eventual execution in the
following cases:
- The task fails repeatedly and hits the maximum retry limit.
- The service crashes or shuts down before retrying.
# Implementation Plan
Stage 1 (this PR):
Introduce per-task transactional leasing in the metastore layer via
`loadTasks(...)`. This enables fine-grained compensation by allowing tasks to
be leased and updated one at a time, avoiding the all-or-nothing semantics of
bulk operations.
Stage 2 (future PR):
Persist failed tasks and introduce a retry mechanism triggered during
Polaris startup and via periodic background checks.
<!--
Possible security vulnerabilities: STOP here and contact
secur...@apache.org instead!
Please update the title of the PR with a meaningful message - do not
leave it "empty" or "generated"
Please update this summary field:
The summary should cover these topics, if applicable:
* the motivation for the change
* a description of the status quo, for example the current behavior
* the desired behavior
* etc
PR checklist:
- Do a self-review of your code before opening a pull request
- Make sure that there's good test coverage for the changes included in
this PR
- Run tests locally before pushing a PR (./gradlew check)
- Code should have comments where applicable. Particularly
hard-to-understand
areas deserve good in-line documentation.
- Include changes and enhancements to the documentation (in
site/content/in-dev/unreleased)
- For Work In Progress Pull Requests, please use the Draft PR feature.
Make sure to add the information BELOW this comment.
Everything in this comment will NOT be added to the PR description.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
