danielhumanmod commented on PR #1523: URL: https://github.com/apache/polaris/pull/1523#issuecomment-2861792421
> > Introduce per-task transactional leasing in the metastore layer via loadTasks(...). This enables fine-grained compensation by allowing tasks to be leased and updated one at a time, avoiding the all-or-nothing semantics of bulk operations (which is also mentioned in TODO). This is important for retry scenarios, where we want to isolate failures and ensure that tasks are independently retried without affecting each other. > > I don't understand how this PR enables isolation of task failures. This PR only reads the tasks from the metastore one at a time, so the only failure would be in loading the task. In a transactional database, the `UPDATE ... WHERE` statement would only update the task state when the task is not currently leased by another client, so I don't see how one or a few tasks would fail to be leased while the others succeed. > > The PR description sounds like it intends to tackle _task execution_ failure - is that right? If so, loading the tasks from the database isn't going to solve that problem. Sorry for the confusion — we actually have a second PR for this feature. The current PR is a draft (still WIP): https://github.com/danielhumanmod/polaris/pull/1 — that’s where pending tasks are loaded from the metastore and executed. Regarding this PR’s changes in the metastore, the goal is to allow each task entity to be read and leased individually. This ensures that if an exception occurs while reading or leasing one task, it won’t affect others. This improvement was also noted in the [TODO](https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java#L1976) comment of the previous implementation. It’s not strictly required, but maybe a “nice-to-have” one for isolating failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
