[
https://issues.apache.org/jira/browse/SAMZA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279101#comment-16279101
]
ASF GitHub Bot commented on SAMZA-1523:
---------------------------------------
GitHub user navina opened a pull request:
https://github.com/apache/samza/pull/379
SAMZA-1523 Cleanup table entries before shutting down the processor
Modified the `TableUtils#deleteProcessorEntity` to provide an option to
disable optimistic locking during a call to Azure Table Storage service.
@sborya @PawasChhokra @nickpan47 Review please?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/navina/samza azure-etag-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/379.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #379
----
commit 28312cb8b91b16349f6d2bc60e9c41ae988b96e0
Author: navina <[email protected]>
Date: 2017-12-05T19:32:09Z
SAMZA-1523 Cleanup table entries before shutting down the processor
----
> Cleanup table entries before shutting down the processor
> --------------------------------------------------------
>
> Key: SAMZA-1523
> URL: https://issues.apache.org/jira/browse/SAMZA-1523
> Project: Samza
> Issue Type: Bug
> Reporter: Navina Ramesh
> Assignee: Navina Ramesh
>
> We want to remove expired entries of the processors from the Azure Table when
> the processor is shutting down. Azure Table service uses optimistic locking
> by default. Hence, when the coordinator thread is cleaning up during
> shutdown, it is possible for the heartbeat thread to update that entry as
> well. This causes a failure in cleanup and throws exceptions in the log.
> Obviously, it also fails to clear the entries :)
> `
> 2017-11-30 15:23:32.804 [JMVersionUpgradeScheduler-0] AzureJobCoordinator
> [INFO] pid=05133d0a-dd85-4178-a97c-2c98dc617308new version 5 of the job model
> got confirmed
> 2017-11-30 15:23:32.833 [HeartbeatScheduler-0] HeartbeatScheduler [INFO]
> Updating heartbeat for processor ID: 05133d0a-dd85-4178-a97c-2c98dc617308 and
> job model version: 4
> 2017-11-30 15:23:32.905 [JMVersionUpgradeScheduler-0] TableUtils [ERROR]
> Azure storage exception while deleting processor entity with job model
> version: 4and pid: 05133d0a-dd85-4178-a97c-2c98dc617308
> com.microsoft.azure.storage.table.TableServiceException: Precondition Failed
> at
> com.microsoft.azure.storage.table.TableServiceException.generateTableServiceException(TableServiceException.java:52)
> at
> com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:319)
> at
> com.microsoft.azure.storage.table.TableOperation$1.preProcessResponse(TableOperation.java:299)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:139)
> at
> com.microsoft.azure.storage.table.TableOperation.performDelete(TableOperation.java:281)
> at
> com.microsoft.azure.storage.table.TableOperation.execute(TableOperation.java:685)
> at
> com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:529)
> at
> com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:496)
> at
> org.apache.samza.util.TableUtils.deleteProcessorEntity(TableUtils.java:157)
> at
> org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelConfirmed(AzureJobCoordinator.java:448)
> at
> org.apache.samza.coordinator.AzureJobCoordinator.onNewJobModelAvailable(AzureJobCoordinator.java:419)
> at
> org.apache.samza.coordinator.AzureJobCoordinator.lambda$createJMVersionUpgradeListener$3(AzureJobCoordinator.java:248)
> at
> org.apache.samza.coordinator.scheduler.JMVersionUpgradeScheduler.lambda$scheduleTask$0(JMVersionUpgradeScheduler.java:81)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-11-30 15:23:32.906 [JMVersionUpgradeScheduler-0] AzureJobCoordinator
> [ERROR] Exception in Job Model Version Upgrade Scheduler. Stopping the
> processor...
> `
> We should disable optimistic locking during the cleanup phase of shutdown.
> Ideal solution is to perhaps have more control over the various schedulers.
> That is beyond the scope of this JIRA though :)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
