[ 
https://issues.apache.org/jira/browse/OAK-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274443#comment-16274443
 ] 

Francesco Mari edited comment on OAK-6884 at 12/1/17 2:10 PM:
--------------------------------------------------------------

[~mduerig], I don't think that's sufficient. {{ShutDown#shutDown}} might be 
called from the main thread after the check on {{ShutDown#shutDownRequested}} 
in the disk space check thread, but before the call to {{FileStore#size}}. 

To implement a safer solution it's necessary to create another synchronization 
primitive in {{ShutDown}} that allows us to do the following in the disk space 
thread.

{noformat}
try (ShutDownCloser ignored = shutDown.tryKeepAlive()) {
    if (shutDown.shutDownRequested()) {
        return;
    }
    checkDiskSpace(gcOptions);
}
{noformat}

The implementation of {{tryKeepAlive}} (naming could be improved) is equivalent 
to {{keepAlive}}, except for the lack of the check on {{shutdownRequested}} and 
the {{IllegalStateException}}. {{tryKeepAlive}} is necessary to properly 
synchronize with {{ShutDown#shutDown}}. Callers of {{tryKeepAlive}} must always 
check {{ShutDown#shutDownRequested}} as soon as they enter in the 
try-with-resource block, and exit if the check is positive. There are only two 
possible reasons why the check could be positive. First, a user asked the 
{{FileStore}} to shut down, but nothing has been closed yet. Second, the 
{{FileStore}} has already been shut down. Whatever the reason, it is necessary 
to bail out either for responsiveness or for correctness.

By the way, while looking at the code I figured out that the flush thread 
suffers from the same race condition.


was (Author: frm):
[~mduerig], I don't think that's sufficient. {{ShutDown#shutDown}} might be 
called from the main thread after the check on {{ShutDown#shutDownRequested}} 
in the disk space check thread, but before the call to {{FileStore#size}}. 

To implement a safer solution it's necessary to create another synchronization 
primitive in {{ShutDown}} that allows us to do the following in the disk space 
thread.

{noformat}
try (ShutDownCloser ignored = shutDown.tryKeepAlive()) {
    if (shutDown.shutDownRequested()) {
        return;
    }
    checkDiskSpace(gcOptions);
}
(noformat}

The implementation of {{tryKeepAlive}} (naming could be improved) is equivalent 
to {{keepAlive}}, except for the lack of the check on {{shutdownRequested}} and 
the {{IllegalStateException}}. {{tryKeepAlive}} is necessary to properly 
synchronize with {{ShutDown#shutDown}}. Callers of {{tryKeepAlive}} must always 
check {{ShutDown#shutDownRequested}} as soon as they enter in the 
try-with-resource block, and exit if the check is positive. There are only two 
possible reasons why the check could be positive. First, a user asked the 
{{FileStore}} to shut down, but nothing has been closed yet. Second, the 
{{FileStore}} has already been shut down. Whatever the reason, it is necessary 
to bail out either for responsiveness or for correctness.

By the way, while looking at the code I figured out that the flush thread 
suffers from the same race condition.

> TarMK disk space check is not synchronized with FileStore opened state
> ----------------------------------------------------------------------
>
>                 Key: OAK-6884
>                 URL: https://issues.apache.org/jira/browse/OAK-6884
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar, upgrade
>    Affects Versions: 1.7.9
>            Reporter: Arek Kita
>            Assignee: Michael Dürig
>            Priority: Minor
>              Labels: concurrency, production
>             Fix For: 1.8, 1.7.14
>
>
> It seems that the disk space check is not properly synchronized with 
> {{FileStore}} as I revealed a race condition while using oak-upgrade during 
> migration to {{segment-tar}}.
> The {{FileStore}} instance is closed while TarMK disk check tries to execute 
> and it seems it is dependent on the state of segment 
> ({{org.apache.jackrabbit.oak.segment.file.FileStore.checkDiskSpace(FileStore.java:541)}}
>  that needs to be opened. 
> {noformat}
> 30.10.2017 11:26:05.834 WARN   o.a.j.o.s.f.Scheduler: The scheduler FileStore 
> background tasks takes too long to shut down
> 30.10.2017 11:26:11.674 INFO   o.a.j.o.s.f.FileStore: TarMK closed: 
> /data/cq/crx-quickstart/repository-segment-tar-20171030-112401/segmentstore
> 30.10.2017 11:26:11.676 ERROR  o.a.j.o.s.f.SafeRunnable: Uncaught exception 
> in TarMK disk space check 
> [/data/cq/crx-quickstart/repository-segment-tar-20171030-112401/segmentstore]
> java.lang.IllegalStateException: already shut down
>     at 
> org.apache.jackrabbit.oak.segment.file.ShutDown.keepAlive(ShutDown.java:42)
>     at 
> org.apache.jackrabbit.oak.segment.file.FileStore.size(FileStore.java:302)
>     at 
> org.apache.jackrabbit.oak.segment.file.FileStore.checkDiskSpace(FileStore.java:541)
>     at 
> org.apache.jackrabbit.oak.segment.file.FileStore.access$300(FileStore.java:102)
>     at 
> org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:237)
>     at 
> org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to