[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987620#action_12987620 ] Dag H. Wanvik commented on DERBY-4741: -- Regressions ran ok, committed derby-4741-raf-stresstest-4 as svn 1064174. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-raf-stresstest-2.diff, derby-4741-raf-stresstest-2.stat, > derby-4741-raf-stresstest-3.diff, derby-4741-raf-stresstest-3.stat, > derby-4741-raf-stresstest-4.diff, derby-4741-raf-stresstest-4.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986710#action_12986710 ] Knut Anders Hatlen commented on DERBY-4741: --- > I think I will commit the patch with the long running time for now. I'd > rather flush out any bugs here sooner rather than later. Sounds good. +1 > I always agonize about whether to include c/s tests or not when the changes > are seemingly embedded/server-side only I don't have a general rule, but in this particular case it looks like we run the entire test inside a stored procedure in order to make sure that only embedded is tested. Sounds like a good indication that embedded only would suffice... :) We could probably simplify the test somewhat and remove the stored procedure indirection too if we made it embedded only. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-raf-stresstest-2.diff, derby-4741-raf-stresstest-2.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986601#action_12986601 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for looking at this, Knut! You are right, it got a bit convoluted. The intention is, as you suggest, to always embedded connections inside the stored procedure. I'll see if I can simplify this a bit! I always agonize about whether to include c/s tests or not when the changes are seemingly embedded/server-side only, maybe we should try to establish some criteria to help us decide when to, and when not to do that. Thanks also for spotting the nits, I converted this from a stand-alone test outside JUnit, so it can be further simplied as you suggest. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986365#action_12986365
]
Knut Anders Hatlen commented on DERBY-4741:
---
The use of the static field thisTest may have some consequences that are not
entirely obvious:
The field is set when the test suite is created, so when the tests start
running it will always point to the last test case that was created, not
necessarily to the running test case. That is, it will always point to a client
test case. Because of this, one may be led to believe that the test that calls
thisTest.openDefaultConnection() always uses client connections. However,
BaseJDBCTestCase.openDefaultConnection() calls TestConfiguration.getCurrent()
to get the current test configuration, so the connection type should be chosen
dynamically based on the active decorators. But this assumption is also not
quite true. The decorators only affect the TestConfiguration for the main
thread, so when running inside a stored procedure in a server thread, the
decorators will have no effect, and an embedded connection will be opened even
if the test case is actually running within a client/server decorator.
I think the intention is to always use embedded connections inside the stored
procedure, and that's also what the code does (so, as mentioned in an earlier
comment, there's probably not much point in running this test in client/server
mode?). But it takes some effort to convince oneself that that's what it
actually does. Do you think it's possible to make it more explicit that the
worker threads always use embedded connections?
And some little nits:
+if (w.e != null) {
+fail("WorkerThread " + i + " saw exception " + w.e);
+}
Use the fail() method that takes an exception to preserve the stack trace?
+ResultSet rs = s.executeQuery("select count(*) from mtTab");
+rs.next();
+assertEquals("too few rows inserted",
+ rs.getLong(1), NO_OF_THREADS * NO_OF_MT_OPS);
+rs.close();
How do you know it must be too few and not too many? ;)
JDBC.assertSingleValueResultSet() could simplify this code.
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-a-01-api-interruptstatus.diff,
> derby-4741-a-01-api-interruptstatus.stat,
> derby-4741-a-02-api-interruptstatus.diff,
> derby-4741-a-02-api-interruptstatus.stat,
> derby-4741-a-03-api-interruptstatus.diff,
> derby-4741-a-03-api-interruptstatus.stat,
> derby-4741-a-04-api-interruptstatus.diff,
> derby-4741-a-04-api-interruptstatus.stat,
> derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat,
> derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff,
> derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat,
> derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff,
> derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff,
> derby-4741-nio-container+log+waits+locks+throws.diff,
> derby-4741-nio-container+log+waits+locks+throws.stat,
> derby-4741-nio-container+log+waits+locks-2.diff,
> derby-4741-nio-container+log+waits+locks-2.stat,
> derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat,
> derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat,
> derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat,
> derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log,
> derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986347#action_12986347 ] Knut Anders Hatlen commented on DERBY-4741: --- Perhaps the difference is whether or not the OS will force changes to disk when doing synchronous writes? > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986212#action_12986212 ] Dag H. Wanvik commented on DERBY-4741: -- Regrressions for *-raf-stresstest-1, and I verified that the modified InterruptResilienceTest works on Windows as well. For some reason it executed much more quickly under Windows, ca 50s (the reported 250 above was on Solaris). I'll see I can figure out why. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985034#action_12985034 ] Dag H. Wanvik commented on DERBY-4741: -- Myrna, I will look into it. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-raf-stresstest-1.diff, derby-4741-raf-stresstest-1.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984852#action_12984852 ] Myrna van Lunteren commented on DERBY-4741: --- I think the latest check-in has caused the following tinderbox failure: derbyall/storeall/storeall.fail:unit/T_RawStoreFactory.unit see: http://dbtg.foundry.sun.com/derby/test/tinderbox_trunk16/jvm1.6/testing/testlog/SunOS-5.10_i86pc-i386/1061516-derbyall_diff.txt: * Diff file derbyall/storeall/storeunit/T_RawStoreFactory.diff *** Start: T_RawStoreFactory jdk1.6.0_18 storeall:storeunit 2011-01-20 23:22:23 *** 2 del < -- Unit Test T_RawStoreFactory finished 2 add > ran out of time > Exit due to time bomb Test Failed. *** End: T_RawStoreFactory jdk1.6.0_18 storeall:storeunit 2011-01-21 00:22:54 *** It failed in the nightly runs with ibm 1.6 also (and 1.4.2 and 1.5). Do we need a separate jira? > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984405#action_12984405 ] Dag H. Wanvik commented on DERBY-4741: -- Committed rev #3 as svn 1061516. This completes the cleanup of interrupt handling in Object#wait and Thread.sleep inside store. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, > derby-4741-sleeps-waits-3.diff, derby-4741-sleeps-waits-3.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983935#action_12983935 ] Dag H. Wanvik commented on DERBY-4741: -- > XactFactory.blockBackupBlockingOperations(): Remove RuntimeException from > throws clause? The explanation added to the javadoc is probably enough for > documentation purposes. I'm ok with that, I was torn on this. > RAFContainer4.java: Why the change from noteAndClearInterrupt() to > setInterrupted()? Just simplification, since in these two locations we know that an interrupt actually did happen, there is no need to test for the flag as noteAndClearInterrupt does. > - Whitespace diff @@ -825,7 +819,7 @@ looks unintended Indeed, thanks. > > ReplicationMessageReceive.java: The while loops are only retried if the > wait() calls are interrupted, and not if the wait() calls experience spurious > wakeups. Any reason for that? (Probably still an improvement compared to the > original code, which didn't retry the wait call in any event.) Yes, I agree the original code is deficient in this repect, I have the code changed to handle this as well. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983697#action_12983697 ] Knut Anders Hatlen commented on DERBY-4741: --- Second revision of sleeps-waits looks good to me. A couple of questions: XactFactory.blockBackupBlockingOperations(): Remove RuntimeException from throws clause? The explanation added to the javadoc is probably enough for documentation purposes. RAFContainer4.java: - Why the change from noteAndClearInterrupt() to setInterrupted()? - Whitespace diff @@ -825,7 +819,7 @@ looks unintended ReplicationMessageReceive.java: The while loops are only retried if the wait() calls are interrupted, and not if the wait() calls experience spurious wakeups. Any reason for that? (Probably still an improvement compared to the original code, which didn't retry the wait call in any event.) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, > derby-4741-sleeps-waits-2.diff, derby-4741-sleeps-waits-2.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982189#action_12982189 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for test running the two patches, Lily. Good to know these run on the IBM VM as well! As for the comment, agreed. I have removed it in my next version of this patch, because it long longer seems to carry important information: earlier practice was to always throw, so then the comment highlighted "unusual" treatment. Now we mostly do not throw, just make a note, so this is code is no longer unusual as I see it. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982135#action_12982135 ] Lily Wei commented on DERBY-4741: - #syncFile - while waiting a bit before retrying a failed sync call Should the comment change a little in turn of the "does not matter..."? I run suites.all against ibm 1.6 jvm along with the LockInterruptTest.java (derby-4967-locking-2.diff) change, test run clean. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981932#action_12981932 ] Dag H. Wanvik commented on DERBY-4741: -- Some details on where patch derby-4741-sleeps-waits-1 changes things. The following locations where we earlier threw 08000 (CONN_INTERRUPT), we now just note that an interrupt occurred: * GenericStatement#prepare: - while waiting for another thread to complete the compilation of the statement * LogToFile #checkpointWithTran: - while waiting for another thread to complete a checkpoint #switchLogFile: - while waiting for a semaphore needed to make sure the log switch can complete "without having to give up the semaphore to a backup or another flusher". #flush: - while waiting for the database to stop being frozen - while waiting for another thread flushing #syncFile - while waiting a bit before retrying a failed sync call #startLogBackup - while waiting for a thread doing checkpointing before we start backup of log as part of online-backup The following locations previously silently swallowed the interrupt, we now (also) note that an interrupt occurred: * LogToFile #recover: - in slave replication mode, while waiting for fail-over. This code swallowed the interrupt #getLogFileAtBeginning: - in slave replication mode, while waiting for SlaveFactory to tell the thread to do recovery. This code swallowed the interrupt > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980984#action_12980984 ] Dag H. Wanvik commented on DERBY-4741: -- suites.All passed. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby-4741-sleeps-waits-1.diff, derby-4741-sleeps-waits-1.stat, derby.log, > derby.log, InterruptResilienceTest.java, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980189#action_12980189 ] Dag H. Wanvik commented on DERBY-4741: -- Committed a followup patch as svn 1057733. Quoting the commit entry: > Followup to patch derby-4741-b-04-nio committed as svn 1038440: > changed the string internal exception string "nospc.U" to "intrp.U". > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980163#action_12980163 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks, Mike. Logged and fixed DERBY-4963 to revert NIO version of RAF#sync. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977862#action_12977862
]
Dag H. Wanvik commented on DERBY-4741:
--
I find that explicit file sync (i.e. not "write sync") is only used if:
a) NIO is not supported (cf. WritableStorageFactory#supportsWriteSync), in
which case we never use FileChannel#force
b) derby.storage.fileSyncTransactionLog is set to true
c) On platforms/VMs that don't support opening a file with mask "rws" (write
sync with metadata) and "rwd" (write sync without metadata), cf. check in
LogToFile#openLogFileInWriteMode which calls checkJvmSyncError to determine
this. The platforms mentioned are (e.g. early versions of 1.4.2 and 1.5 on Mac
OS and FreeBSD).
If this holds, it seems to be we can just skip using the NIO version of
RAF#sync for those platforms. Other platforms do not use the explicit file sync
for the log, cf. the test in LogToFile#flush (snip):
if (isWriteSynced)
{
//LogAccessFile.flushDirtyBuffers() will allow only one write
//sync at a time, flush requests will get queued
logOut.flushDirtyBuffers();
}
else
{
if (!logNotSynced)
logOut.syncLogAccessFile();
}
On most platforms, isWriteSynced is true (cf above), meaning syncLogAccessFile
is not used, so reverting to a interrupt safe, less performant non-NIO sync in
syncLogAccessFile seems acceptable to me. Opinions?
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-a-01-api-interruptstatus.diff,
> derby-4741-a-01-api-interruptstatus.stat,
> derby-4741-a-02-api-interruptstatus.diff,
> derby-4741-a-02-api-interruptstatus.stat,
> derby-4741-a-03-api-interruptstatus.diff,
> derby-4741-a-03-api-interruptstatus.stat,
> derby-4741-a-04-api-interruptstatus.diff,
> derby-4741-a-04-api-interruptstatus.stat,
> derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat,
> derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff,
> derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat,
> derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff,
> derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff,
> derby-4741-nio-container+log+waits+locks+throws.diff,
> derby-4741-nio-container+log+waits+locks+throws.stat,
> derby-4741-nio-container+log+waits+locks-2.diff,
> derby-4741-nio-container+log+waits+locks-2.stat,
> derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java,
> xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an inconsistent state
> and we therefore have to return a database level error. This means the
> applications can no longer access the database without a shutdown and reboot
> including a recovery.
> It would be nice if Derby could somehow detect and finish IO operations
> underway when thread interrupts happen before passing the exception on to the
> application. Derby embedded is sometimes embedded in applications that use
> Thread.interrupt to stop threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977533#action_12977533 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for your thoughts on this, Knut. Here is another issue: The NIO Channel#force call is used to implement sync when we flush the Derby log, cf. this call: LogToFile#flush -> LogAccessFile#syncLogAccessFile -> StorageRandomAccessFile#sync -> getChannel().force(false) The "false" here allows the skipping of flushing metadata, which makes this faster (at least on Solaris) than the non-NIO implementation of StorageRandomAccessFile#sync, which uses getFD().sync(). Unfortunately, an interrupt will make the getChannel().force call throw ClosedChannelException, and I can't see an easy way to recover from this, except assuming the log is corrupt and shutting down the database instance, so we can recover on reboot. If we switch back to using "getFD().sync()", we would lose the performance optimization, but be resilient to interrupts. So, unless somebody can see a way to finesse this problem, should be offer a knob to allow a user to trade this performance optimization for interrupt resilience? Aside: we also use StorageRandomAccessFile#sync in many other places, but in those situations I find I am able to recover by closing the RAF, reopening and writing over again, since the data to be written is known, e.g. in LogToFile#initLogFile. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969711#action_12969711 ] Knut Anders Hatlen commented on DERBY-4741: --- > So the question becomes, would it be sufficient to advise users to run with > this switch set, or should we make Derby tolerate > the interrupts on Solaris here also? How much code do you think will be needed to handle that case? Handling interrupts sent to the engine is somewhat of an edge case in itself. Since this particular problem only affects one platform, and that platform will start behaving the same way as the other platforms in the not so distant future, it sounds acceptable to just advise those who come across the problem to set the flag. Of course it would be nice if Derby was well-behaved by default on all platforms, but given the easy workaround and the expectation that the problem will soon go away by itself, I'd say that it's probably not worth putting a lot of work into and/or adding more complexity to already complex code. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969576#action_12969576 ] Dag H. Wanvik commented on DERBY-4741: -- Committed derby-4741-kristians-01 as svn 1043802 (suitesAll ran ok). > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969530#action_12969530 ] Dag H. Wanvik commented on DERBY-4741: -- While I am looking for the Heisenbug in DERBY-4920, I am considering the next increment of this patch from the prototype patch regarding the issues seen when interrupt hits the logging machinery (mostly not NIO, except for the case of DirRandomAccessFile#sync). Now, it turns out that the vulnerability here only applies for Solaris. Other platforms' "non-NIO" IO is not interruptible, so for those other platforms there is no need to do anything special. For Solaris, the behavior can be tweaked to be impervious to interrupts as well using the special switch: -XX:-UseVMInterruptibleIO on JVMs >= 1.5.0_12. This behavior which will be default in 1.7 also on Solaris[1]. So the question becomes, would it be sufficient to advise users to run with this switch set, or should we make Derby tolerate the interrupts on Solaris here also? I do think we should be able to run correctly with the default settings on all relevant platforms, but I would appricate your opinions. [1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4385444 > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969252#action_12969252 ] Kristian Waagan commented on DERBY-4741: Thanks, Dag. > I guess I have tended to treat them the same. I'll try to improve ;-) Not sure there is something to improve here, I guess this is a question about personal preference. Explicit initialization of instance variables used to result in two assignments (first the default value for the type, and then the explicit value), which is kind of unnecessary if both assignments use the same value. I don't know if this is still the case. And I have become quite accustomed to use the "Step over" or "Run to cursor" in the debugger :) Finally, if the initial value isn't the default for the data type, then you have to assign the correct value anyway - either in the constructor or as an instance variable initializer (i.e. private int i = 3). Thanks for working on the interrupt issue! I think it is of great value for Derby to be able to handle interrupts better. This is especially important when/if other software products used together with Derby use Thread.interrupt() internally. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, derby-4741-kristians-01.diff, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969019#action_12969019
]
Dag H. Wanvik commented on DERBY-4741:
--
Thanks for looking at the new code, Kristian, I'll make a new patch to cater
for your comments.
> o in writeFull the variables beforeOpen and beforeInterrupt are unused. Can
> they be removed?
> o the same issue in readFull.
Absolutely, debugging cruft. Thanks for spotting those.
> o would it make sense to make the monitor object channelCleanupMonitor final?
Absolutely.
> o redundant call to Thread.currentThread when invoking holdsLock.
Thanks, thought I had cleaned out those, but there were two left!
> o missing JavaDoc for recoverContainerAfterInterrupt
Added Javadoc for recoverContainerAfterInterrupt and awaitRestoreChannel.
> A more theoretical question at the end. Is there a reason to
> explicitly define instance variables with the default value for the
> data type? I.e., int i = 0, boolean b = false.
No. I tend to often use explicit initialization due to my paranoid
streak I guess. I see the data flow analysis barks if you use a local
variable uninitialized, although not for a class member, not clear to
me why Java differentiates here.
public class Foo {
public boolean bb;
public static void main(String[] args) {
boolean b;
if (args.length > 0 && args[1].equals("foo")) {
b = true;
}
if (b) { // compiler barks
System.err.println("b is true");
}
Foo f = new Foo();
if (f.bb) { // OK
System.err.println("bb is true");
}
}
}
I guess I have tended to treat them the same. I'll try to improve ;-)
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-a-01-api-interruptstatus.diff,
> derby-4741-a-01-api-interruptstatus.stat,
> derby-4741-a-02-api-interruptstatus.diff,
> derby-4741-a-02-api-interruptstatus.stat,
> derby-4741-a-03-api-interruptstatus.diff,
> derby-4741-a-03-api-interruptstatus.stat,
> derby-4741-a-04-api-interruptstatus.diff,
> derby-4741-a-04-api-interruptstatus.stat,
> derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat,
> derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff,
> derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat,
> derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff,
> derby-4741-c-01-nio.stat,
> derby-4741-nio-container+log+waits+locks+throws.diff,
> derby-4741-nio-container+log+waits+locks+throws.stat,
> derby-4741-nio-container+log+waits+locks-2.diff,
> derby-4741-nio-container+log+waits+locks-2.stat,
> derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java,
> xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an inconsistent state
> and we therefore have to return a database level error. This means the
> applications can no longer access the database without a shutdown and reboot
> including a recovery.
> It would be nice if Derby could somehow detect and finish IO operations
> underway when thread interrupts happen before passing the exception on to the
> application. Derby embedded is sometimes embedded in applications that use
> Thread.interrupt to stop threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968724#action_12968724 ] Kristian Waagan commented on DERBY-4741: Happened to look at parts of the new code, and have a few questions/comments on code in RAF4Container: o in writeFull the variables beforeOpen and beforeInterrupt are unused. Can they be removed? o the same issue in readFull. o would it make sense to make the monitor object channelCleanupMonitor final? o redundant call to Thread.currentThread when invoking holdsLock. o missing JavaDoc for recoverContainerAfterInterrupt A more theoretical question at the end. Is there a reason to explicitly define instance variables with the default value for the data type? I.e., int i = 0, boolean b = false. Just asking, as I find it annoying to step through these in the debugger :) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964723#action_12964723 ] Dag H. Wanvik commented on DERBY-4741: -- Committed derby-4741-c-01-nio as svn 1040086. Please let me know if you still see the hang of DERBY-4920 and, if possible, during execution which test it happens. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, derby-4741-c-01-nio.diff, > derby-4741-c-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935172#action_12935172 ] Dag H. Wanvik commented on DERBY-4741: -- Committed b-04 as svn 1038440. Note: I changed the debug trace flag to "RAF4" to accommodate an exisiting trace not related to recovery. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-b-04-nio.diff, derby-4741-b-04-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934082#action_12934082 ] Dag H. Wanvik commented on DERBY-4741: -- HI Lily! I'll commit it soon after I have verified a debug trace for observing the effects of the giveUpIO logic. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933924#action_12933924 ] Lily Wei commented on DERBY-4741: - Hi Dag: Thank you so much for all the detail message. The code looks good to me. I don't see potential issue. When I run insane build, I got 5.8 second for MicroAPITest and 51.x second for sane build vs 5.07s(insane) and 47.xs(sane) build on clean trunk. It could be just Windows platform issue. +1 to check in to trunk. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933598#action_12933598 ] Dag H. Wanvik commented on DERBY-4741: -- For reviewers of b-03, it may be interesting note that running InterruptResilienceTest with the flag -Dderby.debug.true=RAF4Recovery will likely give one or more instances of this trace in derby.log: DEBUG RAF4Recovery OUTPUT: derby.rawStoreDaemon thread needs to wait for container recovery: 0 showing that this part of the multi threading synchronization is being exercised: The user that was interrupted is doing recovery on the channel so the thread printing this (i.e. RawStoreDaemon) has to wait for recovery to complete to be able to move on. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933589#action_12933589 ] Dag H. Wanvik commented on DERBY-4741: -- I have now run regressions with patch b-03 on Windows Vista SP2 without errors. Since the 10.7 branch has been cut, I intend to commit this patch soon, and move on the the next piece of the puzzle. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932150#action_12932150 ] Dag H. Wanvik commented on DERBY-4741: -- Lily, it seems you reintroduced some of the logic I folded into awaitRestoreChannel, but without throwing to get retry for the case pageNumber == 1 (throw INTERRUPT_DETECTED). I don't think this is correct, not sure why this made a difference. Can you try the latest patch on Windows 7? :) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, derby-4741-b-03-nio.diff, derby-4741-b-03-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931984#action_12931984
]
Lily Wei commented on DERBY-4741:
-
Add the following code in ClosedChannelException block in
RAFContainer4.java->readPage and did not see any hang for Suites.All in Windows
7 enviornment:
if (pageNumber != -1L) {
// Recovery is in progress, wait for another interrupted
thread
// to clean up.
awaitRestoreChannel(e, stealthMode);
} else {
// Called from this.getEmbryonicPage.
// Retry handled in RAFContainer#clean. This is
// necessary since recovery needs the monitor on "this"
// held by clean when it calls getEmbryonicPage via
// writeRAFHeader, so never call awaitRestoreChannel
// here, since then might be stuck on
// channelCleanupMonitor held by another thread
// cleaning up, while still hold onto "this".
if (stealthMode) {
// don't touch threadsInPageIO
} else {
synchronized (channelCleanupMonitor) {
threadsInPageIO--;
}
}
}
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-a-01-api-interruptstatus.diff,
> derby-4741-a-01-api-interruptstatus.stat,
> derby-4741-a-02-api-interruptstatus.diff,
> derby-4741-a-02-api-interruptstatus.stat,
> derby-4741-a-03-api-interruptstatus.diff,
> derby-4741-a-03-api-interruptstatus.stat,
> derby-4741-a-04-api-interruptstatus.diff,
> derby-4741-a-04-api-interruptstatus.stat,
> derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat,
> derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff,
> derby-4741-b-02-nio.stat,
> derby-4741-nio-container+log+waits+locks+throws.diff,
> derby-4741-nio-container+log+waits+locks+throws.stat,
> derby-4741-nio-container+log+waits+locks-2.diff,
> derby-4741-nio-container+log+waits+locks-2.stat,
> derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java,
> xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an inconsistent state
> and we therefore have to return a database level error. This means the
> applications can no longer access the database without a shutdown and reboot
> including a recovery.
> It would be nice if Derby could somehow detect and finish IO operations
> underway when thread interrupts happen before passing the exception on to the
> application. Derby embedded is sometimes embedded in applications that use
> Thread.interrupt to stop threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931535#action_12931535 ] Lily Wei commented on DERBY-4741: - This might sound wired. I am so happy you got this. I woke up to a hanging test for Suites.All and now I know why. Thank you so much for posting this and looking into this. :) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931526#action_12931526 ] Dag H. Wanvik commented on DERBY-4741: -- Found that Derby uses interrupt to stop threads at shutdown. The present logic in RAFContainer4 doesn't take this into account, with the consequence that an application thread got stuck waiting for a container recovery that never took place. We get an exception during execution of closeContainer(); openContainer(currentIdentity); If this exception is not due to an interrupt, we throw FILE_IO_INTERRUPTED. Unfortunately, in this case, the present patch neglects to release any threads waiting for recovery to finish, so they are stuck waiting for "restoreChannelInProgress". Saw this during suitesAll. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, derby-4741-b-02-nio.diff, > derby-4741-b-02-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931263#action_12931263 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for the review, Knut. Uploading a new version of the patch, derby-4741-b-02-nio, details below, rerunning regressions. - In RAFContainer4, folded the cases for pageNumber == -1 (call from getEmbryonicPage) with the case Thread.holdsLock(this). This allows awaitRestoreChannel to throw the retry exception for all cases. I did both static and dynamic analysis to establish invariant, and added a sane check for it. > As far as I can see, the code will behave the same way as before if no > interrupt is detected. So if there are problems with the code, they > will hopefully be limited to the case where the thread is > interrupted. There's an additional container-wide synchronization on > every page read/write, though, which may possibly have a negative > effect on performance in multi-threaded environments. If that turns > out to be an issue, would it be possible to change the code to only > check restoreChannelInProgress after an exception has been thrown, > similar to what we currently do in stealth mode? Yes, I believe we could do that, since it should do no harm to attempt the IO even when recovery is in progress, because all calls to getChannel are synchronized on "this" already, so either the thread sees the old channel (closed), or the new reopened channel. I didn't make this change yet, though, since the monitor hold should short-lived (one boolean check and an integer increment) compared to the IO. - Thread.holdsLock(): done - Created a new subclass of StandardException: InterruptDetectedException, good suggestion, indeed cleaner! - Javadoc for RAFContainer4.readPage: done - The variable "whence": removed - Removed some commented out debugging cruft - Tuned the number of iterations in InterruptDetectedException to make sure we see a a concurrent thread (RawStoreDaemon) having to wait for cleanup before proceeding, at least on my box. Cf. the debug trace for derby.debug.true=RAF4Recovery, which I also added. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931137#action_12931137 ] Knut Anders Hatlen commented on DERBY-4741: --- And a note to others who consider reviewing the patch: I found that it was easier to see exactly what was being changed by running svn diff with the extra option "--extensions=-bw" to disregard whitespace changes in the parts of the code where an extra level of nesting was added. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931129#action_12931129 ] Knut Anders Hatlen commented on DERBY-4741: --- I had a look at the b-01-nio patch, and it looked correct to me. The stealth mode part wasn't entirely obvious, but the comments were helpful and the code appears to be doing the right thing. As far as I can see, the code will behave the same way as before if no interrupt is detected. So if there are problems with the code, they will hopefully be limited to the case where the thread is interrupted. There's an additional container-wide synchronization on every page read/write, though, which may possibly have a negative effect on performance in multi-threaded environments. If that turns out to be an issue, would it be possible to change the code to only check restoreChannelInProgress after an exception has been thrown, similar to what we currently do in stealth mode? Some minor issues found in the patch: 1) Thread.holdsLock() is a static method, but all calls to it are done on Thread.currentThread(). 2) I think the checks for SQLState.INTERRUPT_DETECTED should use getMessageId() instead of getSQLState(), since the SQLState interface, despite its name, contains message ids and not SQLStates (although the message id and the SQLState happen to be the same for this error). And since INTERRUPT_DETECTED is only intended to be used internally, would it make sense, instead of giving INTERRUPT_DETECTED a proper SQLState and also an entry in messages.xml (which will be picked up and translated unnecessarily later), to create a subclass of StandardException for this error? Something like what's done with NoSpaceOnPage, I mean. That could also make the catch blocks used to handle this condition a little bit tidier. 3) Javadoc for RAFContainer4.readPage() should explain how the offset parameter is supposed to be used (-1 for normal reads, actual offset for embryonic pages) 4) The variable "whence" in awaitRestoreChannel() does not seem to be used. Nor is its "pageNumber" parameter, I think. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930913#action_12930913 ] Lily Wei commented on DERBY-4741: - For testing idea, SanityManager + a debug flag sound good. I am trying to picture how it can cover all the code we try to hit. I am assuming we want to hit code like RAFContainer4.readpage. The simple insert case to Blob and Clob data type can be hitting it like the InterruptResilienceTest does. Will BlobSetMethodTest or LobStreamTest do the trick? A far stretch will be utilizing CorruptRandomAccessFile. I am speaking out loud here. :) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930704#action_12930704 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks, Lily, for looking at the latest patch. I agree the code is a bit tricky, so there could be bugs. Feel free to ask detailed questions, I'll try to answer. I did not see any issue with testPartialRowRTStats when I ran the regressions, nor would I expect this patch for increase perm gen space since it doesn't add new classes. But we have seen that perm gen space can be an issue when running regressions. I think I recently upped my pergen space setting for suitesAll to 192m. I have also seen failures in the replication tests, there are some instabilities left, but I don't think they are due to this patch. You are right that the current test, InterruptResilienceTest, is pretty thin right now, but it does exercise the RAF recovery machinery with two concurrent threads (user thread and the raw store daemon). I intend to make more tests; the trick is to make sure the tests actually exercise the code paths we want: just shooting interrupts at threads can land anywhere during execution, and Derby doesn't stand up to it in all places yet. I plan to use the SanityManager + a debug flag to simulate interrupt at selected places in the code. If you have suggestions for good tests, please let's have them :) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-b-01-nio.diff, derby-4741-b-01-nio.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, InterruptResilienceTest.java, MicroAPITest.java, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927970#action_12927970 ] Dag H. Wanvik commented on DERBY-4741: -- Committed derby-4741-a-04-api-interruptstatus as svn 1030630. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927866#action_12927866 ] Lily Wei commented on DERBY-4741: - I test drive a-04-api patch as well. The sane MicroAPITest result is very good. I got result that is 5s - 7s. The insane result is similar to a-03-api which is around 45s--52s vs 27s-28s on clean trunk. However, like we discussed previous for MicroAPITest benchmark is only focus on CPU result and the slowdown is mostly on ASSERT. With so many things to take care of with 10.7 release. +1 to commit. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927797#action_12927797 ] Knut Anders Hatlen commented on DERBY-4741: --- Thanks for the changes and the explanations, Dag. I have no further comments to the a-04-api patch. One observation: After looking at the changes in EmbedBlob.getBytes(long,int) and EmbedBlob.position(), I understand why some people advocate that all methods should have a single exit point. ;) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927660#action_12927660 ] Dag H. Wanvik commented on DERBY-4741: -- Regressions ran ok. Will commit this version soon, if no misgivings emerge. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-a-04-api-interruptstatus.diff, > derby-4741-a-04-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927474#action_12927474 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for looking at the patch, Knut! > Except perhaps in EmbedConnection. It looks to me like most of the calls to > restoreIntrFlagIfSeen() happen within try blocks right after calls to > setupContextStack(). Did you consider it as an option to move the > restoreIntrFlagIfSeen() calls into restoreContextStack(), which is already > called in the finally clauses at those places? I did try to put the call into the finally,although not into restoreContextStack. I found that didn't always work, since during error handling *before* we get to the finally block, calls to handleException(throwable) will depending on severity, potentially take down the connection (and database), causing us to lose the lcc. Hence I had to move it back to just before normal return, and use the no-args variant of restoreIntrFlagIfSeen handling inside the error handling code (cf changes in TransactionResourceImpl#handleException). > - throwIf: Unnecessary return statement. Agreed, will fix that :) > - stackTrace: The initialValue() override does the same as the default > implementation. So by just using a vanilla ThreadLocal object, we'd get the > exact same behaviour, and one less class in derby.jar + permgen. Agreed. > (If the footprint issue is important, the same might be said for > receivedInterruptDuringIO if we change its usage from a TRUE/FALSE check to a > null/non-null check. Actually, the presence of a stack trace is probably a > good enough indicator in itself.) Yes and yes. Originally I used the stacktrace just for debugging with sane, I think. > - Instead of creating an exception, fetching its stack trace and storing the > trace, perhaps it would be easier just to store the exception? Then throwIf() > could throw the stored exception instead of creating a new one and modifying > its stack trace. That's good simplication, yes. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927445#action_12927445 ] Dag H. Wanvik commented on DERBY-4741: -- Knut is right, that was the expensive assert I mentioned before (in sane builds). Since this is a micro benchmark that mainly measures the CPU neeeded to execute ResultSet.next when data is cached in memory, the effect of the assert shows up heavily. Running a profiling I saw the string concatenation there being dominant. No, I did not notice the regression test become slower, but if the turns out to be an issue, it is easy enough to comment out the asset. For now, I'll change the assert code to just call getContextOrNull once. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927441#action_12927441 ] Knut Anders Hatlen commented on DERBY-4741: --- Hi Dag, I went through the a-03-api patch, and I couldn't spot anything that you've missed. There were quite a number of places to check the interrupt status. More than what I had expected. But I cannot think of a better place to short-circuit it, so... Except perhaps in EmbedConnection. It looks to me like most of the calls to restoreIntrFlagIfSeen() happen within try blocks right after calls to setupContextStack(). Did you consider it as an option to move the restoreIntrFlagIfSeen() calls into restoreContextStack(), which is already called in the finally clauses at those places? That may make the code more contained, but perhaps the LCC isn't easily available there? Apart from that, I only have some minor comments to the code in the InterruptStatus class. Very minor, so feel free to ignore. - throwIf: Unnecessary return statement. - stackTrace: The initialValue() override does the same as the default implementation. So by just using a vanilla ThreadLocal object, we'd get the exact same behaviour, and one less class in derby.jar + permgen. (If the footprint issue is important, the same might be said for receivedInterruptDuringIO if we change its usage from a TRUE/FALSE check to a null/non-null check. Actually, the presence of a stack trace is probably a good enough indicator in itself.) - Instead of creating an exception, fetching its stack trace and storing the trace, perhaps it would be easier just to store the exception? Then throwIf() could throw the stored exception instead of creating a new one and modifying its stack trace. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927345#action_12927345 ] Knut Anders Hatlen commented on DERBY-4741: --- The sane jars are not intended for production use, so I wouldn't worry too much about a performance loss that we only see with them. Unless it's getting so slow that it takes an unreasonably long time to run the regression tests with them, that is. I'm guessing that the extra performance loss seen with sane jars is caused by the assert in InterruptStatus.restoreIntrFlagIfSeen(LanguageConnectionContext), which makes us look up the lcc in the context stack. Twice, actually. Perhaps it would help a little to store the result from getContextOrNull() in a local variable so that we get away with a single call. Or perhaps it's the unconditional string concatenation for the message text that's costly. In any case, that assert is the only place I can see that the patch would have a different performance impact on sane and insane. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927284#action_12927284 ] Lily Wei commented on DERBY-4741: - Hi Dag: That is interesting. I was using sane build. Shouldn't sane build have similar performance as insane or they are usually having difference? > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927266#action_12927266 ] Dag H. Wanvik commented on DERBY-4741: -- Lily, did you with a sane or insane build? I see a large difference also (Solaris though) with sane (due to an expensive assert), but with insane is it very small (~<1%). > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926998#action_12926998 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for test driving it, Lily. The new patch does use the lcc if available, it just falls back on the TL approach if not, so the performance drop was unexpected to me. I'll have a look, thanks for noticing. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926841#action_12926841 ] Lily Wei commented on DERBY-4741: - When running MicroAPITest against a-03 patch, it is 45-48 seconds vs clean trunk 27-28 second. I noticed it is not using lcc but the earlier slower approach. Are you seeing the same slow windows performance again a-03 patch. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-a-02-api-interruptstatus.diff, > derby-4741-a-02-api-interruptstatus.stat, > derby-4741-a-03-api-interruptstatus.diff, > derby-4741-a-03-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924571#action_12924571 ] Dag H. Wanvik commented on DERBY-4741: -- Hi Lily, thanks for test driving the patch :) I really appreciate your help looking at this work. Having another set of eyes on this is good, the change are a bit cross cutting and affect much existing code, so it's good to have someone double checking the changes. Two comments: - if the mailjdbc test still deadlocks, I doubt it can be due to this patch, since it doesn't really change any behavior, it just installs the machinery for the behavior changes to come in later patches, as you presumed. - the difference is EmbedBlob with local initial null is just some cruft from an experiment I did with trying to place restoreIntrFlagIfSeen in a finally block. This didn't work out since, error handling may close the connection and with it the lcc, so the check would happen too late. (The current method is to check before return and in the error handling code itself). I'll revert that change, thanks for noticing. My main concern with this patch currently is to verify that I have found all the ways control can return to the user app after Derby code has had control, and possibly set the interrupt status flag. A case in point is the Blob methods that don't require a connection, e.g. EmbedBlob#getBytes if the lob is materialized, we call control.read: if we see an interrupt inside the read, the flag would get stored in a thread local variable, since there is no lcc, so I'll I need to add a call to no-args version of restoreIntrFlagIfSeen() there. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924451#action_12924451 ] Lily Wei commented on DERBY-4741: - Hi Dag: Thank you so much for the write out. All a, b, c, d points are very clear and straight forward. Personally, I like the fact we are checking interrupt at executeBatch, executeQuery, BasicNoPutResultSetImpl(same as timeout) and waiting for lock area. Thank you for add the synchronized (slaveRecoveryMonitor) and adding all the necessary release notes information. I apply the derby-4741-a-01-api-interruptstatus. All the suites.all passed and timing for MicroAPITest is close to clean trunk. It also passed Derby151Test with interrupt check. Mailjdbc is still having deadlock issue. However, like you said, it might or might not cause by the new change. I notice some of the interrupt code is in slight different places in Embed*.java with local initial null value. And, some of interrupt handling code are not in this patch than the previous patch. I am assuming those code are handling machinery of InterruptStatus that save interrupts and it will come in later patch. I view this patch as it will benefit a lot of customers and love to help out if I can. Thank you so much for working so hard on this. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-a-01-api-interruptstatus.diff, > derby-4741-a-01-api-interruptstatus.stat, > derby-4741-all+lenient+resurrect.diff, derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923678#action_12923678 ] Dag H. Wanvik commented on DERBY-4741: -- Hei Lily, thanks for your continued interest in this issue! > Will you elaborate on that in turn of the new patch behavior please? My present thinking is that the goal should be: a) the best user experience will be that for operations (technically JDBC API calls), which detect an interrupt, but manage to retry and continue the operation, and for which we do not cut the operation short, because there is no natural point at which to interrupt it, or we deem it to be so short in duration that we might as well see it through, we: - do not throw an exception - restore the interruped status flag just prior to return from the operation (JDBC API call) b) for operations that we actually cut short due to the interrupt, see current enumeration below, we - throw SQLException 08000, session level (we might change this severity later as discussed) - also resurrect the interrupt status flag just prior to return from the operation (JDBC API call) c) for operations, which detect an interrupt, but manage to retry and continue the operation, but eventually throw an SQLException (not 08000, some other), we do as in a): - restore the interruped status flag just prior to return from the operation (JDBC API call) d) Operations that we may currently cut short if we see an interrupt early enough: - executeBatch (we check between each statement) - executeQuery (we check in the present BasicNoPutResultSetImpl#checkCancellationFlag in the same location that we check for timeout currently) - while waiting for a lock, cf. LockSet.java, ConcurrentLockSet.java Further suggestions for good places to check and throw are welcome :) > .. synchronized (slaveRecoveryMonitor): And, I add it to my client > space and run some tests. It seems not harming anything. Good, we'll add it then. > Will we add a note somewhere about the code not cover for this > interrupt handling? For the time being, we can make notes in this JIRA issue, but eventually I suspect we will want to make a release note to describe the improvements, and what remains to be done. You are right that there is a lot of work, it is also hard to test, unfortunately. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-all+lenient+resurrect.diff, > derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923457#action_12923457 ] Lily Wei commented on DERBY-4741: - I can confirm Knut's number for MicroAPITest .java in turn of 1-3% degradation or some case better with the derby-4741-all+lenient+resurrect.diff patch. Thanks for all the great work, Dag. A lot of customers will benefit from this. Wow!!! It seems like we will throw an error back to the caller indicate they are the one generating the interrupt and we will throw an error in the case of an interrupt during a lock wait with this new approach. Will you elaborate on that in turn of the new patch behavior please? I am still having trouble understand the data flow perspective and would like to see some write out in turn of what exact error I will see from Derby . This is one of the comments you wrote to me. Sorry for getting back to you promptly. >> Do we need InterruptException on synchronized (slaveRecoveryMonitor) in >> LogToFile.recover(...)? >I don't think I changed that part of the code. I see that an interrupt >is just ignored there, so I think we can safely add a call to >"InterruptStatus.setInterrupted" there as well, if that's what you >mean. Yes, this is what I mean. And, I add it to my client space and run some tests. It seems not harming anything. I know you are concentrating on "normal operation" and it is *really* a lot of work. Will we add a note somewhere about the code not cover for this interrupt handling? Such as: backup, export/import, LOB/CLOB streams and replication for during I/O in those areas as well. I run Suites.all on windows and it is looking better than ever!!! > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-all+lenient+resurrect.diff, > derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923367#action_12923367 ] Knut Anders Hatlen commented on DERBY-4741: --- I ran MicroAPITest with derby-4741-all+lenient+resurrect.diff (20 runs with each configuration, insane jars, pageCacheSize=12500). Now I only see a 1% degradation with the patch (p-value=0.07, according to the student t calculator at http://www.bio.miami.edu/rob/Students_t.html), so that definitely looks like a good improvement from the previous patch. I also modified the test so that it reused existing test data instead of creating it on every run. Then I didn't see any degradation. In fact, the results with the patch were 0.5% better than trunk when I ran the test that way. I also saw something similar with the previous patch. I saw an 8% degradation with the unmodified test, but the degradation was only about 2% with the modified test. This is also consistent with the findings Dag posted in the comment dated 15/Oct/10 12:06 PM, where he saw that the full patch gave a 2.5% degradation, whereas if he only applied the parts of the patch that were actually exercised during the data collection phase of the test, he didn't see any degradation. I don't know why the changes in the test setup code alters the result this way. The part of the test where we collect results should exercise the exact same code path with the two variants of the test. This reminds me of the effects we saw in DERBY-4233, which were attributed to JIT at that time. I think the theory at that time was something like this: The just-in-time compiler collects statistics before it compiles the code and uses the statistics to optimize the generated native code. The more time we spend in the test setup code, the more the statistics will be biased towards the setup code. For this test, that could mean that the JIT-compiled native code is optimized for the insert code path, whereas we're really only interested in the select code path in the test. Furthermore, since the patch changes code on the insert code path, the collected statistics and the chosen optimizations may change, and that may ultimately affect how the select code is optimized, even if the select code path barely has been touched by the patch. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-all+lenient+resurrect.diff, > derby-4741-all+lenient+resurrect.stat, > derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923238#action_12923238
]
Dag H. Wanvik commented on DERBY-4741:
--
Note to self: I have found a problem with the interrupt recovery of
RAFContainer4. Its call to openContainer needs to be protected by the
monitor on FileContainer#allocCache, because opening a container
evetually leads to a call on AllocationCache#reset. AllocCache javadoc
states that the the callers need to synchronize themselves since it is
itself not MT safe.
Threads inside RAFContainer4#{readPage, writePage} do not necessarily
own this monitor when recovery is attempted.
I did once see a race condition due to this. In the race condition, a
thread was trying to write to a new page and got a array out of bounds
exception inside AllocationCache.validate (numExtents was suddenly back to 0)
because another thread was doing interrupt recovery by calling
RAFContainer4#recoverContainerAfterInterrupt ->
openContainer -> ... -> AllocationCache#reset (unprotected).
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-all+lenient+resurrect.diff,
> derby-4741-all+lenient+resurrect.stat,
> derby-4741-nio-container+log+waits+locks+throws.diff,
> derby-4741-nio-container+log+waits+locks+throws.stat,
> derby-4741-nio-container+log+waits+locks-2.diff,
> derby-4741-nio-container+log+waits+locks-2.stat,
> derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an inconsistent state
> and we therefore have to return a database level error. This means the
> applications can no longer access the database without a shutdown and reboot
> including a recovery.
> It would be nice if Derby could somehow detect and finish IO operations
> underway when thread interrupts happen before passing the exception on to the
> application. Derby embedded is sometimes embedded in applications that use
> Thread.interrupt to stop threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922716#action_12922716 ] Dag H. Wanvik commented on DERBY-4741: -- I feel pretty confident the main approach will improve things by now, so I'll upload a final version of the experimental patch, and the start producing incremental patches for commit roughly in the following order: a) hooks in the APIs to resurrect interrupt flags before we return to the user's app on the basis of info collected during operation, the new util class InterruptStatus, which is a collection of methods to save away and resurrect interrupt flags, and throw an interrupt SQLexception. I intend to *not* throw any exception if the API method ran to completion, just resurrect the flag. b) the modified RAFContainer4 stuff for reading and writing db cache pages safely with NIO. c) the modified DirRandomAccessFile4 stuff + log retries for safe operation of log writing d) stop execution if intr seen within a batch, when checking query timeout, and when seeing intr while waiting for locks e) make undo safe f) time permitting, make boot and recovery interrupt safe. Please let me know if there any objections or suggestions to this plan! Whether all this gets ready for 10.7 is a possible issue, but hopefully we should not behave worse the before at any step. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922370#action_12922370 ] Dag H. Wanvik commented on DERBY-4741: -- And we are vulnerable during (re)boot: java.sql.SQLException: Failed to start database 'interrupttest' with class loader sun.misc.launcher$appclassloa...@2bbd86, see the next exception for details. : Caused by: java.sql.SQLException: Failed to start database 'interrupttest' with class loader sun.misc.launcher$appclassloa...@2bbd86, see the next exception for details. : Caused by: java.sql.SQLException: Java exception: ': java.io.InterruptedIOException'. : Caused by: java.io.InterruptedIOException at java.io.RandomAccessFile.readBytes(Native Method) at java.io.RandomAccessFile.read(RandomAccessFile.java:322) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) at org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(LogToFile.java:2693) at org.apache.derby.impl.store.raw.log.LogToFile.boot(LogToFile.java:3398) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:2020) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:333) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:572) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(BaseDataFileFactory.java:1747) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(BaseDataFileFactory.java:1235) at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:223) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:2020) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:333) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:572) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:2020) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:333) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:572) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:760) at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:193) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:2020) at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:333) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1857) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1723) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1601) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:1020) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2679) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.l
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922362#action_12922362 ] Dag H. Wanvik commented on DERBY-4741: -- The approach seems to work. This should take care of the performance worries for using InterruptStatus#throwIf in API methods. Sometimes there is no lcc, e.g when creating a new database; we could then fall back on the dedicated thread local variable, cf. during EmbedConnection constructor's call to createDatabase. Another note: undo is still vulnerable, e.g. an interrupt during rollback, usage of StorageRandomAccessFile in cf. Scan.java, cf. BEGIN SHUTDOWN ERROR STACK - ERROR XSLA3: Log Corrupted, has invalid data in the log stream. at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:279) at org.apache.derby.impl.store.raw.log.Scan.getNextRecord(Scan.java:216) at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:939) at org.apache.derby.impl.store.raw.xact.Xact.abort(Xact.java:949) at org.apache.derby.impl.store.raw.xact.XactContext.cleanupOnError(XactContext.java:119) at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(ContextManager.java:333) at org.apache.derby.impl.jdbc.TransactionResourceImpl.cleanupOnError(TransactionResourceImpl.java:419) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:337) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2277) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:81) at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1325) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(EmbedPreparedStatement.java:1683) at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(EmbedPreparedStatement.java:305) at InterruptTest$WorkerThread.run(InterruptTest.java:263) Caused by: java.io.InterruptedIOException at java.io.RandomAccessFile.read(Native Method) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:721) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:758) at org.apache.derby.impl.store.raw.log.Scan.getNextRecordBackward(Scan.java:394) at org.apache.derby.impl.store.raw.log.Scan.getNextRecord(Scan.java:204) (roll-back happens here due to the test trying to insert a duplicate after a commit has been interrupted) > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922096#action_12922096 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks for doing more measurements, Knut! It seems that even in the best case, the movePostion operation is negatively impacted with 1-2% due to the ThreadLocal access, which is not desirable. Knut mentioned that ContextService is already based on a ThreadLocal. I checked around a bit and found that we might be able to get rid of the InterruptStatus thread local check in the API methods in this way: If we were able to retrieve the lcc at the point where we detect the interrupts and save this fact in the lcc at that time, the cost of accessing a ThreadLocal variable would not be incurred unless AN INTERRUPT ACTUALLY HAPPENED. In the API methods, lcc is already available, and checking a boolean flag in lthe cc instead of calling InterruptStatus#throwIf (which accesses a thread local) would be much cheaper. I see we already sometimes do dig out the lcc in store level code, e.g. in store.access.DiskHashtable, in this way: LanguageConnectionContext lcc = (LanguageConnectionContext) ContextService.getContextOrNull( // this call accesses a ThreadLocal LanguageConnectionContext.CONTEXT_ID); I'll look into this approach a bit. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921987#action_12921987 ] Knut Anders Hatlen commented on DERBY-4741: --- Oops... Clean trunk wasn't 8.5% faster than the patched version. It was 7.8% faster. What I should have said, was that the patched version was 8.5% slower than clean trunk. I ran the micro benchmark through a profiler, but I didn't see anything there to explain a ~8% degradation. I did see more time spent in ThreadLocal.get() (no surprise there...), but only enough to explain one or two percent, roughly what Dag saw. I'll try to dig a little more. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921764#action_12921764 ] Lily Wei commented on DERBY-4741: - Thanks for the important finding. The result is hard on issues to keep Derby stand out from other Java database. I humbly believe Derby will keep on shining if performance can keep on improving. Do you have any suggestion in turn of fixing thread interrupts and keep performance no less than 3% slower than current clean trunk? > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921740#action_12921740 ] Knut Anders Hatlen commented on DERBY-4741: --- I've also tried the micro benchmark, but got somewhat different results. In 20 runs with a clean trunk and 20 runs with the derby-4741-nio-container+log+waits+locks+throws.diff patch in random order, clean trunk was on average 8.5% faster than the patched version (3.64 s/query vs 3.95 s/query). I ran the experiment with insane jars on OpenSolaris 2010.05 snv_134b and Java 1.6.0_18, and I had upped derby.storage.pageCacheSize to 12500. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921420#action_12921420 ] Dag H. Wanvik commented on DERBY-4741: -- Running a micro benchmark to test the effect for adding InterruptStatus.throwIf to the APIs were a bit inconclusive. With the latest patch, using the attached MicroAPITest.java (ca 6-7 runs), I saw a ca 2.5 & degradation of performance with just calls to ResultSet.next for a database that fits in memory. However, just adding the InterruptStatus.throwIf calls to the API *without* the rest of the patch seemed to have no significant effect, so perhaps there are other parts of the patch that had made the operation somewhat slower, I'll try to incrementally remove parts of the patch and see what I find. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, MicroAPITest.java, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921418#action_12921418 ] Dag H. Wanvik commented on DERBY-4741: -- Hi Lily, thanks again for looking at this :) > However, I run mailjdbc again the latest > derby-4741-nio-container+log+waits+locks+throws, the test run into > Error ERROR 40001: A lock could not be obtained due to a > deadlock. derby.log is attached for reference. Yes, I saw that such lock errors too after about 18 hours of so. I am not yet sure they have anything to do with the patch changes, but I can't rule it out. I don't know the test app well enough. If you have any insight here which would indicate that locks should not occur under normal operation, what would that be? We could try to run the patch without the changes I made to ActiveLock, ConcurrentLockSet and LockSet, and see if the changes there have something to do with any change of behaviour.. > Do we need > InterruptException on synchronized (slaveRecoveryMonitor) in > LogToFile.recover(...)? I don't think I changed that part of the code. I see that an interrupt is just ignored there, so I think we can safely add a call to "InterruptStatus.setInterrupted" there as well, if that's what you mean. Generally I have just been concentrating on "normal operation" yet, so far I haven't yet looked into things like replication, backup, export/import, or even LOB/CLOB streams yet. I suspect there may be issues with interrupts happening during I/O in those areas as well. > Will change interrupt exception to statement > level rather than session level require more code changes? Such as > code in impl/store/raw/xact/* I don't know that for sure, but I suspected it might, the cleanup on session termination just seemed safer and in line with what we already did for interrupt exceptions. More investigation needed. > I was just thinking statement level is more user friendly to > application users. Yes, I see that point. If we decide that's the way we would ultimately like to go, I think I will still defer that to a next increment (maybe a new JIRA issue?) of this work just to keep things simpler for now. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920998#action_12920998 ] Dag H. Wanvik commented on DERBY-4741: -- A concern with additing the "catch-all" check is performance for fast API operations, like ResultSet#next, which usually only just fetches the next prefetched row (no I/O). I'll run some micro benchmarks to study if adding the interrupt check is too expensive in such a case. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks+throws.diff, > derby-4741-nio-container+log+waits+locks+throws.stat, > derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920790#action_12920790 ] Lily Wei commented on DERBY-4741: - Thanks Dag for spending time to look at mailjdbc. Yes, it is normal and I see it on trunk without the patch too. However, I haven't see the deadlock with the patch on my Windows 7 machine. I had been running 16 hours. It will be great for Derby if the deadlock is not normal. I will run more to see. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920769#action_12920769 ] Dag H. Wanvik commented on DERBY-4741: -- Re mailjdbc: It seems "not switching from an empty log file" in derby.log is normal for this test, at least I see it on trunk without the patch. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920751#action_12920751 ] Dag H. Wanvik commented on DERBY-4741: -- Lily, I ran mailjdbc test with my latest patch on Windows Vista for about 20 hours; I saw deadlocks, but not the read-only container error. I did see several mysterious lines in derby.log, e.g. "not switching from an empty log file (870)" similar to what you had in your derby.log. I have no experience with this test, so I don't know if this is normal for this test or a sign that something is wrong; I'll try to run the test without the patch and compare. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks-2.diff, > derby-4741-nio-container+log+waits+locks-2.stat, > derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920248#action_12920248 ] Dag H. Wanvik commented on DERBY-4741: -- Lily, I looked at derby.log: The stack trace you see first: "ERROR 40XD1: Container was opened in read-only mode." seems to happen while Derby is opening the conglomerate in the normal fashion and finds for some reason that it (or the directory) is not writable. It is not obvious to me that this has to do with the patch, but it is not impossible, I suppose, since the patch does touch RAFContainer(4), e.g. by reopening the container if the channel has been closed by interrupts. But this seems to be a normal open, not a reopen called from RAFContainer4 (which is not on the stack). Cf. RAFContainer#run, action OPEN_CONTAINER_ACTION, ca line 1447. I'll see if I can run the test myself and reproduced the error. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920225#action_12920225 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks, Knut and Lily. Ok, good to know that there is an interrupt involved when mailjdbc fails. Lily, also please note that Derby151Test is its present form will probably change when I start to actually throw Derby exceptions when interrupts are detected. So far, they are mostly only noted in InterruptStatus (thread local) and normal operation continues. The latest patch has started to throw exceptions for interrupts while waiting for locks, though, cf. mods in LockSet.java/ConcurrentLockSet.java, since this seemed the obvious place to throw anyway. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[
https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920218#action_12920218
]
Lily Wei commented on DERBY-4741:
-
Hi Dag:
Thank you. I am so exciting to help out testing and hope I can do more.
Like Knut said, mailjdbc interrupts on Browse.java and more detail on
DERBY-3746 and DERBY-4166, the portion of the code on Browse.java that call
interrupt is:
//Checking whether Refresh thread is running
after doing Browse work
//If Refresh is not running, interrupt the
thread
if (ThreadUtils.isThreadRunning("Refresh
Thread")) {
MailJdbc.logAct.logMsg("
Refresh is running");
} else {
Refresh th = (Refresh) ThreadUtils
.getThread("Refresh
Thread");
th.interrupt();
}
Refresh thread and Brose thread are dealing with the same tables for
mailjdbc application. I am also reading Derby151Test to see whether we should
simplify the mailjdbc test to better fit testing this fix that will benefit a
lot of customers. Thanks again. By the way, the read-only error on Browse
thread and ArrayIndexOutOfBoundsException from backup thread is from running
mailjdbc with embedded server.
> Make Derby work reliably in the presence of thread interrupts
> -
>
> Key: DERBY-4741
> URL: https://issues.apache.org/jira/browse/DERBY-4741
> Project: Derby
> Issue Type: Bug
> Components: Store
>Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0,
> 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0
>Reporter: Dag H. Wanvik
>Assignee: Dag H. Wanvik
> Attachments: derby-4741-nio-container+log+waits+locks.diff,
> derby-4741-nio-container+log+waits+locks.stat,
> derby-4741-nio-container+log+waits.diff,
> derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff,
> derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff,
> derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat,
> derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat,
> derby.log, xsbt0.log.gz
>
>
> When not executing on a small device VM, Derby has been using the Java NIO
> classes java.nio.clannel.* for file io.
> If thread is interrupted while executing blocking IO operations in NIO, the
> ClosedByInterruptException will get thrown. Unfortunately, Derby isn't
> current architected to retry and complete such operations (before passing on
> the interrupt), so the Derby database can be left in an inconsistent state
> and we therefore have to return a database level error. This means the
> applications can no longer access the database without a shutdown and reboot
> including a recovery.
> It would be nice if Derby could somehow detect and finish IO operations
> underway when thread interrupts happen before passing the exception on to the
> application. Derby embedded is sometimes embedded in applications that use
> Thread.interrupt to stop threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920186#action_12920186 ] Knut Anders Hatlen commented on DERBY-4741: --- Hi Dag, There's some discussion about mailjdbc and interrupts on DERBY-3746. The comments there seem to indicate that the test sometimes interrupts the worker threads. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920177#action_12920177 ] Dag H. Wanvik commented on DERBY-4741: -- Hi Lily, thanks for test driving the patches :) Du you know if mailjdbc uses interrupts in any way? I'll try to run it myself, maybe some of the changes changed behavior even in the absence of interrupts. Btw, there are still many holes, I have found more places especially in the logging system that are vulnerable to interrupts, so I am still building out the experimental patch. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > derby.log, xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919805#action_12919805 ] Dag H. Wanvik commented on DERBY-4741: -- Thanks, Knut! I agree it's a bit weird that the JRE is a little schizophrenic about the interrupt flag. I think we should try to be consistent when we throw, no matter where/after which Java API call Derby detects it. If we do retain the interrupt flag, I think it adds to the argument for making the exception session-level, since the user would need to undertake some cleaning up anyway, or else the next Derby call would fail as well. Probably this would be the use case for connection pools. As to arguments for making the error statement level, I guess if one just wants to stop a run-away query, one would possibly prefer that just the statement failed. A thing to consider here is the new JDBC 4.1 method Connection#abort: This would be the future portable way of stopping a run-away transaction. It will also close the the connection. This patch passed regressions, for what it's worth. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919646#action_12919646 ] Knut Anders Hatlen commented on DERBY-4741: --- Thanks Dag. The proposed plan sounds good to me. As to what to do with the interrupt status after a retry, I'd be happy as long as we always either fail or preserve the interrupt status. I don't have any strong opinions on whether or not we should set the interrupt flag raising the exception. I see that the pre-NIO methods that may be interrupted (wait, sleep, join) will throw an exception and *clear* the interrupt status, whereas the interruptible NIO methods will throw an exception and *set* the interrupt status, so the precedence from the class library is ambiguous. I agree, though, that the user's intent is probably to stop the thread, and then preserving the interrupt status sounds reasonable in order to prevent that the thread just runs to the next blocking call and gets blocked there. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits+locks.diff, > derby-4741-nio-container+log+waits+locks.stat, > derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat, derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat, > xsbt0.log.gz > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919376#action_12919376 ] Dag H. Wanvik commented on DERBY-4741: -- Hi Mike, thanks for looking at this! > I was wondering if you have a high level > goal. Basically what should derby do when it encounters an > interrupt. I started off merely trying to make the new IO pieces stand up as well as the old I/O, but I guess my ambitions grew a bit as I found many things could bring Derby down.. My thinking so far is that Derby should: a) avoid having to shut down if a thread has been interrupted b) throw *something* when it finds the thread has been interrupted, preferably as soon as possible, but not before the code has arrived at a "safe place", so we can avoid Derby shutting down. > I think it should always throw some sort of error back to caller, > and would be nice if the error or the nested error indicated clearly > that it is being thrown because of a thread interrupt. Many times > when supporting this issue it takes awhile for the user to realize > that they are the ones generating the interrupt. Agreed. So far I have been thinking of just using the connection level error 08000, wrapping the original NIO channel exception, or InterruptedException. This should make it clear what has happened, and unwrapping the exception would show where Derby detected it. > I think the error should not be database level, and with your retry > on I/O and log it seems like we can avoid that. That's what I am trying to achieve, yes. > I am not sure if it should be session or statement level, I am > leaning to it being session level, but would like to see a > discussion. I know often users are doing the interrupt to stop a > thread. The existing code throws session level error 08000 already in many places, and I felt it's OK to require that the user reconnect when she has done something as drastic as interrupting the thread (?) > Ultimately do you plan on always reenabling the interrupt after > retrying or getting to a safe place? I am still pondering this question. I *think* that by the principle of least surprise to the user, we should set the interrupted flag of the thread again just before we throw the exception from the "safe place". Note: the exisiting code does not resurrect the thread's interrupted flag when it detects an interrupt of a wait and throws StandardException.interrupt. (wait would have cleared the flag). An imminent problem is where would the "safe place" be? It would be nice to *avoid* having to check in all JDBC API methods before we return if Derby has been interrupted during the API call, but I am not yet sure if I am able to determine conclusively which API code paths could lead to Derby being interrupted.. One approach would be to throw as "soon as possible" from a "safe place" on the stack above the method that saw the interrupt, but it may be hard to always determine where that would be in all cases. I am open to suggestions here :) The current experimental patch mostly doesn't throw yet (I didn't get that far) - it just makes a note that an interrupt was detected. Nor does it resurrect the interrupted flag when it does throw - since I was still torn on this. > We should definitely throw an error in the case of an interrupt > during a lock wait, this seems like the one valid case to send an > interrupt to derby if a user thinks it is waiting too > long. Hopefully you can code it to handle it, do processing, get to > safe spot and then throw an error indicating a real user interrupt > during a wait. Yes, I am trying to do exactly that, agreed. I also would like interrupts on queries to be detected no later that at the place where we check for query time-outs already, cf. BasicNoPutResultSetImpl#checkCancellationFlag. > Something to think about in the error throwing is the background > thread. What should we do if it encounters an interrupt. Throwing an > error there might shutdown the db, i am not sure what it does at top > level with an error. Right. A priori, I thought the most common use case would be user threads getting interrupted, but I'll look into that. One approach could be to shutdown, or possibly try to make it impervious and ignore all interrupts.. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log+waits.diff, > derby-4741-nio-container+log+waits.stat,
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918977#action_12918977 ] Dag H. Wanvik commented on DERBY-4741: -- Now, heading back to Solaris after having flushed out NIO issues on Windows, I see that Derby falls over when it gets interrupted during transaction abort (XSTB0), as a result of receving an interrupt during a log flush wait(LogToFile.java:4048). I saw this using InterruptedTest in the patch. The test tries to reconnect (reboot), but eventually I can no longer get a connection, because I hit a NullPointerException in BaseDataFileFactory.java:639 (containerCache empty??). Cf enclosed xsbt0.log. > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Bug > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik >Assignee: Dag H. Wanvik > Attachments: derby-4741-nio-container+log.diff, > derby-4741-nio-container+log.stat, derby-4741-nio-container-2.diff, > derby-4741-nio-container-2.log, derby-4741-nio-container-2.stat, > derby-4741-nio-container-2b.diff, derby-4741-nio-container-2b.stat > > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4741) Make Derby work reliably in the presence of thread interrupts
[ https://issues.apache.org/jira/browse/DERBY-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910067#action_12910067 ] Knut Anders Hatlen commented on DERBY-4741: --- Since Derby used to handle interrupts in a better way before we switched to NIO, would it make sense to change the issue type to bug and flag it as a regression? > Make Derby work reliably in the presence of thread interrupts > - > > Key: DERBY-4741 > URL: https://issues.apache.org/jira/browse/DERBY-4741 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.2.1.6, 10.2.2.0, 10.3.1.4, 10.3.2.1, 10.3.3.0, > 10.4.1.3, 10.4.2.0, 10.5.1.1, 10.5.2.0, 10.5.3.0, 10.6.1.0 >Reporter: Dag H. Wanvik > > When not executing on a small device VM, Derby has been using the Java NIO > classes java.nio.clannel.* for file io. > If thread is interrupted while executing blocking IO operations in NIO, the > ClosedByInterruptException will get thrown. Unfortunately, Derby isn't > current architected to retry and complete such operations (before passing on > the interrupt), so the Derby database can be left in an inconsistent state > and we therefore have to return a database level error. This means the > applications can no longer access the database without a shutdown and reboot > including a recovery. > It would be nice if Derby could somehow detect and finish IO operations > underway when thread interrupts happen before passing the exception on to the > application. Derby embedded is sometimes embedded in applications that use > Thread.interrupt to stop threads. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
