[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845754#action_12845754 ] Knut Anders Hatlen commented on DERBY-646: -- It seems like the fix for this issue went into 10.5. Could we close it now? > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.5.1.1 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689154#action_12689154 ] Knut Magne Solem commented on DERBY-646: You need the log for undo, but for committed transactions I can't see why you need to keep the log records for a non-durable in-memory database. It could be hard to do this because you are doing in-memory stuff on this level. I tried to just discard writing log records to virtualFiles, but rollback failed because it was flushing the log and reading undo records from the storage. So we need to keep some log in virtualFiles to get undo working :) > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689121#action_12689121 ] Kristian Waagan commented on DERBY-646: --- Thanks for testing the new feature, Knut Magne :) I'm currently working on a patch, where I'm considering using java.io.File. That should take care of any other OS that has a different style for paths as well. Feel free to log a bug in Jira. If you don't, I will tomorrow. Regarding memory footprint, you have to balance the memory overhead against performance. Setting the page cache to the minimum size will degrade performance in many configurations. The worst case I saw in my simple testing, was a drop from 28361 (pageCacheSize=1000) tps to 4059 (pageCacheSize=40) tps. This was a single record update test using a page size of 32 K. I'm not quite sure how you would go about discarding the log records, but you do need the UNDO log records to support transactions. Doing something in this area should be possible. I don't know the log system well enough to say something more, but maybe someone else does? > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[
https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1269#action_1269
]
Knut Magne Solem commented on DERBY-646:
Good to see something is happening in this field! Out of curiosity I tried this
out on Windows XP, but it failed in embedded mode when creating an in-memory
database.
The error was this assert:
ERROR XJ001: Java exception: 'ASSERT FAILED serviceName = memory:C:\Documents
and Settings\user\workspace\derby\MyDbTest;storageFactory.getCanonicalName() =
C:\Documents and Settings\user\workspace\derby\MyDbTest:
org.apache.derby.shared.common.sanity.AssertFailure'.
To make it work I altered line 310 in VFMemoryStorageFactory.normalizePath() to
allow windows-style paths. I changed it to this: "} else if (dir.charAt(0) !=
getSeparator() && !dir.toUpperCase().matches("[a-zA-Z]{1}:{1}{1}.*")){"
Appart from that i worked great.
To reduce memory footprint I guess you can reduce the cache to a minimum and
maybe discard flushed log records instead of writing them to memory?
Knut Magne,
(previous user knutmas)
> In-memory backend storage support
> -
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
>Affects Versions: 10.5.0.0
> Environment: All
>Reporter: Stephen Fitch
>Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff,
> derby-646-1a-raw-compiles.stat, derby-646-20090222.diff,
> derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff,
> derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff,
> derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt,
> svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring
> disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688272#action_12688272 ] Kristian Waagan commented on DERBY-646: --- Primer at http://wiki.apache.org/db-derby/InMemoryBackEndPrimer > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683074#action_12683074 ] Rick Hillegas commented on DERBY-646: - I don't think that we can say that this is a finished, full-fledged feature. However, I would like to see the release notes mention that this is an experimental feature which we'd like users to test-drive. We will need some kind of primer (perhaps a wiki page) and it would be great if the release notes could direct users to that primer for more information. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683063#action_12683063 ] Kathey Marsden commented on DERBY-646: -- I think that until this feature is documented and fully supported, it shouldn't be resolved and be included in the release notes. The separate Jiras could be subtasks of this one and then the parent issue resolved for 10.6. Just my opinion. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683062#action_12683062 ] Kristian Waagan commented on DERBY-646: --- The core functionality is in 10.5 already. I'd like to get the following into 10.5 as well (DERBY-4085): a) Add the functional test written by Cheng. b) Enable the new tests as part of a suite (i.e. suites.All and as the ant junit-core target). New features will be logged as separate Jiras and added to the trunk. I do plan to write a Wiki-page documenting the feature to some extent. And then I hope users will give us feedback so that we can consider adding more features for the next Derby feature release. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682899#action_12682899 ] Myrna van Lunteren commented on DERBY-646: -- So the initial commits are in 10.5. How can we proceed with this? New JIRA's for subsequent work? > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681638#action_12681638 ] Kristian Waagan commented on DERBY-646: --- Cheng, thank you for your continued interest :) Since the code branch is cut on the coming Monday, there isn't much more that will be done for the in-memory solution for the upcoming 10.5 release, except testing and stabilization. This leaves plenty of time for an opportunity if you want to contribute later on. Regarding testing, I think that is a very good idea. I have already started extracting the Mog test from your patch, and it runs fine with the current code. The unit tests aren't as easy to adopt, as they test somewhat different characteristics / functionality (for instance peek, and mark/reset) and rely partly on internal variables etc. However, it shouldn't be hard to use your work to write similar tests for the current implementation. I'm sorry I'm charging on without waiting for more feedback from you and other interested parties. The reason is merely my wish to get something committed in time for 10.5. This will allow interested users to more easily test out the feature and give the community valuable feedback before the next feature release. I'm restating my goal for the feature: A basic, working in-memory storage back end that can be used in scenarios where durability isn't required; o basic, it won't have functionality like save on shutdown / JVM exit, or automatic restore on boot. o working, but not tested to the same degree as the rest of the Derby code. o undocumented (in the user docs). > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681623#action_12681623 ] Cheng Che Chen commented on DERBY-646: -- Kristian, you might want to adapt the functional test that I had submitted for your own in-memory storage factory. That way, you can have your changes automatically tested to some extent after every build. If I understand correctly, you tested your initial implementation by manually changing the default storage factory to point to your implementation and then running the Derby unit tests. As for helping out with developing this feature further, I'll see what I can do. In the coming weeks, I don't see much free time to devote to this. So perhaps for the release after the imminent upcoming one, there'll be an opportunity. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681285#action_12681285 ] Kristian Waagan commented on DERBY-646: --- Thanks for the testing, Knut Anders. Seems like I messed up some of the path-handling logic (absolute vs relative) in the patch I committed. I'm looking into the problem now. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681257#action_12681257 ] Knut Anders Hatlen commented on DERBY-646: -- I came across one issue when I experimented with the in-memory backend. When I exit from ij, I see this exception: Exception in thread "main" java.lang.NullPointerException at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(BaseDataFileFactory.java:2684) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.removeStubs(BaseDataFileFactory.java:1591) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.stop(BaseDataFileFactory.java:499) at org.apache.derby.impl.services.monitor.TopService.stop(TopService.java:405) at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:349) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:239) at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:205) at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231) at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:207) at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:413) at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:246) at org.apache.derby.impl.tools.ij.Main.go(Main.java:210) at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:177) at org.apache.derby.impl.tools.ij.Main.main(Main.java:73) at org.apache.derby.tools.ij.main(ij.java:59) To reproduce, execute the following commands in ij: connect 'jdbc:derby:in-memory:mydb;create=true'; exit; Other than that, it seems to be working fine. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681006#action_12681006 ] Francois Orsini commented on DERBY-646: --- +1. Having these (hooks) even as an experiment could be very useful while the in-memory feature is being tested. I would probably wait after 10.5 because allowing for database deletion, but that is just me - just because there needs probably more time to investigate it and its possible side effects. Maybe worth re-enforcing somewhere (notes if any) that the database is locked and its activity blocked while its estimated space usage is computed. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store >Affects Versions: 10.5.0.0 > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Fix For: 10.5.0.0 > > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, > derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680180#action_12680180 ] Kristian Waagan commented on DERBY-646: --- I'm postponing the first commit until tomorrow, mostly because I've run out of time and want to write a overview of the code for the commit message. As soon as the patch is in, users are welcome to test it out. Note that the first revision won't have the features save on shutdown and restore on boot. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680170#action_12680170 ] Adrian Cole commented on DERBY-646: --- JBoss Cache would consider switching onto Derby for some internal work, if this feature is implemented. As of now, our tests take 2 minutes longer with embedded derby then hsqldb due to file i/o. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch >Assignee: Kristian Waagan > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678800#action_12678800 ] Kristian Waagan commented on DERBY-646: --- According to the 10.5 release page on the Wiki, the 10.5 branch will be created 2008-03-16. This leaves very little time to work on the in-memory back end. I'm describing my plan for this user-requested feature below. Most of the elements of the plan originates from the comments on this Jira issue. ** Patch status The community has three patches available; A) derby-646-20090222.diff B) derby-646-2a-vfmem_first_rev.diff C) svn.diff Patch C is the original patch from Stephen Fitch. It was uploaded in March 2006. It is writing the database log to disk. The general state is not very well known, but I believe it is not quite "production ready" int it's current form. The patch demonstrates the overall approach, which is also used by the two other patches. I can't speak for A, but B is more or less a re-implementation of C. This follows from the interfaces that have to be implemented. Stephen Fitch has a CLA on file. Patch A is from Cheng Che Chen, which has written a solution of his own. The solution passes basic ad-hoc testing, and has been successfully used in several of the performance tests in the Derby code repository. This solution seems to be the most feature rich one, and it also has a functional test. Cheng Che Chen has a CLA on file. Patch B is from Kristian Waagan. The test has passed ad-hoc testing, and has been successfully used in several of the performance tests in the Derby code repository. The main difference from patch A, is that it seems to perform better on CMT machines. The solution is able to run most of suites.All without failures. Kristian Waagan has a CLA on file. ** Plan 1) Commit patch B. (reasons: CMT performance, size, readability) 2) Work on the committed code, addressing the TODOs. 3) Incorporate tests from patch A and some additional tests. (additional tests: backup in-mem then restore/re-boot with directory (default) storage engine, simple boot/create test) 4) Incorporate features from patch A. (persist to disk on exit/shutdown, restore from disk on boot, finalizer, more?) 5) Figure out what to do on OOME (if anything). 6) General testing and improvements. 7) Documentation. I hope to address at least steps 1-3 before the branch is cut. It is not clear to me whether an in-memory back end will be a documented and supported feature of 10.5. I like the idea of providing it as-is without any documentation in 10.5, and then solidify it in 10.6. In the meantime, interested users can try it out and give us feedback, and the community can continue to test and improve the feature. Unless someone objects, I will commit the patch on Monday. I would also like to invite those who plan to contribute actively on the feature to let the community know. Feel free to flesh out the details of the plan. Finally, a great thank you to both Stephen Fitch and Cheng Che Chen for contributing their work to the Derby community! I hope continued contribution is of interest :) I would be very happy if Cheng could spare some time to get some of his more advanced features into the Derby code line. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676902#action_12676902 ] Cheng Che Chen commented on DERBY-646: -- I have just emailed scanned images of signed ICLA to [email protected], as instructed by Kathey Marsden. Regarding Rick Hillegas' question about the tests included in my patch, there are three sets of unit tests and one functional test. The unit tests verify the correctness of my internal data structures. They are probably not useful to other implementations, unless, for example, they have similar properties to be verified. The functional test implements an application in both JAVA and SQL and cross-checks the computed results. It can indeed be applied to any implementation of storage factory by simply changing the database name for the SQL portion of the test object. The functional test consists of the following files: "GenMog.java", "SampMog.java", "ClusMog.java", "ClusMogSQL.java", and "MogTest.java" -- the last file is the JUnit test object. However, I have some doubt about whether I have configured/used Derby's test framework properly in "MogTest.java". This file started out as a stand-alone JUnit test. When I was running it stand-alone, it would incur no disk activity. After I converted the file to run under Derby's test framework, the disk activity LED on my computer would go wild every time I ran it. I wonder if I have unwittingly been testing the default storage factory with the converted test. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676785#action_12676785 ] Rick Hillegas commented on DERBY-646: - I also would like to see us deliver an in-memory implementation in 10.5. I recommend that we check in one of these implementations but treat it as experimental code, not supported by official user documentation. A writeup of how to use the feature can appear on our wiki. After we release 10.5, we can solicit user feedback and adjust the api accordingly. Then in 10.6 we can expose this feature as solid and official with supporting user documentation. I think that this feature is important enough that we should consider spinning a 10.6 release a couple months after 10.5 if our users provide fast feedback. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676783#action_12676783 ] Rick Hillegas commented on DERBY-646: - For the first time in the history of the Derby community, two complete implementations of a significant feature have surfaced almost simultaneously. I'm afraid we have to pick one. I hope that the other contributor does not feel snubbed. Thanks to Kristian and Cheng for these two great submissions. I have taken a quick look at the patches and at the commentary on this JIRA. I prefer Kristian's implementation for the following reasons: 1) The coding style is slightly easier for me to understand. 2) According to the performance data which Kristian posted, his implementation is slightly faster. I hope that Cheng's ICLA arrives soon so that we can mine his patch for ideas that can be layered on top of Kristian's work. For instance, A) Cheng's solution lets you snapshot the database to disk at shutdown time. I think some users will find this useful. I understand that we could simulate this behavior using backup and restore, but I would not discount the appeal of this automatic behavior. Perhaps the api here could be an additional subprotocol. B) Cheng's solution has a finalizer method. This seems particularly useful for delivering (A) in the face of a system crash. But winddown logic seems generally useful for a complicated program like an rdbms. C) Cheng provides some additional tests. Some of these tests may be implementation-agnostic and worth running against Kristian's implementation. It would be helpful if Cheng could describe these tests in greater detail so that we can decide whether to incorporate them. Thanks again to both Kristian and Cheng. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676729#action_12676729 ] Kathey Marsden commented on DERBY-646: -- If you have access to a scanner, you can email your scanned ICLA to [email protected] (see http://www.apache.org/licenses/#clas). > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676716#action_12676716 ] Kristian Waagan commented on DERBY-646: --- Here are some results from running 'ant junitreport' with patch 2a (vfmem). The prefix 'org.apache.derbyTesting.' has been removed for brevity. The first number is the number of tests run with the vfmem back end, the second number is the number of tests run with a clean trunk. Note that I changes the memory back end to be the default back end in this run (modified the entry for DIRECTORY storage factory in BaseMonitor), and I also changed DropDatabaseSetup to delete the in-memory database. Suites run without failures: junit.EnvTest (9/9) functionTests.tests.jdbcapi.AutoloadTest (3/3) functionTests.tests.jdbcapi.JDBCDriversEmbeddedTest (8/8) functionTests.tests.jdbcapi.JDBCDriversClientTest (4/4) functionTests.tests.jdbcapi.JDBCDriversAllTest (12/12) functionTests.tests.tools._Suite (87/87) functionTests.tests.demo._Suite (2/2) functionTests.tests.engine._Suite (11/11) functionTests.suites.EncryptionSuite (7/7) functionTests.tests.jdbc4._Suite (896/898) // One test added later functionTests.tests.management._Suite (100/100) The follwing tests had failures and/or errors: functionTests.tests.derbynet._Suite (t=122/f=1/e=0) (f) ShutDownDBWhenNSShutsDownTest.testDatabasesShutDownWhenNSShutdown functionTests.tests.lang._Suite (t=1473/f=4/e=2) (f) DatabaseClassLoadingTest.testCreateDatabaseJar (f) DatabaseClassLoadingTest.testDatabaseInJar (f) ConnectTest.testDBName (f) ConnectTest.testDBName (e) DatabaseClassLoadingTest.testHackedJarReplacedClass (e) DBInJarTest.testConnectDBInJar functionTests.tests.jdbcapi._Suite (t=6301/f=2/e=0) (f) PoolXADSCreateShutdownDBTest.testXADS (f) PoolXADSCreateShutdownDBTest.testXADS functionTests.tests.store._Suite (t=98/f=4/e=19) (f) BootAllTest.testSettingBootAllPropertyWithHomePropertySet (f) OSReadOnlyTest.testOSReadOnly (f) OSReadOnlyTest.testOSReadOnly (f) BackupRestoreTest.testDerby3875 o Ok, test is trying to correct file on disk. (e) OfflineBackupTest.testCreateFromRestoreFrom 6x (e) EncryptionKeyAESTest 6x (e) EncryptionKeyBlowfishTest 6x (e) EncryptionKeyDESTest unitTests.junit._Suite (crashes due to ant/junit problem, will rerun later) (e) AccessControlException functionTests.tests.upgradeTests._Suite (t=420/f=12/e=42) (needs investigation) functionTests.tests.memory._Suite o Ok, fails because it sets the heap size really low and tries to insert data that is bigger than the heap. To sum up, I'm currently getting 23 failures and 65 errors. The time it takes to run the suites is around 45 minutes. I believe the 'ant junitreport' target is roughly equal to running suites.All. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676561#action_12676561 ] V.Narayanan commented on DERBY-646: --- can u please assign the issue to your name? Also for major contributions a icla is generally required and this looks like a major feature. my two paisa! > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676557#action_12676557 ] Cheng Che Chen commented on DERBY-646: -- I believe I have included the Apache license header in all the files and had checked the box granting rights to Apache when I submitted those files. According to the comment by Andrew McIntyre on 16/Mar/07 10:32 AM in this thread, that should be sufficient, no? To have an ICLA on file, I'll have to fill it out and mail it to Apache -- this will take some time. Let me know if this is truly required. Thanks. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676400#action_12676400 ] Kathey Marsden commented on DERBY-646: -- Kristian asked: >We now have two candidate solutions. How do we proceed? I haven't looked at the code but I guess the first thing we need to consider is legal. Is Cheng Che Chen willing to file an ICLA? > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, > svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676272#action_12676272 ] Kristian Waagan commented on DERBY-646: --- We now have two candidate solutions. How do we proceed? I would very much like to get an in-memory back end included in the upcoming 10.5 release. In my opinion this should be pretty safe, as almost no existing code is modified and the user has to take an active step to use the in-memory back end. To start mapping the performance characteristics of the two solutions, I ran some of the performance tests we have. Note that these are based on only a few runs, and a run time of 60 seconds only (pluss 30 seconds warmup). The numbers are average tps (transactions per second). There is some variation in the tests, so where the numbers differ only by a very small percentage, I think we can assume the performance is comparable for now. Hardware #1: 32 hardware execution threads, 1200 MHz. sr_update sr_select bank_tx 1 thread memory 19633059 823 vfmem 23073753 878 8 threads memory 7662 143212163 vfmem 12692 176952294 32 threads memory 7357 139842076 vfmem 12219 165462144 Hardware #2: Dual core Opteron, 2.4 GHz sr_update sr_select bank_tx 1 thread memory12257 202955123 vfmem 12288 206575189 4 threads memory15064 321176120 vfmem 15857 326296115 8 threads memory16682 312575767 vfmem 16357 313615775 Hardware #3: 2xDual core Opteron, 2.8 GHz sr_update sr_select bank_tx 1 thread memory13548 22184 5805 vfmem 13705 21770 5765 4 threads memory29019 5786110443 vfmem 29727 5805310479 8 threads memory26348 46580 9372 vfmem 26676 46805 9347 16 threads memory24500 44249 8976 vfmem 24883 43832 9014 Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page cache size 40 100015000 1 thread memory 9932 12234 19207 vfmem 10075 12223 18995 8 threads memory12898 16540 26393 vfmem 13148 16662 28017 Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page size Page cache size | 40| |1000 | 4 KB32 KB 4 KB32 KB 1 thread memory 99022902 12369 20168 vfmem 103423247 12059 19801 8 threads memory129263599 16523 27333 vfmem 132614059 16592 28361 As far as I can see from these simple performance tests, the only configuration where the two solutions differs significantly, is when running on a CMT (chip multi threading) machine. Of the loads I have tested, this is most clearly visible with update load. I have not investigated this further. Another thing one could look into, is the memory usage of the two solutions. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, > derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675052#action_12675052 ] Kristian Waagan commented on DERBY-646: --- Forgot to say, but if you want to spend time on some of the known weaker spots of the patch, search the diff for "TODO". Also, I'm wondering if it would be wise or not to ditch the self-made path handling and rely on java.io.File instead. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675046#action_12675046 ] Adam Wozniak commented on DERBY-646: Till 2009-02-22 I'm out of office. In urgent matters call my mobile (+48 604 277 044) or use [email protected] address. Regards, Adam Woźniak > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, > derby-646-2a-vfmem_first_rev.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674692#action_12674692 ] Kathey Marsden commented on DERBY-646: -- As Kristian mentioned, you can disable error logging by using derby.stream.error.field or derby.stream.error.method. See http://db.apache.org/derby/docs/10.4/tuning/rtunproper35028.html http://db.apache.org/derby/docs/10.4/tuning/rtunproper33027.html So we probably don't need to file a separate issue for that. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674441#action_12674441 ] Cheng Che Chen commented on DERBY-646: -- I believe "derby.log" is Derby's execution log, and operations on that log would probably not be going through the database storage layer. A run-time option to disable execution logging should be a separate work item (please file a ticket) and not be included in the resolution of this issue, in my opinion. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674339#action_12674339 ] David Sims commented on DERBY-646: -- Thanks, Kristian. In my situation, I would use Derby as a strictly embedded database, and I don't want my users to even be confused by the presence of a derby.log file, however empty it may be. The experience with my users tells me that it's best just to have absolutely nothing. I can deal with the usual SQLException errors that come along. However you and the others might feel that should best be configured in Derby, I leave up to you all. I would just like to chime in and request that there be a way to configure things so that absolutely nothing is on disk, including derby.log -- however that might work out. Thanks, David > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674300#action_12674300 ] Kristian Waagan commented on DERBY-646: --- Thank you for the feedback, David. > For what it's worth, in my situation, the option of having absolutely nothing > written to disk is a huge benefit. What do you want to happen with derby.log, which may contain critical information about errors happening in Derby? Note that the file by default will be overwritten when you restart Derby, so it shouldn't eat up your disk space. Since very little is written to it under normal operation, it shouldn't degrade performance. If you fire up lots of instances in different directories (without setting derby.system.home), you do get 'derby.log' files lying around though. If you want to discard the information, I think you have the possibility to do that yourself using one of the derby.stream.error properties. To make it even easier, we could add a stream class to the Derby engine jar file that simply discards the information (if specifying null causes the information to be printed to std err). My personal choice would be to not discard error information by default though, and I think it should be a conscientious choice made by the user / developer, but feel free to disagree :) > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674279#action_12674279 ] David Sims commented on DERBY-646: -- I'm following this thread with great interest. Many thanks to you guys for working on this. > The only thing my prototype leaves on disk, is derby.log. I think this can be > easily > changed, but often you would want to consult the log anyway. For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit. Thank you for your work, David > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674172#action_12674172 ] Kristian Waagan commented on DERBY-646: --- Yes, me too. However, since I was interested in learning a bit more about Derby and because I didn't know the state of the existing patch, I started from scratch. It is my impression that the existing patch isn't "production ready". I think it would be cool to compare our approaches. Maybe we can draw from both of them and create an even better solution! The only thing my prototype leaves on disk, is derby.log. I think this can be easily changed, but often you would want to consult the log anyway. The goal of my implementation is to avoid creating files/dirs on disk, which for some people is attractive for running tests. I believe there are also performance benefits, especially for update operations, at the cost of increased memory usage of course. The database creation time should also be shorter, but I haven't obtained any numbers on this yet. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674165#action_12674165 ] Cheng Che Chen commented on DERBY-646: -- Sometime back I played a bit with the original solution posted in this thread. I remember that it produced some files and directories on disk although it was supposed to be an in-memory storage factory. But if it works and suits the tasks at hand, we should let it be. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674134#action_12674134 ] Kristian Waagan commented on DERBY-646: --- Cheng, As it happens, I have a working prototype as well. It closely follows the approach taken by Stephen Fitch in the original patch. If possible, I think we should somehow try to coordinate our work. I'd like to spend a few more days cleaning up the initial code contribution, as it is pretty fresh. Regarding testing, the easiest way to do it is to override the storage factory for PersistentService.DIRECTORY in BaseMonitor. Then you can just run the tests as normal, either by using JUnit directly or by running ant junitreport. Currently, I'm getting 24 failures and 66 errors when running junitreport. Somes of these are to be expected, as some tests explicitly tries to access files in the database directory. I also note problems in general with encryption, backup and restoreFrom (in the url) etc, but I suspect these tests require the database files to exist as well. Also, some tests will just fail because all the data is stored in memory, for instance 'ant junit-lowmem'. Lastly, the test framework has a decorator to delete a database and/or a directory. Since it is working against the file system only, it doesn't manage to delete an in-memory database. Note that bugs in the prototype tend to hide if they are triggered during boot. If things go wrong, I recommend to check derby.log before starting debugging. I'll post my code (and the test results) in a day or two, and we should consider how to best cooperate on this issue. Regards, > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[
https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674113#action_12674113
]
Cheng Che Chen commented on DERBY-646:
--
BTW, there appears to be a discrepancy between code in the constructor of
(StorageFactoryService) and documentation for
StorageFactory.newStorageFile(String path).
In the constructor of (StorageFactoryService), there is this block of code:
rootStorageFactory = getStorageFactoryInstance( true, null, null, null);
if( home != null)
{
StorageFile rootDir = rootStorageFactory.newStorageFile( null);
rootDir.mkdirs();
}
Now, according to the javadoc for StorageFactory.newStorageFile(String path),
when the parameter (path) is null, we should return the database directory.
But in the above code block, the parameter representing the database directory
passed into the initialization of the storage factory, is null.
According to the documentation for StorageFactory.init(), when the database
name parameter is null, we will only work with the home directory. So I
believe the author of the above code block intended (rootDir) to represent the
home directory in this case (i.e., when the database parameter is null). This
behavior, however, is not specified in the documentation for
StorageFactory.newStorageFile(String path).
I noticed this discrepancy during manual testing because I was getting
(NullPointerException) at the call to rootDir.mkdirs() with my storage factory
implementation. I worked around the problem by having my implementation of
newStorageFile(String path) return the home directory when the internal
database parameter is null, in the case when the input (path) parameter is null.
In any event, there appears to be a discrepancy between code and documentation,
unless I have misinterpreted the code and/or the documentation.
> In-memory backend storage support
> -
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
>Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff,
> derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring
> disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674106#action_12674106 ] Cheng Che Chen commented on DERBY-646: -- Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes. At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework. I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html). I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works. If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plug things into the Derby test framework. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673865#action_12673865 ] Kristian Waagan commented on DERBY-646: --- If your classes aren't referenced by any other classes used by Derby, they won't be included in the jar file(s). Assuming you are setting properties to wire in your implementation, you probably need to add your "root class(es)" to 'tools/jar/extraDBMSclasses.properties'. As you can see, there are many other classes in there. These are used as roots to determine which classes to include in the jars. Hope this helps, > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[
https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673850#action_12673850
]
Cheng Che Chen commented on DERBY-646:
--
Hi folks,
I've implemented the interfaces
org.apache.derby.io.{StorageFile,StorageRandomAccessFile,WritableStorageFactory}
such that all file system operations are emulated entirely in memory,
producing no file or directory on disk. I'm more than happy to contribute the
implementation.
However, I have a build problem for which I hope to get some help. To start,
for simplicity, I added my source files to
"trunk/java/engine/org/apache/derby/impl/io" -- so that I didn't have to
create/modify any "build.xml" files. I then tried to build the sources and the
JAR files. The sources built with no problem -- all of my classes appeared in
"trunk/classes/org/apache/derby/impl/io". However, for some reason, the JAR
build process did not put my classes into "derby.jar". As a result, I had to
set the JAVA class path to "trunk/classes" during testing.
I'd appreciate any help resolving this build problem.
Thanks.
> In-memory backend storage support
> -
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
>Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff,
> derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring
> disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671849#action_12671849 ] Kristian Waagan commented on DERBY-646: --- I had a quick look at this again, looking at how an in-memory backend could be wired into Derby. >From what I can see, the easiest way is to add a single entry to BaseMonitor >(and the new constant for the subSubProtocol): storageFactories.put( PersistentService.VFMEM, "org.apache.derby.impl.io.VFMemoryStorageFactory"); If VFMemoryStorageFactory (or whatever the class will be called) behaves in the expected way, everything should "just work". The solution won't be optimal. The primary goal is to offer a configuration that doesn't leave any trace in the form of files and directories. Performance may be somewhat better, but the memory usage will be high. Not only will all the data be stored in memory, parts of it will be stored in multiple copies in memory. Note that even though the configuration will implement PersistentService, data won't be persisted beyond the life-time of the JVM running Derby. If this causes any problems with rest of the code, remains to be seen. A new service could be implemented as well, but I think this will require a lot more effort (feel free to comment on this if you have more information). I'll play some more with the idea and post my findings. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514277 ] Ashwin Jayaprakash commented on DERBY-646: -- You can un-JAR the file to a directory, paste your compiled Classes into that folder and use this "exploded" directory with the original classes and your changes in the Classpath instead of having the JAR in the ClassPath. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514254 ] Dave Syer commented on DERBY-646: - Is there an "unsealed" derby.jar available? I guess I just lost interest in testing this very interesting looking feature. I hope someone else can do it. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514237 ] Ashwin Jayaprakash commented on DERBY-646: -- The Derby JAR file is sealed - and this means that all the Derby packages specified in the JAR file's Manifest are sealed and you cannot add a new Class that shares the Package names to the Classpath. Since there are classes in org.apache.derby.impl.io Package in the Derby "sealed" JAR, you cannot append a Class like org.apache.derby.impl.io.PureMemStore that resides outside the JAR to the ClassPath. Unless I'm completely wrong, you will have to use the Source code and add your files there and then recreate the JAR file. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513869 ] Dave Syer commented on DERBY-646: - java.lang.SecurityException: sealing violation: can't seal package org.apache.derby.impl.io: already loaded at java.net.URLClassLoader.defineClass(URLClassLoader.java:235) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:164) at test.EmeddedMemoryTests.setUp(EmeddedMemoryTests.java:41) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196) > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513868 ] Dave Syer commented on DERBY-646: - I trie dit with 10.3.1.2 in case that was the problem. Had to patch the patch slightly to implement two methods that were unimplemented abstract (guessed the implementation, but did get far enough to test it). Now I get a more helpful stack trace (but maybe this is another problem?): > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[
https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512711
]
Dave Syer commented on DERBY-646:
-
I would like to test this storage strategy, but I don't know how to set it up.
I have compiled the code against 10.2.1.6, and now I want to obtain a
connection (ideally through a DataSource). This is what I tried
System.setProperty("derby.subSubProtocol.memory",
"org.apache.derby.impl.io.MemoryStorageFactory");
EmbeddedDataSource ds = new EmbeddedDataSource();
ds.setDatabaseName("memory:test");
ds.setCreateDatabase("create");
Connection connection = ds.getConnection();
It fails with
java.sql.SQLException: Failed to start database 'memory:test', see the next
exception for details.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
...
There is no "next exception" in the stack trace.
I also tried:
System.setProperty("derby.subSubProtocol.memory",
"org.apache.derby.impl.io.MemoryStorageFactory");
DriverManager.registerDriver(new org.apache.derby.jdbc.EmbeddedDriver());
Connection connection = DriverManager.getConnection("jdbc:derby:memory:test")
which fails the same way.
> In-memory backend storage support
> -
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
>Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff,
> derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring
> disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
"Myrna van Lunteren (JIRA)" <[EMAIL PROTECTED]> writes: > Although this issue has had a number of votes, I have to say, it > currently doesn't look like it'll make it for 10.3 - it appears > there has not been any testing of the patch by prospecting users nor > did anyone sign up to drive this to completion. > > If you are interested in this, feel free to give the patch a try and > help driving this into the next release after 10.3. There is also a recent contribution in the form of an uncommitted patch, which may interest you: http://issues.apache.org/jira/browse/DERBY-2798 "A new approach for main-memory database" I have not looked at it yet, so I can't vouch for it. Apparently it is still proof-of-concept code. Dag
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502804 ] Myrna van Lunteren commented on DERBY-646: -- Although this issue has had a number of votes, I have to say, it currently doesn't look like it'll make it for 10.3 - it appears there has not been any testing of the patch by prospecting users nor did anyone sign up to drive this to completion. If you are interested in this, feel free to give the patch a try and help driving this into the next release after 10.3. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502724 ] Cliff Evans commented on DERBY-646: --- I think people are under valuing this request as a useful tool in production code. I'm currently using H2 to implement a multi-threaded matching engine which takes a series of events and matches aggregated information held in an in-memory database in real time. The datasets never get too large but the matching can be quite complex. Being able to use the expressive power of SQL without the overhead of pesisting the information to disk makes the matching process very fast while providing all of the thread synchronisation required in a tried and tested package. My only problem with H2 is the granularity of its locking. Table or nothing. This causes blocking as the tables are updated which an in-memory, row-locked version of Derby would reduce considerably. We're talking manna from heaven here. Are there any plans to include this in the main release? > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12481708 ] Andrew McIntyre commented on DERBY-646: --- "Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?" The answer to this question is yes, sorry I missed it when it was posed several months ago. The 'Grant license to ASF' box was also checked when the original code was submitted, so anyone who wants to work on getting this code complete enough to be committed should feel free to do so. > In-memory backend storage support > - > > Key: DERBY-646 > URL: https://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Attachments: derby-646-1a-raw-compiles.diff, > derby-646-1a-raw-compiles.stat, svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12437075 ] Stefan Arentz commented on DERBY-646: - Has anything happened with this patch? I don't see any comments from the original submitter. > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Assigned To: Stephen Fitch > Attachments: svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
David Van Couvering wrote: > I guess it's not a regression or bug, so not a release note, but I > didn't know how else to indicate that documentation was needed, given > that it's a new feature... A new Jira entry indicating documentation is needed. :-) Dan.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
I guess it's not a regression or bug, so not a release note, but I didn't know how else to indicate that documentation was needed, given that it's a new feature... David Daniel John Debrunner wrote: David Van Couvering wrote: I see your point about new classes added to the public API needing javadoc, that seems reasonable. By release, do you mean a Derby release. I missed an 'e'. You checked the 'Release Note Needed' flag, why would a release note be needed? Dan.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
David Van Couvering wrote: > I see your point about new classes added to the public API needing > javadoc, that seems reasonable. > > By release, do you mean a Derby release. I missed an 'e'. You checked the 'Release Note Needed' flag, why would a release note be needed? Dan.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
I see your point about new classes added to the public API needing javadoc, that seems reasonable. By release, do you mean a Derby release or an ICLA? Assuming the latter, I thought the ICLA issue had been cleared up. The ICLA has been on file for some time, search for Stephen Fitch in http://people.apache.org/~jim/committers.html. David Daniel John Debrunner (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426372 ] Daniel John Debrunner commented on DERBY-646: - Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration. As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for. Why would a release not be needed? In-memory backend storage support - Key: DERBY-646 URL: http://issues.apache.org/jira/browse/DERBY-646 Project: Derby Issue Type: New Feature Components: Store Environment: All Reporter: Stephen Fitch Assigned To: Stephen Fitch Attachments: svn.diff To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426372 ] Daniel John Debrunner commented on DERBY-646: - Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration. As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for. Why would a release not be needed? > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Assigned To: Stephen Fitch > Attachments: svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426368 ] David Van Couvering commented on DERBY-646: --- Are separate tests needed for this feature. Isn't this a configuration option, e.g. couldn't we run the storemats suite with the system configured to use this store module? > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Issue Type: New Feature > Components: Store > Environment: All >Reporter: Stephen Fitch > Assigned To: Stephen Fitch > Attachments: svn.diff > > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12412180 ] Andrew McIntyre commented on DERBY-646: --- The patch is attached to this issue as a diff file. You need to get a local copy of the Derby source, then download the patch from JIRA (generally, right-click -> save as... on the attachment 'svn.diff'), then patch your local copy using 'patch -p0 < svn.diff' or your IDE, if it supports patching. I know Eclipse will apply patches generated by CVS and SVN. You will then need to build a local copy of Derby with the patch applied to try out the patch. For help checking out the Derby source code from Subversion, see: http://db.apache.org/derby/derby_downloads.html#Derby+source+code For help building, see BUILDING.txt at the top of the source tree and/or http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Type: New Feature > Components: Store > Environment: All > Reporter: Stephen Fitch > Assignee: Stephen Fitch > Attachments: svn.diff > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12412140 ] Ashwin Jayaprakash commented on DERBY-646: -- Hi, I would like to try this patch. But, I don't see any JAR or Java files attached. Could you pls tell me where I can download this patch? Thanks, Ashwin. > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Type: New Feature > Components: Store > Environment: All > Reporter: Stephen Fitch > Assignee: Stephen Fitch > Attachments: svn.diff > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
In the meantime you should try out the current system using the durability=test flag, for your unit testing needs. It was implemented specifically for this application from other user requests. It does no sychronous I/O so effectively uses the OS's memory cache. James Synge (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369704 ] James Synge commented on DERBY-646: --- I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great. In-memory backend storage support - Key: DERBY-646 URL: http://issues.apache.org/jira/browse/DERBY-646 Project: Derby Type: New Feature Components: Store Environment: All Reporter: Stephen Fitch Assignee: Stephen Fitch Attachments: svn.diff To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369719 ] Daniel John Debrunner commented on DERBY-646: - I think it would be great to have an in-memory db support like this and thanks for working on it Stephen, but there are a few issues to resolve first. We would need an ICLA on file at apache from Stephen Fitch before this could be committed into Derby. A quick look at the patch showed a couple of extra classes in the org.apache.derby.io package which is the published api for the virtual file system, classes such as VirtualFileData and VirtualFS. I can't tell what the purpose of these classes are since there are no comments for them, thus it makes it impossible for anyone to implement them without guessing what the requirements are. Also seemed to be no tests provided with the patch, so how would a reviewer know that this works or not? Copyrights on all the files seem to have incorrect dates, and file names. The wiki has information on what is required: http://wiki.apache.org/db-derby/DerbyContributorChecklist http://wiki.apache.org/db-derby/DerbyCommitProcess http://wiki.apache.org/db-derby/PatchAdvice > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Type: New Feature > Components: Store > Environment: All > Reporter: Stephen Fitch > Assignee: Stephen Fitch > Attachments: svn.diff > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369704 ] James Synge commented on DERBY-646: --- I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great. > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Type: New Feature > Components: Store > Environment: All > Reporter: Stephen Fitch > Assignee: Stephen Fitch > Attachments: svn.diff > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12359484 ] Mike Perham commented on DERBY-646: --- I would love to see Derby in-memory only support for unit testing purposes. This would solve a major problem of ours. > In-memory backend storage support > - > > Key: DERBY-646 > URL: http://issues.apache.org/jira/browse/DERBY-646 > Project: Derby > Type: New Feature > Components: Store > Environment: All > Reporter: Stephen Fitch > Assignee: Stephen Fitch > Attachments: in-memory.tar, svn.diff > > To allow creation and modification of databases in-memory without requiring > disk access or space to store the database. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
