RE: Question on unloading in an embedded environment
Here is what jhat reports for the is OOME: Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node 26547422 instances of class org.apache.derby.impl.store.raw.data.RecordId 26524427 instances of class [Lorg.apache.derby.iapi.types.DataValueDescriptor; 26524237 instances of class org.apache.derby.iapi.types.SQLTimestamp 26524235 instances of class org.apache.derby.impl.store.access.heap.HeapRowLocation 3323190 instances of class org.apache.derby.iapi.store.raw.PageKey 31521 instances of class org.apache.derby.impl.store.raw.data.StoredRecordHeader ... I am trying to get a stack traceback of where some of these are allocated, but there is definitely a leak in Derby, probably in the implementation of the SYSCS_IMPORT_TABLE somewhere. -Original Message- From: Rick Hillegas [mailto:rick.hille...@oracle.com] Sent: Tuesday, August 16, 2011 9:20 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment On 8/16/11 5:56 AM, Bergquist, Brett wrote: I am writing a database copy utility that is built upon the dblook utility and using the SYSCS_UTIL.SYSCS_EXPORT_TABLE and SYSCS_UTIL.SYSCS_IMPORT_TABLE procedures and I have this mostly working. When running this on a large database (110Gb) I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. I am running this again with the -XX:+HeapDumpOnOutOfMemoryError switch to see if I can get a picture on where it is leaking but it will take about 6 hours to get to this point. In the mean time, assuming the error is not in my utility, I would like to potentially shutdown the database and restart it to try to work around the OOME. I am running the database engine in embedded mode and see the documentation to shutdown the database, but I also see the following in the Derby 10.8 Reference manual page 293: With the embedded driver, if your application shuts down Derby or calls the DriverManager.unload method, and you then want to reload the driver, call the Class.forName().newInstance() method to do so: When I look at the Javadoc for the DriverManager class, I do not find an unload method. So what is this documentation referring to? Hi Brett, That looks like an error in the documentation. I believe it is supposed to refer to DriverManager.deregisterDriver(). I have logged https://issues.apache.org/jira/browse/DERBY-5384 to track this. Thanks for finding this bug, -Rick Thanks Brett
Re: Question on unloading in an embedded environment
Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max. Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used. -Original Message- From: Bryan Pendleton [mailto:bpendleton.de...@gmail.com] Sent: Wednesday, August 17, 2011 9:40 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Here is the whole application. It is still in a rough format but on a smaller database it does indeed work correctly. The only thing that it cannot do when copying the database is handle GENERATED ALWAYS columns. For these, it turns them into GENERATED BY DEFAULT and as the last step determines the next value to use and issues an ALTER TABLE statement to reset the next generated value. It would be useful to somehow allow the SYSCS_IMPORT_TABLE to bypass the constraint that a GENERATED ALWAYS column cannot be inserted with a value or have the ability to change the column type back to GENERATED ALWAYS after the determining the correct next value. -Original Message- From: Bryan Pendleton [mailto:bpendleton.de...@gmail.com] Sent: Wednesday, August 17, 2011 9:40 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide. Main.java Description: Main.java
RE: Question on unloading in an embedded environment
So this is running on Solaris 10 Java 1.6.0_22 with -d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback. -Original Message- From: Bergquist, Brett [mailto:bbergqu...@canoga.com] Sent: Wednesday, August 17, 2011 10:49 AM To: derby-dev@db.apache.org Subject: RE: Question on unloading in an embedded environment This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max. Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used. -Original Message- From: Bryan Pendleton [mailto:bpendleton.de...@gmail.com] Sent: Wednesday, August 17, 2011 9:40 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide.
Re: Question on unloading in an embedded environment
On 8/17/11 11:03 AM, Bergquist, Brett wrote: I have a report generated by MemoryAnalyzer (Eclipse) tool. The report is stored as a zip file that contains the HTML and images. Can I attach this to an email? Hi Brett, If it's big, I would recommend attaching it to a JIRA issue. Thanks, -Rick I have attached a JPG of some of the report here. -Original Message- From: Bergquist, Brett [mailto:bbergqu...@canoga.com] Sent: Wednesday, August 17, 2011 11:16 AM To: derby-dev@db.apache.org Subject: RE: Question on unloading in an embedded environment So this is running on Solaris 10 Java 1.6.0_22 with -d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback. -Original Message- From: Bergquist, Brett [mailto:bbergqu...@canoga.com] Sent: Wednesday, August 17, 2011 10:49 AM To: derby-dev@db.apache.org Subject: RE: Question on unloading in an embedded environment This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max. Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used. -Original Message- From: Bryan Pendleton [mailto:bpendleton.de...@gmail.com] Sent: Wednesday, August 17, 2011 9:40 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Open https://issues.apache.org/jira/browse/DERBY-5387 and attached the Netbeans project for the utility along with two reports from Eclipse MemoryAnalyzer to for the suspected leak. -Original Message- From: Rick Hillegas [mailto:rick.hille...@oracle.com] Sent: Wednesday, August 17, 2011 2:57 PM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment On 8/17/11 11:03 AM, Bergquist, Brett wrote: I have a report generated by MemoryAnalyzer (Eclipse) tool. The report is stored as a zip file that contains the HTML and images. Can I attach this to an email? Hi Brett, If it's big, I would recommend attaching it to a JIRA issue. Thanks, -Rick I have attached a JPG of some of the report here. -Original Message- From: Bergquist, Brett [mailto:bbergqu...@canoga.com] Sent: Wednesday, August 17, 2011 11:16 AM To: derby-dev@db.apache.org Subject: RE: Question on unloading in an embedded environment So this is running on Solaris 10 Java 1.6.0_22 with -d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback. -Original Message- From: Bergquist, Brett [mailto:bbergqu...@canoga.com] Sent: Wednesday, August 17, 2011 10:49 AM To: derby-dev@db.apache.org Subject: RE: Question on unloading in an embedded environment This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max. Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used. -Original Message- From: Bryan Pendleton [mailto:bpendleton.de...@gmail.com] Sent: Wednesday, August 17, 2011 9:40 AM To: derby-dev@db.apache.org Subject: Re: Question on unloading in an embedded environment Instance Counts for All Classes (excluding platform) 38006784 instances of class org.apache.derby.impl.store.access.sort.Node One possibility is that the sorter is confused about how much memory is available. The sorter is very clever, and tries to figure out whether it can perform the sort in-memory, or whether it has to switch to an external (disk-based) sort, using temporary files to hold the partially-sorted data. If the sorter is confused about how much memory is available (i.e., thinks there is more memory available than there actually is), then it might drive the system out of memory. This would certainly be a bug, but not a leak exactly, rather a flaw in the internal-vs-external sort analysis code. thanks, bryan P.S. I'm still wondering if your memory issues are actually external to Derby; that is, if you've configured your JVM with memory sizes that exceed the memory available in the underlying operating system. That would cause Derby to attempt to allocate data structures that the JVM was willing to allow, but which the underlying OS refused to provide.
Re: Question on unloading in an embedded environment
On 8/16/11 5:56 AM, Bergquist, Brett wrote: I am writing a database copy utility that is built upon the dblook utility and using the SYSCS_UTIL.SYSCS_EXPORT_TABLE and SYSCS_UTIL.SYSCS_IMPORT_TABLE procedures and I have this mostly working. When running this on a large database (110Gb) I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. I am running this again with the -XX:+HeapDumpOnOutOfMemoryError switch to see if I can get a picture on where it is leaking but it will take about 6 hours to get to this point. In the mean time, assuming the error is not in my utility, I would like to potentially shutdown the database and restart it to try to work around the OOME. I am running the database engine in embedded mode and see the documentation to shutdown the database, but I also see the following in the Derby 10.8 Reference manual page 293: With the embedded driver, if your application shuts down Derby or calls the DriverManager.unload method, and you then want to reload the driver, call the Class.forName().newInstance() method to do so: When I look at the Javadoc for the DriverManager class, I do not find an unload method. So what is this documentation referring to? Hi Brett, That looks like an error in the documentation. I believe it is supposed to refer to DriverManager.deregisterDriver(). I have logged https://issues.apache.org/jira/browse/DERBY-5384 to track this. Thanks for finding this bug, -Rick Thanks Brett
Re: Question on unloading in an embedded environment
I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. Brett Sometimes, these errors arise externally to Java; e.g., your system runs out of pagefile or swap space, etc. It might be worth checking to ensure that you have adequate system-wide resources. thanks, bryan
