[
https://issues.apache.org/jira/browse/DERBY-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ralf Schubert updated DERBY-7007:
---------------------------------
Description:
Our customer is migrating to a new server platform. We have running several
applications on their old server platform right now, which are running well so
far. But on the new platform some random Derby errors occur reproducably which
we and customer are analysing since several months now. However, the deeper we
get the more clueless we are and it looks more and more like a DERBY bug.
We would be pleased if somebody could look into this and give us some idea if
this is either a bug in derby or if you have some other ideas what could cause
derby to behave like this.
h2. Situation
We have one Application which includes several embedded DERBY databases. After
the server is starting, the application behaves normal for a few minutes. But
after some minutes, one of the Derby DBs (accessed by JAVA Hibernate using
DERBY embedded mode) shows first an error like this on a random derby file (the
files vary each time):
{code:java}
Local derby log (/home/tomcat_i36/derby.log):
------------ Begin Shutdown Error Stack -------------
ERROR XSDG3: Meta-data for Container(0, 33904) could not be accessed to clean
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
at
org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
Source)
at
org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:809)
Caused by: java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
... 8 more
============= begin nested exception, level (1) ===========
java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
at
org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
Source)
at
org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:809)
============= end nested exception, level (1) ===========
------------ End Shutdown Error Stack ------------{code}
After this happens, the DB behaves weird, throwing random errors (e.g. telling
a column is missing in a table although it is there, or telling the DB is
corrupt).
Hint: We do only have READ access on those databases within the application. We
do not write any data to it.
It only happens to one single DB, but this is the most complex one in the
application. Restarting the server will make it WORK for some minutes again!
We deploy the exact same WAR file to Old and new platform for testing.
h2. Already analysed
We already tried several things and did several analysis steps:
# Turning off antivirus solution (Trend Micro Deep Security Agent) did not help
# Exchanging the servers of the new server platform with another set of
servers with same setup does not help
# Comparing a SHA1 hash of the "corrupt" files with the original files turned
out the files are IDENTICAL.
# Copying the "corrupt" DB to another system, testing it there works as
expected without issues.
# Running an integrity check on the DB shows no problems
# Checking the file permissions on the problematic servers shows no problems
{code:java}
# ls -l /opt/apps/tomcat/i36/webapps/XXXX/database/XX/seg0/c8470.dat
-rw-r--r-- 1 tomcat_i36 tomcat 16384 Aug 21 09:32
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
# file /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat: data{code}
# Checking if any linux limits (e.g. open files limit) was reached: nothing
found
# Checking for corrupt file system: Ext3 is used on old and new platform, no
hint about corrupt files found
h2. The server environments
h3. OLD environment (working well)
{code:java}
Linux: SUSE Linux Enterprise Server 11 SP4 (s390x)
Kernel: 3.0.101-91-default #1 SMP Mon Dec 12 13:06:13 UTC 2016 (544b9d1) s390x
s390x s390x GNU/Linux
Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,acl,user_xattr)
Java: IBM 7.1-4.1
Tomcat: 7.0.70{code}
h3. NEW environment (not working)
{code:java}
Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64)
Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989)
x86_64 x86_64 x86_64 GNU/Linux
Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3
(rw,relatime,data=ordered)
Java: IBM 7.1-4.15
Tomcat: 7.0.85{code}
Our customer has to migrate the server platforms very soon so we would be very
glad if someone could assist us in checking and resolving this.
Thanks!
was:
Our customer is migrating to a new server platform. We have running several
applications on their old server platform right now, which are running well so
far. But on the new platform some random Derby errors occur reproducably which
we and customer are analysing since several months now. However, the deeper we
get the more clueless we are and it looks more and more like a DERBY bug.
We would be pleased if somebody could look into this and give us some idea if
this is either a bug in derby or if you have some other ideas what could cause
derby to behave like this.
h2. Situation
We have one Application which includes several embedded DERBY databases. After
the server is starting, the application behaves normal for a few minutes. But
after some minutes, one of the Derby DBs (accessed by JAVA Hibernate using
DERBY embedded mode) shows first an error like this on a random derby file (the
files vary each time):
{code:java}
Local derby log (/home/tomcat_i36/derby.log):
------------ Begin Shutdown Error Stack -------------
ERROR XSDG3: Meta-data for Container(0, 33904) could not be accessed to clean
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
at
org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
Source)
at
org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:809)
Caused by: java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
... 8 more
============= begin nested exception, level (1) ===========
java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown
Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
Source)
at
org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown Source)
at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
Source)
at
org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown Source)
at
org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
Source)
at
org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
Source)
at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
at java.lang.Thread.run(Thread.java:809)
============= end nested exception, level (1) ===========
------------ End Shutdown Error Stack ------------{code}
After this happens, the DB behaves weird, throwing random errors (e.g. telling
a column is missing in a table although it is there, or telling the DB is
corrupt).
Hint: We do only have READ access on those databases within the application. We
do not write any data to it.
It only happens to one single DB, but this is the most complex one in the
application. Restarting the server will make it WORK for some minutes again!
We deploy the exact same WAR file to Old and new platform for testing.
h2. Already analysed
We already tried several things and did several analysis steps:
# Turning off antivirus solution (Trend Micro Deep Security Agent) did not help
# Exchanging the servers of the new server platform with another set of
servers with same setup does not help
# Comparing a SHA1 hash of the "corrupt" files with the original files turned
out the files are IDENTICAL.
# Copying the "corrupt" DB to another system, testing it there works as
expected without issues.
# Running an integrity check on the DB shows no problems
# Checking the file permissions on the problematic servers shows no problems
{code:java}
# ls -l /opt/apps/tomcat/i36/webapps/XXXX/database/XX/seg0/c8470.dat
-rw-r--r-- 1 tomcat_i36 tomcat 16384 Aug 21 09:32
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
# file /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
/opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat: data{code}
# Checking if any linux limits (e.g. open files limit) was reached: nothing
found
# Checking for corrupt file system: Ext3 is used on old and new platform, no
hint about corrupt files found
h2. The server environments
h3. OLD environment (working well)
{code:java}
Linux: SUSE Linux Enterprise Server 11 SP4 (s390x)
Kernel: 3.0.101-91-default #1 SMP Mon Dec 12 13:06:13 UTC 2016 (544b9d1) s390x
s390x s390x GNU/Linux
Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3 (rw,acl,user_xattr)
Java: IBM 7.1-4.1
Tomcat: 7.0.70{code}
h3. NEW environment (not working)
{code:java}
Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64)
Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989)
x86_64 x86_64 x86_64 GNU/Linux
Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3
(rw,relatime,data=ordered)
Java: IBM 7.1-4.15
Tomcat: 7.0.85{code}
Our customer has to migrate the server platforms very soon so we would be very
glad if someone could assist us in checking and resolving this.
Thanks!
> Random IOException: Bad file descriptor on new server platform
> --------------------------------------------------------------
>
> Key: DERBY-7007
> URL: https://issues.apache.org/jira/browse/DERBY-7007
> Project: Derby
> Issue Type: Bug
> Components: Miscellaneous
> Affects Versions: 10.11.1.1
> Environment: Linux: SUSE Linux Enterprise Server for SAP Applications
> 12 SP3 (x86_64)
> Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989)
> x86_64 x86_64 x86_64 GNU/Linux
> Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3
> (rw,relatime,data=ordered)
> Java: IBM 7.1-4.15
> Tomcat: 7.0.85
> Reporter: Ralf Schubert
> Priority: Blocker
>
> Our customer is migrating to a new server platform. We have running several
> applications on their old server platform right now, which are running well
> so far. But on the new platform some random Derby errors occur reproducably
> which we and customer are analysing since several months now. However, the
> deeper we get the more clueless we are and it looks more and more like a
> DERBY bug.
> We would be pleased if somebody could look into this and give us some idea if
> this is either a bug in derby or if you have some other ideas what could
> cause derby to behave like this.
> h2. Situation
> We have one Application which includes several embedded DERBY databases.
> After the server is starting, the application behaves normal for a few
> minutes. But after some minutes, one of the Derby DBs (accessed by JAVA
> Hibernate using DERBY embedded mode) shows first an error like this on a
> random derby file (the files vary each time):
> {code:java}
> Local derby log (/home/tomcat_i36/derby.log):
>
> ------------ Begin Shutdown Error Stack -------------
> ERROR XSDG3: Meta-data for Container(0, 33904) could not be accessed to clean
> /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
> at org.apache.derby.iapi.error.StandardException.newException(Unknown
> Source)
> at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
> Source)
> at
> org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown
> Source)
> at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
> Source)
> at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:809)
> Caused by: java.io.IOException: Bad file descriptor
> at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
> Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown
> Source)
> ... 8 more
> ============= begin nested exception, level (1) ===========
> java.io.IOException: Bad file descriptor
> at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:65)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:210)
> at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:754)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:739)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(Unknown Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer4.getEmbryonicPage(Unknown
> Source)
> at
> org.apache.derby.impl.store.raw.data.RAFContainer.writeRAFHeader(Unknown
> Source)
> at org.apache.derby.impl.store.raw.data.RAFContainer.clean(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.ConcurrentCache.cleanAndUnkeepEntry(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.ConcurrentCache.cleanEntry(Unknown
> Source)
> at
> org.apache.derby.impl.services.cache.BackgroundCleaner.performWork(Unknown
> Source)
> at
> org.apache.derby.impl.services.daemon.BasicDaemon.serviceClient(Unknown
> Source)
> at org.apache.derby.impl.services.daemon.BasicDaemon.work(Unknown
> Source)
> at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:809)
> ============= end nested exception, level (1) ===========
> ------------ End Shutdown Error Stack ------------{code}
> After this happens, the DB behaves weird, throwing random errors (e.g.
> telling a column is missing in a table although it is there, or telling the
> DB is corrupt).
> Hint: We do only have READ access on those databases within the application.
> We do not write any data to it.
> It only happens to one single DB, but this is the most complex one in the
> application. Restarting the server will make it WORK for some minutes again!
> We deploy the exact same WAR file to Old and new platform for testing.
> h2. Already analysed
> We already tried several things and did several analysis steps:
> # Turning off antivirus solution (Trend Micro Deep Security Agent) did not
> help
> # Exchanging the servers of the new server platform with another set of
> servers with same setup does not help
> # Comparing a SHA1 hash of the "corrupt" files with the original files
> turned out the files are IDENTICAL.
> # Copying the "corrupt" DB to another system, testing it there works as
> expected without issues.
> # Running an integrity check on the DB shows no problems
> # Checking the file permissions on the problematic servers shows no problems
> {code:java}
> # ls -l /opt/apps/tomcat/i36/webapps/XXXX/database/XX/seg0/c8470.dat
> -rw-r--r-- 1 tomcat_i36 tomcat 16384 Aug 21 09:32
> /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
>
> # file /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat
> /opt/apps/tomcat/i36/webapps/XXXXX/database/XX/seg0/c8470.dat: data{code}
> # Checking if any linux limits (e.g. open files limit) was reached: nothing
> found
> # Checking for corrupt file system: Ext3 is used on old and new platform, no
> hint about corrupt files found
> h2. The server environments
> h3. OLD environment (working well)
> {code:java}
> Linux: SUSE Linux Enterprise Server 11 SP4 (s390x)
> Kernel: 3.0.101-91-default #1 SMP Mon Dec 12 13:06:13 UTC 2016 (544b9d1)
> s390x s390x s390x GNU/Linux
> Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3
> (rw,acl,user_xattr)
> Java: IBM 7.1-4.1
> Tomcat: 7.0.70{code}
> h3. NEW environment (not working)
> {code:java}
> Linux: SUSE Linux Enterprise Server for SAP Applications 12 SP3 (x86_64)
> Kernel: 4.4.126-94.22-default #1 SMP Wed Apr 11 07:45:03 UTC 2018 (9649989)
> x86_64 x86_64 x86_64 GNU/Linux
> Filesystem: /dev/mapper/appsvg-lvapps on /opt/apps type ext3
> (rw,relatime,data=ordered)
> Java: IBM 7.1-4.15
> Tomcat: 7.0.85{code}
> Our customer has to migrate the server platforms very soon so we would be
> very glad if someone could assist us in checking and resolving this.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
