First, thanks for the reply. I'll try my best to answer all questions.
Yes, I realize that deleting transaction log files is a bad thing, but
the alternative is that the user cannot even open the application if we
don't. I guess it's worth the user having to re-enter what they did
last if need be.
Our 1000 users are 1000 different users using 1000 different instances
of the same thick client running on their desktops. We use Java
Webstart to deploy the swing app to the clients. All single-process to
my knowledge.
This consistency checker you mention... what is it? Where do I find it?
No, I have not been able to reproduce it. We are trying to just gather
as much about each instance as it happens and try to find a pattern. I
will log a JIRA issue to capture as much as I can in one central
location. Part of the problem is that our user base is scattered across
corn fields around the US, and not the most technically savvy folks.
That, and I usually get information second or third-hand. This makes it
hard to get a pattern.
Your questions [my replies in brackets]:
As you are seeing this frequently, some questions:
1) is it possible that the same database is being accessed from within
the same JVM but from 2 different class loaders (ie. DERBY-700)?
[Very unlikely... there is only one app using the derby jars or our
database, and from a single classloader within JavaWebstart]
2) Is the database located on a disk which can be accessed by 2
different machines, and if so is it possible 2 machines are looking
at the db using derby? Note it does not even take 2 updating
accesses to cause problems, just the act of booting a db can
cause writes to the log file.
[No, database must be on the user's PC local disk with no external
access]
3) are you running with derby.system.durability=test?
[No, I'm not familiar with this property. Can you elaborate?]
4) Are the disks where the log files are written set to cache writes,
or do they do proper disk sync when requested. Are they scsi or
ide?
[These are standard IDE or SATA disks 99.9% of the time.]
5) can you post any of the data or the log files, obviously if the
data is private in any way you won't be able to put it in JIRA.
[I will log a JIRA and include as much as I can, xx'ing out urls, etc...
if I have to]
6) Is the log error in derby.log always exactly the same? If not
it may help to post multiple derby.log's, again maybe someone will
see a pattern. For instance is the nested stack
in derby.log always say - it would probably be really interesting if
the number was always 768:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered
for identifier 768.
[I'll try to get as much of a pattern as I can into the JIRA issue]
Thanks for everyone's help.
-Alan-
-----Original Message-----
From: Mike Matrigali [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 16, 2006 12:31 PM
To: Derby Discussion
Subject: Re: java.sql.SQLException: Log Corrupted, has invalid data in
the log stream.
Deleting the log files by hand is an almost guaranteed way to corrupt
the data in the database: uncommitted transactions can't be aborted,
data not flushed from cache will never make it to disk in case of
process crash, machine crash, unclean shutdown
Unless all data from the database cache
was synced to disk and all transactions were committed before the
last shutdown then deleting the log files basically gives you access
to an inconsistent db. The fact that you are running through redo
recovery and getting an error makes it likely that the last shutdown
of the db was not clean, and thus the data on disk is missing updates
described in the log file to make the db consistent. . You can run the
consistency checker to identify some of the possible corruptsions
resulting from removing the log files. The only safe thing to do
in case of log corruption is to go to a backup.
In my support experience if we ever need
to delete log files, we usually recommend unloading all the data
from the db and loading it to a new db to guarantee no ongoing
corruption.
I assume you can't reproduce this yourself. With these kinds of
cases it is probably best to log a single tracking JIRA and load
as much info as possible into it. Maybe someone can see a pattern.
This is the first I have heard of this kind of problem happening
so often. The only cases I can think of are cases where somehow
there are 2 uncoordinated writers to the log file. This can easily
happen with JVM's prior to 1.4 on non-windows OS's.
As you are seeing this frequently, some questions:
1) is it possible that the same database is being accessed from within
the same JVM but from 2 different class loaders (ie. DERBY-700)?
2) Is the database located on a disk which can be accessed by 2
different machines, and if so is it possible 2 machines are looking
at the db using derby? Note it does not even take 2 updating
accesses to cause problems, just the act of booting a db can
cause writes to the log file.
3) are you running with derby.system.durability=test?
4) Are the disks where the log files are written set to cache writes,
or do they do proper disk sync when requested. Are they scsi or
ide?
5) can you post any of the data or the log files, obviously if the
data is private in any way you won't be able to put it in JIRA.
6) Is the log error in derby.log always exactly the same? If not
it may help to post multiple derby.log's, again maybe someone will
see a pattern. For instance is the nested stack
in derby.log always say - it would probably be really interesting if
the number was always 768:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered
for identifier 768.
Can you provide a listing of all the files in the log directory with
size and last modified time for some of log corruption cases. More
than one case may show a pattern. For instance is the nested stack
in derby.log always say:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered
for identifier 768.
What is happening in the one derby.log you posted here is that every
derby log record has a format
id, so the process of reading the log file is to read the format id
and use that format id to instantiate the right class which represents
the subsequent log record. In this case it read 768 - but the
highest format id that derby has is 464 (see MAX_ID_2 in
opensource/java/engine/org/apache/derby/iapi/services/io/StoredFormatIds
.java).
Looking at the state of the particular log.dat file that caused this
error may lend some clues (ie. Is this the last record? Is there any
pattern at this point, is this the last log file?).
Just to get an idea of scope of your app. Are those 1000 users all
using the same database, on the same machine?
If not, how many are likely to be using the same db on the same machine?
If it is one machine is it a multi-processor, if so how many processors?
BALDWIN, ALAN J [AG-Contractor/1000] wrote:
> I have not *noticed* any inconsistencies after deleting these files,
> however, that doesn't mean there aren't any. It just means they
haven't
> been reported.
>
> I have never run the compress table routines. Is that something that
> can be done programmatically at runtime?
>
> The corruption is noticed when we try to start the app and connect to
> the database. It won't let us connect. Also, as expected, we cannot
> connect using tools like Cloudscape Workbench either once this state
is
> reached.
>
> I can't think of anything else that is relevant at this point. It
seems
> to be random, and it does not seem to be related to users killing the
> app halfway through a transaction or anything like that.
>
> -Alan-
>
> -----Original Message-----
> From: Mayuresh Nirhali [mailto:[EMAIL PROTECTED]
> Sent: Thursday, November 16, 2006 6:04 AM
> To: Derby Discussion
> Subject: Re: java.sql.SQLException: Log Corrupted, has invalid data in
> the log stream.
>
> Alan,
>
> As you are aware, it is not recommended to delete the log fils
manually.
> The logCorruptedException has another nestedException and it is
because
> the value read from the log records is not matched with required
> appropriate data.
> But, Did you observe any sort of inconsistencies in your database when
> you boot again after you delete the files. ?
>
> How often do you (or did you ever) run the compress table routines ??
>
> Do you see the log corrupted when you boot the db or during some
> transactions ??
>
> any more information which you think is relevant ??
>
> Mayuresh
>
>
> BALDWIN, ALAN J [AG-Contractor/1000] wrote:
>
>
>>More info:
>>
>>Windows XP
>>
>>JVM 1.5 and JVM 1.4.2
>>
>>
>>
>>I don't see any exceptions being thrown in our application before any
>
> of the
>
>>corruption cases that would indicate that something in the app had
gone
>>awry.
>>
>>
>>
>>Also, attached is my derby.log file.
>>
>>
>>
>>-Alan-
>>
>>
>>
>>
>>
>>-----Original Message-----
>>From: BALDWIN, ALAN J [AG-Contractor/1000]
>>Sent: Thursday, November 16, 2006 9:42 AM
>>To: [email protected]
>>Subject: java.sql.SQLException: Log Corrupted, has invalid data in the
>
> log
>
>>stream.
>>
>>
>>
>>Hi all,
>>
>>We are having log corruption issues with Derby inside of a java swing
>>application.
>>
>>Here is the error:
>>
>>java.sql.SQLException: Log Corrupted, has invalid data in the log
>
> stream.
>
>>We have about 1000 users using this software, and I would guess that
at
>
> any
>
>>given time we see 4-5 with log corruption. Deleting "logXXX.dat"
files
>
> from
>
>>within the log folder fixes the problem for the time being, but I
would
>
> like
>
>>to get at the root of the problem. Is this safe to do as an interim
>>solution? I assume these are transaction logs, so it doesn't seem
like
>
> a
>
>>good idea to delete them, but that is the only thing that seems to
>
> work.
>
>>Also, anecdotally, this seems to be more prevalent in version 10.2
>
> since we
>
>>upgraded from 10.1.3 a few weeks ago.
>>
>>I'm also going to dig around on JIRA to seem if I come up with
>
> anything.
>
>>Any ideas?
>>
>>Thanks,
>>
>>
>>
>
>
>
>
------------------------------------------------------------------------
---------------------------------
> This e-mail message may contain privileged and/or confidential
information, and is intended to be received only by persons entitled to
receive such information. If you have received this e-mail in error,
please notify the sender immediately. Please delete it and all
attachments from any servers, hard drives or any other media. Other use
of this e-mail by you is strictly prohibited.
>
>
> All e-mails and attachments sent and received are subject to
monitoring, reading and archival by Monsanto. The recipient of this
e-mail is solely responsible for checking for the presence of "Viruses"
or other "Malware". Monsanto accepts no liability for any damage caused
by any such code transmitted by or accompanying this e-mail or any
attachment.
>
------------------------------------------------------------------------
---------------------------------
>
>
>
---------------------------------------------------------------------------------------------------------
This e-mail message may contain privileged and/or confidential information, and
is intended to be received only by persons entitled to receive such
information. If you have received this e-mail in error, please notify the
sender immediately. Please delete it and all attachments from any servers, hard
drives or any other media. Other use of this e-mail by you is strictly
prohibited.
All e-mails and attachments sent and received are subject to monitoring,
reading and archival by Monsanto. The recipient of this e-mail is solely
responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto accepts no liability for any damage caused by any such code
transmitted by or accompanying this e-mail or any attachment.
---------------------------------------------------------------------------------------------------------
