BALDWIN, ALAN J [AG-Contractor/1000] wrote:
First, thanks for the reply. I'll try my best to answer all questions.
Yes, I realize that deleting transaction log files is a bad thing, but
the alternative is that the user cannot even open the application if we
don't. I guess it's worth the user having to re-enter what they did
last if need be.
Our 1000 users are 1000 different users using 1000 different instances
of the same thick client running on their desktops. We use Java
Webstart to deploy the swing app to the clients. All single-process to
my knowledge.
thanks, I was wondering if this was some highly multi-threaded log bug.
This consistency checker you mention... what is it? Where do I find it?
http://db.apache.org/derby/docs/dev/adminguide/cadminconsist01.html
No, I have not been able to reproduce it. We are trying to just gather
as much about each instance as it happens and try to find a pattern. I
will log a JIRA issue to capture as much as I can in one central
location. Part of the problem is that our user base is scattered across
corn fields around the US, and not the most technically savvy folks.
That, and I usually get information second or third-hand. This makes it
hard to get a pattern.
Your questions [my replies in brackets]:
As you are seeing this frequently, some questions:
1) is it possible that the same database is being accessed from within
the same JVM but from 2 different class loaders (ie. DERBY-700)?
[Very unlikely... there is only one app using the derby jars or our
database, and from a single classloader within JavaWebstart]
2) Is the database located on a disk which can be accessed by 2
different machines, and if so is it possible 2 machines are looking
at the db using derby? Note it does not even take 2 updating
accesses to cause problems, just the act of booting a db can
cause writes to the log file.
[No, database must be on the user's PC local disk with no external
access]
ok, we have had problems in the past where a users home directory was
actually some shared filesystem resource. And could be easily accessed
from multiple machines by the user. In this case Derby can't coordinate
the proper access to the single db from multiple machines.
3) are you running with derby.system.durability=test?
[No, I'm not familiar with this property. Can you elaborate?]
You should not be using this, just wanted to make sure. you can
search derby docs on the apache site. I always just use the search box
in this page: http://db.apache.org/derby/manuals/index.html
4) Are the disks where the log files are written set to cache writes,
or do they do proper disk sync when requested. Are they scsi or
ide?
[These are standard IDE or SATA disks 99.9% of the time.]
There is a software option called "cache writes" available on XP on
disks. When enabled, writes to the disk which have been requested
by the application to not return until they are ensured on disk,
return immediately rather than waiting until they hit the drive. I
believe this option is often the default. Derby's ability to recover
after a system crash depends on log data hitting disk when we ask it
to. One can check the setting of this on XP by going to:
start->settings->control panel->system->hardware->device manager
o find disk drive you are interested in, and double click - go to
policies tab, last check box option for me is:
[] Enable write caching on the disk
This setting enables write caching to improve disk performance,
but a power outage or equipment failure might result in data loss
or corruption.
In Derby's case this same warning from microsoft applies to writes to
the log file.
5) can you post any of the data or the log files, obviously if the
data is private in any way you won't be able to put it in JIRA.
[I will log a JIRA and include as much as I can, xx'ing out urls, etc...
if I have to]
best chance to understand would be to post an entire database with the
complete set of logs for multiple of these cases.
second best would be at least the posting the set of log files in the
corrupted db. It is sometimes hard to understand the logs without
the db, but better than nothing.
third would be to at least post complete derby.log from each error.
6) Is the log error in derby.log always exactly the same? If not
it may help to post multiple derby.log's, again maybe someone will
see a pattern. For instance is the nested stack
in derby.log always say - it would probably be really interesting if
the number was always 768:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered
for identifier 768.
[I'll try to get as much of a pattern as I can into the JIRA issue]
I'd like to think about what more information derby can dump into the
derby.log when it encounters this issue, for this it is important to
understand what errors you are seeing. It will help a lot if you
can initially include as many different derby.log's as you have. Note
that once you get this error it is likely to reproduce everytime you
boot the same db, so multiple derby.log instances from the same db
are not that interesting. I always retry the boot at least once to
see if anything changes, but 99.9% you are just going to get the same
error.
Thanks for everyone's help.
-Alan-
snipped ...
I don't see any exceptions being thrown in our application before any
of the
corruption cases that would indicate that something in the app had
gone
awry.
I am not sure how your app logs this kind of info. It is likely that
whatever went wrong happened the LAST time the app shutdown, not this
time. If at all possible it would be interesting to ask those seeing
the corruptions if anything drastic happened the last time - like OS
crash, machine crash, blue screen, disk error.
Also when your app is finished with derby, to you do a "clean" shutdown
like "shutdown=true". If so, the fact that we are running through redo
means that it is likely the last time this app shutdown derby it did not
do a shutdown=true. Derby should still recover if you don't do a clean
shutdown, but it will save startup time by flushing all dirty cache
pages to disk and then marking the log so redo is not necessary. Note
that a clean shutdown with uncommitted, active transactions will result
in redo log records being processed the next time you boot.
Also, attached is my derby.log file.
-Alan-
-----Original Message-----
From: BALDWIN, ALAN J [AG-Contractor/1000]
Sent: Thursday, November 16, 2006 9:42 AM
To: [email protected]
Subject: java.sql.SQLException: Log Corrupted, has invalid data in the
log
stream.
Hi all,
We are having log corruption issues with Derby inside of a java swing
application.
Here is the error:
java.sql.SQLException: Log Corrupted, has invalid data in the log
stream.
We have about 1000 users using this software, and I would guess that
at
any
given time we see 4-5 with log corruption. Deleting "logXXX.dat"
files
from
within the log folder fixes the problem for the time being, but I
would
like
to get at the root of the problem. Is this safe to do as an interim
solution? I assume these are transaction logs, so it doesn't seem
like
a
good idea to delete them, but that is the only thing that seems to
work.
Also, anecdotally, this seems to be more prevalent in version 10.2
since we
upgraded from 10.1.3 a few weeks ago.
I'm also going to dig around on JIRA to seem if I come up with
anything.
Any ideas?
Thanks,
------------------------------------------------------------------------
---------------------------------
This e-mail message may contain privileged and/or confidential
information, and is intended to be received only by persons entitled to
receive such information. If you have received this e-mail in error,
please notify the sender immediately. Please delete it and all
attachments from any servers, hard drives or any other media. Other use
of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to
monitoring, reading and archival by Monsanto. The recipient of this
e-mail is solely responsible for checking for the presence of "Viruses"
or other "Malware". Monsanto accepts no liability for any damage caused
by any such code transmitted by or accompanying this e-mail or any
attachment.
------------------------------------------------------------------------
---------------------------------
---------------------------------------------------------------------------------------------------------
This e-mail message may contain privileged and/or confidential information, and
is intended to be received only by persons entitled to receive such
information. If you have received this e-mail in error, please notify the
sender immediately. Please delete it and all attachments from any servers, hard
drives or any other media. Other use of this e-mail by you is strictly
prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by
Monsanto. The recipient of this e-mail is solely responsible for checking for the presence of
"Viruses" or other "Malware". Monsanto accepts no liability for any damage
caused by any such code transmitted by or accompanying this e-mail or any attachment.
---------------------------------------------------------------------------------------------------------
