Even if we cannot guarantee durability, it would be nice if we could
guarantee a consistent index. It sounds like the only problem in a
machine with a lying drive is that you could lose a number of committed
transactions. I would much prefer that to a corrupted index. I can
always re-add what was lost much quicker than rebuilding a 5 million doc
archive. In either case, I have my choice between the two as long as the
index is guaranteed to be corruption free.
robert engels wrote:
Usually you can configure the drives so that sync() ALWAYS syncs -
drive jumpers, driver setup, or other methods. Some drives that are
battery backed and such do not need it.
Without sync() truly being a sync you could never write a database
that was resilient.
It will exact a heavier toll on performance that you might think. In
order to do it properly, all filesystem metadata must be sync;d as
well. The biggest difference is that you lose the degree of
multi-processing that is inherent when sync'ing is disabled - as the
drive (or OS) does the physical write asynchronously while the system
does other work - with sync() this is lost.
This is why in a db system, the only file that is sync'd is the log
file - all other files can be made "in sync" from the log file - and
this file is normally striped for optimum write performance. Some
systems have special "log file drives" (some even solid state, or
battery backed ram) to aid the performance.
On Nov 4, 2007, at 8:30 AM, Yonik Seeley wrote:
On 11/4/07, Michael McCandless <[EMAIL PROTECTED]> wrote:
The problem is, on a hard shutdown (kill -9 or JVM/machine crashes),
apparently future operations may have completed while some past
operations have not. For example, the new segments_N file was
successfully written while say the _X.fdx file of the just-flushed
segment was not successfully written, even though Lucene had written &
closed _X.fdx before segments_N.
That should be impossible except for a machine crash. Kill -9 or a
JVM crash should have no effect on data already written.
But a sync option would be both simple and useful for people trying to
take live snapshots of an index, or to protect against machine
crashes. This isn't an absolute 100% guarantee either (so don't test
for it) - the drives often lie to the OS about data being flushed.
It's the best we can do at our level though.
http://www.google.com/search?q=fsync+drive+lies
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
