Then how can the commit during reopen be an issue?
I am not very family with this new code, but it seems that you need
to write segments.XXX.new and then rename to segments.XXX.
As long as the files are sync'd, even on nfs the reopen should not
see segments.XXX until is is ready.
Although lockless commits are beneficial in their own rite, I still
think that people's understanding of NFS limitations are flawed. Read
the section below on "close to open" consistency. There should be no
problem using Lucene across NFS - even the old version.
The write-once nature of Lucene makes this trivial. The only problem
was the segments file, which is lucene used the read/write lock and
close(0 correctly never would have been a problem.
According to the NFS docs:
NFS Version 2 requires that a server must save all the data in a
write operation to disk before it replies to a client that the write
operation has completed. This can be expensive because it breaks
write requests into small chunks (8KB or less) that must each be
written to disk before the next chunk can be written. Disks work best
when they can write large amounts of data all at once.
NFS Version 3 introduces the concept of "safe asynchronous writes." A
Version 3 client can specify that the server is allowed to reply
before it has saved the requested data to disk, permitting the server
to gather small NFS write operations into a single efficient disk
write operation. A Version 3 client can also specify that the data
must be written to disk before the server replies, just like a
Version 2 write. The client specifies the type of write by setting
the stable_how field in the arguments of each write operation to
UNSTABLE to request a safe asynchronous write, and FILE_SYNC for an
NFS Version 2 style write.
Servers indicate whether the requested data is permanently stored by
setting a corresponding field in the response to each NFS write
operation. A server can respond to an UNSTABLE write request with an
UNSTABLE reply or a FILE_SYNC reply, depending on whether or not the
requested data resides on permanent storage yet. An NFS protocol-
compliant server must respond to a FILE_SYNC request only with a
FILE_SYNC reply.
Clients ensure that data that was written using a safe asynchronous
write has been written onto permanent storage using a new operation
available in Version 3 called a COMMIT. Servers do not send a
response to a COMMIT operation until all data specified in the
request has been written to permanent storage. NFS Version 3 clients
must protect buffered data that has been written using a safe
asynchronous write but not yet committed. If a server reboots before
a client has sent an appropriate COMMIT, the server can reply to the
eventual COMMIT request in a way that forces the client to resend the
original write operation. Version 3 clients use COMMIT operations
when flushing safe asynchronous writes to the server during a close
(2) or fsync(2) system call, or when encountering memory pressure.
A8. What is close-to-open cache consistency?
A. Perfect cache coherency among disparate NFS clients is very
expensive to achieve, so NFS settles for something weaker that
satisfies the requirements of most everyday types of file sharing.
Everyday file sharing is most often completely sequential: first
client A opens a file, writes something to it, then closes it; then
client B opens the same file, and reads the changes.
So, when an application opens a file stored in NFS, the NFS client
checks that it still exists on the server, and is permitted to the
opener, by sending a GETATTR or ACCESS operation. When the
application closes the file, the NFS client writes back any pending
changes to the file so that the next opener can view the changes.
This also gives the NFS client an opportunity to report any server
write errors to the application via the return code from close().
This behavior is referred to as close-to-open cache consistency.
Linux implements close-to-open cache consistency by comparing the
results of a GETATTR operation done just after the file is closed to
the results of a GETATTR operation done when the file is next opened.
If the results are the same, the client will assume its data cache is
still valid; otherwise, the cache is purged.
Close-to-open cache consistency was introduced to the Linux NFS
client in 2.4.20. If for some reason you have applications that
depend on the old behavior, you can disable close-to-open support by
using the "nocto" mount option.
There are still opportunities for a client's data cache to contain
stale data. The NFS version 3 protocol introduced "weak cache
consistency" (also known as WCC) which provides a way of checking a
file's attributes before and after an operation to allow a client to
identify changes that could have been made by other clients.
Unfortunately when a client is using many concurrent operations that
update the same file at the same time, it is impossible to tell
whether it was that client's updates or some other client's updates
that changed the file.
For this reason, some versions of the Linux 2.6 NFS client abandon
WCC checking entirely, and simply trust their own data cache. On
these versions, the client can maintain a cache full of stale file
data if a file is opened for write. In this case, using file locking
is the best way to ensure that all clients see the latest version of
a file's data.
A system administrator can try using the "noac" mount option to
achieve attribute cache coherency among multiple clients. Almost
every client operation checks file attribute information. Usually the
client keeps this information cached for a period of time to reduce
network and server load. When "noac" is in effect, a client's file
attribute cache is disabled, so each operation that needs to check a
file's attributes is forced to go back to the server. This permits a
client to see changes to a file very quickly, at the cost of many
extra network operations.
Be careful not to confuse "noac" with "no data caching." The "noac"
mount option will keep file attributes up-to-date with the server,
but there are still races that may result in data incoherency between
client and server. If you need absolute cache coherency among
clients, applications can use file locking, where a client purges
file data when a file is locked, and flushes changes back to the
server before unlocking a file; or applications can open their files
with the O_DIRECT flag to disable data caching entirely.
For a better understanding of the compromises faced in the design of
NFS caching, see Callaghan's "NFS Illustrated."
On Nov 12, 2007, at 3:47 PM, Yonik Seeley wrote:
On Nov 12, 2007 4:43 PM, robert engels <[EMAIL PROTECTED]> wrote:
Why doesn't reopen get the 'read' lock, since commit has the write
lock, it should wait...
After lockless commits, there is no read lock!
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
