GitHub user nwellnhof opened a pull request:
https://github.com/apache/lucy/pull/54
Improve locking code (v2)
[Diff to previous pull
request](https://gist.github.com/nwellnhof/f262bcd53c3d69281748571410a0bdbd).
Fixes [LUCY-323](https://issues.apache.org/jira/browse/LUCY-323).
- Major rewrite of FilePurger, making it do less work.
- Eliminate global deletion lock, making PolyReader and FilePurger never
wait for a lock.
- Redesign Lock API in preparation for native locks.
- Remove LockFactory.
- Fix locking bug with non-default merge lock timeout.
- Fix locking bug with concurrent BackgroundMergers.
- Port some tests to C.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nwellnhof/lucy improve-locking-v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucy/pull/54.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #54
----
commit cb6004330a8e211c2944bb470bdea936d9d69feb
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-15T04:10:07Z
Separate purging of snapshots and aborted merges
Purge aborted merge when starting an Indexer/BGMerger session and
purge snapshots at the end of Commit.
Rework FilePurger to use a single hash for purged and spared files
when purging snapshots. Optimize Folder handling assuming that
Snapshots only contain entries local to the index folder.
Make sure to list the contents of the real directory, not the
virtual compound file directory. CFReaderDirHandle lists both real
and virtual files. Close the CFReader before deleting a segment to
allow immediate deletion on Windows.
Port BGMerger tests to C.
TODO: Move Perl BGMerge test to t/binding.
commit 38d296557923e07c3717b54cf271f43daa55dba2
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-16T14:58:45Z
Merge SharedLock into LockFileLock
Supplying the lock type when requesting a lock better matches other
file lock APIs. Merging the implementations allows to convert
Maybe_Delete_File into a static function.
Make shared and exclusive locks check for locks of the other type
except for exclusive locks created with `exclusive_only`.
Clear_Stale is broken for shared locks but will be removed with the
following commit.
commit a99bbdb44a67240f8759e9e38a3e607e657dbafc
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-16T15:47:03Z
Always check for stale locks in Is_Locked
Note that checking for stale locks isn't necessary when requesting
shared locks.
commit d9c661d070e34a5150e0d4513d01575bbc9d6ccf
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-16T16:47:33Z
Remove LockFactory and make Lock a private class
If there are users who really use custom locks, this can be reverted.
commit 09e8393bf60fbe211cd19ec0c7bfb8be5041f86e
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T12:53:59Z
Don't allow double obtain/release
commit 571bb9b771acc1f82dc31925d90e143d35f9dd4c
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T16:19:43Z
Rework snapshot and deletion locks
Request exclusive or shared snapshot locks instead of the global
deletion lock. This removes the need to retry lock requests and
reduces contention.
commit 53cbc1088e816229a4c106d8eefd24e189c0a6cb
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T17:25:35Z
Improve merge lock handling
Don't check the merge lock with Is_Locked in
FilePurger_Purge_Aborted_Merge. This prevented background mergers from
cleaning up because they already held the merge lock. Require that
callers hold the merge lock instead.
Don't check the merge lock with Is_Locked in Indexer_init. Since we
hold the write lock, and the merge.json file from aborted merge has
been removed, the presence of a merge.json file is a reliable indicator
that it belongs to an active background merger. Move the merge lock
request form S_maybe_merge to Indexer_init.
Also, S_maybe_merge used to Obtain the merge lock instead of merely
Requesting it. Waiting for a potentially long-running background merger
slows things down unnecessarily and increases the chance for write lock
contention. This didn't affect the default configuration which has a
zero merge lock timeout.
commit 1924910df5d7629aed736d6a2766cb6650ab432a
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T17:34:31Z
Change lock acquitision order in BGMerger_init
Obtain the merge lock before the write lock so that an already running
background merger won't abort when failing to acquire the write lock
during commit.
Release the merge lock at the very end of BGMerger_Commit. It doesn't
really matter when we release the merge lock after we acquired the
write lock in BGMerger_Prepare_Commit. I think it makes the code
clearer to release it at the very end of BGMerger_Commit.
commit 7c7b560388e97b778ddeb83ef46fd3ff4ac6b548
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T18:13:59Z
Remove Lock_Is_Locked
commit 0674cdd78844152bef89c15898f4550b7dc2d1d8
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T19:26:13Z
Don't hide I/O errors behind LockErr
Only return a LockErr if there's lock contention, not on unrelated
I/O errors.
commit 7232b5807cc97b14335ddd158ff33b32f034322e
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-17T20:38:56Z
Port LockFileLock tests to C
commit f7dd386e06f327a12b282a6e5bb78f3b2c6b7b82
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-19T13:09:02Z
Release locks on destruction
commit 240a1fbfa2fa88048d17daca57ef47f786cea092
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-19T15:21:00Z
Move ivars from Lock to LockFileLock
commit 87652fb7ad7e363e41f4f08dd38ead57cbc24a1f
Author: Nick Wellnhofer <[email protected]>
Date: 2017-02-19T16:01:50Z
Move LockFileLock to separate file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
