Thanks to all for responding.
OK, to summarise (and in the interests of keeping things simple):
merging/updating indexers:
-------------------------------------
touch $idx_lockfile
rm -rf $index
mv $fresh $index
unlink $idx_lockfile
touch $idx_has_changed
# ie, don't bother with symlinks and whatnot since active searchers
have cached everything.
searchers:
--------------
loop
$lockfound = 0;
while (-f $idx_lockfile) {
# pause for 100ms
select(undef, undef, undef, 0.1); }
$lockfound = 1;
}
if ($lockfound) {
$lockfound = 0;
# restart_searcher
goto restart;
}
end
As I'm typing this though, I'm thinking, what about
Lucy::Search::IndexSearcher->serve?
It's in a loop hiding it's internals - how do I get *it* to honour my
$idx_lockfile?
Hack LucyX/Remote/SearchServer.pm's sub serve()? Perhaps between
read( $client_sock, $buf, $len );
+ sleep while $idx_lockfile is extant
my $response = $dispatch{$method}->( $self, thaw($buf) );
My final question: Peter says "For full index swap-outs, I just let
existing Searchers finish and re-open
themselves when they realize the underlying index is different" --
peter are you saying LucyX::Remote::SearchServer does this
automatically on it's own without intervention needed, or should I
have a wrapper with LucyX::Remote::SearchServer ensconced which
monitors the indexes and kills/restarts LucyX::Remote::SearchServer
when a changed index is detected (-f $idx_has_changed)?
urgh, this is getting dirty ugly.
/throws toys out cot, sobs
On 11/2/11, Peter Karman <[email protected]> wrote:
> Marvin Humphrey wrote on 11/1/11 12:36 PM:
>> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>>> I'm pondering about the best approach to handle fresh/stale indexes
>>> while searches are happening (ie, we don't want to disrupt an active
>>> search).
>>
>> Once an IndexSearcher has opened successfully, it no longer needs access
>> to
>> the index dir[1] -- you can actually wipe the index dir and the
>> IndexSearcher
>> will keep functioning indefinitely. This is achieved by caching open
>> filehandles to all files the IndexSearcher could ever need access to, so
>> that
>> it will still have access even if the files are deleted[2].
>>
>
> fwiw, this is exactly what I do.
>
> For incremental updates, my Searcher wrapper code[1] detects when the index
> has
> changed and re-opens itself.
>
> For full index swap-outs, I just let existing Searchers finish and re-open
> themselves when they realize the underlying index is different. When the
> open
> filehandles eventually close, then the filesystem actually removes the old
> index.
>
> [1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher
>
> --
> Peter Karman . http://peknet.com/ . [email protected]
>
