In my application I was queuing IDs of appropriate record in Database
not whole document. Document was created right before adding it to
index. All this work was done in separated thread, so other threads
responded very quickly.
It depends on your application and at what speed your new data comes.
If application is killed or crashes then you'll lost all documents that
was buffered in memory (IndexWriter.minMergeDocs property).
Actually I don't know how JVM's shutdown hooks reacts to different kill
signals. My though is that application should not be killed. :)
Also if you need to enshure that your data is indexed properly then on
startup you can do special check what data was already added to database
but wasn't added to index and enqueue it into queue... But it also
depends...
Rob Staveley (Tom) wrote:
This is a good idea. I had been worried about the additional heap
requirement maintaining a queue, without being able to serialize/deserialize
Documents (i.e. a build up of Lucene Documents in RAM). I have been
marshalling addDocument() calls using a synchronized object; the same
threads have been taking responsibility for creating Documents
(unsynchronized) and adding them to the index writer (synchronized). I guess
I could have a one Document queue feeding a single addDocument thread, which
would effectively be the same approach, but which would make it easier to
ensure that only the create Document thread is killed when I get a SIGTERM
and the addDocument thread is left to run its course (assuming it hasn't
hanged!).
Having said that, I'm not sure what I could do in a shutdown hook, which
wouldn't already have been done by a SIGTERM to get the hanged thread to
terminate. The reason for SIGKILL was that the daemon wouldn't be killed by
SIGTERM. I guess I'd feel more confident about using SIGKILL, if I knew that
the uninterruptible hanged thread was creating a Document, which I could
interrupt without corrupting the index, rather than adding the document to
the index, which is liable to result in orphaned files and/or a corrupted
index, if killed.
-----Original Message-----
From: Volodymyr Bychkoviak [mailto:[EMAIL PROTECTED]
Sent: 06 June 2006 10:54
To: java-user@lucene.apache.org
Subject: Re: Compound / non-compound index files and SIGKILL
If your content handlers should respond quickly then you should move
indexing process to separate thread and maintain items in queue.
Rob Staveley (Tom) wrote:
This is a real eye-opener, Volodymyr. Many thanks. I guess that means
that my orphan-producing hangs must be addDocument() calls, and not in
the content handlers, as I'd previously assumed. I'll put some debug
before and after my addDocument() calls to confirm (and point my
writer's infoStream to System.out).
-----Original Message-----
From: Volodymyr Bychkoviak [mailto:[EMAIL PROTECTED]
Sent: 05 June 2006 18:33
To: java-user@lucene.apache.org
Subject: Re: Compound / non-compound index files and SIGKILL
Hi.
My five cents :)
It might be helpful to know how lucene is working with compound files.
When segment is flushed to disk it is written uncompound and after
that is merged into single .cfs file. If you don't change default
setting for using compound files (which is on) this is only place (I
guess) for these files to appear.
If you're working with large indexes, than merging segments can take a
while (Maybe here is your problem? :) ) (merging happens on
addDocument() call). If you will kill indexing process during such
merge you'll get many orphaned files...
You can just run optimize on this index. You'll get three files:
segments, deletable, <segment>.cfs; you can look name of segment in
'segments' file. Everything else is 'garbage' - you can delete it.
Rob Staveley (Tom) wrote:
I've been indexing live data into a compound index from an MTA. I'm
resolving a bunch of problems unrelated to Lucene (disparate hangs in
my content handlers). When I get a hang, I typically need to kill my
daemon, alas more often than not using kill -9 (SIGKILL).
However, these SIGKILLs are leaving large temporary(?) files, which I
guess
are non-compound index files transiently extracted from the working
.cfs
files:
-rw-r--r-- 1 373138432 Jun 2 13:42 _18hup.fdt
-rw-r--r-- 1 5054464 Jun 2 13:42 _18hup.fdx
-rw-r--r-- 1 426 Jun 2 13:42 _18hup.fnm
-rw-r--r-- 1 457253888 Jun 2 09:22 _15djq.fdt
-rw-r--r-- 1 6205440 Jun 2 09:22 _15djq.fdx
-rw-r--r-- 1 426 Jun 2 09:21 _15djq.fnm
They are left intact after restarting my daemon. Presumably they are
not treated as being part of the compound index. I see no
corresponding .cfs file for them.
As a consequence of these - I suspect - I am getting a very large
overall disk requirement for my index, presumably because of
replicated field
data.
My guess is that the field data in the orphaned .fdt files needs to
be regenerated.
In another index directory from a previous test run (again with
SIGKILLs),
I
have 98 GB of index files, with only 12 BG devoted to compound files
for
the
field index (.cfs). The rest of the disk space is used by orphaned
uncompounded index files; I see 51 GB devoted to uncompounded field
data (.fdt), 13 BG devoted to term positions (.prx) and 13 BG devoted
to term frequencies (.frq).
Here's my question:
How can I attempt to merge these orphaned into the compound index,
using IndexWriter.addIndexes(), or would I be foolish attempting this?
--
regards,
Volodymyr Bychkoviak
--
regards,
Volodymyr Bychkoviak
