date:20161012

Bug#724610: marked as done (incrementally updated databases eventually become corrupt)

2016-10-12 Thread Debian Bug Tracking System

Your message dated Thu, 13 Oct 2016 15:24:35 +1300
with message-id <20161013022435.wgl64whzrsd2d...@survex.com>
and subject line Re: Bug#724610: Acknowledgement (incrementally updated
databases eventually become corrupt)
has caused the Debian Bug report #724610,
regarding incrementally updated databases eventually become corrupt
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)

--
724610: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724610
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: libsearch-xapian-perl
Version: 1.2.10.0-1
Severity: normal

ikiwiki's search plugin incrementally updates a xapian index as a wiki
is edited. Certian of my larger wikis tend to corrupt themselves every
month or two, preventing xapian-omega from finding anything.
xapian-check says there is a btree error.

I see this frequently on my main ikiwiki server. But I've heard from
others who have also had this kind of corruption elsewhere so I don't
think it's bad hardware. (I've also migrated my main ikiwiki server to
new hardware at least twice in the years since I started experiencing
this problem.)

It used to be the case that this could cause
Search::Xapian::WritableDatabase->new to throw an exception, as seen
here:

http://www.branchable.com/bugs/Exception:_Cannot_open_tables_at_consistent_revisions_at___47__usr__47__lib__47__perl5__47__Search__47__Xapian__47__WritableDatabase.pm_line_41./#comment-c159ea3f9be35fcd9ed0eeedb162e816

I have not seen that behavior since upgrading to wheezy. Now when the
database is corrupt, further changes to it don't cause a crash (which is
good); xapian-omega just never finds any matches.

ikiwiki calls methods like add_term and replace_document_by_term and
delete_document_by_term. It may be exercising the incremental update
of xapian databases more frequently than is typical and exposing some
bug. It's also the only thing in Debian that seems to use this perl
library, so it could be that the bug in in this library and not in
xapian itself.

I suppose it's possible that the bug is in ikiwiki. For example, it
could be that two ikiwiki processes end up manipulating the same xapian
database concurrently. I don't know if that is allowed, or likely to
corrupt it. But ikiwiki is supposed to use a lock to prevent two processes
from both making stateful changes at the same time, and I have audited
it and cannot find anywhere that it updates the xapian database without
first taking that lock.

I also wonder if this could be a problem with the flushing of the
database. Ikiwiki relies on the automatic flushing behavior, and assumes
that the database will be closed and flushed automatically when the
xapian database object is destroyed at program termination. It also
could be the case that ikiwiki sometimes crashes, and this could
interfere with that.

--
see shy jo

signature.asc
Description: Digital signature
--- End Message ---
--- Begin Message ---
Control: fixed -1 1.2.21-1
Control: fixed -1 1.4.0-1

On Wed, Jul 01, 2015 at 02:21:23PM +1200, Olly Betts wrote:
> It's possible it's a bug which has since been fixed - e.g. 1.2.21
> fixed a bug where the cursor could end up with bad data, which could
> cause DatabaseCorruptError to be thrown when the database on disk was
> fine. The bad data could theoretically end up written back, though I've
> never actually seen that happen before something notices the data isn't
> right and throws an exception.

I'm increasingly believing this to be a manifestation of the bug
referred to above (which was fixed in xapian-core 1.2.21-1, and that fix
is also in 1.4.0-1).

I'd like to have had a report from someone experiencing this before
1.2.21 and it going away upon upgrading. We don't have that, but there
have been no further reports of this in over a year, and the newest
version it has been reported in was 1.2.19-1 (or at least using
libsearch-xapian-perl 1.2.19.0-1, and that's released upstream on the
same schedule as xapian-core and typically updated in Debian at a
similar time), and keeping this open indefinitely isn't useful, so I
think it's time to close this bug.

88 matches

Mail list logo