[
https://issues.apache.org/jira/browse/SOLR-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310894#comment-17310894
]
Jason Gerlowski commented on SOLR-13608:
----------------------------------------
Since committing the bulk of this work, there has been a bit of ongoing
flakiness in the backup/restore tests that I've been digging into. There are
at least two types of failures I've run into.
The first is a ClassCastException in the {{TrackingBackupRepository}} test
utility. I haven't had any luck understanding or reproducing this yet. I've
noticed it mostly on 'main'.
The second failure is characterized by a CorruptIndexException thrown by the
backup code that checks copied files for corruption. This problem I understand
and think I've fixed with the commits in the preceding 2 comments. In short:
backups compute a checksum of each file as it's being copied, and compares that
checksum against an "expected" value that Lucene puts in the footer of most
files. The Solr backup code uses Lucene's {{CodecUtil}} to extract this
expected checksum value, and CodecUtil assumes a particular footer format in
each index file. But it turns out that Solr's test randomization sometimes
chooses SimpleTextCodec - an instructional codec that doesn't use the standard
footer format. CodecUtil isn't smart enough to recognize this, so
{{CodecUtil.validateFooter}} throws a CorruptIndexException when given an
inputstream for a SimpleTextCodec file.
I've "handled" this in the commits above by suppressing this codec in Solr's
backup tests, which seemed acceptable since SimpleTextCodec doesn't really aim
to be a fully compatible codec and there's already precedent for suppressing it
based on that. It'd be awesome down the road if CodecUtil was smart enough to
detect these oddball codecs that don't contain the standard footer or if Lucene
had some more general API to accomplish the same task, but I'm not going to
take that on here.
I'm still working on the other ClassCast failure that occurs sporadically in
various backup tests. Hopefully this flakiness can be all knocked out soon!
> Incremental backup for Solr
> ---------------------------
>
> Key: SOLR-13608
> URL: https://issues.apache.org/jira/browse/SOLR-13608
> Project: Solr
> Issue Type: Sub-task
> Reporter: Cao Manh Dat
> Assignee: Jason Gerlowski
> Priority: Major
> Fix For: main (9.0), 8.9
>
> Time Spent: 82h 40m
> Remaining Estimate: 14h 20m
>
> SIP-12 lays out a plan for adding support for incremental backups to Solr.
> At a high level, the idea is that Solr will be able to store multiple backups
> in the same location, and backups beyond the first one will only upload those
> files that were not uploaded by previous backups.
> This involves changes to the file structure within a particular backup
> location. It also entails changes to some of the backup/restore API
> parameters and semantics, to accommodate storing multiple backups in the same
> place, etc.
> This ticket covers the changes required for this functionality, as described
> in SIP-12 unless mentioned specifically below. It does not implement all of
> [SIP-12.|https://cwiki.apache.org/confluence/display/SOLR/SIP-12%3A+Incremental+Backup+and+Restore]
> Same-collection-restoration, support for popular proprietary blob stores,
> etc. are left for separate tickets in an attempt to keep PRs manageable and
> conceptually cohesive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
