[
https://issues.apache.org/jira/browse/LUCENE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603827#action_12603827
]
Christopher Morris commented on LUCENE-1292:
--------------------------------------------
Not sure I understand the question. There is no database or in-memory data.
There are three indexes: the static index (s), the pseudo index (p), and the
tag index (t). The index p doesn't exist - it is spoofed from t. The user view
is of a ParallelReader wrapping s and p.
Example:
Populate index s with four documents. It is a requirement that the documents
have a primary key field (a field known to contain a unique integer). The PKs
for these documents are 1,2,3,and 4; ascending order from the document with
docID 0.
The user tags documents with PK 1, 3, 4 with the Term("tag","foo").
The index p looks like:
Document 0
==========
tag : foo
Document 2
==========
tag : foo
Document 3
==========
tag : foo
The index t looks like the following (where {n} means the term is stored as
occuring at term position 'n'; the term position is (ab)used here to store the
primary key for use by p.termDocs()) :
Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo
The user now deletes the Term("tag", "foo") from the PK 3.
The index p looks like:
Document 0
==========
tag : foo
Document 3
==========
tag : foo
The index t looks like the following:
Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo
Document: 0
==========
tag : foo_DEL{3}
PK_tag : 3_foo
The method p.docFreq(new Term("tag", "foo")) runs the following block of code
(removed some fail-fast tests and tidy up code for clarity):
int docFreq = 0;
TermDocs additions = t.termDocs(new Term("tag","foo_ADD")); // A TermDocs with
one doc. The doc has a freq() of 3.
while (additions.next()) // true; once
{
docFreq += additions.freq(); // +3
}
deletions = t.termDocs(new Term("tag","foo_DEL")); // A TermDocs with one doc.
The doc has a freq() of 1.
while(deletions.next()) // true; once
{
docFreq -= additions.freq(); // -1
}
return docFreq; // 2
I'd hoped to have test results for very dynamic terms, but I forgot to check
that the server had any disk space available before I started the test.
> Tag Index
> ---------
>
> Key: LUCENE-1292
> URL: https://issues.apache.org/jira/browse/LUCENE-1292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Affects Versions: 2.3.1
> Reporter: Jason Rutherglen
> Attachments: lucene-1292.patch
>
>
> The problem the tag index solves is slow field cache loading and range
> queries, and reindexing an entire document to update fields that are not
> tokenized.
> The tag index holds untokenized terms with a docfreq of 1 in a term
> dictionary like index file. The file also stores the docs per term, similar
> to LUCENE-1278. The index also has a transaction log and in memory index for
> realtime updates to the tags. The transaction log is periodically merged
> into the existing tag term dictionary index file.
> The TagIndexReader extends IndexReader and is unified with a regular index by
> ParallelReader. There is a doc id to terms skip pointer file for the
> IndexReader.document method. This file contains a pointer for looking up the
> terms for a document.
> There is a higher level class that encapsulates writing a document with tag
> fields to IndexWriter and TagIndexWriter. This requires a hook into
> IndexWriter to coordinate doc ids and flushing segments to disk.
> The writer class could be as simple as:
> {code}
> public class TagIndexWriter {
>
> public void add(Term term, DocIdSetIterator iterator) {
> }
>
> public void delete(Term term, DocIdSetIterator iterator) {
> }
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
