Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

Uwe Schindler Fri, 10 Feb 2023 02:06:59 -0800

Hi,

the reason for this is that files in Lucene are always write-once. Wenever ever change a file after it was written and committed in the2-phase-commit. If you write some own index files, e.g. as part of anIndex Codec you must adhere this rule. See Docvalues or Livedocsimplementation for an example how "changes" are done in later commits:it creates new files with similar name and different suffix having somedelta-like content.

In general I would really avoid to deal with custom index files. SinceLucene 2 there were so many new features so it is never a good idea tohave your own index file format. Often Docvalues is the solution to allyour problems you had in early Lucene versions. If you add your ownstuff not knowing how the transactional model of Lucene work then youare possibly causing index corrumption. Index file formats need acarefully designed file format with thoughts on transactional safety andperformance.

If you want to just deal with termporary files, the Directory API allowsyou to maintain temporary files, too.


Uwe

Am 10.02.2023 um 06:49 schrieb Saha, Rajib:

Hi Uwe,

Thanks for the clarification.
We may have to rewrite the whole logic related to it, as seek functionality is 
no more for IndexOutput.

BTW, I have one more query related to it.
On playing around, I see, directory.createOutput(String name, IOContext 
context) API throwing FileAlreadyExistsException in case the file[say 
output.index] already exists in 8.11.2.
Now, I wondering, if my process is closed. And the in new process I want to use 
the same file[output.index] to keep appending to write. How, I can achieve it?

My Sample code:
========================================
Try {
        SimpleFSDirectory directory = new SimpleFSDirectory(new 
File("E:\\Lucene-index").toPath());
        IndexOutput output = directory.createOutput("output.index", 
IOContext.DEFAULT);
        output.writeInt(223344);
        output.writeString("Testing Testing");
        output.close();
} catch(Exception e) {
        e.printStackTrace();
}
==============================================



Regards
Rajib

-----Original Message-----
From: Uwe Schindler <u...@thetaphi.de>
Sent: 06 February 2023 16:46
To: java-user@lucene.apache.org
Subject: Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

Hi,

Since around Lucene 4 (maybe already in 3) there is no way to write
index files in random access anymore. All data must be written in
sequence (aka input stream). This is especially important as Lucene
index files use checksums since around Lucene 5.

Uwe

Am 06.02.2023 um 11:57 schrieb Saha, Rajib:

Hi Mikhail,

Thanks for all you’re your suggestions in one shot.
It helped us a lot.
Thank you very much once again. 😊

Need one more suggestion for below API.
==========================
IndexOutput.seek(long pos)
==========================

We have used it extensively in around 40-50 places.
Currently, this API is not there.

Could you please suggest, how we can handle the API in 8.11.2?

Regards
Rajib


-----Original Message-----
From: Mikhail Khludnev <m...@apache.org>
Sent: 01 February 2023 12:22
To: java-user@lucene.apache.org
Subject: Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

Hello, Rajib.

On Mon, Jan 30, 2023 at 4:07 PM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:

Hi Mikhail,

Thanks for your suggestion. It solved lots of cases today in my end. 😊

I need some more suggestions from your end. I am putting together as below
one by one:
================================
In 2.4, we have used couple of cases with APIs:

Field(String name, String value, Field.Store store, Field.Index index)
Field(String name, String value, Field.Store store, Field.Index index,
Field.TermVector termVector)

Check org.apache.lucene.document.StringField/TextField and its FieldType
constants.

In 8.11, I can see suitable API corresponding to it as :
Field(String name, Reader reader, IndexableFieldType type)

But, I am not clear, how can I use IndexableFieldType for Field.Store,
Field.Index, Field.TermVector.
Can you please suggest here?

check usages for org.apache.lucene.document.Field.Store
org.apache.lucene.document.FieldType#setIndexOptions
org.apache.lucene.document.FieldType#setStoreTermVectors

=================================

In 2.4, there was an API:
IndexReader.indexExists(File file)
This checks, if index files exists in the path.

In 8.11, any API, which can do the same job?

org.apache.lucene.index.DirectoryReader#indexExists

==================================
In 2.4, there was an API:
IndexReader.isLocked(FSDirectory fsdir)
IndexReader.unlock(Directory directory)

In 8.11, are IndexReader and IndexWritter synchronized enough internally
for not using the APIs?

org.apache.lucene.store.BaseDirectory#obtainLock
Lock.close()

IndexWriters are mutually exclusive via lock factory.
org.apache.lucene.index.DirectoryReader#open(org.apache.lucene.index.IndexWriter)
opens NRT reader i.e. search what not yet committed.

Or is there any other class contain the suitable similar APIs?

==================================
If I have to delete a document from Index file with Doc Id, which API to
use?

Previously there was an API
IndexReader.deleteDocument(docID)

org.apache.lucene.index.IndexWriter#deleteDocuments(org.apache.lucene.index.Term...)

==================================
IndexWritter. addIndexesNoOptimize(FSDirectory[])
IndexWriter.optimize()

Is there any similar concept in 8.11? If so, can you please help with APIs

org.apache.lucene.index.IndexWriter#addIndexes(org.apache.lucene.store.Directory...)
But it kicks merge underneath. Should be fine.

===================================

Regards
Rajib




-----Original Message-----
From: Mikhail Khludnev <m...@apache.org>
Sent: 29 January 2023 18:05
To: java-user@lucene.apache.org
Subject: Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

Hello,
You can use
        // gather list of valid fields from lucene
        Collection<String> fields = FieldInfos.getIndexedFields(ir);
to loop field names.
And then obtain terms per field vis

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucene.apache.org%2Fcore%2F8_11_2%2Fcore%2Forg%2Fapache%2Flucene%2Findex%2FMultiTerms.html%23getTerms-org.apache.lucene.index.IndexReader-java.lang.String-&data=05%7C01%7Crajib.saha%40sap.com%7C0bf8263294c94ff6180108db08338af0%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638112789690761979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=M5ixyr%2BiAcNqDZ4ggbmG6tKqnsHsqmmacU5wP7iEvlE%3D&reserved=0

On Sun, Jan 29, 2023 at 2:08 PM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:

Hi Mikhail,

Thanks for the reference link.
It really helped me.

In One of my requirement, I need to extract, all the Terms in a
IndexReader.
I was trying the reference code " Fields fields = reader.fields();" in
your reference link.

But, there is no "reader.fields()" in 8.11.2.

Could you please suggest someway to extract all the Terms with an
IndexReader or some alternative ways?

Regards
Rajib

-----Original Message-----
From: Mikhail Khludnev <m...@apache.org>
Sent: 19 January 2023 04:26
To: java-user@lucene.apache.org
Subject: Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

[You don't often get email from m...@apache.org. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]

Hello, Rajib.
API were evolved since 2.4, but it should be clear

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucene.apache.org%2Fcore%2F8_11_2%2Fcore%2Forg%2Fapache%2Flucene%2Findex%2Fpackage-summary.html%23fields&data=05%7C01%7Crajib.saha%40sap.com%7C0bf8263294c94ff6180108db08338af0%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638112789690761979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HMCAtPvnCu4ApbDxt%2BegbcFXKWfJYNz8xOMuRCDEgNc%3D&reserved=0

On Wed, Jan 18, 2023 at 1:11 PM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:

Hi All,

We are in a process for conversion of Lucene from  2.4.0 to 8.11.2 for

our

platform code.
We have used extensively Lucene in our code.

We have replaced several of our code to Lucene 8.11.2 APIs.

But, few places, we are stuck of which New Lucene APIs to use, as not
getting any suitable match.

Can somebody help me, how we can convert below code using Lucene 2.4.0

to

8.11.2?


ProcessDocs(IndexReader reader, Term t) {

                final TermDocs termDocs = reader.termDocs();
                termDocs.seek(t);
                while (termDocs.next()) {
              //Some internal function to process the doc.
              forEach.process(termDocs.doc());
        }

}

Regards
Rajib

--
Sincerely yours
Mikhail Khludnev

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft.me%2FMUST_SEARCH&data=05%7C01%7Crajib.saha%40sap.com%7C0bf8263294c94ff6180108db08338af0%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638112789690761979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xIiE98I1EzpBl3El%2FhKjlGIBW8BNBvgfGXdvpUnMkwA%3D&reserved=0

A caveat: Cyrillic!

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

--
Sincerely yours
Mikhail Khludnev

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft.me%2FMUST_SEARCH&data=05%7C01%7Crajib.saha%40sap.com%7C0bf8263294c94ff6180108db08338af0%7C42f7676cf455423c82f6dc2d99791af7%7C0%7C0%7C638112789690761979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xIiE98I1EzpBl3El%2FhKjlGIBW8BNBvgfGXdvpUnMkwA%3D&reserved=0
A caveat: Cyrillic!

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Re: Need help for conversion code from Lucene 2.4.0 to 8.11.2

Reply via email to