Re: Suggestion needed for a case of Lucene Migration with TokenStream

Fri, 25 Apr 2025 05:50:10 -0700 

Hi,
I'd like to mention the following: You are trying to upgrade Lucene from a really ancient version. Of course, basic concepts are still the same, but the serach engine and its APIs have changed dramatically, so just trying to "compile code and fix random stuff until it compiles" will not bring you to a working product. On top, it may make the product worse than before the update. 


To do the upgrade correctly, it is recommended to have somebody 
available (ideally the person who wrote the code originally) and then go 
though it line-by line and rewrite it. I am explicitely mentioning 
"rewrite" because that's what you should do! If you don't have a person 
that undertstands Lucene enough, I'd suggest to get help from outside. 
You need to understand every line of code when rewriting it. In addition 
there are many new features that make all that sepcial cases like 
PayLoads on Tokenstreams obsolete. I'd not recommend to use something 
payloads on terms nowadays.


Uwe

Am 24.04.2025 um 12:29 schrieb Mikhail Khludnev:

Right. TextField.TYPE_NOT_STORED should be used then.

On Thu, Apr 24, 2025 at 10:37 AM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:


Thanks Mikhail for the suggestion.
Now the previous exception has gone. But a new exception has come from
Field.java.
Here below are the exception details.
========
java.lang.IllegalArgumentException: TokenStream fields cannot be stored
         at org.apache.lucene.document.Field.<init>(Field.java:155)
=========

Can you please suggest here too?

Regards
Rajib


-----Original Message-----
From: Mikhail Khludnev <m...@apache.org>
Sent: 24 April 2025 12:10
To: java-user@lucene.apache.org
Subject: Re: Suggestion needed for a case of Lucene Migration with
TokenStream

Hi
Use TextField.TYPE_STORED as the third argument in new Field()
see

https://github.com/apache/lucene-solr/blob/e27f44e3d78dfcec230c97e0a1240e3751daeff9/lucene/core/src/java/org/apache/lucene/document/TextField.java#L35C33-L35C44


On Thu, Apr 24, 2025 at 8:37 AM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:


Hi Experts,

We are migrating Lucene from 2.4.1 to 8.11.2.

During Migration for a part of code, we are getting below exception in
8.11.2 based changes from Red line colored.
=============
java.lang.IllegalArgumentException: TokenStream fields must be indexed

and

tokenized
at org.apache.lucene.document.Field.<init>(Field.java:152)

I tied few options. But, could not able to resolve the error. Beiiw
Can somebody of you please help me to identify, where it is going as

wrong?

We had code based on 2.4.1 as like below:
===================================
Int currentVal<
http://10.238.236.101:8080/source/s?defs=currentVal&project=2025_RTM> =
//some value
PayloadTokenStream<


http://10.238.236.101:8080/source/s?defs=PayloadTokenStream&project=2025_RTM

tokenStream<
http://10.238.236.101:8080/source/s?refs=tokenStream&project=2025_RTM> =
new PayloadTokenStream<


http://10.238.236.101:8080/source/s?defs=PayloadTokenStream&project=2025_RTM

();

tokenStream<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#tokenStream

.setPayload<

http://10.238.236.101:8080/source/s?defs=setPayload&project=2025_RTM

(currentVal<

http://10.238.236.101:8080/source/s?defs=currentVal&project=2025_RTM>);
lucField<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#lucField

= new Field<
http://10.238.236.101:8080/source/s?defs=Field&project=2025_RTM>(config<
http://10.238.236.101:8080/source/s?defs=config&project=2025_RTM

.payloadUid<

http://10.238.236.101:8080/source/s?defs=payloadUid&project=2025_RTM

().name<http://10.238.236.101:8080/source/s?defs=name&project=2025_RTM

,
tokenStream<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#tokenStream

);

doc<http://10.238.236.101:8080/source/s?defs=doc&project=2025_RTM>.add<
http://10.238.236.101:8080/source/s?defs=add&project=2025_RTM>(lucField<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#lucField

);

......
public class PayloadTokenStream<


http://10.238.236.101:8080/source/s?refs=PayloadTokenStream&project=2025_RTM

extends TokenStream<
http://10.238.236.101:8080/source/s?defs=TokenStream&project=2025_RTM>{
public static String<
http://10.238.236.101:8080/source/s?defs=String&project=2025_RTM>
UID_PAYLOAD_START_VAL<


http://10.238.236.101:8080/source/s?refs=UID_PAYLOAD_START_VAL&project=2025_RTM

= "_UID_";
               private Token<
http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM> token<
http://10.238.236.101:8080/source/s?refs=token&project=2025_RTM> = new
Token<http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM

(UID_PAYLOAD_START_VAL<

http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#UID_PAYLOAD_START_VAL

,0,0);

               private byte[] buffer<
http://10.238.236.101:8080/source/s?refs=buffer&project=2025_RTM> = new
byte[4];
               private boolean returnToken<
http://10.238.236.101:8080/source/s?refs=returnToken&project=2025_RTM> =
false;

               public void setPayload<
http://10.238.236.101:8080/source/s?refs=setPayload&project=2025_RTM
(int
uid<http://10.238.236.101:8080/source/s?refs=uid&project=2025_RTM>){
                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[0]
= (byte)uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

;

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[1]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

8);

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[2]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

16);

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[3]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

24);

                              token<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#token

.setPayload<

http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#setPayload

(new
Payload<

http://10.238.236.101:8080/source/s?defs=Payload&project=2025_RTM

(buffer<

http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

));

                              returnToken = true;
               }
               public Token<
http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM> next<
http://10.238.236.101:8080/source/s?refs=next&project=2025_RTM>() throws
IOException<
http://10.238.236.101:8080/source/s?defs=IOException&project=2025_RTM>{
                              if (returnToken<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#returnToken

){
returnToken<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#returnToken

= false; return token<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#token

;
}
                              else { return null<
http://10.238.236.101:8080/source/s?defs=null&project=2025_RTM>; }

               }
}


We have made code based on 8.11.2 as like below:
==========================================
PayloadTokenStream tokenStream = new PayloadTokenStream();
tokenStream.setPayload(currentVal);
FieldType fieldType = new FieldType();
lucField = new Field(config.payloadUid().name, tokenStream, fieldType);
doc.add(lucField);
----
public class PayloadTokenStream<


http://10.238.236.101:8080/source/s?refs=PayloadTokenStream&project=2025_RTM

extends TokenStream<
http://10.238.236.101:8080/source/s?defs=TokenStream&project=2025_RTM>{
public static String<
http://10.238.236.101:8080/source/s?defs=String&project=2025_RTM>
UID_PAYLOAD_START_VAL<


http://10.238.236.101:8080/source/s?refs=UID_PAYLOAD_START_VAL&project=2025_RTM

= "_UID_";
private byte[] buffer<
http://10.238.236.101:8080/source/s?refs=buffer&project=2025_RTM> = new
byte[4];
               private boolean returnToken<
http://10.238.236.101:8080/source/s?refs=returnToken&project=2025_RTM> =
false;

               public void setPayload<
http://10.238.236.101:8080/source/s?refs=setPayload&project=2025_RTM
(int
uid<http://10.238.236.101:8080/source/s?refs=uid&project=2025_RTM>){
                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[0]
= (byte)uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

;

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[1]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

8);

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[2]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

16);

                              buffer<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer

[3]
= (byte)(uid<


http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid

24);

                              PayloadAttributeImpl attributeImpl = new
PayloadAttributeImpl(new BytesRef(buffer));
                             addAttributeImpl(attributeImpl);
                              returnToken = true;
               }
               public boolean incrementToken() throws IOException {
                              if (returnToken){
                                            returnToken = false;
                                            return true;
                              }
                              else {
                                            return false;
                              }
               }
}

Regards
Rajib



--
Sincerely yours
Mikhail Khludnev




--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org























					Reply via email to