Thanks Mikhail for the suggestion.
Now the previous exception has gone. But a new exception has come from
Field.java.
Here below are the exception details.
========
java.lang.IllegalArgumentException: TokenStream fields cannot be stored
at org.apache.lucene.document.Field.<init>(Field.java:155)
=========
Can you please suggest here too?
Regards
Rajib
-----Original Message-----
From: Mikhail Khludnev <m...@apache.org>
Sent: 24 April 2025 12:10
To: java-user@lucene.apache.org
Subject: Re: Suggestion needed for a case of Lucene Migration with TokenStream
Hi
Use TextField.TYPE_STORED as the third argument in new Field()
see
https://github.com/apache/lucene-solr/blob/e27f44e3d78dfcec230c97e0a1240e3751daeff9/lucene/core/src/java/org/apache/lucene/document/TextField.java#L35C33-L35C44
On Thu, Apr 24, 2025 at 8:37 AM Saha, Rajib <rajib.s...@sap.com.invalid>
wrote:
> Hi Experts,
>
> We are migrating Lucene from 2.4.1 to 8.11.2.
>
> During Migration for a part of code, we are getting below exception in
> 8.11.2 based changes from Red line colored.
> =============
> java.lang.IllegalArgumentException: TokenStream fields must be indexed and
> tokenized
> at org.apache.lucene.document.Field.<init>(Field.java:152)
>
> I tied few options. But, could not able to resolve the error. Beiiw
> Can somebody of you please help me to identify, where it is going as wrong?
>
> We had code based on 2.4.1 as like below:
> ===================================
> Int currentVal<
> http://10.238.236.101:8080/source/s?defs=currentVal&project=2025_RTM> =
> //some value
> PayloadTokenStream<
> http://10.238.236.101:8080/source/s?defs=PayloadTokenStream&project=2025_RTM>
> tokenStream<
> http://10.238.236.101:8080/source/s?refs=tokenStream&project=2025_RTM> =
> new PayloadTokenStream<
> http://10.238.236.101:8080/source/s?defs=PayloadTokenStream&project=2025_RTM
> >();
> tokenStream<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#tokenStream
> >.setPayload<
> http://10.238.236.101:8080/source/s?defs=setPayload&project=2025_RTM
> >(currentVal<
> http://10.238.236.101:8080/source/s?defs=currentVal&project=2025_RTM>);
> lucField<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#lucField>
> = new Field<
> http://10.238.236.101:8080/source/s?defs=Field&project=2025_RTM>(config<
> http://10.238.236.101:8080/source/s?defs=config&project=2025_RTM
> >.payloadUid<
> http://10.238.236.101:8080/source/s?defs=payloadUid&project=2025_RTM
> >().name<http://10.238.236.101:8080/source/s?defs=name&project=2025_RTM>,
> tokenStream<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#tokenStream
> >);
> doc<http://10.238.236.101:8080/source/s?defs=doc&project=2025_RTM>.add<
> http://10.238.236.101:8080/source/s?defs=add&project=2025_RTM>(lucField<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/serviceplugins/src/com/sap/businessobjects/platform/search/lucene/index/engine/LuceneIndexEngine.java#lucField
> >);
>
> ......
> public class PayloadTokenStream<
> http://10.238.236.101:8080/source/s?refs=PayloadTokenStream&project=2025_RTM>
> extends TokenStream<
> http://10.238.236.101:8080/source/s?defs=TokenStream&project=2025_RTM>{
> public static String<
> http://10.238.236.101:8080/source/s?defs=String&project=2025_RTM>
> UID_PAYLOAD_START_VAL<
> http://10.238.236.101:8080/source/s?refs=UID_PAYLOAD_START_VAL&project=2025_RTM>
> = "_UID_";
> private Token<
> http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM> token<
> http://10.238.236.101:8080/source/s?refs=token&project=2025_RTM> = new
> Token<http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM
> >(UID_PAYLOAD_START_VAL<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#UID_PAYLOAD_START_VAL
> >,0,0);
> private byte[] buffer<
> http://10.238.236.101:8080/source/s?refs=buffer&project=2025_RTM> = new
> byte[4];
> private boolean returnToken<
> http://10.238.236.101:8080/source/s?refs=returnToken&project=2025_RTM> =
> false;
>
> public void setPayload<
> http://10.238.236.101:8080/source/s?refs=setPayload&project=2025_RTM>(int
> uid<http://10.238.236.101:8080/source/s?refs=uid&project=2025_RTM>){
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[0]
> = (byte)uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >;
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[1]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>8);
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[2]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>16);
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[3]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>24);
> token<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#token
> >.setPayload<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#setPayload>(new
> Payload<http://10.238.236.101:8080/source/s?defs=Payload&project=2025_RTM
> >(buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer
> >));
> returnToken = true;
> }
> public Token<
> http://10.238.236.101:8080/source/s?defs=Token&project=2025_RTM> next<
> http://10.238.236.101:8080/source/s?refs=next&project=2025_RTM>() throws
> IOException<
> http://10.238.236.101:8080/source/s?defs=IOException&project=2025_RTM>{
> if (returnToken<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#returnToken>){
> returnToken<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#returnToken>
> = false; return token<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#token>;
> }
> else { return null<
> http://10.238.236.101:8080/source/s?defs=null&project=2025_RTM>; }
>
> }
> }
>
>
> We have made code based on 8.11.2 as like below:
> ==========================================
> PayloadTokenStream tokenStream = new PayloadTokenStream();
> tokenStream.setPayload(currentVal);
> FieldType fieldType = new FieldType();
> lucField = new Field(config.payloadUid().name, tokenStream, fieldType);
> doc.add(lucField);
> ----
> public class PayloadTokenStream<
> http://10.238.236.101:8080/source/s?refs=PayloadTokenStream&project=2025_RTM>
> extends TokenStream<
> http://10.238.236.101:8080/source/s?defs=TokenStream&project=2025_RTM>{
> public static String<
> http://10.238.236.101:8080/source/s?defs=String&project=2025_RTM>
> UID_PAYLOAD_START_VAL<
> http://10.238.236.101:8080/source/s?refs=UID_PAYLOAD_START_VAL&project=2025_RTM>
> = "_UID_";
> private byte[] buffer<
> http://10.238.236.101:8080/source/s?refs=buffer&project=2025_RTM> = new
> byte[4];
> private boolean returnToken<
> http://10.238.236.101:8080/source/s?refs=returnToken&project=2025_RTM> =
> false;
>
> public void setPayload<
> http://10.238.236.101:8080/source/s?refs=setPayload&project=2025_RTM>(int
> uid<http://10.238.236.101:8080/source/s?refs=uid&project=2025_RTM>){
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[0]
> = (byte)uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >;
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[1]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>8);
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[2]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>16);
> buffer<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#buffer>[3]
> = (byte)(uid<
> http://10.238.236.101:8080/source/xref/2025_RTM/platform.services.search.java/framework/java/sdk/src/com/sap/businessobjects/platform/search/sdk/index/PayloadTokenStream.java#uid
> >>>24);
> PayloadAttributeImpl attributeImpl = new
> PayloadAttributeImpl(new BytesRef(buffer));
> addAttributeImpl(attributeImpl);
> returnToken = true;
> }
> public boolean incrementToken() throws IOException {
> if (returnToken){
> returnToken = false;
> return true;
> }
> else {
> return false;
> }
> }
> }
>
> Regards
> Rajib
>
>
--
Sincerely yours
Mikhail Khludnev
