Re: Regarding extracting Token as String from TokenStream.

2024-01-25 Thread Mikhail Khludnev
Hello Rajib

To extract tokens as strings from a Lucene TokenStream, you can use the
CharTermAttribute class. This attribute holds the current token's text.
Here's how you can modify your code:

ArrayList hotTokens = new ArrayList<>();
TokenStream stream = analyzer.tokenStream(field, new StringReader(query));
stream.reset();

CharTermAttribute charTermAttribute =
stream.addAttribute(CharTermAttribute.class);

while (stream.incrementToken()) {
String termText = charTermAttribute.toString();
hotTokens.add(termText);
}

In this code, we first add the CharTermAttribute to the TokenStream. Then,
inside the loop where we iterate over the tokens, we call
charTermAttribute.toString(), which gives us the current token as a string.
We then add this string to our list of hot tokens.


On Thu, Jan 25, 2024 at 2:26 PM Saha, Rajib 
wrote:
>
> Hi Experts,
>
> I need to get the Token as String from TokenStream for some further
processing.
> I have similar code as below.
> Can somebody please help me, how I can get the Token as String[termText]
in below code?
>
>
> ArrayList hotTokens = new ArrayList();
> TokenStream stream = analyzer.tokenStream(field, new StringReader(query));
> stream.reset();
>
> while (stream.incrementToken()) {
>   String termText = //Extract the Term from the stream. Need Help
here, how to extract it.
> hotTokens.add(termText);
> }
>
>
> Please let me know, if needed any further information from my side.
>
> Thanks In Advance.
>
> Regards
> Rajib
>
>


--
Sincerely yours
Mikhail Khludnev


Regarding extracting Token as String from TokenStream.

2024-01-25 Thread Saha, Rajib
Hi Experts,

I need to get the Token as String from TokenStream for some further processing.
I have similar code as below.
Can somebody please help me, how I can get the Token as String[termText] in 
below code?


ArrayList hotTokens = new ArrayList();
TokenStream stream = analyzer.tokenStream(field, new StringReader(query));
stream.reset();

while (stream.incrementToken()) {
  String termText = //Extract the Term from the stream. Need Help here, how 
to extract it.
hotTokens.add(termText);
}


Please let me know, if needed any further information from my side.

Thanks In Advance.

Regards
Rajib