akashorabek commented on code in PR #34559:
URL: https://github.com/apache/beam/pull/34559#discussion_r2031006636
##########
sdks/python/apache_beam/ml/rag/chunking/langchain_test.py:
##########
@@ -167,7 +167,7 @@ def test_huggingface_tokenizer_splitter(self):
def check_token_lengths(chunks):
for chunk in chunks:
# Verify each chunk's token length is within limits
- num_tokens = len(tokenizer.encode(chunk.content.text))
+ num_tokens = len(tokenizer.tokenize(chunk.content.text))
Review Comment:
Apparently, this is a known
[issue](https://github.com/langchain-ai/langchain/issues/30184) where the
`encode` function adds 2 extra tokens. Replaced it with `tokenize` since it
solved the problem
[here](https://github.com/langchain-ai/langchain/pull/30185/files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
