liferoad commented on issue #31285:
URL: https://github.com/apache/beam/issues/31285#issuecomment-2780713185
```
chunks = [Chunk(content=Content(text='This is a simple test document. It has
multiple'), id='0791e41b-2607-49af-938c-36a2ee8804...c splitting.'),
id='63f67fb4-8347-4c3a-8829-673602f5a439', index=2, metadata={'source':
'simple.txt'}, embedding=None)]
def check_token_lengths(chunks):
for chunk in chunks:
# Verify each chunk's token length is within limits
num_tokens = len(tokenizer.encode(chunk.content.text))
if not num_tokens <= 10:
> raise BeamAssertException(
f"Chunk has {num_tokens} tokens, expected <= 10")
E apache_beam.testing.util.BeamAssertException: Chunk has 12 tokens,
expected <= 10 [while running 'assert_that/Match']
apache_beam/ml/rag/chunking/langchain_test.py:172: BeamAssertException```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
