Adalennis opened a new issue, #9283: URL: https://github.com/apache/hop/issues/9283
### Apache Hop version? 2.20.0 ### Java version? Java 21 ### Operating system macOS ### What happened? Found while testing the Text chunker on 2.20.0 (code unchanged on main). **1. Overlap starts in the middle of a word** The next chunk starts at `chunkEnd - overlap` with no word alignment. Lorem ipsum, Character, size 100, overlap 20: chunks 2-4 start with `ncididunt`, `ostrud`, `is aute`. The page says words are not cut, and #8415 notes that mid-word cuts retrieve poorly. The existing `0115` character golden encodes the current behaviour. **2. Overlap can degenerate to one chunk per character** When a chunk backs off to a space and ends up shorter than the overlap, the next chunk only moves forward by 1 character. 20 x "supercalifragilistic" (419 characters), Character, size 25, overlap 15: 228 chunks (`alifragilistic `, `lifragilistic `, `ifragilistic `, …). The pipeline check shows no warning. **3. Paragraph and Structure apply the overlap** The page says overlap is "Character strategy only", and the check warns "The Paragraph strategy does not apply chunk overlap; the configured overlap is ignored". But both strategies pass the overlap to the character fallback for oversized paragraphs and sections: with size 100 and overlap 20, Paragraph chunks start with `ncididunt`, `ostrud`, `is aute` and Structure chunks with `empor incididunt`, `eniam, quis`. Structure gets no check warning. **4. Positions in split Structure sections ignore the heading** `# Title` + blank line + 319 characters of words, Structure, size 100: chunk_start_position is 0, 88, 176, 264. The text of chunk 2 (`word011`) starts at 97 in the input: every chunk after the first is 9 too low (the length of `# Title\n\n`). The position is the heading position plus the offset in the stripped body. **5. Whitespace-only chunks in Character and Paragraph** `"a" + 30 spaces + "b"`, Character, size 10: chunks 2 and 3 are 10 spaces each. Paragraph with separator `---` and size 6: `one--- ---two` gives `one`, ` `, `two`. #8751 removed whitespace-only chunks for Structure only; is that intended for the other two strategies? **6. Character can cut a surrogate pair** `😀😁😂`, Character, size 5: chunks `😀😁?` and `?`. The split is on Java char positions, so an emoji is cut in half. ### Issue Priority Priority: 3 ### Issue Component Component: Hop Gui -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
