Adalennis opened a new issue, #9282:
URL: https://github.com/apache/hop/issues/9282
### Apache Hop version?
2.20.0
### Java version?
Java 21
### Operating system
macOS
### What happened?
Found while testing the Text chunker on 2.20.0 (code unchanged on main). All
with the Structure strategy, chunk size 1000, overlap 0.
**1. AsciiDoc block delimiters become headings, and the text after them is
lost**
In `AsciiDocDocumentParser` the heading pattern `^(=+)\s+(.+)$` lets `\s+`
match a line break, so a `====` example block delimiter is read as a heading
and the next line as its title. Those sections have an empty body and are
dropped.
Input (Content type ASCIIDOC, same result with AUTO):
= Title / Intro. / == Section / Before the block. / ==== / Example text
inside the block. / ==== / After the block. / == Next / More text.
Chunks: `Title`, `Title > Section: Before the block.`, `Title > Next: More
text.`. "Example text inside the block." and "After the block." are in no chunk.
**2. `#` lines inside a Markdown code fence become headings**
# Guide / Intro. / ```bash / # download the zip / curl -O … / ``` /
After the block.
gives two chunks: `Guide: Intro. ```bash` and `download the zip: curl -O …
``` After the block.` The fence is split in two and a shell comment becomes a
section title.
**3. A skipped heading level loses its parent**
`# Guide` followed by `### Detail` gives the breadcrumb `Detail` instead of
`Guide > Detail`. When the stack has no entry for the level just above, the
parser falls back to the root instead of the nearest ancestor.
**4. Closing hashes stay in the title**
`## Install ##` gives the title `Install ##`. In CommonMark the closing
sequence is not part of the heading.
**5. Content type AUTO picks the wrong parser**
- Markdown that mentions `<pipeline>` or `<workflow>`, or that starts with
`{` (for example a Hugo shortcode `{{< note >}}`), is detected as Hop XML or
metadata JSON. The parse fails (`[Fatal Error] :1:1: Content is not allowed in
prolog.` on stderr) and the whole text becomes one chunk with the `#` markers
still in it.
- A single line that starts with `= `, such as `= 42`, makes AUTO pick
AsciiDoc. In `# Guide / The answer: / = 42 / ## Next / More text.` the `#`
headings are ignored and `42` becomes the only section title.
### Issue Priority
Priority: 3
### Issue Component
Component: Hop Gui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]