hansva opened a new issue, #9451: URL: https://github.com/apache/hop/issues/9451
### Apache Hop version? 2.20.0 ### Java version? 21 ### Operating system Other ### What happened? Two bugs and a performance problem in the HTML to text transform (`plugins/transforms/html2text`). **Bugs** 1. **Rows with empty HTML are dropped.** When the HTML field is null or blank, `process()` returns without writing the row, so the row disappears from the stream. It should pass on with an empty text. 2. **Parallel mode can wait forever.** With "Use parallelism" on, a row that fails in a worker is logged, but its job is never counted down. At the end of the input the transform waits for a count that never reaches zero. The failure should stop the transform, or send the row to the error hop when error handling is on. **Performance** 3. **A new parser for every row.** `Jsoup.parseBodyFragment(html)` builds a new parser and tree builder for each row. In clean mode the safelist and the cleaner are rebuilt for each row as well. Parsing is almost all of what this transform does, so that set-up is a visible share of every row. ### Proposed changes - **Empty HTML:** write the row with an empty text in the output field. - **Parallel mode:** always count a job down when it ends. On a failure, send the row to the error hop when error handling is on; otherwise stop the transform with the error. Count a job before it is handed to the pool, not after. - **One parser:** keep one jsoup `Parser` per transform copy and parse each fragment with it. A `Parser` takes a lock for every parse, so in parallel mode give each worker thread its own, or the workers queue up behind one parser. - **One safelist and cleaner:** in clean mode, build them once from the selected safelist. - **Error handling:** the conversion itself is not covered by the error hop today, only the row-level log line is. Put the conversion inside it. The output text does not change. Reusing the parser gives the same result as today for all three modes: whole text, normalised text and clean. Not part of this ticket: in clean mode the HTML is parsed, written back to HTML and parsed again. Cleaning the first document directly would be much faster, but it gives different output, so it is left as it is. ### Behavior that changes - A pipeline whose input has rows with empty HTML now gets those rows in the output. Before, they were silently removed. - A failing row in parallel mode now fails the pipeline or goes to the error hop. Before, the pipeline hung. ### Measured gain jsoup 1.23.2 on 200,000 small HTML documents, new parser per row against one reused parser: | Mode | Before | After | Gain | |---|---|---|---| | Whole text | 237k rows/s | 287k rows/s | 1.21x | | Normalised text | 239k rows/s | 275k rows/s | 1.15x | | Clean, basic safelist | 66k-77k rows/s | 83k-90k rows/s | 1.18-1.25x | Results were identical for every document in all three modes. The transform itself was not benchmarked yet; parsing is close to its whole cost, so about 1.2x is expected on benchmark B150. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
