This is an automated email from the ASF dual-hosted git repository.

hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git


The following commit(s) were added to refs/heads/main by this push:
     new 0e8ee5589e Text chunker: overlap cuts words, can produce a chunk per 
character, wrong positions in split Structure sections, whitespace-only chunks 
and split surrogate pairs, fixes #9283 (#9364)
0e8ee5589e is described below

commit 0e8ee5589e1d8105571ae738be12252ced7eb862
Author: Bart Maertens <[email protected]>
AuthorDate: Wed Nov 11 12:39:57 2026 +0100

    Text chunker: overlap cuts words, can produce a chunk per character, wrong 
positions in split Structure sections, whitespace-only chunks and split 
surrogate pairs, fixes #9283 (#9364)
---
 .../pages/pipeline/transforms/textchunker.adoc     |  10 +-
 ...hpl => 0115-text-chunker-character-overlap.hpl} |  24 ++--
 .../0115-text-chunker-paragraph-separator.hpl      |   3 +
 ...r.hpl => 0115-text-chunker-structure-split.hpl} |  18 +--
 .../golden-text-chunker-character-overlap.csv      |  14 +++
 .../datasets/golden-text-chunker-character.csv     |   4 +-
 .../golden-text-chunker-paragraph-separator.csv    |   2 +
 .../golden-text-chunker-structure-split.csv        |  16 +++
 .../transforms/main-0115-text-chunker.hwf          |   6 +
 .../golden-text-chunker-character-overlap.json     |  39 ++++++
 .../golden-text-chunker-structure-split.json       |  39 ++++++
 .../0115-text-chunker-character-overlap UNIT.json  |  43 +++++++
 .../0115-text-chunker-structure-split UNIT.json    |  43 +++++++
 .../transforms/chunker/TextChunkerMeta.java        |  15 ++-
 .../chunking/CharacterChunkingStrategy.java        | 116 +++++++++++++----
 .../chunking/ParagraphChunkingStrategy.java        |  10 +-
 .../chunking/StructureChunkingStrategy.java        |  15 ++-
 .../transforms/chunker/document/DocumentNode.java  |  49 +++++++-
 .../chunker/document/HeadingParserSupport.java     |  20 ++-
 .../chunker/messages/messages_en_US.properties     |   4 +-
 .../transforms/chunker/TextChunkerMetaTest.java    |  33 +++++
 .../chunking/CharacterChunkingStrategyTest.java    | 140 +++++++++++++++++++--
 .../chunking/ParagraphChunkingStrategyTest.java    |  58 +++++++++
 .../chunking/StructureChunkingStrategyTest.java    |  76 +++++++++++
 24 files changed, 725 insertions(+), 72 deletions(-)

diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
index cb16bf1a64..e329402eca 100644
--- 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
+++ 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
@@ -36,7 +36,7 @@ The transform has no dependency on an AI provider or a model. 
It is plain text p
 |Strategy |Behaviour
 
 |Character
-|Split on a fixed character count, backing off to the nearest word boundary so 
words are not cut in half. Overlap carries the tail of each chunk into the 
next, which keeps a sentence that straddles a boundary retrievable from both 
sides.
+|Split on a fixed character count, backing off to the nearest word boundary so 
words are not cut in half. Only a word longer than the chunk size is cut, and 
never inside a surrogate pair such as an emoji. Overlap carries the last whole 
words of each chunk into the next, which keeps a sentence that straddles a 
boundary retrievable from both sides.
 
 |Paragraph
 |Split on paragraph breaks, packing whole paragraphs up to the chunk size. A 
paragraph longer than the chunk size falls back to character splitting. By 
default a paragraph break is a blank line (any `\LF`, `\CR` or `\CRLF` line 
ending); the optional *Paragraph separator* option replaces it with a literal 
separator.
@@ -79,7 +79,7 @@ The Hop-native parsers make a pipeline, a workflow or a 
metadata file retrievabl
 |Content type|Document format used by the Structure strategy.
 |Content type field|Optional field naming the content type per row, for a 
stream that mixes formats.
 |Chunk size|Maximum chunk size in characters. For Paragraph and Structure this 
is a ceiling that triggers the character fallback.
-|Chunk overlap|Characters carried from the end of one chunk into the start of 
the next. Character strategy only.
+|Chunk overlap|Maximum number of characters carried from the end of one chunk 
into the start of the next. The overlap starts at a word boundary, so it holds 
whole words and can be shorter than this value, or empty when no word starts in 
that span. Paragraph and Structure only apply it inside a paragraph or section 
that is split with the character fallback.
 |Include metadata|Add the chunk metadata fields below to the output.
 |Chunk index field|Output field for the position of the chunk within its 
document.
 |Chunk start position field|Output field for the character offset of the chunk 
in the source text.
@@ -89,8 +89,10 @@ The Hop-native parsers make a pipeline, a workflow or a 
metadata file retrievabl
 
 == Notes
 
-Chunk overlap applies to the Character strategy. The Paragraph and Structure 
strategies split on natural boundaries, where an overlap would duplicate whole 
paragraphs or sections between neighbouring chunks.
+Chunk overlap never crosses a paragraph or section boundary. The Paragraph and 
Structure strategies split on natural boundaries, where an overlap would 
duplicate whole paragraphs or sections between neighbouring chunks. They only 
apply it between the pieces of a paragraph or section that is longer than the 
chunk size. Every chunk adds text the previous chunk did not hold: when an 
overlap would only repeat the previous chunk, the next chunk starts where the 
previous one ended.
 
-The Structure strategy drops chunks that contain only whitespace. The chunk 
index and total chunk count cover only the chunks that are emitted. Text that 
yields no chunks produces a single row with an empty chunk and a total of 0, 
the same as empty text.
+The chunk start position is the character offset in the input text. For a 
section of the Structure strategy, the first chunk starts at the section 
heading and the chunks a long section is split into start where their own text 
is.
+
+All strategies drop chunks that contain only whitespace. The chunk index and 
total chunk count cover only the chunks that are emitted. Text that yields no 
chunks produces a single row with an empty chunk and a total of 0, the same as 
empty text.
 
 Keep *Include metadata* enabled when the chunks are going into a vector store: 
the document ID and chunk index are what let a later search merge neighbouring 
chunks back into a passage, and re-index a single document without touching the 
rest.
diff --git 
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl 
b/integration-tests/transforms/0115-text-chunker-character-overlap.hpl
similarity index 79%
copy from integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
copy to integration-tests/transforms/0115-text-chunker-character-overlap.hpl
index e880461384..62000c8dd2 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-character-overlap.hpl
@@ -19,9 +19,9 @@ limitations under the License.
 -->
 <pipeline>
   <info>
-    <name>0115-text-chunker-paragraph-separator</name>
+    <name>0115-text-chunker-character-overlap</name>
     <name_sync_with_filename>Y</name_sync_with_filename>
-    <description>Paragraph chunking with a custom paragraph separator instead 
of blank lines.</description>
+    <description>Character overlap starts at a word, never repeats a whole 
chunk, drops whitespace-only chunks and keeps surrogate pairs together 
(#9283).</description>
     <extended_description/>
     <pipeline_version/>
     <pipeline_type>Normal</pipeline_type>
@@ -32,9 +32,9 @@ limitations under the License.
     
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
     
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
     <created_user>-</created_user>
-    <created_date>2026/10/03 12:00:00.000</created_date>
+    <created_date>2026/09/16 12:00:00.000</created_date>
     <modified_user>-</modified_user>
-    <modified_date>2026/10/03 12:00:00.000</modified_date>
+    <modified_date>2026/09/16 12:00:00.000</modified_date>
     <key_for_session_key/>
     <is_key_private>N</is_key_private>
   </info>
@@ -57,7 +57,16 @@ limitations under the License.
     </partitioning>
     <data>
       <line>
-        <item>First part.&lt;PARA&gt;Second part.&lt;PARA&gt;Third part.</item>
+        <item>supercalifragilistic supercalifragilistic supercalifragilistic 
supercalifragilistic supercalifragilistic</item>
+      </line>
+      <line>
+        <item>Apache Hop is a data orchestration platform for building 
pipelines</item>
+      </line>
+      <line>
+        <item>a                                                            
b</item>
+      </line>
+      <line>
+        <item>😀😀😀😀😀😀😀😀😀😀😀😀😀</item>
       </line>
     </data>
     <fields>
@@ -88,10 +97,9 @@ limitations under the License.
     </partitioning>
     <inputField>content</inputField>
     <outputChunkField>chunk_text</outputChunkField>
-    <chunkingStrategy>PARAGRAPH</chunkingStrategy>
-    <paragraphSeparator>&lt;PARA&gt;</paragraphSeparator>
+    <chunkingStrategy>CHARACTER</chunkingStrategy>
     <chunkSize>25</chunkSize>
-    <chunkOverlap>0</chunkOverlap>
+    <chunkOverlap>15</chunkOverlap>
     <includeMetadata>Y</includeMetadata>
     <chunkIndexField>chunk_index</chunkIndexField>
     <chunkStartPosField>chunk_start_position</chunkStartPosField>
diff --git 
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl 
b/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
index e880461384..8ec6bcbf07 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
@@ -59,6 +59,9 @@ limitations under the License.
       <line>
         <item>First part.&lt;PARA&gt;Second part.&lt;PARA&gt;Third part.</item>
       </line>
+      <line>
+        <item>one&lt;PARA&gt;                    &lt;PARA&gt;two</item>
+      </line>
     </data>
     <fields>
       <field>
diff --git 
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl 
b/integration-tests/transforms/0115-text-chunker-structure-split.hpl
similarity index 79%
copy from integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
copy to integration-tests/transforms/0115-text-chunker-structure-split.hpl
index e880461384..465f223eb7 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-structure-split.hpl
@@ -19,9 +19,9 @@ limitations under the License.
 -->
 <pipeline>
   <info>
-    <name>0115-text-chunker-paragraph-separator</name>
+    <name>0115-text-chunker-structure-split</name>
     <name_sync_with_filename>Y</name_sync_with_filename>
-    <description>Paragraph chunking with a custom paragraph separator instead 
of blank lines.</description>
+    <description>A Structure section longer than the chunk size: the chunks 
after the first start where their text is, after the heading, and the overlap 
starts at a word (#9283).</description>
     <extended_description/>
     <pipeline_version/>
     <pipeline_type>Normal</pipeline_type>
@@ -32,9 +32,9 @@ limitations under the License.
     
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
     
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
     <created_user>-</created_user>
-    <created_date>2026/10/03 12:00:00.000</created_date>
+    <created_date>2026/09/16 12:00:00.000</created_date>
     <modified_user>-</modified_user>
-    <modified_date>2026/10/03 12:00:00.000</modified_date>
+    <modified_date>2026/09/16 12:00:00.000</modified_date>
     <key_for_session_key/>
     <is_key_private>N</is_key_private>
   </info>
@@ -57,7 +57,7 @@ limitations under the License.
     </partitioning>
     <data>
       <line>
-        <item>First part.&lt;PARA&gt;Second part.&lt;PARA&gt;Third part.</item>
+        <item># Title&#10;&#10;word001 word002 word003 word004 word005 word006 
word007 word008 word009 word010 word011 word012 word013 word014 word015 word016 
word017 word018 word019 word020 word021 word022 word023 word024 word025 word026 
word027 word028 word029 word030 word031 word032 word033 word034 word035 word036 
word037 word038 word039 word040</item>
       </line>
     </data>
     <fields>
@@ -88,10 +88,10 @@ limitations under the License.
     </partitioning>
     <inputField>content</inputField>
     <outputChunkField>chunk_text</outputChunkField>
-    <chunkingStrategy>PARAGRAPH</chunkingStrategy>
-    <paragraphSeparator>&lt;PARA&gt;</paragraphSeparator>
-    <chunkSize>25</chunkSize>
-    <chunkOverlap>0</chunkOverlap>
+    <chunkingStrategy>STRUCTURE</chunkingStrategy>
+    <contentType>MARKDOWN</contentType>
+    <chunkSize>100</chunkSize>
+    <chunkOverlap>20</chunkOverlap>
     <includeMetadata>Y</includeMetadata>
     <chunkIndexField>chunk_index</chunkIndexField>
     <chunkStartPosField>chunk_start_position</chunkStartPosField>
diff --git 
a/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
 
b/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
new file mode 100644
index 0000000000..c0851d54cb
--- /dev/null
+++ 
b/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
@@ -0,0 +1,14 @@
+chunk_text,chunk_index,chunk_start_position,total_chunks
+"supercalifragilistic ",0,0,5
+"supercalifragilistic ",1,21,5
+"supercalifragilistic ",2,42,5
+"supercalifragilistic ",3,63,5
+"supercalifragilistic",4,84,5
+"Apache Hop is a data ",0,0,4
+"orchestration platform ",1,21,4
+"platform for building ",2,35,4
+"for building pipelines",3,44,4
+"a                        ",0,0,2
+"           b",1,50,2
+"😀😀😀😀😀😀😀😀😀😀😀😀",0,0,2
+"😀😀😀😀😀😀😀😀😀",1,8,2
diff --git 
a/integration-tests/transforms/datasets/golden-text-chunker-character.csv 
b/integration-tests/transforms/datasets/golden-text-chunker-character.csv
index c979ed665d..c0bbd313f8 100644
--- a/integration-tests/transforms/datasets/golden-text-chunker-character.csv
+++ b/integration-tests/transforms/datasets/golden-text-chunker-character.csv
@@ -1,4 +1,4 @@
 chunk_text,chunk_index,chunk_start_position,total_chunks
 "Apache Hop is a data orchestration ",0,0,3
-"estration platform for building ",1,25,3
-" building pipelines and workflows.",2,47,3
+"platform for building pipelines and ",1,35,3
+"and workflows.",2,67,3
diff --git 
a/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
 
b/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
index 5fd8616245..2a423cbb83 100644
--- 
a/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
+++ 
b/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
@@ -2,3 +2,5 @@ chunk_text,chunk_index,chunk_start_position,total_chunks
 "First part.",0,0,3
 "Second part.",1,17,3
 "Third part.",2,35,3
+"one",0,0,2
+"two",1,35,2
diff --git 
a/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv 
b/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv
new file mode 100644
index 0000000000..fcc170f128
--- /dev/null
+++ 
b/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv
@@ -0,0 +1,16 @@
+chunk_text,chunk_index,chunk_start_position,total_chunks
+"Title
+
+word001 word002 word003 word004 word005 word006 word007 word008 word009 
word010 word011 ",0,0,5
+"Title
+
+word010 word011 word012 word013 word014 word015 word016 word017 word018 
word019 word020 ",1,81,5
+"Title
+
+word019 word020 word021 word022 word023 word024 word025 word026 word027 
word028 word029 ",2,153,5
+"Title
+
+word028 word029 word030 word031 word032 word033 word034 word035 word036 
word037 word038 ",3,225,5
+"Title
+
+word037 word038 word039 word040",4,297,5
diff --git a/integration-tests/transforms/main-0115-text-chunker.hwf 
b/integration-tests/transforms/main-0115-text-chunker.hwf
index 00d468a3e4..e1c26aaac9 100644
--- a/integration-tests/transforms/main-0115-text-chunker.hwf
+++ b/integration-tests/transforms/main-0115-text-chunker.hwf
@@ -66,6 +66,12 @@ limitations under the License.
         <test_name>
           <name>0115-text-chunker-structure-markdown UNIT</name>
         </test_name>
+        <test_name>
+          <name>0115-text-chunker-character-overlap UNIT</name>
+        </test_name>
+        <test_name>
+          <name>0115-text-chunker-structure-split UNIT</name>
+        </test_name>
         <test_name>
           <name>0117-text-chunker-redaction UNIT</name>
         </test_name>
diff --git 
a/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
 
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
new file mode 100644
index 0000000000..4e44e2cd1d
--- /dev/null
+++ 
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
@@ -0,0 +1,39 @@
+{
+  "base_filename": "golden-text-chunker-character-overlap.csv",
+  "name": "golden-text-chunker-character-overlap",
+  "description": "Character overlap on word boundaries, no repeated chunks, no 
whitespace-only chunks, surrogate pairs kept together",
+  "dataset_fields": [
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 2,
+      "field_precision": -1,
+      "field_name": "chunk_text",
+      "field_format": ""
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "chunk_index",
+      "field_format": "0"
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "chunk_start_position",
+      "field_format": "0"
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "total_chunks",
+      "field_format": "0"
+    }
+  ]
+}
diff --git 
a/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
 
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
new file mode 100644
index 0000000000..efb56b588c
--- /dev/null
+++ 
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
@@ -0,0 +1,39 @@
+{
+  "base_filename": "golden-text-chunker-structure-split.csv",
+  "name": "golden-text-chunker-structure-split",
+  "description": "Structure: positions and word-aligned overlap of a section 
split with the character fallback",
+  "dataset_fields": [
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 2,
+      "field_precision": -1,
+      "field_name": "chunk_text",
+      "field_format": ""
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "chunk_index",
+      "field_format": "0"
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "chunk_start_position",
+      "field_format": "0"
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 5,
+      "field_precision": 0,
+      "field_name": "total_chunks",
+      "field_format": "0"
+    }
+  ]
+}
diff --git 
a/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
 UNIT.json 
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
 UNIT.json
new file mode 100644
index 0000000000..59039144d8
--- /dev/null
+++ 
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
 UNIT.json     
@@ -0,0 +1,43 @@
+{
+  "variableValues": [],
+  "database_replacements": [],
+  "autoOpening": false,
+  "basePath": "",
+  "golden_data_sets": [
+    {
+      "field_mappings": [
+        {
+          "transform_field": "chunk_text",
+          "data_set_field": "chunk_text"
+        },
+        {
+          "transform_field": "chunk_index",
+          "data_set_field": "chunk_index"
+        },
+        {
+          "transform_field": "chunk_start_position",
+          "data_set_field": "chunk_start_position"
+        },
+        {
+          "transform_field": "total_chunks",
+          "data_set_field": "total_chunks"
+        }
+      ],
+      "field_order": [
+        "chunk_text",
+        "chunk_index",
+        "chunk_start_position",
+        "total_chunks"
+      ],
+      "transform_name": "OUTPUT",
+      "data_set_name": "golden-text-chunker-character-overlap"
+    }
+  ],
+  "input_data_sets": [],
+  "name": "0115-text-chunker-character-overlap UNIT",
+  "description": "Character overlap on word boundaries, no repeated chunks, no 
whitespace-only chunks, surrogate pairs kept together",
+  "trans_test_tweaks": [],
+  "persist_filename": "",
+  "pipeline_filename": "./0115-text-chunker-character-overlap.hpl",
+  "test_type": "UNIT_TEST"
+}
diff --git 
a/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
 UNIT.json 
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
 UNIT.json
new file mode 100644
index 0000000000..a5eeedfc22
--- /dev/null
+++ 
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
 UNIT.json       
@@ -0,0 +1,43 @@
+{
+  "variableValues": [],
+  "database_replacements": [],
+  "autoOpening": false,
+  "basePath": "",
+  "golden_data_sets": [
+    {
+      "field_mappings": [
+        {
+          "transform_field": "chunk_text",
+          "data_set_field": "chunk_text"
+        },
+        {
+          "transform_field": "chunk_index",
+          "data_set_field": "chunk_index"
+        },
+        {
+          "transform_field": "chunk_start_position",
+          "data_set_field": "chunk_start_position"
+        },
+        {
+          "transform_field": "total_chunks",
+          "data_set_field": "total_chunks"
+        }
+      ],
+      "field_order": [
+        "chunk_text",
+        "chunk_index",
+        "chunk_start_position",
+        "total_chunks"
+      ],
+      "transform_name": "OUTPUT",
+      "data_set_name": "golden-text-chunker-structure-split"
+    }
+  ],
+  "input_data_sets": [],
+  "name": "0115-text-chunker-structure-split UNIT",
+  "description": "Structure: positions and word-aligned overlap of a section 
split with the character fallback",
+  "trans_test_tweaks": [],
+  "persist_filename": "",
+  "pipeline_filename": "./0115-text-chunker-structure-split.hpl",
+  "test_type": "UNIT_TEST"
+}
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
index addbf66392..4056aaf175 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
@@ -147,7 +147,10 @@ public class TextChunkerMeta extends 
BaseTransformMeta<TextChunker, TextChunkerD
   @HopMetadataProperty(key = "chunkSize", injectionKey = "CHUNK_SIZE")
   private String chunkSize = "1000";
 
-  /** The number of characters to overlap between chunks (for CHARACTER 
strategy). */
+  /**
+   * The maximum number of characters, in whole words, to overlap between 
chunks. Paragraph and
+   * Structure only apply it inside a paragraph or section they split with the 
character fallback.
+   */
   @GuiWidgetElement(
       order = "0800",
       type = GuiElementType.TEXT,
@@ -421,8 +424,14 @@ public class TextChunkerMeta extends 
BaseTransformMeta<TextChunker, TextChunkerD
       warning(remarks, transformMeta, "TextChunker.Validation.OverlapWarning");
     }
 
-    if (chunkingStrategy == ChunkingStrategyType.PARAGRAPH && resolvedOverlap 
> 0) {
-      warning(remarks, transformMeta, 
"TextChunker.Validation.ParagraphOverlapIgnored");
+    if ((chunkingStrategy == ChunkingStrategyType.PARAGRAPH
+            || chunkingStrategy == ChunkingStrategyType.STRUCTURE)
+        && resolvedOverlap > 0) {
+      remarks.add(
+          new CheckResult(
+              ICheckResult.TYPE_RESULT_COMMENT,
+              BaseMessages.getString(PKG, 
"TextChunker.Validation.OverlapOnlyWhenSplit"),
+              transformMeta));
     }
 
     if (!Utils.isEmpty(paragraphSeparator) && chunkingStrategy != 
ChunkingStrategyType.PARAGRAPH) {
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
index 7351c93fd5..c32d4c5197 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
@@ -23,6 +23,11 @@ import org.apache.hop.pipeline.transforms.chunker.Chunk;
 /**
  * Chunking strategy that splits text on fixed character count while 
respecting word boundaries.
  * This ensures that chunks don't split words in the middle.
+ *
+ * <p>The overlap carries whole words, at most {@code overlap} characters, 
from the end of one chunk
+ * into the next. Only a word longer than the chunk size, which has to be cut 
anyway, is overlapped
+ * on characters. Every chunk ends past the end of the previous one, chunks 
that hold only
+ * whitespace are dropped, and a surrogate pair is never cut in half.
  */
 public class CharacterChunkingStrategy implements ChunkingStrategy {
 
@@ -35,50 +40,117 @@ public class CharacterChunkingStrategy implements 
ChunkingStrategy {
     }
 
     if (overlap < 0) {
-      chunks.add(new Chunk(text, 0, 0, text.length()));
+      if (!text.isBlank()) {
+        chunks.add(new Chunk(text, 0, 0, text.length()));
+      }
       return chunks;
     }
 
     overlap = Math.min(overlap, maxSize - 1);
 
+    int textLength = text.length();
     int start = 0;
+    int previousEnd = 0;
     int chunkIndex = 0;
-    int textLength = text.length();
 
     while (start < textLength) {
-      int end = Math.min(start + maxSize, textLength);
+      int chunkEnd = chunkEnd(text, start, maxSize);
 
-      if (end == textLength) {
-        String chunkContent = text.substring(start, end);
-        chunks.add(new Chunk(chunkContent, chunkIndex++, start, end));
-        break;
+      if (start < previousEnd && chunkEnd <= previousEnd) {
+        // The overlap would only repeat text the previous chunk already 
holds: no new text would
+        // reach this chunk. Continue where the previous chunk ended instead.
+        start = previousEnd;
+        continue;
+      }
+
+      String chunkContent = text.substring(start, chunkEnd);
+      if (!chunkContent.isBlank()) {
+        chunks.add(new Chunk(chunkContent, chunkIndex++, start, chunkEnd));
       }
 
-      int separatorPos = -1;
-      for (int i = end - 1; i >= start; i--) {
-        if (Character.isWhitespace(text.charAt(i))) {
-          separatorPos = i;
-          break;
-        }
+      if (chunkEnd >= textLength) {
+        break;
       }
 
-      int chunkEnd = separatorPos >= start ? separatorPos + 1 : end;
+      previousEnd = chunkEnd;
+      start = nextStart(text, start, chunkEnd, overlap);
+    }
+
+    return chunks;
+  }
+
+  /**
+   * The end of the chunk that starts at {@code start}: just after the last 
whitespace within {@code
+   * maxSize} characters, or a hard cut when there is none, moved so it does 
not split a surrogate
+   * pair.
+   */
+  private static int chunkEnd(String text, int start, int maxSize) {
+    int textLength = text.length();
+    int end = Math.min(start + maxSize, textLength);
+    if (end == textLength) {
+      return end;
+    }
 
-      if (chunkEnd <= start) {
-        chunkEnd = Math.min(start + maxSize, textLength);
+    for (int i = end - 1; i >= start; i--) {
+      if (Character.isWhitespace(text.charAt(i))) {
+        return i + 1;
       }
+    }
 
-      String chunkContent = text.substring(start, chunkEnd);
-      chunks.add(new Chunk(chunkContent, chunkIndex++, start, chunkEnd));
+    if (splitsSurrogatePair(text, end)) {
+      // Keep the pair together: in this chunk if there is room for something 
before it,
+      // otherwise in this chunk even though it then holds one character more 
than the maximum.
+      return end - 1 > start ? end - 1 : end + 1;
+    }
+    return end;
+  }
 
-      if (chunkEnd >= textLength) {
-        break;
+  /**
+   * Where the chunk after {@code [start, chunkEnd)} starts: the first word 
that begins in the last
+   * {@code overlap} characters of the chunk, or {@code chunkEnd} when no word 
begins there. When
+   * the chunk was cut inside a word that is too long to back off from, the 
overlap stays on
+   * characters, as no word boundary is available.
+   */
+  private static int nextStart(String text, int start, int chunkEnd, int 
overlap) {
+    if (overlap <= 0) {
+      return chunkEnd;
+    }
+
+    int candidate = Math.max(start + 1, chunkEnd - overlap);
+
+    int wordStart = candidate;
+    if (!Character.isWhitespace(text.charAt(wordStart - 1))) {
+      // Inside a word: skip to its end.
+      while (wordStart < chunkEnd && 
!Character.isWhitespace(text.charAt(wordStart))) {
+        wordStart++;
       }
+    }
+    while (wordStart < chunkEnd && 
Character.isWhitespace(text.charAt(wordStart))) {
+      wordStart++;
+    }
+    if (wordStart < chunkEnd) {
+      return wordStart;
+    }
 
-      start = overlap > 0 ? Math.max(start + 1, chunkEnd - overlap) : chunkEnd;
+    boolean cutInsideWord =
+        chunkEnd < text.length()
+            && !Character.isWhitespace(text.charAt(chunkEnd - 1))
+            && !Character.isWhitespace(text.charAt(chunkEnd));
+    if (!cutInsideWord) {
+      return chunkEnd;
+    }
+    if (splitsSurrogatePair(text, candidate)) {
+      return candidate - 1 > start ? candidate - 1 : candidate + 1;
     }
+    return candidate;
+  }
 
-    return chunks;
+  /** True when {@code position} falls between the two halves of a surrogate 
pair. */
+  private static boolean splitsSurrogatePair(String text, int position) {
+    return position > 0
+        && position < text.length()
+        && Character.isHighSurrogate(text.charAt(position - 1))
+        && Character.isLowSurrogate(text.charAt(position));
   }
 
   @Override
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
index 9bb9b949cd..2747e08aeb 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
@@ -29,7 +29,7 @@ import org.apache.hop.pipeline.transforms.chunker.Chunk;
  * #setSeparator(String)} when the text uses a custom multi-byte marker 
instead. A paragraph that on
  * its own exceeds {@code maxSize} is split further with {@link 
CharacterChunkingStrategy}, because
  * a chunk larger than the limit would be rejected downstream by the embedding 
model rather than
- * simply being large.
+ * simply being large. The overlap applies only within that split, never 
between paragraphs.
  *
  * <p>Consecutive paragraphs are packed into one chunk while they fit within 
{@code maxSize}, which
  * is the usual contract for an embedding chunker: it keeps related text 
together and avoids
@@ -103,7 +103,7 @@ public class ParagraphChunkingStrategy implements 
ChunkingStrategy {
   public List<Chunk> chunk(String text, int maxSize, int overlap) {
     List<Chunk> chunks = new ArrayList<>();
 
-    if (text == null || text.isEmpty()) {
+    if (text == null || text.isBlank()) {
       return chunks;
     }
     if (maxSize <= 0) {
@@ -183,8 +183,12 @@ public class ParagraphChunkingStrategy implements 
ChunkingStrategy {
     return paragraphs;
   }
 
+  /**
+   * Whitespace-only paragraphs are left out, so they never become a chunk on 
their own. Packing
+   * still slices the source text, so a chunk that spans one keeps it.
+   */
   private static void addParagraph(List<Paragraph> paragraphs, String text, 
int start, int end) {
-    if (end > start) {
+    if (end > start && !text.substring(start, end).isBlank()) {
       paragraphs.add(new Paragraph(text.substring(start, end), start));
     }
   }
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
index 65347dff12..c5912173b8 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
@@ -70,24 +70,33 @@ public class StructureChunkingStrategy implements 
ChunkingStrategy {
                 sectionText,
                 chunks.size(),
                 section.getStartPosition(),
-                section.getStartPosition() + sectionText.length()));
+                section.getBodyStartPosition() + section.getBody().length()));
       } else {
         String breadcrumb = section.breadcrumb();
         String prefix = breadcrumb.isEmpty() ? "" : breadcrumb + "\n\n";
         String body = section.getBody();
         int bodyMax = prefix.isEmpty() ? maxSize : Math.max(1, maxSize - 
prefix.length());
         List<Chunk> split = fallback.chunk(body, bodyMax, overlap);
+        boolean firstPart = true;
         for (Chunk part : split) {
           if (part.getContent().isBlank()) {
             continue;
           }
           String content = prefix.isEmpty() ? part.getContent() : prefix + 
part.getContent();
+          // The first part starts the section, heading included, like an 
unsplit section does.
+          // The others start where their text is in the source: the body is 
stripped and follows
+          // the heading, so their offsets are relative to the body, not to 
the section.
+          int start =
+              firstPart
+                  ? section.getStartPosition()
+                  : section.getBodyStartPosition() + part.getStartPosition();
           chunks.add(
               new Chunk(
                   content,
                   chunks.size(),
-                  section.getStartPosition() + part.getStartPosition(),
-                  section.getStartPosition() + part.getEndPosition()));
+                  start,
+                  section.getBodyStartPosition() + part.getEndPosition()));
+          firstPart = false;
         }
       }
     }
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
index d1d826115d..8980d14a05 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
@@ -28,12 +28,31 @@ public final class DocumentNode {
   private final String title;
   private final String body;
   private final int startPosition;
+  private final int bodyStartPosition;
   private final List<DocumentNode> children;
 
+  /**
+   * A node whose body starts at {@code startPosition}: a document root, or a 
body that is not a
+   * slice of the source text, such as a summary built by a Hop pipeline or 
metadata parser.
+   */
   public DocumentNode(String title, String body, int startPosition, 
List<DocumentNode> children) {
+    this(title, body, startPosition, startPosition, children);
+  }
+
+  /**
+   * @param startPosition where the section starts in the source text, its 
heading included
+   * @param bodyStartPosition where {@code body} starts in the source text
+   */
+  public DocumentNode(
+      String title,
+      String body,
+      int startPosition,
+      int bodyStartPosition,
+      List<DocumentNode> children) {
     this.title = title != null ? title : "";
     this.body = body != null ? body : "";
     this.startPosition = Math.max(0, startPosition);
+    this.bodyStartPosition = Math.max(this.startPosition, bodyStartPosition);
     this.children = children != null ? List.copyOf(children) : List.of();
   }
 
@@ -57,6 +76,10 @@ public final class DocumentNode {
     return startPosition;
   }
 
+  public int getBodyStartPosition() {
+    return bodyStartPosition;
+  }
+
   public List<DocumentNode> getChildren() {
     return children;
   }
@@ -76,7 +99,16 @@ public final class DocumentNode {
     }
 
     if (!body.isBlank()) {
-      sections.add(new DocumentSection(List.copyOf(currentPath), body.strip(), 
startPosition));
+      int leadingWhitespace = 0;
+      while (Character.isWhitespace(body.charAt(leadingWhitespace))) {
+        leadingWhitespace++;
+      }
+      sections.add(
+          new DocumentSection(
+              List.copyOf(currentPath),
+              body.strip(),
+              startPosition,
+              bodyStartPosition + leadingWhitespace));
     }
 
     for (DocumentNode child : children) {
@@ -89,11 +121,22 @@ public final class DocumentNode {
     private final List<String> path;
     private final String body;
     private final int startPosition;
+    private final int bodyStartPosition;
 
     public DocumentSection(List<String> path, String body, int startPosition) {
+      this(path, body, startPosition, startPosition);
+    }
+
+    /**
+     * @param startPosition where the section starts in the source text, its 
heading included
+     * @param bodyStartPosition where {@code body} starts in the source text
+     */
+    public DocumentSection(
+        List<String> path, String body, int startPosition, int 
bodyStartPosition) {
       this.path = path != null ? List.copyOf(path) : List.of();
       this.body = body != null ? body : "";
       this.startPosition = startPosition;
+      this.bodyStartPosition = bodyStartPosition;
     }
 
     public List<String> getPath() {
@@ -108,6 +151,10 @@ public final class DocumentNode {
       return startPosition;
     }
 
+    public int getBodyStartPosition() {
+      return bodyStartPosition;
+    }
+
     public String breadcrumb() {
       return String.join(" > ", path);
     }
diff --git 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
index 226af2d584..fa39b9815e 100644
--- 
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
+++ 
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
@@ -48,7 +48,7 @@ final class HeadingParserSupport {
     }
 
     String preamble = text.substring(0, headings.get(0).start).strip();
-    MutableNode root = new MutableNode("", preamble, 0);
+    MutableNode root = new MutableNode("", preamble, 0, skipWhitespace(text, 
0));
     MutableNode[] stack = new MutableNode[7];
     stack[0] = root;
 
@@ -59,7 +59,8 @@ final class HeadingParserSupport {
       String body = text.substring(bodyStart, bodyEnd).strip();
 
       int level = Math.min(Math.max(heading.level, 1), stack.length - 1);
-      MutableNode node = new MutableNode(heading.title, body, heading.start);
+      MutableNode node =
+          new MutableNode(heading.title, body, heading.start, 
skipWhitespace(text, bodyStart));
       stack[level] = node;
       for (int j = level + 1; j < stack.length; j++) {
         stack[j] = null;
@@ -75,6 +76,15 @@ final class HeadingParserSupport {
     return root.toImmutable();
   }
 
+  /** The first position at or after {@code position} that is not whitespace. 
*/
+  private static int skipWhitespace(String text, int position) {
+    int result = position;
+    while (result < text.length() && 
Character.isWhitespace(text.charAt(result))) {
+      result++;
+    }
+    return result;
+  }
+
   private static final class HeadingMatch {
     private final int level;
     private final String title;
@@ -93,12 +103,14 @@ final class HeadingParserSupport {
     private final String title;
     private final String body;
     private final int startPosition;
+    private final int bodyStartPosition;
     private final List<MutableNode> children = new ArrayList<>();
 
-    private MutableNode(String title, String body, int startPosition) {
+    private MutableNode(String title, String body, int startPosition, int 
bodyStartPosition) {
       this.title = title;
       this.body = body;
       this.startPosition = startPosition;
+      this.bodyStartPosition = bodyStartPosition;
     }
 
     private DocumentNode toImmutable() {
@@ -106,7 +118,7 @@ final class HeadingParserSupport {
       for (MutableNode child : children) {
         childNodes.add(child.toImmutable());
       }
-      return new DocumentNode(title, body, startPosition, childNodes);
+      return new DocumentNode(title, body, startPosition, bodyStartPosition, 
childNodes);
     }
   }
 }
diff --git 
a/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
 
b/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
index 091974a3eb..37df05af3d 100644
--- 
a/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
+++ 
b/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
@@ -47,7 +47,7 @@ TextChunker.chunkSize.Label=Chunk size
 TextChunker.chunkSize.Tooltip=Maximum size for each chunk in characters 
(approximate for paragraphs)
 
 TextChunker.chunkOverlap.Label=Chunk overlap
-TextChunker.chunkOverlap.Tooltip=Number of characters to overlap between 
chunks (for continuity)
+TextChunker.chunkOverlap.Tooltip=Maximum number of characters, in whole words, 
carried from the end of one chunk into the next (for continuity). Paragraph and 
Structure only use it inside a paragraph or section that has to be split
 
 TextChunker.includeMetadata.Label=Include metadata
 TextChunker.includeMetadata.Tooltip=Add metadata fields (chunk index, 
position, document ID) to output
@@ -85,5 +85,5 @@ TextChunker.Log.EmptyText=No text to chunk, emitting a single 
empty chunk row
 TextChunker.Validation.OutputChunkFieldRequired=Output chunk field name must 
be specified
 TextChunker.Validation.SourceDocumentIdFieldNotFound=Source document ID field 
''{0}'' not found in the input stream
 TextChunker.Validation.ContentTypeFieldNotFound=Content type field ''{0}'' not 
found in the input stream
-TextChunker.Validation.ParagraphOverlapIgnored=The Paragraph strategy does not 
apply chunk overlap; the configured overlap is ignored
+TextChunker.Validation.OverlapOnlyWhenSplit=The Paragraph and Structure 
strategies only apply the chunk overlap inside a paragraph or section that is 
longer than the chunk size and has to be split
 TextChunker.Validation.ParagraphSeparatorIgnored=The paragraph separator only 
applies to the Paragraph strategy; the configured separator is ignored
diff --git 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
index 3c9e22cece..d1285beaef 100644
--- 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
+++ 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
@@ -107,6 +107,39 @@ class TextChunkerMetaTest {
         "a separator set for the Character strategy must produce a warning");
   }
 
+  /**
+   * Paragraph and Structure apply the overlap only inside a paragraph or 
section they split. The
+   * check says so for both, as a comment: it used to warn, for Paragraph 
only, that the overlap was
+   * ignored.
+   */
+  @Test
+  void checkExplainsWhereParagraphAndStructureApplyTheOverlap() {
+    for (ChunkingStrategyType type : ChunkingStrategyType.values()) {
+      TextChunkerMeta meta = new TextChunkerMeta();
+      meta.setDefault();
+      meta.setInputField("body");
+      meta.setChunkingStrategy(type);
+      meta.setChunkOverlap("20");
+
+      IRowMeta prev = new RowMeta();
+      prev.addValueMeta(new ValueMetaString("body"));
+
+      List<ICheckResult> remarks = new ArrayList<>();
+      meta.check(remarks, null, new TransformMeta(), prev, null, null, null, 
new Variables(), null);
+
+      boolean explained =
+          remarks.stream()
+              .anyMatch(
+                  r ->
+                      r.getType() == ICheckResult.TYPE_RESULT_COMMENT
+                          && r.getText().contains("overlap"));
+      assertEquals(type != ChunkingStrategyType.CHARACTER, explained, 
type.name());
+      assertTrue(
+          remarks.stream().noneMatch(r -> r.getType() == 
ICheckResult.TYPE_RESULT_WARNING),
+          type + " with a valid overlap must not produce a warning");
+    }
+  }
+
   /** Chunk index, start position and total count are numbers, not strings. */
   @Test
   void exposesChunkMetadataAsIntegers() throws Exception {
diff --git 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
index 1c252a674f..36dc3fc9d1 100644
--- 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
+++ 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
@@ -20,6 +20,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
+import java.util.Collections;
 import java.util.List;
 import org.apache.hop.pipeline.transforms.chunker.Chunk;
 import org.junit.jupiter.api.BeforeEach;
@@ -82,23 +83,121 @@ public class CharacterChunkingStrategyTest {
     String text = "This is a test string for chunking with overlap.";
     List<Chunk> chunks = strategy.chunk(text, 15, 5);
 
-    assertTrue(chunks.size() > 1, "Should have multiple chunks");
+    // "test " is the only whole word within the last 5 characters of the 
first chunk. No word
+    // starts
+    // within the last 5 characters of the second ("ring ") or third ("king ") 
one, so nothing is
+    // carried there.
+    assertEquals(
+        List.of("This is a test ", "test string ", "for chunking ", "with 
overlap."),
+        contents(chunks));
+    assertAddressesTheSource(text, chunks);
+  }
+
+  /** #9283: the overlap must start at a word, not in the middle of one. */
+  @Test
+  public void testOverlapStartsAtAWordBoundary() {
+    List<Chunk> chunks = strategy.chunk(LOREM, 100, 20);
 
-    // Verify overlap exists between chunks
-    for (int i = 0; i < chunks.size() - 1; i++) {
+    assertTrue(chunks.size() > 3);
+    assertAddressesTheSource(LOREM, chunks);
+    for (int i = 1; i < chunks.size(); i++) {
+      Chunk previous = chunks.get(i - 1);
       Chunk current = chunks.get(i);
-      Chunk next = chunks.get(i + 1);
+      int start = current.getStartPosition();
+      assertTrue(
+          Character.isWhitespace(LOREM.charAt(start - 1)),
+          "chunk " + i + " starts in the middle of a word: " + 
current.getContent());
+      assertTrue(
+          previous.getEndPosition() - start <= 20,
+          "chunk " + i + " overlaps more than the configured 20 characters");
+    }
+    // The overlap window of the first chunk starts inside "incididunt", which 
used to be carried
+    // over as "ncididunt". Now the next whole word is.
+    assertTrue(chunks.get(1).getContent().startsWith("ut labore "), 
chunks.get(1).getContent());
+  }
 
-      String currentContent = current.getContent();
-      String nextContent = next.getContent();
+  /**
+   * #9283: when a chunk backs off to a space and ends up shorter than the 
overlap, the next chunk
+   * used to move forward by one character: 228 chunks for 419 characters.
+   */
+  @Test
+  public void testOverlapDoesNotProduceAChunkPerCharacter() {
+    String text = String.join(" ", Collections.nCopies(20, 
"supercalifragilistic"));
+    List<Chunk> chunks = strategy.chunk(text, 25, 15);
 
-      // The beginning of the next chunk should overlap with the end of the 
current chunk
-      String overlapPart = nextContent.substring(0, Math.min(5, 
nextContent.length()));
-      String endOfCurrent = currentContent.substring(Math.max(0, 
currentContent.length() - 5));
+    assertEquals(20, chunks.size());
+    assertAddressesTheSource(text, chunks);
+    for (int i = 1; i < chunks.size(); i++) {
+      assertTrue(chunks.get(i).getEndPosition() > chunks.get(i - 
1).getEndPosition());
+      
assertTrue(chunks.get(i).getContent().startsWith("supercalifragilistic"));
+    }
+  }
+
+  /** Every chunk has to add text the previous one did not hold. */
+  @Test
+  public void testNoChunkIsContainedInThePreviousOne() {
+    String text = "aaaa bbbb cccc dddd eeee " + "x".repeat(30) + " ffff gggg";
+    List<Chunk> chunks = strategy.chunk(text, 25, 10);
 
-      // At least some overlap should exist
+    assertAddressesTheSource(text, chunks);
+    for (int i = 1; i < chunks.size(); i++) {
       assertTrue(
-          endOfCurrent.contains(overlapPart.substring(0, Math.min(2, 
overlapPart.length()))));
+          chunks.get(i).getEndPosition() > chunks.get(i - 1).getEndPosition(),
+          "chunk " + i + " only repeats the previous chunk: " + 
chunks.get(i).getContent());
+    }
+  }
+
+  /** #9283: a run of whitespace longer than the chunk size gave chunks of 
only spaces. */
+  @Test
+  public void testWhitespaceOnlyChunksAreDropped() {
+    String text = "a" + " ".repeat(30) + "b";
+    List<Chunk> chunks = strategy.chunk(text, 10, 0);
+
+    assertAddressesTheSource(text, chunks);
+    for (int i = 0; i < chunks.size(); i++) {
+      assertFalse(chunks.get(i).getContent().isBlank(), "chunk " + i + " is 
whitespace only");
+      assertEquals(i, chunks.get(i).getIndex());
+    }
+    assertEquals(2, chunks.size());
+    assertTrue(chunks.get(0).getContent().startsWith("a"));
+    assertTrue(chunks.get(1).getContent().endsWith("b"));
+  }
+
+  @Test
+  public void testWhitespaceOnlyTextYieldsNoChunks() {
+    assertTrue(strategy.chunk(" \n\t ", 10, 2).isEmpty());
+    assertTrue(strategy.chunk(" \n\t ", 10, -1).isEmpty());
+  }
+
+  /** #9283: the split was on char positions, so an emoji was cut into two 
lone surrogates. */
+  @Test
+  public void testSurrogatePairIsNotCut() {
+    String text = "\uD83D\uDE00\uD83D\uDE01\uD83D\uDE02";
+    List<Chunk> chunks = strategy.chunk(text, 5, 0);
+
+    assertEquals(List.of("\uD83D\uDE00\uD83D\uDE01", "\uD83D\uDE02"), 
contents(chunks));
+    assertAddressesTheSource(text, chunks);
+  }
+
+  @Test
+  public void testSurrogatePairIsNotCutWhenItIsTheOnlyCharacter() {
+    String text = "\uD83D\uDE00\uD83D\uDE01";
+    List<Chunk> chunks = strategy.chunk(text, 1, 0);
+
+    assertEquals(List.of("\uD83D\uDE00", "\uD83D\uDE01"), contents(chunks));
+  }
+
+  @Test
+  public void testOverlapInsideALongWordDoesNotCutASurrogatePair() {
+    String text = "\uD83D\uDE00".repeat(20);
+    List<Chunk> chunks = strategy.chunk(text, 9, 3);
+
+    assertAddressesTheSource(text, chunks);
+    for (Chunk chunk : chunks) {
+      assertFalse(Character.isLowSurrogate(chunk.getContent().charAt(0)), 
chunk.toString());
+      assertFalse(
+          
Character.isHighSurrogate(chunk.getContent().charAt(chunk.getLength() - 1)),
+          chunk.toString());
     }
   }
 
@@ -179,6 +278,25 @@ public class CharacterChunkingStrategyTest {
     assertEquals(text, reconstructFromChunks(text, chunks));
   }
 
+  private static final String LOREM =
+      "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod 
tempor incididunt"
+          + " ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis 
nostrud exercitation"
+          + " ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis 
aute irure dolor in"
+          + " reprehenderit in voluptate velit esse cillum dolore eu fugiat 
nulla pariatur.";
+
+  private static List<String> contents(List<Chunk> chunks) {
+    return chunks.stream().map(Chunk::getContent).toList();
+  }
+
+  /** Each chunk is the slice of the source text its positions name, and the 
chunks cover it. */
+  private static void assertAddressesTheSource(String text, List<Chunk> 
chunks) {
+    for (Chunk chunk : chunks) {
+      assertEquals(
+          text.substring(chunk.getStartPosition(), chunk.getEndPosition()), 
chunk.getContent());
+    }
+    assertEquals(text.strip(), reconstructFromChunks(text, chunks).strip());
+  }
+
   private static String reconstructFromChunks(String text, List<Chunk> chunks) 
{
     if (chunks.isEmpty()) {
       return "";
diff --git 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
index 17826d0f51..37a119431b 100644
--- 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
+++ 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
@@ -137,6 +137,64 @@ public class ParagraphChunkingStrategyTest {
     assertEquals(text, chunks.get(0).getContent());
   }
 
+  /** #9283: a whitespace-only paragraph between two separators came out as a 
chunk. */
+  @Test
+  public void testWhitespaceOnlyParagraphsAreNotEmitted() {
+    strategy.setSeparator("---");
+    String text = "one--- ---two";
+
+    List<Chunk> chunks = strategy.chunk(text, 6, 0);
+
+    assertEquals(2, chunks.size());
+    assertEquals("one", chunks.get(0).getContent());
+    assertEquals(0, chunks.get(0).getIndex());
+    assertEquals("two", chunks.get(1).getContent());
+    assertEquals(1, chunks.get(1).getIndex());
+    assertEquals(text.indexOf("two"), chunks.get(1).getStartPosition());
+  }
+
+  @Test
+  public void testWhitespaceOnlyParagraphIsKeptInsideAPackedChunk() {
+    strategy.setSeparator("---");
+    String text = "one--- ---two";
+
+    List<Chunk> chunks = strategy.chunk(text, 100, 0);
+
+    assertEquals(1, chunks.size());
+    assertEquals(text, chunks.get(0).getContent());
+  }
+
+  @Test
+  public void testWhitespaceOnlyTextYieldsNoChunks() {
+    assertTrue(strategy.chunk(" \n\n \t ", 100, 0).isEmpty());
+    strategy.setSeparator("---");
+    assertTrue(strategy.chunk("  ", 100, 0).isEmpty());
+  }
+
+  /** The overlap inside an oversized paragraph starts at a word, the same as 
for Character. */
+  @Test
+  public void testOverlapInsideAnOversizedParagraphStartsAtAWord() {
+    String text =
+        "Short one.\n\nLorem ipsum dolor sit amet, consectetur adipiscing 
elit, sed do eiusmod"
+            + " tempor incididunt ut labore et dolore magna aliqua. Ut enim ad 
minim veniam, quis"
+            + " nostrud exercitation ullamco laboris nisi ut aliquip ex ea 
commodo consequat.";
+
+    List<Chunk> chunks = strategy.chunk(text, 100, 20);
+
+    assertTrue(chunks.size() > 3);
+    for (int i = 2; i < chunks.size(); i++) {
+      Chunk chunk = chunks.get(i);
+      assertEquals(
+          text.substring(chunk.getStartPosition(), chunk.getEndPosition()), 
chunk.getContent());
+      assertTrue(
+          Character.isWhitespace(text.charAt(chunk.getStartPosition() - 1)),
+          "chunk " + i + " starts mid-word: " + chunk.getContent());
+      assertTrue(
+          chunk.getStartPosition() < chunks.get(i - 1).getEndPosition(),
+          "the overlap still applies inside an oversized paragraph");
+    }
+  }
+
   @Test
   public void testEmptyParagraphs() {
     String text = "First\n\n\n\nSecond";
diff --git 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
index 44f5452f83..fc6f30d76c 100644
--- 
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
+++ 
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
@@ -97,6 +97,82 @@ class StructureChunkingStrategyTest {
     assertTrue(chunks.get(chunks.size() - 1).getContent().contains("beta"));
   }
 
+  /**
+   * #9283: the chunks a long section is split into reported the heading 
position plus their offset
+   * in the stripped body, so every chunk after the first was off by the 
length of the heading.
+   */
+  @Test
+  void positionsOfASplitSectionPointAtTheirTextInTheSource() {
+    StringBuilder body = new StringBuilder();
+    for (int i = 1; i <= 40; i++) {
+      body.append(String.format("word%03d ", i));
+    }
+    String text = "# Title\n\n" + body.toString().strip();
+
+    List<Chunk> chunks = strategy.chunk(text, 100, 0, ContentType.MARKDOWN);
+
+    assertTrue(chunks.size() > 2);
+    assertEquals(0, chunks.get(0).getStartPosition(), "the first chunk starts 
at the heading");
+    for (int i = 1; i < chunks.size(); i++) {
+      Chunk chunk = chunks.get(i);
+      String partText = chunk.getContent().substring("Title\n\n".length());
+      assertEquals(
+          partText,
+          text.substring(chunk.getStartPosition(), chunk.getEndPosition()),
+          "chunk " + i + " does not address its text in the source");
+    }
+    assertEquals(text.indexOf("word012"), chunks.get(1).getStartPosition());
+  }
+
+  @Test
+  void positionsOfASplitSectionSkipBlankLinesAfterTheHeading() {
+    String text = "## A\n\n\n   " + "alpha beta gamma delta 
".repeat(10).strip();
+
+    List<Chunk> chunks = strategy.chunk(text, 40, 0, ContentType.MARKDOWN);
+
+    assertTrue(chunks.size() > 2);
+    for (int i = 1; i < chunks.size(); i++) {
+      Chunk chunk = chunks.get(i);
+      assertEquals(
+          chunk.getContent().substring("A\n\n".length()),
+          text.substring(chunk.getStartPosition(), chunk.getEndPosition()));
+    }
+  }
+
+  /** An unsplit section keeps starting at its heading. */
+  @Test
+  void anUnsplitSectionStartsAtItsHeading() {
+    String text = "# Title\n\nIntro line.\n\n## Section A\n\nBody A.";
+
+    List<Chunk> chunks = strategy.chunk(text, 200, 0, ContentType.MARKDOWN);
+
+    assertEquals(2, chunks.size());
+    assertEquals(0, chunks.get(0).getStartPosition());
+    assertEquals(text.indexOf("## Section A"), 
chunks.get(1).getStartPosition());
+    assertEquals(text.length(), chunks.get(1).getEndPosition());
+  }
+
+  /** The overlap inside a split section starts at a word, the same as for 
Character. */
+  @Test
+  void overlapInsideASplitSectionStartsAtAWord() {
+    StringBuilder body = new StringBuilder();
+    for (int i = 1; i <= 40; i++) {
+      body.append(String.format("word%03d ", i));
+    }
+    String text = "# Title\n\n" + body.toString().strip();
+
+    List<Chunk> chunks = strategy.chunk(text, 100, 20, ContentType.MARKDOWN);
+
+    assertTrue(chunks.size() > 2);
+    for (int i = 1; i < chunks.size(); i++) {
+      String partText = 
chunks.get(i).getContent().substring("Title\n\n".length());
+      assertTrue(partText.startsWith("word"), "chunk " + i + " starts 
mid-word: " + partText);
+      assertTrue(
+          chunks.get(i).getStartPosition() < chunks.get(i - 
1).getEndPosition(),
+          "chunk " + i + " does not overlap the previous one");
+    }
+  }
+
   @Test
   void strategyTypeIsStructure() {
     assertEquals(ChunkingStrategyType.STRUCTURE, strategy.getType());

Reply via email to