[
https://issues.apache.org/jira/browse/CAMEL-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092309#comment-18092309
]
Raymond commented on CAMEL-23264:
---------------------------------
Good call to embed this within the Splitter EIP.
As now work on the splitter is in progress, I would like to make a more broader
point:
By default the properties:
*streaming*
*shareUnitOfWork*
*parallelProcessing*
Are all false. Then users of our platform use the split pattern (and we use the
same defaults as Camel), and then complain that the memory is high, or
splitting is slow. Then I recommended to set these options to true. Then they
are satisfied with the performance, but then they complain that the split_size
is empty, and the the (aggregation) order isn't respected.
I would like to have default that is fast and low on resources, while still
being fully featured. Otherwise it's always needed to tune and explain. Though
technically it's all understandable, I have the feeling that there could be
better defaults, with less or no trade-offs (also for less technical users).
Another side note:
The documentation of ShareUnitOfWork:
"Shares the org.apache.camel.spi.UnitOfWork with the parent and each of the sub
messages. Splitter will by default not share unit of work between the parent
exchange and each split exchange. This means each split exchange has its own
individual unit of work."
This is clear what is does, but it would be nice to explain why to enable it,
and what are the benifits and consequences of doing this. It's a good idea to
take a close look at all the properties of the split EIP here, and see if the
"why" is explained thoroughly enough.
> Enhance Splitter EIP with chunking, error threshold, failure tracking, and
> watermark support
> --------------------------------------------------------------------------------------------
>
> Key: CAMEL-23264
> URL: https://issues.apache.org/jira/browse/CAMEL-23264
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Guillaume Nodet
> Assignee: Guillaume Nodet
> Priority: Major
> Fix For: 4.22.0
>
>
> Enhance the Splitter EIP to support common bulk processing patterns that
> currently require significant manual assembly.
> This replaces the proposed camel-bulk component (CAMEL-23240), which was
> closed after analysis showed these features are better served as Splitter EIP
> enhancements rather than a standalone component. See PR #22159 for the full
> analysis: https://github.com/apache/camel/pull/22159#issuecomment-4144168967
> h3. Enhancements
> *1. Chunking — {{group(int)}}*
> Add a {{group}} option to the Splitter that works with any Iterable (not just
> tokenized strings). The iterator is wrapped with a chunking iterator, so each
> split exchange body becomes a {{List}} of up to N items.
> {code:java}
> .split(body()).group(100)
> .to("direct:processChunk") // body is List of up to 100 items
> .end();
> {code}
> *2. Error Threshold — {{errorThreshold(double)}} / {{maxFailedRecords(int)}}*
> Generalizes {{stopOnException()}} from boolean (stop on first failure) to
> ratio/count-based abort. The Splitter tracks failures internally and aborts
> mid-stream when the threshold is exceeded.
> {code:java}
> .split(body()).errorThreshold(0.1).maxFailedRecords(50)
> .to("direct:process")
> .end();
> {code}
> * {{errorThreshold(0.1)}} — abort if more than 10% of items fail
> * {{maxFailedRecords(50)}} — abort after more than 50 failures
> * Both can be combined — abort when either is exceeded
> *3. Failure Tracking — built-in {{SplitResult}}*
> When error threshold is configured and no custom {{AggregationStrategy}} is
> set, the Splitter produces a {{SplitResult}} containing:
> * {{totalItems}}, {{successCount}}, {{failureCount}}, {{duration}},
> {{aborted}}
> * {{failures}} list with {{(index, item, exception)}} for each failed item
> * Output headers: {{CamelSplitTotal}}, {{CamelSplitSuccess}},
> {{CamelSplitFailed}}, etc.
> *4. Watermark Tracking via ResumeStrategy*
> Add {{resumeStrategy}}, {{watermarkKey}}, and {{watermarkExpression}} options
> for resume-from-last-position patterns. Uses Camel's existing
> {{ResumeStrategy}} SPI (from {{camel-api}}) for offset persistence.
> {code:java}
> ResumeStrategy strategy = ...; // any ResumeStrategy implementation
> .split(body())
> .resumeStrategy(strategy, "importJob")
> .to("direct:process")
> .end();
> {code}
> * *Index-based*: tracks how many items have been processed, skips
> already-processed items on subsequent runs (assumes stable ordering)
> * *Value-based* (with {{watermarkExpression}}): evaluates an expression on
> each successful item (e.g., a timestamp), stores the last value via
> {{ResumeStrategy.updateLastOffset()}} for upstream filtering
> * Watermark reads use lazy loading from the strategy's {{ResumeCache}} on
> each exchange
> * Watermark is not updated on abort, allowing retry from the same position
> h3. What This Replaces
> The proposed camel-bulk component (CAMEL-23240) provided these features as a
> standalone component. After review, we concluded:
> * Most features are natural extensions of the Splitter EIP
> * Users benefit from enhancing what they already know rather than learning a
> new component
> * The features are independently useful and compose with existing Splitter
> options ({{parallelProcessing}}, {{streaming}}, {{shareUnitOfWork}}, etc.)
> * The Bulk component's multi-step accept policy is already expressible with
> {{doTry/doCatch}} inside the split route
> h3. PR
> https://github.com/apache/camel/pull/22300
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
