GitHub user JoergKe opened a pull request:
https://github.com/apache/camel/pull/667
Implement chunking of payloads in Camel JMS
Change-Id: Id8409275395bd49a2148ed21c3dfa4ea7a23708b
Hi,
this is about the JMS component available in Camel. What is currently
missing in this component is that it can deal with brokers who limit the
message size for queueing. The broker we are using limits the message size but
we have to process larger messages. The idea is to cut the message in chunks in
the component when the message is sent to the broker and to reassemble it when
it is received. This feature can also be used to "stream" messages through
brokers that do not support streaming since it is not supported in the JMS
standard. The basic idea is that a large message is cut into a head message and
several sub messages. The sub messages are later identified by a JMS property
in form of GUID. The head messages is also indentified by a special JMS
property. It also contains the identifier for the sub messages and it contains
the count of the sub messages. The head message is stored last to the queue
defined by the user. This ensures that the sub messages are already available
when the head message is selected (by a selector). The sub messages are also
stored to the queue. The Camel producer uses a selector on the JMS property to
process the head messages. During processing of a head message the sub messages
are pulled from the queue using a selector and the complete message is
assembled.
The implementation has some limitations.
- It only works for Camel payloads that can be converted into an input
stream because this format provides means to cut the message into chunks.
- It only works in the transacted mode because only this mode garantees
that a message can cut into pieces and can be reassembled without the risk of
data loss.
- It works only for queues and not for topics since it is difficult if not
impossible to garantee that no sub messages can be lost
- For request response pattern only the request is chunked.
The consumer endpoint is configured as
"jms:queue:hello?maxChunkSize=1000000"
where maxChunkSize is in bytes and defines the maximum message size.
The provider endpoint is the defined as
"jms:queue:hello?chunking=true"
where chunking enables that first only the head messages are selected.
The overhead for message having a size that is smaller than maxChunkSize is
low.
The impact on the existing code is surprisingly low. I had to create
several 'mini' classes that implement the message creator interface. Before
that extension these were anonymous classes. To avoid code duplication those
classes were created (I think I also removed some code duplication in the
existing code). This is the only bigger change if this feature is not enabled.
I hope you like this extension and I can contribute it to Camel.
Jörg
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoergKe/camel master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/camel/pull/667.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #667
----
commit 0bd5c29b58bf88825b29517e280671e9b9ec1342
Author: Jörg KeÃler <[email protected]>
Date: 2015-10-20T05:45:39Z
Implement chunking of payloads in Camel JMS
Change-Id: Id8409275395bd49a2148ed21c3dfa4ea7a23708b
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
