[
https://issues.apache.org/jira/browse/CAMEL-9274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15004852#comment-15004852
]
Aaron Whiteside edited comment on CAMEL-9274 at 11/13/15 10:53 PM:
-------------------------------------------------------------------
I've noticed a lot of inconsistency, some DataFormat's just return a
InputStream others that could return an InputStream read it into a buffer then
return that buffer, and then we have StreamCaching, which is intended to solve
the issue of buffering too much data in memory/heap. But a lot of those places
we use StreamCaching we could also just return the InputStream in question..
I've been thinking of adding support to the DataFormat's I care about to
configure the desired behavior, return byte[], raw InputStream or stream cached
InputStream.
And I must admit I was tempted to change the behavior of existing DataFormat's
to return their raw InputStream's instead of fixing StreamCaching, but I
figured that might break someone else.
What are your thoughts?
My use case is, decrypt, unzip, and split file content using a streaming
iterator. So having a bunch of wrapped InputStreams would be much more
efficient than buffering things in a StreamCache twice (decrypt, unzip).
was (Author: aaronjwhiteside):
I've noticed a lot of inconsistency, some DataFormat's just return a
InputStream others that could return an InputStream read it into a buffer then
return that buffer, and then we have StreamCaching, which is intended to solve
the issue of buffering too much data in memory/heap. But a lot of those places
we use StreamCaching we could also just return the InputStream in question..
I've been thinking of adding support to the DataFormat's I care about to
configure the desired behavior, return byte[], raw InputStream or stream cached
InputStream.
And I must admit I was tempted to change the behavior of existing DataFormat's
to return their raw InputStream's instead of fixing StreamCaching, but I
figured that might break someone else.
What are your thoughts?
> StreamCache does not work correctly with MarshalProcessor and pretty much all
> DataFormats
> -----------------------------------------------------------------------------------------
>
> Key: CAMEL-9274
> URL: https://issues.apache.org/jira/browse/CAMEL-9274
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.16.0
> Reporter: Aaron Whiteside
> Assignee: Claus Ibsen
> Fix For: 2.17.0
>
> Attachments: CAMEL_9274.patch, CAMEL_9274_v2.patch,
> CAMEL_9274_v3.patch
>
>
> StreamCache does not work correctly with MarshalProcessor and pretty much all
> DataFormats
> # Documentation states that stream caching can be turned on, on a per route
> basis. The places were we actually try and utilize stream caching only checks
> if the global option is turned on (MarshalProcessor + PGPKeyAccessDataFormat,
> I can't find any other meaningful attempts to use stream caching anywhere
> else) not the per route setting.
> # Secondly, if we ignore the first problem, only one data formatter tries to
> use stream caching for unmarshaling: PGPKeyAccessDataFormat. If you try and
> unzip a large zip file it's contents will be decompressed into memory. The
> same goes for any other large data formatter conversion (unmarshaling, only).
> Attached is a patch that corrects the first problem and part of the second
> problem (ZipFileDataFormat, ZipDataFormat and PGPKeyAccessDataFormat).
> A separate issue should probably be opened to fix all the remaining
> DataFormat implementation to support Stream Caching.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
