Hello Bobby,
If it's possible to enable a TupleSerializer while submitting
topologies, without touching Storm's internals it would be really wise
and a fast way to implement compression. Actually I have no idea if it's
possible this way.
Other than that, If I got the Netty's gist right, to enable Zlib
compression one needs only 2 lines of code for each pipeline to enable
it. It's first class citizen of netty. Looking into
https://github.com/netty/netty/tree/master/codec/src/main/java/io/netty/handler/codec/compression
a good chunk of compression algorithms are also available.
From
https://github.com/netty/netty/blob/ed4a89082bb29b9e7d869c5d25d6b9ea8fc9d25b/example/src/main/java/io/netty/example/factorial/FactorialClientInitializer.java:
// Enable stream compression (you can remove these two if
unnecessary)
pipeline.addLast(ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP));
pipeline.addLast(ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP));
On 09/18/2015 08:32 PM, Bobby Evans wrote:
Compression was just not something that we really though about all that much.
The fastest route is probably to replace the tuple serializer with one that can
handle compression. We did something similar for encryption.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/security/serialization/BlowfishTupleSerializer.java
But compression is generic enough it might be nice to make it a part of the
Real TupleSerializer.
https://github.com/EsotericSoftware/kryo#compression-and-encryption
I would also suggest that you look at Snappy and LZO or LZ4 for your
compression. As they tend to be much faster and still get good compression
ratios. - Bobby
On Friday, September 18, 2015 11:51 AM, Onur Yalazı
<[email protected]> wrote:
Hello,
I'm very new to storm world and the list, so Hello from Turkey.
Because of a recent incident we had to increase our openstack network
bandwidth soft limits from 1gb/s to 2gb/s.
And of course even though the problem resides in our tuples' size and
topology size, I thought if storm's netty was using ZlibEncoding.
Looking into backtype.storm.messaging.netty.StormServerPipelineFactory
and client counterpart, pipline seems to not have any compression handlers.
Is it an intentional decision to not include compression in the
pipeline? I know it would need more processing power and reduce topology
performance, but I would like to know it it was considered before, and
if not raise the issue.
Thank you.
