Re: Transport compression (not store compression)

On Fri, Feb 16, 2018 at 4:22 AM, Vladimir Ozerov wrote: > Dmitry, > > IMO we should not compress individual messages. Instead, we should compress > stream of enqueued messages. This way we will reduce number of array > copying and improve performance. Makes sense? > Yes,

Re: Transport compression (not store compression)

Dmitry, IMO we should not compress individual messages. Instead, we should compress stream of enqueued messages. This way we will reduce number of array copying and improve performance. Makes sense? On Fri, Feb 16, 2018 at 12:05 AM, Dmitriy Setrakyan wrote: > Vova, I

Re: Transport compression (not store compression)

Vladimir Ozerov, I also agree that your solution is good. I will check this flag before adding a client to map of clients. If one of the nodes have the flag then the session will be marked "compressed". At the nearest time, I will provide a solution. Dmitriy Setrakyan, I will implement and test

Re: Transport compression (not store compression)

Vova, I think your solution is fine, but I think we will always have some messages compressed and others not. For example, in many cases, especially when messages are relatively small, compressing them will introduce an unnecessary overhead, and most likely slow down the cluster. Why not have

Re: Transport compression (not store compression)

I think that we should not guess on how the clients are used. They could be used in any way - in the same network, in another network, in Docker, in hypervisor, etc.. This holds for both thin and thick clients. It is essential that we design configuration API in a way that compression could be

Re: Transport compression (not store compression)

Hello, Igniters. I have not seen such use-cases, where heavy client ignite node placed in a much worse network than the server. I'm not sure we should encourage a bad cluster architecture. Usually, in my use-cases, the servers and clients locate in the same network. And if the cluster has SSL

Re: Transport compression (not store compression)

Thanks for your comments, I will try to separate network compression for clients and servers. It makes sense to enable compression on servers if we have SSL turned on. I tested rebalancing time and compression+ssl is faster. SSL throughput is limited by 800 Mbits/sec per connection and if enable

Re: Transport compression (not store compression)

I think Igor is right. Ususally servers connected via fast local network. But clients could be in external and slow network. In this scenario compression will be very useful. Once I had such scenario - client connected to cluster via 300 kb/s network and tries to transfer ~10Mb of uncumpressed

Re: Transport compression (not store compression)

Nikita, Yes, you're right. Maybe I wasn't clear enough. Usually server nodes are placed in the same fast network segment (one datacenter); in any case we need an ability to setup compression per connection using some filter like useCompression(ClusterNode, ClusterNode) to compress traffic only

Re: Transport compression (not store compression)

Thanks for your comments, 1. At first, I would define final design and then I will make it pluggable. 2. We cannot prioritize messages at this network communication level. If compression utilizes processor poorly then we can increase count of selectors (make more threads when compression

Re: Transport compression (not store compression)

Nikita, I took a look at the changes and see two issues. The first one is additional dependencies in core module. That means you should whether make the filter plugable and provide some additional module for it or put all the necessary classes to ignite-core. The second is possible bottleneck

Re: Transport compression (not store compression)

Hello, Igniters! I implement network compression as described earlier [1]. I implemented Zstd, LZ4, and Deflater [2, 3]. LZ4 have good speed (1.2 G/s per connection) but compression ratio is not more than 2. Zstd has good compression ratio, up to 5, but relatively slow throughputs (0.28 G/s per

Re: Transport compression (not store compression)

I would start with communication only, and put discovery and clients aside for now. Let's confirm that communication works well with compression first in terms of performance. On Thu, Nov 30, 2017 at 10:36 AM, Nikita Amelchev wrote: > Hello, everybody. > I propose the

Re: Transport compression (not store compression)

Hello, everybody. I propose the following design for network compression. I suggest to implement it like SSL implementation that already works. I add compression filter to a chain of NIO filters. It changes the logic of TcpCommunicationSpi.safeTcpHandshake() and

Re: Transport compression (not store compression)

Hi, I've filed a ticket [1]. I'll try to share design details in a couple of days. 1. https://issues.apache.org/jira/browse/IGNITE-7024 2017-11-23 18:31 GMT+03:00 Denis Magda : > Nikita, > > Sounds like a good plan. Please share the design details prior getting > down to the

Re: Transport compression (not store compression)

Nikita, Sounds like a good plan. Please share the design details prior getting down to the implementation. — Denis > On Nov 23, 2017, at 4:38 AM, Nikita Amelchev wrote: > > Hi Igniters! > > I’m working on the similar feature for my own project. > I would like to

Re: Transport compression (not store compression)

Hi Igniters! I’m working on the similar feature for my own project. I would like to suggest use in-line compression and write encoded bytes in network channel by bytes array buffer. It allows us avoiding expensive memory allocation. The described design may be implemented in TcpCommunicationSpi

Re: Transport compression (not store compression)

Denis, Regarding zipped marshaller - this would be inefficient, because compression rate will be lower. On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda wrote: > Nikita, > > Your solution sounds reasonable from the first glance. However, the > communication layer processes a

Re: Transport compression (not store compression)

Nikita, Your solution sounds reasonable from the first glance. However, the communication layer processes a dozen of small system messages that should be excluded from the compression. Guess, that we will spend more time on compressing/decompressing them thus diminishing the positive effect of

Re: Transport compression (not store compression)

I think it is very useful feature. I also have experience when server nodes connected via fast network. But client nodes via very slow network. I implemeted GridClientZipOptimizedMarshaller and that solved my issue. But this marshaller works only with old and

Re: Transport compression (not store compression)

Hello, Igniters! I think it is a useful feature. I suggest to implement it to communication SPI like SSL encryption implemented. I have experience with this feature and I can try to develop it. 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin : > Forwarding to DEV list:

Re: Transport compression (not store compression)

Forwarding to DEV list: Ignite developers, could you please share your thoughts on how hard it is to extend Ignite to compress data on the network. On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) < gordon.r...@ninemilefinancial.com> wrote: > Hi Igniters, > > > > I see there is a lot of