On 11.02.2021 16:09, Daniil Zakhlystov wrote:
Hi!
On 09.02.2021 09:06, Konstantin Knizhnik wrote:
Sorry, but my interpretation of your results is completely different:
permanent compression is faster than chunked compression (2m15 vs. 2m27)
and consumes less CPU (44 vs 48 sec).
Size of RX data is slightly larger - 0.5Mb but TX size is smaller - 5Mb.
So permanent compression is better from all points of view: it is
faster, consumes less CPU and reduces network traffic!
From my point of view your results just prove my original opinion that
possibility to control compression on the fly and use different
compression algorithm for TX/RX data
just complicates implementation and given no significant advantages.
When I mentioned the lower CPU usage, I was referring to the pgbench test
results in attached
google doc, where chunked compression demonstrated lower CPU usage compared to
the permanent compression.
I made another (a little bit larger) pgbench test to demonstrate this:
Pgbench test parameters:
Data load
pgbench -i -s 100
Run configuration
pgbench --builtin tpcb-like -t 1500 --jobs=64 --client==600"
Pgbench test results:
No compression
latency average = 247.793 ms
tps = 2421.380067 (including connections establishing)
tps = 2421.660942 (excluding connections establishing)
real 6m11.818s
user 1m0.620s
sys 2m41.087s
RX bytes diff, human: 703.9221M
TX bytes diff, human: 772.2580M
Chunked compression (compress only CopyData and DataRow messages)
latency average = 249.123 ms
tps = 2408.451724 (including connections establishing)
tps = 2408.719578 (excluding connections establishing)
real 6m13.819s
user 1m18.800s
sys 2m39.941s
RX bytes diff, human: 707.3872M
TX bytes diff, human: 772.1594M
Permanent compression
latency average = 250.312 ms
tps = 2397.005945 (including connections establishing)
tps = 2397.279338 (excluding connections establishing)
real 6m15.657s
user 1m54.281s
sys 2m37.187s
RX bytes diff, human: 610.6932M
TX bytes diff, human: 513.2225M
As you can see in the above results, user CPU time (1m18.800s vs 1m54.281s) is
significantly smaller in
chunked compression because it doesn’t try to compress all of the packets.
Well, but permanent compression provides some (not so large) reducing of
traffic, while
for chunked compression network traffic is almost the same as with
no-compression, but it consumes more CPU.
Definitely pgbench queries are not the case where compression should be
used: both requests and responses are too short to make compression
efficient.
So in this case compression should not be used at all.
From my point of view, "chunked compression" is not a good compromise
between no-compression and permanent-compression cases,
but it combines drawbacks of two approaches: doesn't reduce traffic but
consume more CPU.
Here is the summary from my POV, according to these and previous tests results:
1. Permanent compression always brings the highest compression ratio
2. Permanent compression might be not worthwhile in load different from COPY
data / Replication / BLOBs/JSON queries
3. Chunked compression allows to compress only well compressible messages and
save the CPU cycles by not compressing the others
4. Chunked compression introduces some traffic overhead compared to the
permanent (1.2810G vs 1.2761G TX data on pg_restore of IMDB database dump,
according to results in my previous message)
5. From the protocol point of view, chunked compression seems a little bit more
flexible:
- we can inject some uncompressed messages at any time without the need to
decompress/compress the compressed data
- we can potentially switch the compression algorithm at any time (but I
think that this might be over-engineering)
Given the summary above, I think it’s time to make a decision on which path we
should take and make the final list of goals that need to be reached in this
patch to make it committable.
Thanks,
Daniil Zakhlystov
