kpumuk commented on code in PR #3547:
URL: https://github.com/apache/thrift/pull/3547#discussion_r3314601407
##########
lib/rb/lib/thrift/transport/header_transport.rb:
##########
@@ -370,13 +385,48 @@ def parse_header_format(buf)
payload = buf.read
transforms.each do |transform_id|
if transform_id == HeaderTransformID::ZLIB
- payload = Zlib::Inflate.inflate(payload)
+ payload = bounded_inflate(payload)
end
end
StringIO.new(payload)
end
+ # Inflates +compressed+ with a running byte-count check against
@max_decompressed_size.
+ # Raises TransportException::SIZE_LIMIT if the decompressed output would
exceed the limit.
+ def bounded_inflate(compressed)
+ inflater = Zlib::Inflate.new
+ total = 0
+ chunks = []
Review Comment:
For the 16mb field read 1000 times, the results are:
| Variant | Wall | CPU | Peak RSS |
|---|---:|---:|---:|
| pre-commit unbounded `Zlib::Inflate.inflate` | 81.9s | 83.3s | 153 MB |
| array + `join` | 103.5s | 104.0s | 192 MB |
| append with preallocated capacity | 95.7s | 96.2s | 171 MB |
| **append, no preallocation, final patch** | 103.9s | 104.1s | 131 MB |
| single inflate then size check | 80.0s | 81.3s | 169 MB |
I am currently testing shared chunk buffer (caused segfault in my Ruby,
trying to figure out if I did it, or I should report it to Ruby) :-)
So far current option (empty string buffer) seems to be the best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
