[
https://issues.apache.org/jira/browse/AVRO-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107406#comment-18107406
]
Ismaël Mejía edited comment on AVRO-3802 at 8/24/26 2:06 PM:
-------------------------------------------------------------
This appears to already be fixed on main. The deflate-decompression memory
issues described here were addressed in commit 8eab8f14b5 ("AVRO-3802: [Csharp]
Fix memory leak on deflate codec decompression", PR #2439).
The current DeflateCodec (lang/csharp/src/apache/main/File/DeflateCodec.cs) now
wraps the MemoryStreams and DeflateStream in using blocks and uses the
framework's built-in Stream.CopyTo, which resolves the leak/inefficiency in
CopyTo and the un-disposed inStream/outStream reported here.
I am marking this one then as "Fixed". Please reopen if the leak still
reproduces on a current release.
was (Author: iemejia):
This appears to already be fixed on main. The deflate-decompression memory
issues described here were addressed in commit 8eab8f14b5 ("AVRO-3802: [Csharp]
Fix memory leak on deflate codec decompression", PR #2439).
The current DeflateCodec (lang/csharp/src/apache/main/File/DeflateCodec.cs) now
wraps the MemoryStreams and DeflateStream in using blocks and uses the
framework's built-in Stream.CopyTo, which resolves the leak/inefficiency in
CopyTo and the un-disposed inStream/outStream reported here.
Recommend closing as "Fixed". Please reopen if the leak still reproduces on a
current release.
> Memory leak on decompressing deflate codec avro files
> -----------------------------------------------------
>
> Key: AVRO-3802
> URL: https://issues.apache.org/jira/browse/AVRO-3802
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
> Affects Versions: 1.11.1, 1.12.0, 1.11.2, 1.11.3
> Reporter: amy myo
> Priority: Major
> Labels: pull-request-available
> Attachments: call tree.PNG, deflate decompression memory profiler.png
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Hi,
> We are using apache.avro C# library for compression and decompression of
> large avro files. Each file is 2-3MB and decompression of 20 files is using
> 2.1GB memory and memory keep increasing around 200-300MB for follow up calls.
> Attached is the memory profiler of three calls. Each call decompress the same
> 20 files. First call uses 2.1GB , increase to 2.37GB and then 2.68GB.
> Call tree is showing most of the memory are used by `DeflatCodec.CopyTo`
> method.
> # `CopyTo` method is not using Array pool. Default Microsoft's stream
> `copyTo` method is much more efficient.
> # `inStream` and `outStream` memory streams are not in using statement.
> # On `outstream` object creation, object can be initialized with defined
> length/stream capacity.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
