[
https://issues.apache.org/jira/browse/IGNITE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759868#comment-17759868
]
Pavel Tupitsyn commented on IGNITE-20291:
-----------------------------------------
[~apolovtcev] looks good to me.
> Possible memory leak in StreamerBuffer
> --------------------------------------
>
> Key: IGNITE-20291
> URL: https://issues.apache.org/jira/browse/IGNITE-20291
> Project: Ignite
> Issue Type: Bug
> Reporter: Aleksandr Polovtcev
> Assignee: Aleksandr Polovtcev
> Priority: Major
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I was running a stress test that simply inserted a lot of rows using
> {{RecordView#streamData}} method with the following options:
> {code:java}
> var options = DataStreamerOptions.builder()
> .perNodeParallelOperations(1)
> .batchSize(10_000)
> .autoFlushFrequency(100_000)
> .build();
> {code}
> After about 30 minutes I started getting Java OOM errors. Having inspected
> the heap dump, I found that all memory has been consumed by the Tuple
> instances that I have been inserting. The reason for that is the following:
> # {{StreamerBuffer}} accumulates batches of data and then sends them to the
> server. If the server is slow for any reason, batch sending is placed into a
> "queue" by chaining {{flushFut}} one after another.
> # {{StreamerSubscriber}} is the class that controls back-pressure. It uses
> two variables for this purpose: {{pendingItemCount}} which represents that
> amount of items, requested by the Subscriber, and {{inFlightItemCount}} which
> represents the amount of items being sent to the server. However,
> {{StreamerSubscriber}} does not know about the "queue" present in the
> {{StreamerBuffer}} and it increases the {{inFlightItemCount}} only when a
> particular flush future is being executed. This means that futures sitting in
> the "queue" do not contribute to the overall {{inFlightItemCount}} and
> {{StreamerSubscriber}} keeps requesting more items even though a lot of items
> are simply stuck in a different place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
