Peter Georgantas created SAMZA-2044:
---------------------------------------
Summary: EOSMessage causes out of memory Exceptions related to
WindowOperatorImpl
Key: SAMZA-2044
URL: https://issues.apache.org/jira/browse/SAMZA-2044
Project: Samza
Issue Type: Bug
Components: container
Affects Versions: 0.14.0, 1.0
Reporter: Peter Georgantas
The contract of the handleEndOfStream method dictates that a collection of
results be returned. In the case of WindowOperatorImpl which has a backing
RocksDB store, this effectively causes the entirety of the store to be pulled
into memory. In many cases, this will cause out of memory exceptions (otherwise
why not keep the store in memory in the first place).
Since this is a protected api, I have a relatively simple change to propose
which could allow data to be consumed downstream as it is brought out of the
store:
{{protected void handleEndOfStream(Consumer<WindowPane<K, Object>> consumer,
MessageCollector collector, TaskCoordinator coordinator) { ... }}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
