Xinyu Liu created SAMZA-834:
-------------------------------
Summary: Heap usage increases dramatically after container timer
metrics changes to nanosecond
Key: SAMZA-834
URL: https://issues.apache.org/jira/browse/SAMZA-834
Project: Samza
Issue Type: Bug
Reporter: Xinyu Liu
Assignee: Xinyu Liu
After the change of container timer metrics (chooseNs, windowNs, processNs, and
commitNs) from millisecond to nanosecond, we noticed a dramatic increase of
memory heap usage in one of our production job. After investigation we found
that the SlidingTimeWindowReservoir.update(duration) will be called much more
frequently due to the duration is non-zero after the nanosecond change (In
contrast, it is often zero when using millisecond). Within the 5-minute window,
the storage inside SlidingTimeWindowReservoir increases a lot for a high qps
job (for our job with around 10K qps, it increases the heap from <5M to 100M).
It causes long GCs and degrades the job performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
