----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41071/ -----------------------------------------------------------
Review request for samza and Yi Pan (Data Infrastructure). Repository: samza Description ------- After the change of container timer metrics (chooseNs, windowNs, processNs, and commitNs) from millisecond to nanosecond, we noticed a dramatic increase of memory heap usage in one of our production job. After investigation we found that the SlidingTimeWindowReservoir.update(duration) will be called much more frequently due to the duration is non-zero after the nanosecond change (In contrast, it is often zero when using millisecond). Within the 5-minute window, the storage inside SlidingTimeWindowReservoir increases a lot for a high qps job (for our job with around 10K qps, it increases the heap from <5M to 100M). It causes long GCs and degrades the job performance. The fix will make the SlidingTimeWindowReservoir collision buffer default to be 1 and configurable from the constructor. Diffs ----- samza-api/src/main/java/org/apache/samza/metrics/SlidingTimeWindowReservoir.java df543599771b7cda3be6ea702a85091cf61883d7 samza-api/src/main/java/org/apache/samza/metrics/Timer.java b49d14758116660ac6b26cc7e2459b293343e47e samza-api/src/test/java/org/apache/samza/metrics/TestSlidingTimeWindowReservoir.java d392b3258b7aca4323d663159c4e545e113277bb samza-api/src/test/java/org/apache/samza/metrics/TestTimer.java 63c183f9283c58006de23a342e61717e062030c3 samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala b4d6f351f97a7c3a29c133b4ba3876ce4b6baca2 Diff: https://reviews.apache.org/r/41071/diff/ Testing ------- Thanks, Xinyu Liu
