----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48080/ -----------------------------------------------------------
(Updated June 13, 2016, 10:28 p.m.) Review request for samza. Changes ------- More consistent use of "entry" where "policy" was previously used as policy now means a collection of policy entries. Repository: samza Description ------- This change introduces a ThrottlingExecutor which is used to control the rate of execution in the main run loop. The DiskQuotaEnforcer houses the rules for switching from one DiskQuotaPolicy to the next as new disk usage samples arrive. By default, no throttling will occur. New policies can be added using the following form: ``` container.disk.quota.bytes=XXX container.disk.quota.policy.count=2 container.disk.quota.policy.0.lowWaterMark=0.4 container.disk.quota.policy.0.highWaterMark=0.5 container.disk.quota.policy.0.workFactor=0.5 container.disk.quota.policy.1.lowWaterMark=0.05 container.disk.quota.policy.1.highWaterMark=0.1 container.disk.quota.policy.1.workFactor=0.05 ``` See ThrottlingExecutor for details about how the work factor works and DiskQuotaPolicy for details about how the low and high water mark configuration work. Diffs (updated) ----- samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaPolicy.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaPolicyFactory.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskSpaceMonitor.java 2a565be7858a4d3a6adbc49989b43b71ca3f6721 samza-core/src/main/java/org/apache/samza/container/disk/NoThrottlingDiskQuotaPolicy.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/NoThrottlingDiskQuotaPolicyFactory.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/WatermarkDiskQuotaPolicy.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/WatermarkDiskQuotaPolicyFactory.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/HighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/SystemHighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 6916c5c71e479d43a7435fa4987565d93ed437ac samza-core/src/main/scala/org/apache/samza/container/SameThreadExecutor.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala cf3c4c0ab08a59760bc899c6f2027755e933b350 samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala 9e6641c3628290dc05e1eb5537e86bff9d37f92c samza-core/src/main/scala/org/apache/samza/util/Util.scala fc3d085d7fff9f7dcec766ba48e550eb0052e99d samza-core/src/test/java/org/apache/samza/container/disk/TestDiskQuotaPolicyEntry.java PRE-CREATION samza-core/src/test/java/org/apache/samza/container/disk/TestWatermarkDiskQuotaPolicy.java PRE-CREATION samza-core/src/test/java/org/apache/samza/util/TestThrottlingExecutor.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala 05b4e5c37578340eefe6d412f5a76f540bec6fa6 Diff: https://reviews.apache.org/r/48080/diff/ Testing ------- - Added new unit tests - Ran existing tests with gradle test - Verified throttling behavior and instrumentation with local deployment - Verified average latency impact of feature to be < 150ns for Linux and OSX Thanks, Chris Pettitt
