[
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875854#comment-14875854
]
ASF GitHub Bot commented on STORM-886:
--------------------------------------
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r39872212
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -1024,6 +1024,48 @@
/**
+ * Whether to enable backpressure in for a certain topology
+ */
+ public static final String TOPOLOGY_BACKPRESSURE_ENABLE =
"topology.backpressure.enable";
+ public static final Object TOPOLOGY_BACKPRESSURE_ENABLE_SCHEMA =
Boolean.class;
+
+ /**
+ * This signifies the tuple congestion in a worker's out-going queue.
+ * When the used ratio of a worker's outgoing queue is higher than the
high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+ public static final String
BACKPRESSURE_WORKER_HIGH_WATERMARK="backpressure.worker.high.watermark";
+ public static final Object BACKPRESSURE_WORKER_HIGH_WATERMARK_SCHEMA
=ConfigValidation.PositiveNumberValidator;
+
+ /**
+ * This signifies a state that a worker has left the congestion.
+ * If the used ratio of a worker's outgoing queue is lower than the
low watermark,
+ * it notifies the worker to check whether all its executors have also
left congestion,
+ * if yes, it will unset the worker's backpressure flag on the
Zookeeper
+ */
+ public static final String
BACKPRESSURE_WORKER_LOW_WATERMARK="backpressure.worker.low.watermark";
+ public static final Object BACKPRESSURE_WORKER_LOW_WATERMARK_SCHEMA
=ConfigValidation.PositiveNumberValidator;
+
+ /**
+ * This signifies the tuple congestion in an executor's receiving
queue.
+ * When the used ratio of an executor's receiving queue is higher than
the high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+ public static final String
BACKPRESSURE_EXECUTOR_HIGH_WATERMARK="backpressure.executor.high.watermark";
+ public static final Object BACKPRESSURE_EXECUTOR_HIGH_WATERMARK_SCHEMA
=ConfigValidation.PositiveNumberValidator;
+
+ /**
+ * This signifies a state that an executor has left the congestion.
+ * If the used ratio of an execuotr's receive queue is lower than the
low watermark,
+ * it may notify the worker to check whether all its executors have
also left congestion,
+ * if yes, the worker's backpressure flag will be unset on the
Zookeeper
+ */
+ public static final String
BACKPRESSURE_EXECUTOR_LOW_WATERMARK="backpressure.executor.low.watermark";
+ public static final Object BACKPRESSURE_EXECUTOR_LOW_WATERMARK_SCHEMA
=ConfigValidation.PositiveNumberValidator;
--- End diff --
Looking through the code it looks like these two configs are never used.
BACKPRESSURE_EXECUTOR_HIGH_WATERMARK and
BACKPRESSURE_EXECUTOR_LOW_WATERMARK. I don't see a reason to have different
configs for different queues. Perhaps we can just remove these configs. and
make the description for the other configs more generic.
> Automatic Back Pressure
> -----------------------
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG
> since different components may have unmatched tuple processing speed.
> Currently, the tuples may get dropped if the downstream components can not
> process as quickly, thereby causing a waste of network bandwidth and
> processing capability. In addition, it is difficult to tune the
> max.spout.pending parameter for best backpressure performance. Therefore, an
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that does not rely on acking or max
> spout pending. Instead spouts throttle not only when max.spout.pending is
> hit, but also if any bolt has gone over a high water mark in their input
> queue, and has not yet gone below a low water mark again. There is a lot of
> room for potential improvement here around control theory and having spouts
> only respond to downstream bolts backing up, but a simple bang-bang
> controller like this is a great start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
