Github user acommuni commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-195988246
In case external system is down, it could be interesting to pause the Spout
for an amount of time. The state "OPEN" of the CB could be directly linked to
the spout. I do
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-146874511
@rsltrifork That does sound like a very interesting approach. I would love
to see some numbers on how it would perform.
---
If your project is set up for it, you can re
Github user rsltrifork commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-146779690
Thanks for the quick reply, revans2.
> How do you know that the bolt is waiting in a controlled manner?
A Bolt sending to an external system can includ
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-144723285
@rsltrifork, No this does not solve that issue. The timeout is still a
hard coded value. The backpressure just means that the spout will not be
outputting new values.
Github user rsltrifork commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-144703338
Does this solve the problem of tuple timeout, when a bolt is completely
stalled waiting for an external component to get back up?
I believe waiting for too long in
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/700
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabl
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-141544473
@zhuoliu I looked over the code, and I ran some tests with a word count
program that does not sleep, and there are not enough split sentence bolts.
This kept the workers
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-141514572
Thanks, Bobby @revans2 . I addressed all the comments.
Actually I have such examples already written and tested. See:
https://github.com/zhuoliu/storm/blob/888/e
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-141499766
For the most part things look really good. I would also love to see
something added to Examples. Like a Word Count that just goes as fast as it
can. That way we can se
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r39872683
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -35,6 +35,7 @@
import java.util.HashMap;
import java.util.Map;
impor
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r39872538
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -138,7 +150,21 @@ private void consumeBatchToCursor(long cursor,
EventHandler hand
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r39872212
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -1024,6 +1024,48 @@
/**
+ * Whether to enable backpressure in for a
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r39871444
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -602,6 +640,7 @@
(log-message "Activating spout " component-id "
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-141491743
@zhuoliu looks like you missed adding in a file, We also need to upmerge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-139678312
2. Test with a congested topology.
WordCountTopology2 (3 times more workers and executors than original
WordCount; in every minute, for the wordcount bolt: in the fir
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-139676554
Initial tests on 5 node openstack cluster (1 zk, 1 nimbus, 3 supervisors).
1. Test a normal topology (no congestion happen)
WordCountTopology3 (3 times more worker
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-137588941
Hi, Bobby @revans2 , all comments and concerns have been addressed. Ready
for another round of review. Thanks!
---
If your project is set up for it, you can reply to thi
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38438290
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,38 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.se
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38437006
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,38 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135577332
Addressed comments:
a. Removed the config of suspend time, reuse the empty emit and put the
judgement together with max.spout.pending
b. Separate functions for cre
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38152708
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,34 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.s
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38128803
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -137,9 +159,14 @@
(.add remote (TaskMessage. task (.serialize
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38126936
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -811,9 +841,18 @@
(setup-metrics! executor-data)
(let [r
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135422890
@arunmahadevan It is the intention to throttle the entire topology, all
spouts. This is what Heron does and is intended to be a last resort, which is
better then nothing
Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135359036
It appears that the spouts across all the workers are throttled if an
executor queue fills up. For instance if there are multiple spouts, even the
ones not causing
Github user vesense commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135248186
+1 Looks very exciting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38023281
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,34 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.se
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135145296
This looks very interesting. I would really like to see some unit tests,
especially around the disruptor queue to show that the callback is working.
Perhaps we can also
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38023405
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -510,15 +554,21 @@
))
)
credentials (a
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38022357
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,34 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.se
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38022111
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -671,7 +689,19 @@
user-context (:user-context ta
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38021285
--- Diff: conf/defaults.yaml ---
@@ -140,6 +140,14 @@ task.heartbeat.frequency.secs: 3
task.refresh.poll.secs: 10
task.credentials.poll.secs: 30
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38021586
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -602,7 +607,15 @@
(log-message "Activating spout " component-id
GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/700
[STORM-886] Automatic Back Pressure (ABP)
This new feature is aimed for automatic flow control through the topology
DAG since different components may have unmatched tuple processing speed.
Currentl
34 matches
Mail list logo
