GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/739
fix same name topologies bug
When a topology is submitted repeatedly very closely, or nimbus is too busy
to response the repeated submit RPC calls from same topology, there is very
high
Github user danny0405 commented on the pull request:
https://github.com/apache/storm/pull/739#issuecomment-140941136
I have filed a JIRA issue, thanks for your nice response.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/1874
@arunmahadevan thx for your nice review, i have checked the code
The fact is that, now for storm, we will pass in a `:scratch-topology-id`
to execute mk-assignments function for rebalance
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/1874
@arunmahadevan There are over 2500+ topologies on our cluster, in most
cases, we wanna increase component executors too when we add workers to a
topology in order to make full use of these workers
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/1874
@erikdw thx for your nice review, i have fixed the headline and ticket
title.
For this PR, the ratio is still 1:1 for tasks and executors for a
component, but not for the Storm before
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/838
Now pacemaker is a stand-alone service and no HA is supported. When it goes
down, all the workers's heartbeats will be lost. It will take a long time to
recover even if pacemaker goes up
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/1874
Strom Rebalance command should support arbitrary component parallelism
For legacy reasons, config TOPOLOGY-TASKS is considered first when
schedule a topology, for a component, if user donâ
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/1851
@erikdw thx for your nice review, i have change the style ^_^
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/1851
@HeartSaVioR thx, i have changed the title
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/1851
Kafka spout should consume from latest when zk offset bigger than latest
offset
Kafka spout should consume from latest when ZK offset bigger than latest
offset[ an TopicOffsetOutOfRangeException
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@revans2
I have removed the rocks db backend
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@revans2 really thx for your review work
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138791919
--- Diff:
storm-core/src/jvm/org/apache/storm/assignments/AssignmentDistributionService.java
---
@@ -0,0 +1,232 @@
+/**
+ * Licensed
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138780747
--- Diff: storm-core/src/clj/org/apache/storm/cluster.clj ---
@@ -244,15 +255,19 @@
;; Watches should be used for optimization. When ZK
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138781163
--- Diff:
storm-core/src/jvm/org/apache/storm/assignments/AssignmentDistributionService.java
---
@@ -0,0 +1,232 @@
+/**
+ * Licensed
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138781061
--- Diff: storm-core/src/clj/org/apache/storm/cluster.clj ---
@@ -244,15 +255,19 @@
;; Watches should be used for optimization. When ZK
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138780949
--- Diff: storm-core/src/clj/org/apache/storm/cluster.clj ---
@@ -123,6 +133,7 @@
(def SUPERVISORS-SUBTREE (str "/" SUPERVISORS-ROOT))
Github user danny0405 closed the pull request at:
https://github.com/apache/storm/pull/2318
---
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2319
[STORM-2693] Nimbus assignments promotion
Storm now doesn't support large cluster[ for example thousand of
supervisors] very well, for our production, topology submission/killing is very
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138091372
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -758,6 +752,35 @@
(catch Exception e
(log-error e "
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138092651
--- Diff: storm-core/pom.xml ---
@@ -331,6 +331,10 @@
commons-codec
+org.rocksdb
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138097813
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -961,14 +1015,25 @@
;; tasks figure out what tasks to talk to by looking
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138103588
--- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
@@ -961,14 +1015,25 @@
;; tasks figure out what tasks to talk to by looking
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2318
[STORM-2693] Nimbus assignments assign promotion
Storm now didn't support large cluster[ for example thousand of
supervisors] very well, for our production, topology submission/killing is very
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2319#discussion_r138090333
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -62,26 +63,17 @@
(defn- assignments-snapshot [storm-cluster-state
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@HeartSaVioR @revans2
i have change the default backend to memory one, also i change the
distribute mode to asynchronous
i chose a rocks-db backend because when the cluster goes to large
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2203
@HeartSaVioR @revans2 I have read most of the tickets and issues of
STORM-2153, there are many discussions about the metrics V2 and the now
built-in metrics. But i didn't see any about
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@revans2 hi, is this PR okey for merging now? I have another heartbeats
promotion which is based on this, and i expect for your review too
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153036746
--- Diff: storm-client/src/jvm/org/apache/storm/Config.java ---
@@ -1293,6 +1318,44 @@
@isPositiveNumber
public static final String
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153036973
--- Diff:
storm-client/src/jvm/org/apache/storm/cluster/StormClusterStateImpl.java ---
@@ -43,13 +44,15 @@
private static Logger LOG
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153037176
--- Diff:
storm-client/src/jvm/org/apache/storm/cluster/StormClusterStateImpl.java ---
@@ -147,21 +153,52 @@ protected void
issueMapCallback
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153037892
--- Diff: storm-client/src/jvm/org/apache/storm/stats/StatsUtil.java ---
@@ -1589,18 +1570,16 @@ public static ComponentPageInfo aggCompExecsStats
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152716336
--- Diff:
storm-client/src/jvm/org/apache/storm/assignments/ILocalAssignmentsBackend.java
---
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152716491
--- Diff:
storm-client/src/jvm/org/apache/storm/cluster/StormClusterStateImpl.java ---
@@ -43,13 +44,15 @@
private static Logger LOG
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717501
--- Diff: storm-core/test/clj/org/apache/storm/nimbus_test.clj ---
@@ -792,15 +795,18 @@
(bind _ (log-message "ass1, t76, pre beat: &quo
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152716651
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java
---
@@ -382,6 +374,33 @@ public void establishLogSettingCallback
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717096
--- Diff: storm-client/src/jvm/org/apache/storm/stats/StatsUtil.java ---
@@ -1547,26 +1533,21 @@ public static ComponentPageInfo aggCompExecsStats
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717747
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -231,7 +183,15 @@
@VisibleForTesting
public
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717599
--- Diff: storm-server/pom.xml ---
@@ -130,7 +130,7 @@
maven-checkstyle-plugin
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717373
--- Diff: storm-client/src/jvm/org/apache/storm/stats/StatsUtil.java ---
@@ -1589,18 +1570,16 @@ public static ComponentPageInfo aggCompExecsStats
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717439
--- Diff:
storm-client/test/jvm/org/apache/storm/assignments/LocalAssignmentsBackendTest.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r152717836
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -1672,7 +1703,7 @@ private int fragmentedCpu() {
long
Github user danny0405 closed the pull request at:
https://github.com/apache/storm/pull/2431
---
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2433
[STORM-2693] Heartbeats and assignments promotion for storm2.0
For large cluster support, good scheduling performance for scheduling and
assignments distribution.
For nimbus heartbeats
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@HeartSaVioR @revans2 i have moved all the features to
https://github.com/apache/storm/pull/2389 for 1.1.x-branch
https://github.com/apache/storm/pull/2433 for 2.0-branch
so
Github user danny0405 closed the pull request at:
https://github.com/apache/storm/pull/2319
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153705766
--- Diff:
storm-server/src/main/java/org/apache/storm/nimbus/TimeOutWorkerHeartbeatsRecoveryStrategy.java
---
@@ -0,0 +1,86 @@
+/**
+ * Licensed
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153778405
--- Diff:
storm-server/src/main/java/org/apache/storm/nimbus/TimeOutWorkerHeartbeatsRecoveryStrategy.java
---
@@ -0,0 +1,86 @@
+/**
+ * Licensed
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r153774053
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/supervisor/Supervisor.java
---
@@ -228,6 +257,51 @@ private void launchDaemon
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2431
heartbeats and assignments promotion for storm2.0
For large cluster support, good scheduling performance for scheduling and
assignments distribution.
This is PR for 1.1.x-branch
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2431
@HeartSaVioR hi, i have make PR for storm2.0, please help me to review this
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@HeartSaVioR i have made a PR for storm 2.0, really thx for your review
work, this is the link:
https://github.com/apache/storm/pull/2431/
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2389
@HeartSaVioR hi, i have add support for local cluster and fix the unit
tests problems, thx for your nice review work
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2319
@HeartSaVioR ok, i will do the 2.0.0 version, the promotion is running on
our production cluster for a while and it works great, i will start to work on
the 2.0.0 branch now, is the master branch
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2389#discussion_r148777382
--- Diff:
storm-core/test/jvm/org/apache/storm/nimbus/NimbusHeartbeatsPressureTest.java
---
@@ -0,0 +1,208 @@
+package org.apache.storm.nimbus
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2389#discussion_r148776617
--- Diff:
storm-core/test/jvm/org/apache/storm/nimbus/NimbusHeartbeatsPressureTest.java
---
@@ -0,0 +1,208 @@
+package org.apache.storm.nimbus
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2389#discussion_r148777617
--- Diff:
storm-core/test/jvm/org/apache/storm/nimbus/NimbusHeartbeatsPressureTest.java
---
@@ -0,0 +1,208 @@
+package org.apache.storm.nimbus
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2389
@revans2 @HeartSaVioR can you help me to review this?
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2433#discussion_r154558915
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -231,7 +183,15 @@
@VisibleForTesting
public
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2389
Storm heartbeats promotion
Storm now doesn't support large cluster[ for example thousand of
supervisors] very well, for our production, topology submission/killing is very
ineffective when
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2433
@HeartSaVioR i have fixed the checkstyle issue already, yeah the storm-core
building is slow, i will check the reason.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2647
@revans2
Thx for your work, the storm-core travis check still fails, we should fix
that.
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2647#discussion_r186665919
--- Diff:
storm-server/src/main/java/org/apache/storm/scheduler/Cluster.java ---
@@ -763,6 +773,7 @@ public void setAssignments
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2647#discussion_r186665266
--- Diff:
storm-server/src/main/java/org/apache/storm/scheduler/Cluster.java ---
@@ -48,6 +49,9 @@
public class Cluster implements
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2647
@revans2
I approve with you promotion totally.
The only concern are all kinds of cache we use here, now storm has many
caches not just for scheduling. I just think if we can make
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2640
@revans2
I have fixed the conflicts and rebase, thx for your review work.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2640
@revans2
Hi, Bobby, kindly reminder.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2714
@srdo
I kind of saw the Apache Flink metrics system, is has a internal metrics
layer on top of the Yammer Metrics, every Role in Flink has a MetricsGroup, one
MetricsGroup has many Metrics
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2721
For security, I think we need to keep the if user check, we should fix it
but not through deleting it.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2723
@HeartSaVioR @revans2
Please help me to review this if you have time, thx in advance
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2636
@cfriaszapater
Ok, look forward to your master branch patch.
---
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2723
[STORM-3112] Incremental scheduling supports
Incremental scheduling supports.
JIRA: [STORM-3112](https://issues.apache.org/jira/browse/STORM-3112).
You can merge this pull request into a Git
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2710#discussion_r194212295
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java ---
@@ -410,9 +420,11 @@ static DynamicState
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2710
Sorry, i didn't see any practical changes for this patch but only some
comment java doc. Can you describe it?
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2718
@agresch
Yes the
[https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/StormTimer.java#L173](https://github.com/apache/storm/blob/master/storm-client/src/jvm
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2636#discussion_r191669691
--- Diff: external/storm-cassandra/README.md ---
@@ -24,6 +24,12 @@ The following properties may be passed to storm
configuration
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2636#discussion_r191669779
--- Diff: external/storm-cassandra/README.md ---
@@ -24,6 +24,12 @@ The following properties may be passed to storm
configuration
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2636#discussion_r191670022
--- Diff:
external/storm-cassandra/src/main/java/org/apache/storm/cassandra/client/SslProps.java
---
@@ -0,0 +1,92 @@
+/**
+ * Licensed
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2636
@cfriaszapater
You should first build and install your dependencies modules successfully.
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2636#discussion_r191670402
--- Diff: integration-test/pom.xml ---
@@ -22,7 +22,7 @@
storm
org.apache.storm
-1.2.0-SNAPSHOT
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2701
+ 1, thx for your contribution.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2700
@Ethanlm
Sorry i have not did the benchmark test, but we can see the promotion
through the storm-core module building time, it's about 10 seconds faster on my
laptop.
I can add
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2699
Looks good, thx for your contribution.
---
GitHub user danny0405 opened a pull request:
https://github.com/apache/storm/pull/2700
[STORM-3093] Cache the storm id to executors mapping on master to avoâ¦
# What this patch for
Now nimbus will collect all the topologies's conf/topology-ser/storm-base
to compute
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2702
+1 thx for your contribution.
---
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2703
Sorry i don't think put the validation at Client side is a good idea, cause
many kind of clients may interact with Master through thrift RPC, we must keep
all the clients sync on this validation
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193270487
--- Diff: pom.xml ---
@@ -82,7 +82,7 @@
Committer
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193270523
--- Diff: pom.xml ---
@@ -109,7 +109,7 @@
Committer
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2704
@srdo
Great work, i will take my time to review this patch as soon as possible.
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193434753
--- Diff:
storm-client/src/jvm/org/apache/storm/messaging/netty/SaslStormClientHandler.java
---
@@ -41,80 +38,88 @@ public SaslStormClientHandler
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193438481
--- Diff:
storm-server/src/main/java/org/apache/storm/pacemaker/PacemakerServer.java ---
@@ -83,48 +88,51 @@ public PacemakerServer(IServerMessageHandler
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193438066
--- Diff: storm-core/pom.xml ---
@@ -456,29 +456,29 @@
- org.apache.storm
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193437756
--- Diff: storm-core/pom.xml ---
@@ -140,7 +140,7 @@
data.codec
test
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193437909
--- Diff: storm-core/pom.xml ---
@@ -365,17 +365,17 @@
${basedir}/src/resources
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2703
@revans2
You are right, i just thought if we can keep all these clients sync in
rule, but fail fast just for this is ok.
---
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193436151
--- Diff:
storm-client/src/jvm/org/apache/storm/messaging/netty/StormServerPipelineFactory.java
---
@@ -12,28 +12,40 @@
package
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193437371
--- Diff: storm-client/src/jvm/org/apache/storm/utils/TransferDrainer.java
---
@@ -40,94 +41,39 @@ public void add(TaskMessage taskMsg
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193437501
--- Diff: storm-client/src/jvm/org/apache/storm/utils/TransferDrainer.java
---
@@ -40,94 +41,39 @@ public void add(TaskMessage taskMsg
Github user danny0405 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2704#discussion_r193436981
--- Diff:
storm-client/src/jvm/org/apache/storm/pacemaker/PacemakerClient.java ---
@@ -100,23 +99,25 @@ public PacemakerClient(Map config,
String host
Github user danny0405 commented on the issue:
https://github.com/apache/storm/pull/2723
@revans2
Hi, bobby, i have already updated the test data for this patch.
As for NeedsFullTopologiesScheduler, i agree to remove it. We can tweak the
IsolationScheduler or cache
1 - 100 of 242 matches
Mail list logo
