Github user danny0405 commented on a diff in the pull request:

    https://github.com/apache/storm/pull/2319#discussion_r138097813
  
    --- Diff: storm-core/src/clj/org/apache/storm/daemon/nimbus.clj ---
    @@ -961,14 +1015,25 @@
         ;; tasks figure out what tasks to talk to by looking at topology at 
runtime
         ;; only log/set when there's been a change to the assignment
         (doseq [[topology-id assignment] new-assignments
    -            :let [existing-assignment (get existing-assignments 
topology-id)
    -                  topology-details (.getById topologies topology-id)]]
    +            :let [existing-assignment (get existing-assignments 
topology-id)]]
           (if (= existing-assignment assignment)
             (log-debug "Assignment for " topology-id " hasn't changed")
             (do
               (log-message "Setting new assignment for topology id " 
topology-id ": " (pr-str assignment))
               (.set-assignment! storm-cluster-state topology-id assignment)
               )))
    +
    +    ;; grouping assignment by node to see the nodes diff, then notify 
nodes/supervisors to synchronize its owned assignment
    +    ;; because the number of existing assignments is small for every 
scheduling round,
    +    ;; we expect to notify supervisors at almost the same time.
    +    (->> new-assignments
    +         (map (fn [[tid new-assignment]]
    +           (let [existing-assignment (get existing-assignments tid)]
    +             (assignment-changed-nodes existing-assignment new-assignment 
))))
    +         (apply concat)
    +         (into #{})
    +         (notify-supervisors-assignments conf new-assignments))
    --- End diff --
    
    Really thx for your suggestions, i agree with you that putting a blocking 
method into nimbus's main loop is not a good idea, an thread poll is a better 
choice.
    Because only assignments changed nodes will be notified in a round, the 
nodes num will not be very large[ except that we submit really large topologies 
or submit large number of topologies at almost the same time ]
    Actually supervisors will synchronize its assignments every 10 seconds, if 
we make nimbus mk-assignments totally asynchronous, it will be okey.


---

Reply via email to