[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/667


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[knusbaum]

Github user knusbaum commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-193484745
  
JStorm's cgroups merged with #1053. Closing this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-168877269
  
I agree with @d2r, or would at least answer that, yes, we should hold off 
on cgroups support until the jstorm merge. I believe that @revans2 might have 
already covered this in JIRA. If not, please open a new issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[bastiliu]

Github user bastiliu commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-168880465
  
@ptgoetz  Yes, @revans2  has created one. Following is the JIRA for jstorm 
merge.
https://issues.apache.org/jira/browse/STORM-1336


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[d2r]

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-167626299
  
@revans2 @hustfxj @jerrypeng @caishize 

Should we add support for cgroups with the jstorm merge, or should we try 
to add support for it now?

@caishize It would be good to create a Jira issue for this change and 
resolve the conflicts with the master branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[hustfxj]

Github user hustfxj commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-154820435
  
it is great that storm begin to support cgroup function just like jstorm


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[jerrypeng]

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-154126726
  
any updates on this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[caishize]

Github user caishize commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-146106627
  
Yes,there's an update haven't been  submitted, but it still only works on 
an Linux operating system which has the  cgroup installed..

Sent from my iPhone

> On Oct 3, 2015, at 11:46 PM, Robert (Bobby) Evans 
 wrote:
> 
> @caishize Any update on this? I really would like to have cgroup support, 
and this should work really well with the resource aware scheduling we are 
working on, on another JIRA.
> 
> —
> Reply to this email directly or view it on GitHub.
> 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-145260043
  
@caishize Any update on this?  I really would like to have cgroup support, 
and this should work really well with the resource aware scheduling we are 
working on, on another JIRA. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/667#issuecomment-136175840
  
@caishize please file a JIRA here 
https://issues.apache.org/jira/browse/STORM . I haven't taken a close look at 
the changes but does any of these changes interfere with running on windows?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[caishize]

Github user caishize commented on a diff in the pull request:

https://github.com/apache/storm/pull/667#discussion_r36604416
  
--- Diff: storm-core/src/clj/backtype/storm/cgroup/cgroup_manager.clj ---
@@ -0,0 +1,109 @@
+(ns backtype.storm.cgroup.cgroup-manager
+  (:import [java.io File BufferedReader FileReader])
+  (:use [backtype.storm log]
+[backtype.storm.cgroup constant])
+  (:require [backtype.storm.util :as util]))
+
+(defn- get-dir [dir file-dir]
+  (util/normalize-path (str dir file-dir)))
+
+(defn get-tasks [dir]
+  (if-let [string-tasks (util/read-file (get-dir dir TASKS))]
+(into #{} (map #(util/parse-int %) string-tasks
+
+(defn add-task [dir task]
+  (util/append-line (get-dir dir TASKS) (str task)))
+
+(defn set-cpu-shares [dir weight]
+  (util/append-line (get-dir dir CPU-SHARES) (str weight)))
+
+(defn set-physical-usage-limit [dir value]
+  (util/append-line (get-dir dir MEMORY-LIMIT-IN-BYTES) (str value)))
+
+(defn set-with-swap-usage-limit [dir value]
+  (util/append-line (get-dir dir MEMORY-MEMSW-LIMIT-IN-BYTES) (str value)))
+
+(defn set-oom-control [dir flag]
+  (util/append-line (get-dir dir MEMORY-OOM-CONTROL) (if flag 0 1)))
+
+(defn analyse-subsystem [subsystems]
+  (loop [systems #{}
+ rest-systems (.split subsystems ,)]
+(if-let [system (keyword (first rest-systems))]
+  (recur (if (contains? SUBSYSTEM-TYPE system)
+   (conj systems system)
+   systems)
+(rest rest-systems))
+  systems)))
+
+(defn get-hierarchies []
+  (with-open [^FileReader file (FileReader. MOUNT-STATUS-FILE)
+  ^BufferedReader br (BufferedReader. file)]
+(loop [hierarchies {}
+   line (.readLine br)]
+  (if line
+(let [fields (.split line  )]
+  (if (= (get fields 2) cgroup)
+(let [name (get fields 0)
+  type (get fields 3)
+  dir (get fields 1)
+  hierarchy {:name name :subsystems (analyse-subsystem 
type) :dir dir :parent nil :is-root? true}]
+  (recur (conj hierarchies {type hierarchy}) (.readLine br)))
+(recur hierarchies (.readLine br
+(vals hierarchies)
+
+(defn busy [subsystem]
+  (loop [_hierarchy (get-hierarchies)]
+(if-let [hierarchy (first _hierarchy)]
+  (if (contains? (:subsystems hierarchy) subsystem)
+hierarchy
+(recur (rest _hierarchy))
+
+(defn check-cgroup
+  return true means cgroup service is ok, else means that cgroup hasn't 
been install or hasn't been started.
+  [conf]
+  (log-message checking cgroup mode)
+  (if (true? (conf CGROUP-ENABLE))
+(if-not (.exists (File. CGROUP-STATUS-FILE))
+  (log-error Cgroup error, please check /proc/cgroups, maybe hasn't 
been started.)
+  (if-let [root-dir (conf CGROUP-ROOT-DIR)]
+(let [_files (map #(File. (util/normalize-path (str % 
File/separator root-dir))) [CPU-HIERARCHY-DIR MEMORY-HIERARCHY-DIR])]
+  (loop [files _files]
+(if-let [file (first files)]
+  (if-not (.exists file)
+(log-error (.getAbsolutePath file)  is not existing.)
+(recur (rest files)))
+  true)))
+
+(defn mounted [hierarchy]
+  (let [dir (:dir hierarchy)
+name (:name hierarchy)]
+(if (util/exists-dir? dir)
+  (loop [a_hierarchy (get-hierarchies)]
+(if-let [_hierarchy (first a_hierarchy)]
+  (if (and (= dir (:dir _hierarchy)) (= name (:name _hierarchy)))
+_hierarchy
+(recur (rest a_hierarchy
+
+
+;(defn mount [hierarchy]
--- End diff --

somebody may want to mount cgroup hierarchy in his code, not just with 
linux command.. i will delete it later


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[lazyval]

Github user lazyval commented on a diff in the pull request:

https://github.com/apache/storm/pull/667#discussion_r36594606
  
--- Diff: storm-core/src/clj/backtype/storm/cgroup/cgroup_manager.clj ---
@@ -0,0 +1,109 @@
+(ns backtype.storm.cgroup.cgroup-manager
+  (:import [java.io File BufferedReader FileReader])
+  (:use [backtype.storm log]
+[backtype.storm.cgroup constant])
+  (:require [backtype.storm.util :as util]))
+
+(defn- get-dir [dir file-dir]
+  (util/normalize-path (str dir file-dir)))
+
+(defn get-tasks [dir]
+  (if-let [string-tasks (util/read-file (get-dir dir TASKS))]
+(into #{} (map #(util/parse-int %) string-tasks
+
+(defn add-task [dir task]
+  (util/append-line (get-dir dir TASKS) (str task)))
+
+(defn set-cpu-shares [dir weight]
+  (util/append-line (get-dir dir CPU-SHARES) (str weight)))
+
+(defn set-physical-usage-limit [dir value]
+  (util/append-line (get-dir dir MEMORY-LIMIT-IN-BYTES) (str value)))
+
+(defn set-with-swap-usage-limit [dir value]
+  (util/append-line (get-dir dir MEMORY-MEMSW-LIMIT-IN-BYTES) (str value)))
+
+(defn set-oom-control [dir flag]
+  (util/append-line (get-dir dir MEMORY-OOM-CONTROL) (if flag 0 1)))
+
+(defn analyse-subsystem [subsystems]
+  (loop [systems #{}
+ rest-systems (.split subsystems ,)]
+(if-let [system (keyword (first rest-systems))]
+  (recur (if (contains? SUBSYSTEM-TYPE system)
+   (conj systems system)
+   systems)
+(rest rest-systems))
+  systems)))
+
+(defn get-hierarchies []
+  (with-open [^FileReader file (FileReader. MOUNT-STATUS-FILE)
+  ^BufferedReader br (BufferedReader. file)]
+(loop [hierarchies {}
+   line (.readLine br)]
+  (if line
+(let [fields (.split line  )]
+  (if (= (get fields 2) cgroup)
+(let [name (get fields 0)
+  type (get fields 3)
+  dir (get fields 1)
+  hierarchy {:name name :subsystems (analyse-subsystem 
type) :dir dir :parent nil :is-root? true}]
+  (recur (conj hierarchies {type hierarchy}) (.readLine br)))
+(recur hierarchies (.readLine br
+(vals hierarchies)
+
+(defn busy [subsystem]
+  (loop [_hierarchy (get-hierarchies)]
+(if-let [hierarchy (first _hierarchy)]
+  (if (contains? (:subsystems hierarchy) subsystem)
+hierarchy
+(recur (rest _hierarchy))
+
+(defn check-cgroup
+  return true means cgroup service is ok, else means that cgroup hasn't 
been install or hasn't been started.
+  [conf]
+  (log-message checking cgroup mode)
+  (if (true? (conf CGROUP-ENABLE))
+(if-not (.exists (File. CGROUP-STATUS-FILE))
+  (log-error Cgroup error, please check /proc/cgroups, maybe hasn't 
been started.)
+  (if-let [root-dir (conf CGROUP-ROOT-DIR)]
+(let [_files (map #(File. (util/normalize-path (str % 
File/separator root-dir))) [CPU-HIERARCHY-DIR MEMORY-HIERARCHY-DIR])]
+  (loop [files _files]
+(if-let [file (first files)]
+  (if-not (.exists file)
+(log-error (.getAbsolutePath file)  is not existing.)
+(recur (rest files)))
+  true)))
+
+(defn mounted [hierarchy]
+  (let [dir (:dir hierarchy)
+name (:name hierarchy)]
+(if (util/exists-dir? dir)
+  (loop [a_hierarchy (get-hierarchies)]
+(if-let [_hierarchy (first a_hierarchy)]
+  (if (and (= dir (:dir _hierarchy)) (= name (:name _hierarchy)))
+_hierarchy
+(recur (rest a_hierarchy
+
+
+;(defn mount [hierarchy]
--- End diff --

commented code? why would you want it? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: add cgroup function that can limit cpu share o...

[caishize]

GitHub user caishize opened a pull request:

https://github.com/apache/storm/pull/667

add cgroup function that can limit cpu share of each worker

add cgroup function that can limit cpu share of each worker, and it really 
help indeed

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/caishize/storm-1 master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #667


commit 72631f47a0678195a7a3b2db931e7fab728e459c
Author: darwin caish...@outlook.com
Date:   2015-08-06T07:01:29Z

add cgroup

commit 14e984812877309734f851c133005ad556f8d1ef
Author: darwin caish...@outlook.com
Date:   2015-08-06T07:03:23Z

add cgroup




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---