[
https://issues.apache.org/jira/browse/STORM-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rick Kellogg updated STORM-55:
------------------------------
Component/s: storm-core
> Replace Storm's scheduler with a constraint logic programming engine
> --------------------------------------------------------------------
>
> Key: STORM-55
> URL: https://issues.apache.org/jira/browse/STORM-55
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/383
> CLP seems to be a great fit for Storm's resource scheduling. We want to be
> able to declaratively specify constraints, such as:
> 1. Topology A's slots should be <= 10 and as close to 10 as possible
> (minimize the delta between assigned slots and 10)
> 2. All topologies should use less than 200 CPU's and less than 600 GB of
> memory
> 3. Topology B should run at most 2 workers on each host
> 4. Each worker for topology C should run at most one task for component X and
> one task for component Y
> 5. Should minimize the amount of reassignment to running topologies in order
> to satisfy constraints
> 6. Should only be allowed to reassign workers for an individual topology
> whose individual constraints are satisfied once every 10 minutes
> And then the logic engine should do an efficient search and optimize the
> constraints.
> What's going to make this especially powerful is exposing the logic engine to
> users (so people can plug in functions that return new constraints), making
> it really easy for people to add sophisticated scheduling logic with minimal
> effort.
> Clojure's core.logic may be a great fit for this. According to this thread it
> doesn't do minimization goals (
> https://groups.google.com/group/clojure/browse_thread/thread/a050a4e6b390514a
> ), but it may in the future.
> -------------------------------------------
> @nathanmarz: There also appears to be a fair amount of research in this area
> that we should look at:
> http://scholar.google.com/scholar?q=resource+scheduling+constraint+logic+programming&hl=en&as_sdt=0&as_vis=1&oi=scholart&sa=X&ei=X1uIUK6gA4rBigL5yoCgBg&ved=0CD0QgQMwAA
> -------------------------------------------
> bgedik: It would be good if the default scheduler understands placement
> constraints. Would be very helpful if the placement constraints can be
> attached to the topology, relieving the developer from having to write a
> scheduler to get the placement she wants.
> The placement constraints can come in various forms. Here are some examples:
> Isolation constraint: The host/spout task cannot share its host with any
> other task
> Co-location constraint: A group of host/spout tasks has to be placed on the
> same host
> Ex-location constraint: A group of host/spout tasks cannot be placed on the
> same host
> Pool placement: A group of host/spout tasks are sprayed over a pool of hosts
> Explicit placement: A host/spout task is placed on a user-specified host
> Pools can be defined using host properties. For instance, configuration files
> can be used to assign properties to hosts. As a specific example, some hosts
> can have GPUs installed. As an application developer, I can request 4 such
> hosts (with gpu=true property) to create a pool in my topology construction
> code (without hardcoding the identities of the hosts). Then I can spray some
> of my bolt instances across the hosts in my GPU pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
