Repository: storm Updated Branches: refs/heads/master 31db7dc2f -> fbcd074c6
Adding documentation for Trident RAS API Project: http://git-wip-us.apache.org/repos/asf/storm/repo Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/3b56c6b1 Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/3b56c6b1 Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/3b56c6b1 Branch: refs/heads/master Commit: 3b56c6b1847e5388e94b5c195b0965d55198d81c Parents: df3867f Author: Kyle Nusbaum <[email protected]> Authored: Thu Mar 24 14:42:17 2016 -0500 Committer: Kyle Nusbaum <[email protected]> Committed: Thu Mar 24 14:42:17 2016 -0500 ---------------------------------------------------------------------- docs/Resource_Aware_Scheduler_overview.md | 2 ++ docs/Trident-RAS-API.md | 49 ++++++++++++++++++++++++++ docs/index.md | 1 + 3 files changed, 52 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/storm/blob/3b56c6b1/docs/Resource_Aware_Scheduler_overview.md ---------------------------------------------------------------------- diff --git a/docs/Resource_Aware_Scheduler_overview.md b/docs/Resource_Aware_Scheduler_overview.md index 591b37c..30e0eeb 100644 --- a/docs/Resource_Aware_Scheduler_overview.md +++ b/docs/Resource_Aware_Scheduler_overview.md @@ -16,6 +16,8 @@ The user can switch to using the Resource Aware Scheduler by setting the followi ## API Overview +For use with Trident, please see the [Trident RAS API](Trident-RAS-API.html) + For a Storm Topology, the user can now specify the amount of resources a topology component (i.e. Spout or Bolt) is required to run a single instance of the component. The user can specify the resource requirement for a topology component by using the following API calls. ### Setting Memory Requirement http://git-wip-us.apache.org/repos/asf/storm/blob/3b56c6b1/docs/Trident-RAS-API.md ---------------------------------------------------------------------- diff --git a/docs/Trident-RAS-API.md b/docs/Trident-RAS-API.md new file mode 100644 index 0000000..30a3889 --- /dev/null +++ b/docs/Trident-RAS-API.md @@ -0,0 +1,49 @@ +--- +title: Trident RAS API +layout: documentation +documentation: true +--- + +## Trident RAS API + +The Trident RAS (Resource Aware Scheduler) API provides a mechanism to specify the resource consumption of their topology. The API looks exactly like the base RAS API, only it is called on Trident Streams instead of Bolts and Spouts. + +In order to avoid duplication and inconsistency in documentation, the purpose and effects of resource setting are not described here, but are instead found in the [Resource Aware Scheduler Overview](Resource_Aware_Scheduler_overview.html) + +### Use + +First an example: + +```java + TridentTopology topo = new TridentTopology(); + TridentState wordCounts = + topology + .newStream("words", feeder) + .parallelismHint(5) + .setCPULoad(20) + .setMemoryLoad(512,256) + .each( new Fields("sentence"), new Split(), new Fields("word")) + .setCPULoad(10) + .setMemoryLoad(512) + .each(new Fields("word"), new BangAdder(), new Fields("word!")) + .parallelismHint(10) + .setCPULoad(50) + .setMemoryLoad(1024) + .groupBy(new Fields("word!")) + .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")) + .setCPULoad(100) + .setMemoryLoad(2048); +``` + +Resources can be set per operation (except for grouping, shuffling, partitioning). +Operations that are combined by Trident into single Bolts have their resources summed. + +Every Bolt is given **at least** the default resources, regardless of user settings. + +In the above case, we end up with + * a spout and spout coordinator with a CPU load of 20% each, and a memory load of 512MiB on heap and 256MiB off heap. + * a bolt with 60% cpu load (10% + 50%) and a memory load of 1536MiB (1024 + 512) on heap from the combined `Split` and `BangAdder` + * a bolt with 100% cpu load and a memory load of 2048MiB. + +The methods can be called for every operation (or some of the operations) or used in the same manner as `parallelismHint()`. +That is, resource declarations have the same *boundaries* as parallelismHints. They don't cross any groupings, shufflings, or any other kind of repartitioning. http://git-wip-us.apache.org/repos/asf/storm/blob/3b56c6b1/docs/index.md ---------------------------------------------------------------------- diff --git a/docs/index.md b/docs/index.md index ff36693..56dcfd4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -28,6 +28,7 @@ Trident is an alternative interface to Storm. It provides exactly-once processin * [Trident API Overview](Trident-API-Overview.html) -- operations for transforming and orchestrating data * [Trident State](Trident-state.html) -- exactly-once processing and fast, persistent aggregation * [Trident spouts](Trident-spouts.html) -- transactional and non-transactional data intake +* [Trident RAS API](Tridnet-RAS-API.html) -- using the Resource Aware Scheduler with Trident. ### Setup and Deploying
