Repository: reef Updated Branches: refs/heads/master 8b997771b -> 6627d7017
[REEF-1116] Fix typos in REEF website JIRA: [REEF-1116](https://issues.apache.org/jira/browse/REEF-1116) Pull Request: This closes #751 Project: http://git-wip-us.apache.org/repos/asf/reef/repo Commit: http://git-wip-us.apache.org/repos/asf/reef/commit/6627d701 Tree: http://git-wip-us.apache.org/repos/asf/reef/tree/6627d701 Diff: http://git-wip-us.apache.org/repos/asf/reef/diff/6627d701 Branch: refs/heads/master Commit: 6627d7017665eaf5e41c9183eddfdbedabd45f1b Parents: 8b99777 Author: Dongjoon Hyun <[email protected]> Authored: Tue Dec 29 22:40:45 2015 -0800 Committer: Mariia Mykhailova <[email protected]> Committed: Wed Dec 30 12:43:42 2015 -0800 ---------------------------------------------------------------------- website/src/site/markdown/glossary.md | 28 ++++++++++---------- website/src/site/markdown/introduction.md | 8 +++--- website/src/site/markdown/mailing-list.md | 6 ++--- website/src/site/markdown/tang.md | 36 +++++++++++++------------- website/src/site/markdown/wake.md | 8 +++--- 5 files changed, 43 insertions(+), 43 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/reef/blob/6627d701/website/src/site/markdown/glossary.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/glossary.md b/website/src/site/markdown/glossary.md index d424d71..11372c8 100644 --- a/website/src/site/markdown/glossary.md +++ b/website/src/site/markdown/glossary.md @@ -25,13 +25,13 @@ under the License. ###<a name="context"></a>Context -Contexts are a way to structure the state and Configuration of an Evaluator. A Context exists on one and only one individual Evaluator. Each Evaluator has at least one Context, which we refer to as the *root* Context. This root context is special, as it is *required* on all Evaluators and because closing it is synonymous to releasing the Evaluator. In many simple REEF programs, this is the only Context used and it is therefore convenient to think of it as synonymous to "the Evaluator": The Driver can submit Tasks to it, is notifified when they are done and can close the root context when it wants to dispose of the Evaluator. +Contexts are a way to structure the state and Configuration of an Evaluator. A Context exists on one and only one individual Evaluator. Each Evaluator has at least one Context, which we refer to as the *root* Context. This root context is special, as it is *required* on all Evaluators and because closing it is synonymous to releasing the Evaluator. In many simple REEF programs, this is the only Context used and it is therefore convenient to think of it as synonymous to "the Evaluator": The Driver can submit Tasks to it, is notified when they are done and can close the root context when it wants to dispose of the Evaluator. -Contexts are formed by calls to `submitContext()` to the event types that allow this (`AllocatedEvaluator` and `ActiveContext`) Contexts are the main way for an Evaluator to be exposed and accessed. For instance, Tasks are submitted to an `ActiveContext` which represents the top Context on the Evaluator. +Contexts are formed by calls to `submitContext()` to the event types that allow this (`AllocatedEvaluator` and `ActiveContext`). Contexts are the main way for an Evaluator to be exposed and accessed. For instance, Tasks are submitted to an `ActiveContext` which represents the top Context on the Evaluator. -Beyond this, a Driver can submit a Context to the root, or in fact any, Context, as long as the resulting structure is that of a stack: The root Context forms the bottom of the stack, the top-most Context is called *active*, hence the `ActiveContext` event. The two can be one and the same, and often are: The root Context is the subject of the first `ActiveContext` event on an Evaluator. +Beyond this, a Driver can submit a Context to the root, or in fact any, Context, as long as the resulting structure is that of a stack: the root Context forms the bottom of the stack, the top-most Context is called *active*, hence the `ActiveContext` event. The two can be one and the same, and often are: the root Context is the subject of the first `ActiveContext` event on an Evaluator. -Nomenclature: When Context B is submitted to an already existing Context A, we say that Context A is the parent Context of Context B. Also, Context B is the child of Context A. +Nomenclature: when Context B is submitted to an already existing Context A, we say that Context A is the parent Context of Context B. Also, Context B is the child of Context A. It is only the `ActiveContext` that allows the submission of Tasks or child Contexts. Hence, one can think of the whole Evaluator structure as that of a stack: the root Context at the bottom, layers of Contexts in the middle and either the current `ActiveContext` or the current Task at the top. @@ -39,15 +39,15 @@ It is only the `ActiveContext` that allows the submission of Tasks or child Cont It is convenient to think of a Context as a `Configuration` that gets merged with the `Configuration` supplied for Tasks and child Contexts. While not entirely true (see below), this view allows us to show just *why* Contexts are a convenient construct. -It is often the case that subsequent tasks that get executed on an Evaluator want access to the same Configuration variables and / or the same objects. Consider a simple `LinkedList` bound to a named parameter. If that linked list is part of the subsequent Task `Configurations` submited, each Task will get its very *own* `LinkedList`. If the named parameter is bound in the Context `Configuration`, all Tasks subsequently submitted to the Context will get the very *same* `LinkedList` instance. +It is often the case that subsequent tasks that get executed on an Evaluator want access to the same Configuration variables and / or the same objects. Consider a simple `LinkedList` bound to a named parameter. If that linked list is part of the subsequent Task `Configurations` submitted, each Task will get its very *own* `LinkedList`. If the named parameter is bound in the Context `Configuration`, all Tasks subsequently submitted to the Context will get the very *same* `LinkedList` instance. ####Contexts are (Tang) Injectors -This mechanism is implemented by using Tang's `Injector`s. On the Evaluator, a Task is launched by first *forking* the Context's `Injector` with the Task`Configuration` and then requesting an instance of the `Task` interface from that forked `Injector`. By this mechanism and the fact that objects are singletons with respect to an `Injector` in Tang, object sharing can be implemented. All objects already instantiated on the Context `Injector` will also be referenced by the Task`Injector`. Hence, the `LinkedList` in the example above would be shared amongst subsequent Task `Injectors` in the construction of the `Task` instance. +This mechanism is implemented by using Tang's `Injector`s. On the Evaluator, a Task is launched by first *forking* the Context's `Injector` with the Task `Configuration` and then requesting an instance of the `Task` interface from that forked `Injector`. By this mechanism and the fact that objects are singletons with respect to an `Injector` in Tang, object sharing can be implemented. All objects already instantiated on the Context `Injector` will also be referenced by the Task `Injector`. Hence, the `LinkedList` in the example above would be shared amongst subsequent Task `Injectors` in the construction of the `Task` instance. ###<a name="driver"></a>Driver -REEF imposes a centralized control flow design on applications: All events are routed to the master node, called the Driver. REEF also prescribes event-driven programming for the Driver. In that sense, the application provided Driver is a collection of event handlers for the various events exposed in `DriverConfiguration`. While most of these deal with occurrences during the computation (Evaluator allocation, Task launch, ...), several stand out as life-cycle events of the Driver, and therefore the application: +REEF imposes a centralized control flow design on applications: all events are routed to the master node, called the Driver. REEF also prescribes event-driven programming for the Driver. In that sense, the application-provided Driver is a collection of event handlers for the various events exposed in `DriverConfiguration`. While most of these deal with occurrences during the computation (Evaluator allocation, Task launch, ...), several stand out as life-cycle events of the Driver, and therefore the application: ####ON_START @@ -67,7 +67,7 @@ Hence, the `ON_STOP` event can be used to prevent Driver shutdown, e.g. in appli ####Evaluators and Tasks -The Evaluator is the runtime environment for Tasks. On one Evaluator, there is either no or one Task executing at any given point in time. Different or multiple executions of the same Tasks can be executed in sequence on an Evaluator. The Evaluator and Task lifecycle are decoupled: Whenever a Task finishes, the Driver receives the CompletedTask event, which contains a reference to the Evaluator the Task executed on. It is then up to the Driver to decide whether to return the Evaluator to the resource manager or to make other use of it, e.g. by submitting another task. +The Evaluator is the runtime environment for Tasks. On one Evaluator, there is either no or one Task executing at any given point in time. Different or multiple executions of the same Tasks can be executed in sequence on an Evaluator. The Evaluator and Task lifecycle are decoupled: whenever a Task finishes, the Driver receives the CompletedTask event, which contains a reference to the Evaluator the Task executed on. It is then up to the Driver to decide whether to return the Evaluator to the resource manager or to make other use of it, e.g. by submitting another task. ####Evaluators and Contexts @@ -77,7 +77,7 @@ Contexts are REEF's form of state management inside of the Evaluator. See the [C On typical resource managers, an Evaluator is a process executing inside a container. Depending on the resource manager, that process may or may not be guarded by a resource or security isolation layer. -This also means that the Evaluator, not the Task, is the unit of resource consumption: while an Evaluator is occupying a Container, that Container is "used" from the perspective of the Resource Manager.That is true even if the Evaluator is idle from the perspective of the Driver, i.e. when no Task is running on it. +This also means that the Evaluator, not the Task, is the unit of resource consumption: while an Evaluator is occupying a Container, that Container is "used" from the perspective of the Resource Manager. That is true even if the Evaluator is idle from the perspective of the Driver, i.e. when no Task is running on it. ###<a name="task"></a>Task @@ -87,7 +87,7 @@ A Task in REEF is a unit of work to be executed on an Evaluator. In its simplest public byte[] call(byte[] input); -From REEF's perspective, a Task is therefore a single threaded method call. It starts when entering the call method. It is a `RunningTask` while it hasn't returned from it and is a `CompletedTask` when it has. Should there be an Exception thrown by `call()`, we call it a `FailedTask`. +From REEF's perspective, a Task is therefore a single-threaded method call. It starts when entering the call method. It is a `RunningTask` while it hasn't returned from it and is a `CompletedTask` when it has. Should there be an Exception thrown by `call()`, we call it a `FailedTask`. Task identity is established by a user-defined string set in `TaskConfiguration.IDENTIFIER`. All subsequent task-related events in the Driver will carry that ID. Note that REEF doesn't take any particular precautions to ensure unique Task identifiers. It is up to the application to do so. While technically feasible to assign the same identifier to multiple Tasks, this isn't advised as it makes error handling, debugging and logging unnecessarily hard. @@ -95,7 +95,7 @@ Task identity is established by a user-defined string set in `TaskConfiguration. The return value of the `call` method will be made available to the Driver as part of the `CompletedTask` event. Note that it isn't advised to return large values in this fashion, but merely small control flow or status information. Sending large data on this channel creates the risk of overloading the Driver at scale. The networking APIs provided by REEF IO are much better suited for data transmissions than this channel. -The parameter given to the call method is also to be used in a similar fashion: The Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter. +The parameter given to the call method is also to be used in a similar fashion: the Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter. ####Communicating between a Task and a Driver @@ -103,7 +103,7 @@ REEF provides some facilities to communicate between a Driver and a Task. These #####Sending a message from the Driver to a Task -REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered. +REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered: - Upon some schedule (which may also vary at runtime due to load conditions on the Driver), each Evaluator will report its current status to the Driver. This is used by the Driver to maintain health status and load statistics of the Evaluators. @@ -113,8 +113,8 @@ Whenever the Evaluator performs a heartbeat, it will ask the Task whether it has ####Multithreaded Tasks -Just because REEF views a Task as a method call doesn't restrict the Task to be single threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations: +Just because REEF views a Task as a method call doesn't restrict the Task to be single-threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations: - All Threads spawned need to exit before the `Task.call()` method returns. Otherwise, you run the risk of resource leakage. -- Exceptions on spawned Threads need to be caught and re-thrown by the `Thread.call()` method. Before that, all spawned threads need to be shut down, just like during a normal exit of `Task.call()`. If an exception from an another thread isn't caught, REEF's JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: The Driver will then have to allocate another Evaluator and try again. \ No newline at end of file +- Exceptions on spawned Threads need to be caught and re-thrown by the `Thread.call()` method. Before that, all spawned threads need to be shut down, just like during a normal exit of `Task.call()`. If an exception from an another thread isn't caught, REEF's JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: the Driver will then have to allocate another Evaluator and try again. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/reef/blob/6627d701/website/src/site/markdown/introduction.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/introduction.md b/website/src/site/markdown/introduction.md index 8a74d2d..9699d00 100644 --- a/website/src/site/markdown/introduction.md +++ b/website/src/site/markdown/introduction.md @@ -22,7 +22,7 @@ It is instructive to first remind us of how MapReduce works: Hadoop Map-Reduce s From the perspective of the scheduler, a number of issues arise that must be appropriately handled in order to scale-out to massive datasets. First, each map task should be scheduled close to where the input block resides; ideally on the same machine or rack. Second, failures can occur at the task level at any step; requiring backup tasks to be scheduled or the job being aborted. Third, performance bottlenecks can cause an imbalance in the task-level progress. The scheduler must react to these stragglers by scheduling clones and incorporating the logical task that crosses the finish line first. -Anyone of these issues can limit the scale-out degree of a Map-Reduce job. In what follows, we perscribe a scheduler framework that provides task life-cycle management mechanisms. Using this framework, we developed a complete version of the Map-Reduce runtime that addresses the above issues. Our framework is designed around three components. +Anyone of these issues can limit the scale-out degree of a Map-Reduce job. In what follows, we prescribe a scheduler framework that provides task life-cycle management mechanisms. Using this framework, we developed a complete version of the Map-Reduce runtime that addresses the above issues. Our framework is designed around three components. 1. `Driver`: captures the client code embodying the scheduler 2. `Evaluator`: provides a runtime environment on a container. @@ -54,10 +54,10 @@ The above figure presents the REEF components in terms of a running application, <br></br> -The Figure above describes the state transitions for (a) `Evaluator` and `Context` and (b) `Task` components. Each state transition is associated with an object reference that is surfaced to the `Driver` in an Rx-style interface. For instance, when the YARN-RM notifies the REEF-AM of an allocated container, the `Driver` is given an `AllocatedEvaluator` object; containing methods for adding configurations (i.e., for data services, see below) and file resources, and submit methods that bootstraps the `Evaluator` runtime on the YARN-NM. When an `Evaluator` bootstrap successfully completes, the `Driver` is given an `ActiveContext` object, which can be used to launch `Tasks` or to initiate a close, which triggers a shutdown event at the `Evaluator` runtime and a subsequent container release at the YARN-RM. If at any point a failure occurs, the `Driver` is passed a `FailedEvaluator` object; containing an exception trace when possible. +The Figure above describes the state transitions for (a) `Evaluator` and `Context` and (b) `Task` components. Each state transition is associated with an object reference that is surfaced to the `Driver` in an Rx-style interface. For instance, when the YARN-RM notifies the REEF-AM of an allocated container, the `Driver` is given an `AllocatedEvaluator` object; containing methods for adding configurations (i.e., for data services, see below) and file resources, and submit methods that bootstraps the `Evaluator` runtime on the YARN-NM. When an `Evaluator` bootstrap successfully completes, the `Driver` is given an `ActiveContext` object, which can be used to launch `Tasks` or to initiate a close, which triggers a shutdown event at the `Evaluator` runtime and a subsequent container release at the YARN-RM. If at any point a failure occurs, the `Driver` is passed a `FailedEvaluator` object, containing an exception trace when possible. -Recall that the `Driver` launches a `Task` on a submit method call from the `ActiveContext` reference. This state transition is denoted in the Figure above by the edge labeled submit; spanning the two state machines. The REEF-AM passes a `RunningTask` object to the `Driver` after receiving confirmation of a successful `Task` start or resume. The `Driver` may use the `RunningTask` reference to close or suspend the execution; triggering a `CompletedTask` or `SuspendedTask` object reference to the `Driver`. The `SuspendedTask` object contains a memento used to resume the execution on some (possibly alternative) `ActiveContext`. Exceptions during the `Task` execution are surfaced to the `Driver` in the form of a `FailedTask`, which contains the actual exception object. +Recall that the `Driver` launches a `Task` on a submit method call from the `ActiveContext` reference. This state transition is denoted in the Figure above by the edge labeled submit, spanning the two state machines. The REEF-AM passes a `RunningTask` object to the `Driver` after receiving confirmation of a successful `Task` start or resume. The `Driver` may use the `RunningTask` reference to close or suspend the execution, triggering a `CompletedTask` or `SuspendedTask` object reference to the `Driver`. The `SuspendedTask` object contains a memento used to resume the execution on some (possibly alternative) `ActiveContext`. Exceptions during the `Task` execution are surfaced to the `Driver` in the form of a `FailedTask`, which contains the actual exception object. ###Task Component -A `Task` encapsulates the task work of a job. The client interface contains a single synchronous call method that takes an optional memento argument and returns a byte array, which will be packaged with the `CompletedTask` object surfaced to the `Driver`. An exception may be thrown at any point during the call method; returning control back to the `Evaluator`, which packages the exception and sends it to the `Driver` where it is surfaced as a `FailedTask`. The `Evaluator` periodically performs a heartbeat with the REEF-AM to convey its status information. A `Task` can optionally implement a method interface that, when called, returns a (bounded) byte array, which the `Evaluator` includes in its heartbeat to the REEF-AM and surfaced to the `Driver`. \ No newline at end of file +A `Task` encapsulates the task work of a job. The client interface contains a single synchronous call method that takes an optional memento argument and returns a byte array, which will be packaged with the `CompletedTask` object surfaced to the `Driver`. An exception may be thrown at any point during the call method, returning control back to the `Evaluator`, which packages the exception and sends it to the `Driver` where it is surfaced as a `FailedTask`. The `Evaluator` periodically performs a heartbeat with the REEF-AM to convey its status information. A `Task` can optionally implement a method interface that, when called, returns a (bounded) byte array, which the `Evaluator` includes in its heartbeat to the REEF-AM and surfaced to the `Driver`. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/reef/blob/6627d701/website/src/site/markdown/mailing-list.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/mailing-list.md b/website/src/site/markdown/mailing-list.md index 60d16a2..d894730 100644 --- a/website/src/site/markdown/mailing-list.md +++ b/website/src/site/markdown/mailing-list.md @@ -21,14 +21,14 @@ under the License. - [REEF Dev](#reef-dev) - [Commits](#commits) -**Note:** These email addresses are automated. Just writing "Subscribe" or "Unsusbscribe" in both the subject line and the body will suffice and a confirmation email with further instructions will be sent as a reply to your email after some delay. +**Note:** These email addresses are automated. Just writing "Subscribe" or "Unsubscribe" in both the subject line and the body will suffice and a confirmation email with further instructions will be sent as a reply to your email after some delay. ###<a name="reef-dev"></a>REEF Dev Mailing List This gets traffic from both the JIRAs filed on REEF as well as the general development discussions - [Subscribe](mailto:[email protected]) ([email protected]) -- [Unsusbscribe](mailto:[email protected]) ([email protected]) +- [Unsubscribe](mailto:[email protected]) ([email protected]) - [Archive](http://mail-archives.apache.org/mod_mbox/reef-dev/) ###<a name="commits"></a>Commits Mailing List @@ -36,5 +36,5 @@ This gets traffic from both the JIRAs filed on REEF as well as the general devel This email list contains gets all commits made to REEF - [Subscribe](mailto:[email protected]) ([email protected]) -- [Unsusbscribe](mailto:[email protected]) ([email protected]) +- [Unsubscribe](mailto:[email protected]) ([email protected]) - [Archive](http://mail-archives.apache.org/mod_mbox/reef-commits/) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/reef/blob/6627d701/website/src/site/markdown/tang.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/tang.md b/website/src/site/markdown/tang.md index 5f96c1a..f72089a 100644 --- a/website/src/site/markdown/tang.md +++ b/website/src/site/markdown/tang.md @@ -18,9 +18,9 @@ under the License. --> #Tang -Tang is a configuration managment and checking framework that emphasizes explicit documentation and automatic checkability of configurations and applications instead of ad-hoc, application-specific configuration and bootstrapping logic. It supports distributed, multi-language applications, but gracefully handles simpler use cases as well. +Tang is a configuration management and checking framework that emphasizes explicit documentation and automatic checkability of configurations and applications instead of ad-hoc, application-specific configuration and bootstrapping logic. It supports distributed, multi-language applications, but gracefully handles simpler use cases as well. -Tang makes use of dependency injection to automatically instantiate applications. Dependency injectors can be thought of as "make for objects" -- given a request for some type of object, and information that explains how dependencies between objects should be resolved, dependency injectors automatically instantiate the requested object and all of the objects it dependes upon. Tang makes use of a few simple wire formats to support remote and even cross-language dependency injection. +Tang makes use of dependency injection to automatically instantiate applications. Dependency injectors can be thought of as "make for objects" -- given a request for some type of object, and information that explains how dependencies between objects should be resolved, dependency injectors automatically instantiate the requested object and all of the objects it depends upon. Tang makes use of a few simple wire formats to support remote and even cross-language dependency injection. Outline ------- @@ -48,7 +48,7 @@ Tang is our attempt to address these problems. It consists of a dependency inje Tang leverages existing language type systems, allowing unmodified IDEs such as Eclipse or NetBeans to surface configuration information in tooltips, provide auto-complete of configuration parameters, and to detect a wide range of configuration problems as you edit your code. Since such functionality is surfaced in the tools you are already familiar with, there is no need to install (or learn) additional development software to get started with Tang. Furthermore, we provide a set of sophisticated build time and runtime tools that detect a wide range of common architectural problems and configuration errors. -This documentation consists of tutorials that present prefered Tang design patterns. By structuring your application according to the patterns we suggest throughout the tutorials, you will allow our static analysis framework, Tint ("Tang Lint"), to detect problematic design patterns and high-level configuration problems as part of your build. These patterns provide the cornerstone for a number of more advanced features, such as interacting with legacy configuration systems, designing for cross-language applications, and multi-tenancy issues, such as secure injections of untrusted application code. To the best of our knowledge, implementing such tools and addressing these real-world implementation constraints would be difficult, or even impossible, atop competing frameworks. +This documentation consists of tutorials that present preferred Tang design patterns. By structuring your application according to the patterns we suggest throughout the tutorials, you will allow our static analysis framework, Tint ("Tang Lint"), to detect problematic design patterns and high-level configuration problems as part of your build. These patterns provide the cornerstone for a number of more advanced features, such as interacting with legacy configuration systems, designing for cross-language applications, and multi-tenancy issues, such as secure injections of untrusted application code. To the best of our knowledge, implementing such tools and addressing these real-world implementation constraints would be difficult, or even impossible, atop competing frameworks. <a name="design-principles"></a>Design principles ================= @@ -57,30 +57,30 @@ Tang encourages application developers to specify default implementations and co In the process of building complicated systems built atop Tang, we found that, as the length of configurations that are passed around at runtime increased, it rapidly became impossible to debug or maintain our higher-level applications. In an attempt to address this problem, traditional dependency injection systems actually compound this issue. They encourage the developers of each application-level component to implement hand-written "Modules" that are executed at runtime. Hand-written modules introspect on the current runtime configuration, augment and modify it, and then return a new configuration that takes the new application component into account. -In other systems, developers interact with modules by invoking ad-hoc builder methods, and passing configurations (in the correct order) from module to module. Modules frequently delgate to each other, either via inheritance or wrappers. This makes it difficult for developers and end-users to figure out which value of a given parameter will be used, or even to figure out why it was (or was not) set. +In other systems, developers interact with modules by invoking ad-hoc builder methods, and passing configurations (in the correct order) from module to module. Modules frequently delegate to each other, either via inheritance or wrappers. This makes it difficult for developers and end-users to figure out which value of a given parameter will be used, or even to figure out why it was (or was not) set. Tang provides an alternative called `ConfigurationModule`s: -- `Configurations` and `ConfigurationModules` are "just data," and can be read and written in human readable formats. +- `Configurations` and `ConfigurationModules` are "just data", and can be read and written in human readable formats. - Interfaces and configuration parameters are encouraged to specify defaults, significantly shortening the configurations generated at runtime, and making it easy to see what was "strange" about a given run of the application. - Tang's static analysis and documentation tools sanity check `ConfigurationModule`s, and document their behavior and any extra parameters they export. -- Configuration options can be set at most once. This avoids (or at least detects) situations in which users and application-level code inadvertantly "fight" over the setting of a particular option. +- Configuration options can be set at most once. This avoids (or at least detects) situations in which users and application-level code inadvertently "fight" over the setting of a particular option. -The last property comes from Tang's use of _monotonic_ set oriented primitives. This allows us to leverage recent theoretical results in commtative data types; particularly CRDTs, and the CALM theorem. Concretely: +The last property comes from Tang's use of _monotonic_ set oriented primitives. This allows us to leverage recent theoretical results in commutative data types, particularly CRDTs, and the CALM theorem. Concretely: - A large subset of Tang's public API is commutative, which frees application-level configuration and bootstrapping logic from worrying about the order in which configuration sources are processed at runtime. - Tang can detect configuration and injection problems much earlier than is possible with other approaches. Also, upon detecting a conflict, Tang lists the configuration sources that contributed to the problem. -Finally, Tang is divided into a set of "core" primtives, and higher-level configuration "formats". Tang's core focuses on dependency injection and static checking of configurations. The formats provide higher-level configuration languages primitives, such as distributed, cross-language injection, configuration files, and `ConfigurationModule`. Each Tang format imports and/or exports standard Tang `Configuration` objects, which can then be composed with other configuration data at runtime. +Finally, Tang is divided into a set of "core" primitives, and higher-level configuration "formats". Tang's core focuses on dependency injection and static checking of configurations. The formats provide higher-level configuration languages primitives, such as distributed, cross-language injection, configuration files, and `ConfigurationModule`. Each Tang format imports and/or exports standard Tang `Configuration` objects, which can then be composed with other configuration data at runtime. Improvements to these formats are planned, such as command-line tab completion, and improved APIs for extremely complex applications that are built by composing multiple Tang configurations to inject arbitrary object graphs. Furthermore, Tang formats include documentation facilities, and automatic command line and configuration file parsing. From an end-user perspective, this takes a lot of the guesswork out of configuration file formats. Although Tang surfaces a text-based interface for end-users of the applications built atop it, all configuration options and their types are specified in terms of Java classes and annotations. As with the core Tang primitives, this allows the Java compiler to statically check Tang formats for problems such as inconsistent usage of configuration parameters, naming conflicts and so on. This eliminates broad classes of runtime errors. These checks can be run independently of the application's runtime environment, and can find problems both in the Java-level implementation of the system, and with user-provided configuration files. The tools that perform these checks are designed to run as a post-processing step of projects built atop Tang. Like the Java compiler checks, this prevents such errors from making it to production environments. It also prevents such errors from being exposed to application logic or end-users, greatly simplifying applications built atop Tang. -Taken together, these properties greatly simplify dependency injection in distributed environments. We expect Tang to be used in environments that are dominated by "plugin"-style APIs with many alternative implementations. Tang cleanly separates concerns over configuration management, dependency injection and object implementations, which hides most of the complexity of dependency injection from plugin implementers. It also prevents plugin implementations from inadvertently conflicting with each other or their runtime environements. Such clean semantics are crucial in distributed, heterogeneous environments. +Taken together, these properties greatly simplify dependency injection in distributed environments. We expect Tang to be used in environments that are dominated by "plugin"-style APIs with many alternative implementations. Tang cleanly separates concerns over configuration management, dependency injection and object implementations, which hides most of the complexity of dependency injection from plugin implementers. It also prevents plugin implementations from inadvertently conflicting with each other or their runtime environments. Such clean semantics are crucial in distributed, heterogeneous environments. <a name="tutorial-getting-started"></a>Tutorial: Getting started ========================= @@ -179,7 +179,7 @@ Configuration modules allow applications to perform most configuration generatio In the example below, we extend the Timer API to include a second implementation that simply outputs the amount of time a real timer would have slept to stderr. In a real unit testing example, it would likely interact with a scheduler based on logical time. Of course, in isolation, having the ability to specify configuration parameters is not particularly useful; this example also adds a `main()` method that invokes Tang, and instantiates an object. -The process of instantiting an object with Tang is called _injection_. As with configurations, Tang's injection process is designed to catch as many potential runtime errors as possible before application code begins to run. This simplifies debugging and eliminates many types of runtime error handling code, since many configurations can be caught before running (or examining) application-specific initialization code. +The process of instantiating an object with Tang is called _injection_. As with configurations, Tang's injection process is designed to catch as many potential runtime errors as possible before application code begins to run. This simplifies debugging and eliminates many types of runtime error handling code, since many configurations can be caught before running (or examining) application-specific initialization code. package org.apache.reef.tang.examples.timer; @@ -265,7 +265,7 @@ Again, there are a few things going on here: - First, we push the implementation of `Timer` into a new class, `TimerImpl`. The `@DefaultImplementation` tells Tang to use `TimerImpl` when no other implementation is explicitly provided. - We leave the `Sleep` class in the Timer interface. This, plus the `@DefaultImplementation` annotation maintain backward compatibility with code that used Tang to inject the old `Timer` class. - The `TimerMock` class includes a dummy implementation of Timer, along with a `ConfigurationModule` final static field called `CONF`. - - The main method uses `CONF` to generate a configuration. Rather than set `Timer.Sleep` directly, it sets `MOCK_SLEEP_TIME`. In a more complicated example, this would allow `CONF` to route the sleep time to testing infrastructure, or other classes that are specific to the testing environment or implemenation of `TimerMock`. + - The main method uses `CONF` to generate a configuration. Rather than set `Timer.Sleep` directly, it sets `MOCK_SLEEP_TIME`. In a more complicated example, this would allow `CONF` to route the sleep time to testing infrastructure, or other classes that are specific to the testing environment or implementation of `TimerMock`. `ConfigurationModule`s serve a number of purposes: @@ -300,7 +300,7 @@ Here are some sample Tint errors. These (and others) can be run by passing `--t Named parameter org.apache.reef.tang.implementation.UnannotatedName is missing its @NamedParameter annotation. Field org.apache.reef.tang.formats.MyMissingBindConfigurationModule.BAD_CONF: Found declared options that were not used in binds: { FOO_NESS } -<a name="injnecting-objects-with-getInstance"></a>Injecting objects with getInstance() +<a name="injecting-objects-with-getinstance"></a>Injecting objects with getInstance() -------------------------------------- Above, we explain how to register constructors with Tang, and how to configure Tang to inject the desired objects at runtime. This section explains how Tang actually instantiates objects, and how the primitives it provides can be combined to support sophisticated application architectures. @@ -345,7 +345,7 @@ In order to inject an instance of `A`, Tang first injects an instance of `B` by Therefore, along with `forkInjector()` and `bindVolatile()`, this allows Tang to inject arbitrary graphs of objects. This pattern avoids non-final fields (once set, all fields of all objects are constant), and it also avoids boiler plate error handling code that checks to see if `B`'s instance of `A` has been set. -When `get()` is called after the application-level call to `getInstance()` returns, it is guranteed to return a non-null reference to an injected instance of the object. However, if `get()` is called _before_ the constructor it was passed to returns, then it is guaranteed to throw an exception. In between these two points in time, `get()`'s behavior is undefined, but, for the sake of race-detection and forward compatibility it makes a best-effort attempt to throw an exception. +When `get()` is called after the application-level call to `getInstance()` returns, it is guaranteed to return a non-null reference to an injected instance of the object. However, if `get()` is called _before_ the constructor it was passed to returns, then it is guaranteed to throw an exception. In between these two points in time, `get()`'s behavior is undefined, but, for the sake of race-detection and forward compatibility it makes a best-effort attempt to throw an exception. Following Tang's singleton semantics, the instance returned by `get()` will be the same instance the injector would pass into other constructors or return from `getInstance()`. @@ -356,7 +356,7 @@ Tang provides a number of so-called _formats_ that interface with external confi <a name="raw-configuration-api"></a>Raw configuration API --------- -Tang also provides a lower level configurtion API for applications that need more dynamic control over their configurations: +Tang also provides a lower level configuration API for applications that need more dynamic control over their configurations: ... import org.apache.reef.tang.Tang; @@ -389,8 +389,8 @@ Tang also provides a lower level configurtion API for applications that need mor The first step in using Tang is to get a handle to a Tang object by calling "Tang.Factory.getTang()". Having obtained a handle, we run through each of the phases of a Tang injection: - * We use `ConfigurationBuilder` objects to tell Tang about the class hierarchy that it will be using to inject objects and (in later examples) to register the contents of configuration files, override default configuration values, and to set default implementations of classes. `ConfigurationBuilder` and `ConfigurationModuleBuider` export similar API's. The difference is that `ConfigurationBuilder` produces `Configuration` objects directly, and is designed to be used at runtime. `ConfigurationModuleBuilder` is desgined to produce data structures that will be generated and analyzed during the build, and at class load time. - * `bindNamedParameter()` overrides the default value of Timer.Sleep, setting it to 5. Tang inteprets the 5 as a string, but allows instances of Number to be passed in as syntactic sugar. + * We use `ConfigurationBuilder` objects to tell Tang about the class hierarchy that it will be using to inject objects and (in later examples) to register the contents of configuration files, override default configuration values, and to set default implementations of classes. `ConfigurationBuilder` and `ConfigurationModuleBuilder` export similar API's. The difference is that `ConfigurationBuilder` produces `Configuration` objects directly, and is designed to be used at runtime. `ConfigurationModuleBuilder` is designed to produce data structures that will be generated and analyzed during the build, and at class load time. + * `bindNamedParameter()` overrides the default value of Timer.Sleep, setting it to 5. Tang interprets the 5 as a string, but allows instances of Number to be passed in as syntactic sugar. * We call `.build()` on the `ConfigurationBuilder`, creating an immutable `Configuration` object. At this point, Tang ensures that all of the classes it has encountered so far are consistent with each other, and that they are suitable for injection. When Tang encounters conflicting classes or configuration files, it throws a `BindException` to indicate that the problem is due to configuration issues. Note that `ConfigurationBuilder` and `Configuration` do not determine whether or not a particular injection will succeed; that is the business of the _Injector_. * To obtain an instance of Injector, we pass our Configuration object into `tang.newInjector()`. * `injector.isInjectable(Timer.class)` checks to see if Timer is injectable without actually performing an injection or running application code. (Note that, in this example, the Java classloader may have run application code. For more information, see the advanced tutorials on cross-language injections and securely building configurations for untrusted code.) @@ -402,7 +402,7 @@ The second type of configuration option, _implementation bindings_, are used to New parameters are created and passed into constructors as in the examples above, by creating implementations of `Name<T>`, and adding `@NamedParameter`, `@Parameter` and `@Inject` annotations as necessary. Specifying implementations for interfaces is a bit more involved, as a number of subtle use cases arise. -However, all configuration settings in Tang can be unambiguously represented as a `key=value` pair that can be interpreted either asan `interface=implementation` pair or a `configuration_parameter=value` pair. This maps well to Java-style properties files. For example: +However, all configuration settings in Tang can be unambiguously represented as a `key=value` pair that can be interpreted either as an `interface=implementation` pair or a `configuration_parameter=value` pair. This maps well to Java-style properties files. For example: com.examples.Interface=com.examples.Implementation @@ -419,7 +419,7 @@ See the `ConfigurationFile` API for more information about processing configurat ###<a name="injectionPlan"></a>InjectionPlan InjectionPlan objects explain what Tang would do to instantiate a new object, but don't actually instantiate anything. -Add the following lines to the Timer example; +Add the following lines to the Timer example: import org.apache.reef.tang.implementation.InjectionPlan; import org.apache.reef.tang.implementation.InjectorImpl; http://git-wip-us.apache.org/repos/asf/reef/blob/6627d701/website/src/site/markdown/wake.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/wake.md b/website/src/site/markdown/wake.md index fa01eee..a0d8ed3 100644 --- a/website/src/site/markdown/wake.md +++ b/website/src/site/markdown/wake.md @@ -26,7 +26,7 @@ Background ---------- Wake applications consist of asynchronous *event handlers* that run inside of *stages*. Stages provide scheduling primitives such as thread pool sizing and performance isolation between event handlers. In addition to event handler and stage APIs, Wake includes profiling tools and a rich standard library of primitives for system builders. -Event driven processing frameworks improve upon the performance of threaded architectures in two ways: (1) Event handlers often have lower memory and context switching overhead than threaded solutions, and (2) event driven systems allow applications to allocate and monitor computational and I/O resources in an extremely fine-grained fashion. Modern threading packages have done much to address the first concern, and have significantly lowered concurrency control and other implementation overheads in recent years. However, fine grained resource allocation remains a challenge in threaded systems, and is Wake's primary advantage over threading. +Event driven processing frameworks improve upon the performance of threaded architectures in two ways: (1) event handlers often have lower memory and context switching overhead than threaded solutions, and (2) event driven systems allow applications to allocate and monitor computational and I/O resources in an extremely fine-grained fashion. Modern threading packages have done much to address the first concern, and have significantly lowered concurrency control and other implementation overheads in recent years. However, fine-grained resource allocation remains a challenge in threaded systems, and is Wake's primary advantage over threading. Early event driven systems such as SEDA executed each event handler in a dedicated thread pool called a stage. This isolated low-latency event handlers (such as cache lookups) from expensive high-latency operations, such as disk I/O. With a single thread pool, high-latency I/O operations can easily monopolize the thread pool, causing all of the CPUs to block on disk I/O, even when there is computation to be scheduled. With separate thread pools, the operating system schedules I/O requests and computation separately, guaranteeing that runnable computations will not block on I/O requests. @@ -66,7 +66,7 @@ The latter approach can be implemented by registering separate event handlers fo void onCompleted(); } -The `Observer` is designed for stateful event handlers that need to be explicitly torn down at exit, or when errors occor. Such event handlers may maintain open network sockets, write to disk, buffer output, and so on. As with `onNext()`, neither `onError()` nor `onCompleted()` throw exceptions. Instead, callers should assume that they are asynchronously invoked. +The `Observer` is designed for stateful event handlers that need to be explicitly torn down at exit, or when errors occur. Such event handlers may maintain open network sockets, write to disk, buffer output, and so on. As with `onNext()`, neither `onError()` nor `onCompleted()` throw exceptions. Instead, callers should assume that they are asynchronously invoked. `EventHandler` and `Observer` implementations should be threadsafe and handle concurrent invocations of `onNext()`. However, it is illegal to call `onCompleted()` or `onError()` in race with any calls to `onNext()`, and the call to `onCompleted()` or `onError()` must be the last call made to the object. Therefore, implementations of `onCompleted()` and `onError()` can assume they have a lock on `this`, and that `this` has not been torn down and is still in a valid state. @@ -86,7 +86,7 @@ or they can contain `Observable`s, as [RxStage](https://github.com/apache/reef/b public interface RxStage<T> extends Observer<T>, Stage { } -In both cases, the stage simply exposes the same API as the event handler that it manages. This allows code that produces events to treat downstream stages and raw `EventHandlers` / `Observers` interchangebly. Recall that Wake implements thread sharing by allowing EventHandlers and Observers to directly invoke each other. Since Stages implement the same interface as raw EventHandlers and Observers, this pushes the placement of thread boundaries and other scheduling tradeoffs to the code that is instantiating the application. In turn, this simplifies testing and improves the reusability of code written on top of Wake. +In both cases, the stage simply exposes the same API as the event handler that it manages. This allows code that produces events to treat downstream stages and raw `EventHandlers` / `Observers` interchangeably. Recall that Wake implements thread sharing by allowing EventHandlers and Observers to directly invoke each other. Since Stages implement the same interface as raw EventHandlers and Observers, this pushes the placement of thread boundaries and other scheduling tradeoffs to the code that is instantiating the application. In turn, this simplifies testing and improves the reusability of code written on top of Wake. #### close() vs. onCompleted() @@ -96,7 +96,7 @@ In contrast, `close()` is synchronous, and is not allowed to return until all qu `Observer` implementations do not expose a `close()` method, and generally do not invoke `close()`. Instead, when `onCompleted()` is invoked, it should arrange for `onCompleted()` to be called on any `Observer` instances that `this` directly invokes, free any resources it is holding, and then return. Since the downstream `onCompleted()` calls are potentially asynchronous, it cannot assume that downstream cleanup completes before it returns. -In a thread pool `Stage`, the final `close()` call will block until there are no more outstanding events queued in the stage. Once `close()` has been called (and returns) on each stage, no events are left in any queues, and no `Observer` or `EventHandler` objects are holding resources or scheduled on any cores, so shutdown is compelete. +In a thread pool `Stage`, the final `close()` call will block until there are no more outstanding events queued in the stage. Once `close()` has been called (and returns) on each stage, no events are left in any queues, and no `Observer` or `EventHandler` objects are holding resources or scheduled on any cores, so shutdown is complete. Helper libraries ----------------
