Author: buildbot
Date: Mon Nov 2 17:24:47 2015
New Revision: 971096
Log:
Staging update by buildbot for slider
Modified:
websites/staging/slider/trunk/content/ (props changed)
websites/staging/slider/trunk/content/design/architecture.html
websites/staging/slider/trunk/content/design/index.html
websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
websites/staging/slider/trunk/content/design/registry/index.html
websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
websites/staging/slider/trunk/content/design/registry/references.html
websites/staging/slider/trunk/content/design/registry/registry-model.html
websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
websites/staging/slider/trunk/content/design/rolehistory.html
websites/staging/slider/trunk/content/design/specification/cli-actions.html
websites/staging/slider/trunk/content/design/specification/index.html
websites/staging/slider/trunk/content/design/specification/slider-model.html
websites/staging/slider/trunk/content/design/ssl_implementation.html
websites/staging/slider/trunk/content/developing/building.html
websites/staging/slider/trunk/content/developing/chaosmonkey.html
websites/staging/slider/trunk/content/developing/debugging.html
websites/staging/slider/trunk/content/developing/demo.html
websites/staging/slider/trunk/content/developing/functional_tests.html
websites/staging/slider/trunk/content/developing/index.html
websites/staging/slider/trunk/content/developing/manual_testing.html
websites/staging/slider/trunk/content/developing/releasing.html
websites/staging/slider/trunk/content/developing/releasing_process_v1.html
websites/staging/slider/trunk/content/developing/style.html
websites/staging/slider/trunk/content/developing/submitting_patches.html
websites/staging/slider/trunk/content/developing/testing.html
websites/staging/slider/trunk/content/developing/windows.html
websites/staging/slider/trunk/content/docs/api/index.html
websites/staging/slider/trunk/content/docs/api/slider_REST_api_v2.html
websites/staging/slider/trunk/content/docs/api/slider_REST_v1.html
websites/staging/slider/trunk/content/docs/client-configuration.html
websites/staging/slider/trunk/content/docs/configuration/appconfig.html.html
websites/staging/slider/trunk/content/docs/configuration/core.html
websites/staging/slider/trunk/content/docs/configuration/index.html
websites/staging/slider/trunk/content/docs/configuration/internal.html
websites/staging/slider/trunk/content/docs/configuration/resources.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/index.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/redesign.html
websites/staging/slider/trunk/content/docs/configuration/revision-1/specification.html
websites/staging/slider/trunk/content/docs/examples.html
websites/staging/slider/trunk/content/docs/exitcodes.html
websites/staging/slider/trunk/content/docs/getting_started.html
websites/staging/slider/trunk/content/docs/high_availability.html
websites/staging/slider/trunk/content/docs/index.html
websites/staging/slider/trunk/content/docs/manpage.html
websites/staging/slider/trunk/content/docs/security.html
websites/staging/slider/trunk/content/docs/slider_specs/application_configuration.html
websites/staging/slider/trunk/content/docs/slider_specs/application_definition.html
websites/staging/slider/trunk/content/docs/slider_specs/application_instance_configuration.html
websites/staging/slider/trunk/content/docs/slider_specs/application_needs.html
websites/staging/slider/trunk/content/docs/slider_specs/application_package.html
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_docker.html
websites/staging/slider/trunk/content/docs/slider_specs/application_pkg_upgrade.html
websites/staging/slider/trunk/content/docs/slider_specs/canonical_scenarios.html
websites/staging/slider/trunk/content/docs/slider_specs/creating_app_definitions.html
websites/staging/slider/trunk/content/docs/slider_specs/hello_world_slider_app.html
websites/staging/slider/trunk/content/docs/slider_specs/index.html
websites/staging/slider/trunk/content/docs/slider_specs/simple_pkg.html
websites/staging/slider/trunk/content/docs/slider_specs/specifying_exports.html
websites/staging/slider/trunk/content/docs/slider_specs/writing_app_command_scripts.html
websites/staging/slider/trunk/content/docs/ssl.html
websites/staging/slider/trunk/content/docs/troubleshooting.html
websites/staging/slider/trunk/content/downloads/index.html
websites/staging/slider/trunk/content/index.html
websites/staging/slider/trunk/content/mailing_lists.html
websites/staging/slider/trunk/content/release_notes/release-0.30.0.html
websites/staging/slider/trunk/content/release_notes/release-0.40.0.html
websites/staging/slider/trunk/content/release_notes/release-0.50.2.html
websites/staging/slider/trunk/content/release_notes/release-0.60.0.html
websites/staging/slider/trunk/content/release_notes/release-0.61.0.html
websites/staging/slider/trunk/content/release_notes/release-0.70.1.html
websites/staging/slider/trunk/content/release_notes/release-0.80.0.html
websites/staging/slider/trunk/content/team.html
Propchange: websites/staging/slider/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Mon Nov 2 17:24:47 2015
@@ -1 +1 @@
-1708722
+1712094
Modified: websites/staging/slider/trunk/content/design/architecture.html
==============================================================================
--- websites/staging/slider/trunk/content/design/architecture.html (original)
+++ websites/staging/slider/trunk/content/design/architecture.html Mon Nov 2
17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-architecture">Apache Slider Architecture</h1>
-<h2 id="summary">Summary</h2>
+<h1 id="apache-slider-architecture">Apache Slider Architecture<a
class="headerlink" href="#apache-slider-architecture" title="Permanent
link">¶</a></h1>
+<h2 id="summary">Summary<a class="headerlink" href="#summary" title="Permanent
link">¶</a></h2>
<p>Slider is a YARN application to deploy non-YARN-enabled applications in a
YARN cluster</p>
<p>Slider consists of a YARN application master, the "Slider AM", and a client
application which communicates with YARN and the Slider AM via remote procedure
@@ -204,10 +215,10 @@ options and parameters.</p>
for each component; It then monitors the state of the application instance,
receiving messages
from YARN when a remotely executed process finishes. It then deploys another
instance of
that component.</p>
-<h2 id="slider-packaging">Slider Packaging</h2>
+<h2 id="slider-packaging">Slider Packaging<a class="headerlink"
href="#slider-packaging" title="Permanent link">¶</a></h2>
<p>A key goal of Slider is to support the deployment of existing applications
into
a YARN application instance, without having to extend Slider itself. </p>
-<h2 id="am-architecture">AM Architecture</h2>
+<h2 id="am-architecture">AM Architecture<a class="headerlink"
href="#am-architecture" title="Permanent link">¶</a></h2>
<p>The application master consists of</p>
<ol>
<li>The AM engine which handles all integration with external services,
specifically YARN and any Slider clients</li>
@@ -231,14 +242,14 @@ actual configuration and execution of th
<p>To summarize: Slider is not an classic YARN analysis application, which
allocates and schedules work across the cluster in short-to-medium life
containers with the lifespan of a query or an analytics session, but instead
for an application with a lifespan of days to months. Slider works to keep the
actual state of its application cluster to match the desired state, while the
application has the tasks of recovering from node failure, locating peer nodes
and working with data in an HDFS filesystem. </p>
<p>As such it is one of the first applications designed to use YARN as a
platform for long-lived services -Samza being the other key example. These
application's needs of YARN are different, and their application manager
design is focused around maintaining the distributed application in its desired
state rather than the ongoing progress of submitted work.</p>
<p>The clean model-view-controller split was implemented to isolate the model
and aid mock testing of large clusters with simulated scale, and hence increase
confidence that Slider can scale to work in large YARN clusters and with larger
application instances. </p>
-<h3 id="failure-model">Failure Model</h3>
+<h3 id="failure-model">Failure Model<a class="headerlink"
href="#failure-model" title="Permanent link">¶</a></h3>
<p>The application master is designed to be a <a
href="https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf";>crash-only
application</a>, clients are free to terminate
the application instance by asking YARN directly. </p>
<p>There is an RPC call to stop the application instance - this is a nicety
which includes a message in the termination log, and
could, in future, perhaps warn the provider that the application instance is
being torn down. That is a potentially dangerous feature
to add -as provider implementors may start to expect the method to be called
reliably. Slider is designed to fail without
warning, to rebuild its state on a YARN-initiated restart, and to be manually
terminated without any advance notice.</p>
-<h3 id="rpc-interface">RPC Interface</h3>
+<h3 id="rpc-interface">RPC Interface<a class="headerlink"
href="#rpc-interface" title="Permanent link">¶</a></h3>
<p>The RPC interface allows the client to query the current application state,
and to update it by pushing out a new JSON specification. </p>
<p>The core operations are</p>
<ul>
@@ -251,9 +262,9 @@ warning, to rebuild its state on a YARN-
it triggers a re-evaluation of the application state, building up the list of
container add and release requests to make of
the YARN resource manager.</p>
<p>The final operation, <code>stopCluster()</code>, stops the application
instance. </p>
-<h3 id="security-and-identity">Security and Identity</h3>
+<h3 id="security-and-identity">Security and Identity<a class="headerlink"
href="#security-and-identity" title="Permanent link">¶</a></h3>
<p>Slider's security model is described in detail in <a
href="/docs/security.html">an accompanying document</a></p>
-<h3 id="agent-to-application-master-secure-communication">Agent to Application
Master Secure Communication</h3>
+<h3 id="agent-to-application-master-secure-communication">Agent to Application
Master Secure Communication<a class="headerlink"
href="#agent-to-application-master-secure-communication" title="Permanent
link">¶</a></h3>
<p>By default, one-way SSL is leveraged to secure the communication between
Slider agents and the Application Master. However, two-way SSL can be enabled.
A more detailed discussion of the SSL implementation in Slider can be found <a
href="/design/ssl_implementation.html">in the SSL documentation</a>.</p>
</div>
Modified: websites/staging/slider/trunk/content/design/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/index.html (original)
+++ websites/staging/slider/trunk/content/design/index.html Mon Nov 2 17:24:47
2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-architecture">Apache Slider Architecture</h1>
+<h1 id="apache-slider-architecture">Apache Slider Architecture<a
class="headerlink" href="#apache-slider-architecture" title="Permanent
link">¶</a></h1>
<ul>
<li><a href="/design/architecture.html">Architecture</a></li>
<li><a href="/docs/slider_specs/application_needs.html">Application
Needs</a></li>
Modified:
websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
==============================================================================
---
websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
(original)
+++
websites/staging/slider/trunk/content/design/registry/a_YARN_service_registry.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="a-yarn-service-registry-for-apache-slider">A YARN Service Registry for
Apache Slider</h1>
-<h2 id="april-2014">April 2014</h2>
-<h1 id="introduction">Introduction</h1>
+<h1 id="a-yarn-service-registry-for-apache-slider">A YARN Service Registry for
Apache Slider<a class="headerlink"
href="#a-yarn-service-registry-for-apache-slider" title="Permanent
link">¶</a></h1>
+<h2 id="april-2014">April 2014<a class="headerlink" href="#april-2014"
title="Permanent link">¶</a></h2>
+<h1 id="introduction">Introduction<a class="headerlink" href="#introduction"
title="Permanent link">¶</a></h1>
<p>This document looks at the needs and options of a service registry.</p>
<p>The core issue is that as the location(s) of a dynamically deployed
application are unknown, the standard Hadoop and Java configuration model of
some form of text files containing hostnames, ports and URLS no longer works.
You cannot define up-front where a service will be.</p>
<p>Some Hadoop applications -HBase and Accumulo -have solved this with custom
ZK bindings. This works for the specific clients, but requires hbase and
accumulo client JARs in order to be able to work with the content. (or a
re-implementation with knowledge of the non-standard contents of the ZK
nodes)</p>
@@ -195,10 +206,10 @@ Latest release: <strong>0.80.0-incubatin
<p>YARN provides some minimal publishing of AM remote endpoints: a URL to what
is assumed to be a Web UI (not a REST API), and an IPC port. The URL is
displayed in the YARN UI -in which case it is accessed via a proxy which
(currently) only support HTTP GET operations. The YARN API call to list all
applications can be used to locate a named instance of an application by (user,
application-type, name), and then obtain the raw URL and IPC endpoints. This
enumeration process is an O(apps) operation on the YARN RM and only provides
access to those two endpoints. Even with the RAW URL, REST operations have
proven "troublesome", due to a web filter which redirects all direct requests
to the proxy -unless it comes from the same host as the proxy.</p>
<p>Hadoop client applications tend to retrieve all their configuration
information from files in the local filesystem, hadoop-site.xml, hdfs-site.xml,
hbase-site.xml, etc. This requires the configuration files to be present on all
systems. Tools such as Ambari can keep the files in the server up to date
-assuming a low rate of change- ---but these tools do nothing for the client
applications themselves. It is up to the cluster clients to (somehow) retrieve
these files, and to keep their copies up to date. <em>This is a problem that
exists with today's applications</em>. </p>
<p>As an example, if a YARN client does not know the value of
"yarn.application.classpath", it cannot successfully deploy any application in
the YARN cluster which needs the cluster-side Hadoop and YARN JARs on its
application master's classpath. This is not a theoretical problem, as some
clusters have a different classpath from the default: without a correct value
the Slider AM does not start. And, as it is designed to be run remotely, it
cannot rely on a local installation of YARN to provide the correct values (<a
href="https://issues.apache.org/jira/browse/YARN-973";>YARN-973</a>).</p>
-<h1 id="what-do-we-need">What do we need?</h1>
+<h1 id="what-do-we-need">What do we need?<a class="headerlink"
href="#what-do-we-need" title="Permanent link">¶</a></h1>
<p><strong>Discovery</strong>: An IPC and URL discovery system for
service-aware applications to use to look up a service to which it wishes to
talk to. This is not an ORB -it's not doing redirection -, but it is something
that needs to be used before starting IPC or REST communications. </p>
<p><strong>Configuration</strong>: A way for clients of a service to retrieve
more configuration data than simply the service endpoints. For example:
everything needed to create a site.xml document.</p>
-<h2 id="client-side">Client-side</h2>
+<h2 id="client-side">Client-side<a class="headerlink" href="#client-side"
title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Allow clients of a YARN application to locate the service instance and its
service ports (web, IPC, REST...) efficiently even on a large YARN cluster. </p>
@@ -216,7 +227,7 @@ Latest release: <strong>0.80.0-incubatin
<p>cross-language support.</p>
</li>
</ul>
-<h2 id="for-all-services">For all Services</h2>
+<h2 id="for-all-services">For all Services<a class="headerlink"
href="#for-all-services" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Allow services to publish their binding details for the AM and of code
running in the containers (which may be published by the containers)</p>
@@ -234,7 +245,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Services to be able to act as clients to other services</p>
</li>
</ul>
-<h2 id="for-slider-services-and-presumably-others">For Slider Services (and
presumably others)</h2>
+<h2 id="for-slider-services-and-presumably-others">For Slider Services (and
presumably others)<a class="headerlink"
href="#for-slider-services-and-presumably-others" title="Permanent
link">¶</a></h2>
<ul>
<li>
<p>Ability to publish information about configuration documents that can be
retrieved -and URLs</p>
@@ -246,7 +257,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Ability to use service paths as a way to ensure a single instance of a
named service can be deployed by a user</p>
</li>
</ul>
-<h2 id="management-and-paas-uis">Management and PaaS UIs</h2>
+<h2 id="management-and-paas-uis">Management and PaaS UIs<a class="headerlink"
href="#management-and-paas-uis" title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Retrieve lists of web UI URLs of AM and of deployed components</p>
@@ -267,7 +278,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Listen to changes in the service mix -the arrival and departure of service
instances, as well as changes in their contents.</p>
</li>
</ul>
-<h2 id="other-needs">Other Needs</h2>
+<h2 id="other-needs">Other Needs<a class="headerlink" href="#other-needs"
title="Permanent link">¶</a></h2>
<ul>
<li>
<p>Registry-configured applications. In-cluster applications should be able to
subscribe to part of the registry
@@ -281,9 +292,9 @@ applications on which they depend themse
<p>Cloud-based deployments: work on virtual infrastructures where hostnames
are unpredictable.</p>
</li>
</ul>
-<h1 id="open-source-registry-code">Open Source Registry code</h1>
+<h1 id="open-source-registry-code">Open Source Registry code<a
class="headerlink" href="#open-source-registry-code" title="Permanent
link">¶</a></h1>
<p>What can we use to implement this from ASF and ASF-compatible code? </p>
-<h2 id="zookeeper">Zookeeper</h2>
+<h2 id="zookeeper">Zookeeper<a class="headerlink" href="#zookeeper"
title="Permanent link">¶</a></h2>
<p>We'd need a good reason not to use this. There are still some issues</p>
<ol>
<li>
@@ -299,7 +310,7 @@ applications on which they depend themse
<p>Impact of other ZK load on the behaviour of the service registry -will it
cause problems if overloaded -and are they recoverable?</p>
</li>
</ol>
-<h2 id="apache-curator">Apache Curator</h2>
+<h2 id="apache-curator">Apache Curator<a class="headerlink"
href="#apache-curator" title="Permanent link">¶</a></h2>
<p>Netflix's core curator -now <a href="http://curator.apache.org/";>Apache
Curator</a>- framework adds a lot to make working with ZK easier, including
pluggable retry policies, binding tools and other things.</p>
<p>There is also its "experimental" <a
href="http://curator.apache.org/curator-x-discovery-server/index.html";>service
discovery framework</a>, which</p>
<ol>
@@ -350,7 +361,7 @@ Jackson (1.8.8) in Hadoop 2.4. The upgra
}</p>
</li>
</ul>
-<h2 id="helix-service-registry"><a
href="http://helix.apache.org/0.7.0-incubating-docs/recipes/service_discovery.html";>Helix
Service Registry</a></h2>
+<h2 id="helix-service-registry"><a
href="http://helix.apache.org/0.7.0-incubating-docs/recipes/service_discovery.html";>Helix
Service Registry</a><a class="headerlink" href="#helix-service-registry"
title="Permanent link">¶</a></h2>
<p>This is inside Helix somewhere, used in LI in production at scale -and
worth looking at. LI separate their Helix Zookeeper Quorum from their
application-layer quorum, to isolate load.</p>
<p>Notable features</p>
<ol>
@@ -358,24 +369,24 @@ Jackson (1.8.8) in Hadoop 2.4. The upgra
<li>The deployed application can pick up changes to its state the same way,
allowing for live application manipulation.</li>
<li>Tracks nodes that continually join/leave the group and drops them as
unreliable.</li>
</ol>
-<h2 id="twill-service-registry">Twill Service Registry</h2>
+<h2 id="twill-service-registry">Twill Service Registry<a class="headerlink"
href="#twill-service-registry" title="Permanent link">¶</a></h2>
<p>Twill's <a
href="http://twill.incubator.apache.org/apidocs/index.html";>service registry
code</a>, lets applications register a <a
href="http://twill.incubator.apache.org/apidocs/org/apache/twill/discovery/Discoverable.html";>(hostname,
port)</a> pair in the registry by a name, a name by which clients can look up
and enumerate all services with a specific name.</p>
<p>Clients can subscribe to changes in the list of services with a specific
name -so picking up the arrival and departure of instances, and probe to see if
a previously discovered entity is still registered.</p>
<p>Zookeeper- and in-memory registry implementations are provided.</p>
<p>One nice feature about this architecture -and Twill in general- is that its
general single-method callback model means that it segues nicely into Java-8
lambda-expressions. This is something to retain in a YARN-wide service
registry.</p>
<p>Comparing it to curator, it offers a proper subset of curator's registered
services <a
href="http://curator.apache.org/apidocs/org/apache/curator/x/discovery/ServiceInstance.html";>ServiceInstance</a>
-implying that you could publish and retrieve Curator-registered services via
a new implementation of Twill's DiscoveryService. This would require extensions
to the curator service discovery client allow ZK nodes to be watched for
changes. This is a feature that would be useful in many use cases -such as
watching service availability across a cluster, or simply blocking until a
dependent service was launched.</p>
<p>As with curator, the amount of information that can be published isn't
enough for management tools to make effective use of the service registration,
while for slider there's no way to publish configuration data. However a YARN
registry will inevitably be a superset of the Twill client's enumerated and
retrieved data -so if its registration API were sufficient to register a
minimal service, supporting the YARN registry via Twill's existing API should
be straightforward.</p>
-<h2 id="twitter-commons-service-registration">Twitter Commons Service
Registration</h2>
+<h2 id="twitter-commons-service-registration">Twitter Commons Service
Registration<a class="headerlink" href="#twitter-commons-service-registration"
title="Permanent link">¶</a></h2>
<p><a href="https://github.com/twitter/commons";>Twitter Commons</a> has a
service registration library, which allows for registration of sets of servers,
<a
href="http://twitter.github.io/commons/apidocs/com/twitter/common/service/registration/package-tree.html";>publishing
the hostname and port of each</a>., along with a map of string properties.</p>
<p>Zookeeper based, it suffices if all servers are identical and only
publishing single (hostname, port) pairs for callers.</p>
-<h2 id="airbnb-smartstack">AirBnB Smartstack</h2>
+<h2 id="airbnb-smartstack">AirBnB Smartstack<a class="headerlink"
href="#airbnb-smartstack" title="Permanent link">¶</a></h2>
<p>SmartStack is <a
href="http://nerds.airbnb.com/smartstack-service-discovery-cloud/";>Air BnB's
cloud-based service discovery system</a>.</p>
<p>It has two parts, <em>Nerve</em> and <em>Synapse</em>:</p>
<p><a href="https://github.com/airbnb/nerve";><strong>Nerve</strong></a> is a
ruby agent designed to monitor processes and register healthy instances in ZK
(or to a mock reporter). It includes <a
href="https://github.com/airbnb/nerve/tree/master/lib/nerve/service_watcher";>probes
for TCP ports, HTTP and rabbitMQ</a>. It's <a
href="https://github.com/airbnb/nerve/blob/master/lib/nerve/service_watcher.rb";>a
fairly simple liveness monitor</a>.</p>
<p><a href="https://github.com/airbnb/synapse";><strong>Synapse</strong></a>
takes the data and uses it to configure <a
href="http://haproxy.1wt.eu/";>HAProxy instances</a>. HAProxy handles the load
balancing, queuing and integrating liveness probes into the queues. Synapse
generates all the configuration files for an instance -but also tries to
reconfigure the live instances via their socket APIs, </p>
<p>Alongside these, AirBnB have another published project on Github, <a
href="https://github.com/airbnb/optica";>Optica</a>, which is a web application
for nodes to register themselves with (POST) and for others to query. It
publishes events to RabbitMQ, and again uses ZK to store state.</p>
<p>AirBnB do complain a bit about ZK and its brittleness. They do mention that
they suspect it is due to bugs in the Ruby ZK client library. This may be
exacerbated by in-cloud deployments. Hard-coding the list of ZK nodes may work
for a physical cluster, but in a virtualized cluster, the hostnames/IP
Addresses of those nodes may change -leading to a meta-discovery problem: how
to find the ZK quorum -especially if you can't control the DNS servers.</p>
-<h2 id="apache-directory"><a
href="http://directory.apache.org/apacheds/";>Apache Directory</a></h2>
+<h2 id="apache-directory"><a
href="http://directory.apache.org/apacheds/";>Apache Directory</a><a
class="headerlink" href="#apache-directory" title="Permanent
link">¶</a></h2>
<p>This is an embeddable LDAP server</p>
<ul>
<li>
Modified: websites/staging/slider/trunk/content/design/registry/index.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/index.html (original)
+++ websites/staging/slider/trunk/content/design/registry/index.html Mon Nov 2
17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry">Apache Slider Service Registry</h1>
-<h2 id="important-this-document-is-obsolete">Important: This document is
obsolete</h2>
+<h1 id="apache-slider-service-registry">Apache Slider Service Registry<a
class="headerlink" href="#apache-slider-service-registry" title="Permanent
link">¶</a></h1>
+<h2 id="important-this-document-is-obsolete">Important: This document is
obsolete<a class="headerlink" href="#important-this-document-is-obsolete"
title="Permanent link">¶</a></h2>
<p>This secion has been supplanted by the YARN Service Registry; a
cross-application
service registry built into Apache Hadoop, with special integration with
the YARN Resource Manager, security in a secure cluster, and other
features.</p>
@@ -194,7 +205,7 @@ the YARN Resource Manager, security in a
so we are confident that it works for Slider as well as other applications.
Indeed, for the Hadoop 2.6 release, Slider was the functional test suite for
the registry.</p>
-<h2 id="introduction">Introduction</h2>
+<h2 id="introduction">Introduction<a class="headerlink" href="#introduction"
title="Permanent link">¶</a></h2>
<p>The service registry is a core part of the Slider Architecture -it is how
dynamically generated configurations are published for clients to pick up.</p>
<p>The need for a service registry goes beyond Slider, however. We effectively
@@ -209,7 +220,7 @@ to be on all YARN clusters, so code coul
including HBase, Accumulo, Oozie, -applications deployed by management tools.
Unless/until these applications self-published their binding data, it would
be the duty of the management tools to do the registration.</p>
-<h2 id="contents">Contents</h2>
+<h2 id="contents">Contents<a class="headerlink" href="#contents"
title="Permanent link">¶</a></h2>
<ol>
<li><a href="the_YARN_application_registration_and_binding_problem.html">YARN
Application Registration and Binding: the Problem</a></li>
<li><a href="a_YARN_service_registry.html">A YARN Service Registry</a></li>
Modified:
websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
==============================================================================
---
websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
(original)
+++
websites/staging/slider/trunk/content/design/registry/initial_registry_design.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="april-2014-initial-registry-design-for-apache-slider">April 2014
Initial Registry Design for Apache Slider</h1>
+<h1 id="april-2014-initial-registry-design-for-apache-slider">April 2014
Initial Registry Design for Apache Slider<a class="headerlink"
href="#april-2014-initial-registry-design-for-apache-slider" title="Permanent
link">¶</a></h1>
<p>This is the plan for the initial registry design.</p>
<ol>
<li>
@@ -213,7 +224,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM to serve up curator service (later -host in RM? elsewhere?)</p>
</li>
</ol>
-<h3 id="forwards-compatilibity">forwards-compatilibity</h3>
+<h3 id="forwards-compatilibity">forwards-compatilibity<a class="headerlink"
href="#forwards-compatilibity" title="Permanent link">¶</a></h3>
<ol>
<li>
<p>This initial design will hide the fact that Apache Curator is being used to
discover services,
@@ -225,7 +236,7 @@ can use the same datatype -or map to it-
<p>The first implementation will not support watching for changes.</p>
</li>
</ol>
-<h3 id="initial-templates">Initial templates</h3>
+<h3 id="initial-templates">Initial templates<a class="headerlink"
href="#initial-templates" title="Permanent link">¶</a></h3>
<ul>
<li>
<p>hadoop XML conf files</p>
@@ -237,7 +248,7 @@ can use the same datatype -or map to it-
<p>HTML listing of services</p>
</li>
</ul>
-<h2 id="example-curator-service-entry">Example Curator Service Entry</h2>
+<h2 id="example-curator-service-entry">Example Curator Service Entry<a
class="headerlink" href="#example-curator-service-entry" title="Permanent
link">¶</a></h2>
<p>This is the prototype's content</p>
<p>Toplevel</p>
<div class="codehilite"><pre><span class="n">service</span> <span
class="n">CuratorServiceInstance</span><span class="p">{</span><span
class="n">name</span><span class="p">=</span><span
class="s">'slider'</span><span class="p">,</span> <span
class="n">id</span><span class="p">=</span><span
class="s">'stevel.test_registry_am'</span><span class="p">,</span>
<span class="n">address</span><span class="p">=</span><span
class="s">'192.168.1.101'</span><span class="p">,</span> <span
class="n">port</span><span class="p">=</span>62552<span class="p">,</span>
<span class="n">sslPort</span><span class="p">=</span><span
class="n">null</span><span class="p">,</span> <span
class="n">payload</span><span class="p">=</span><span class="n">org</span><span
class="p">.</span><span class="n">apache</span><span class="p">.</span><span
class="n">slider</span><span class="p">.</span><span class="n">core</span><span
class="p">.</span><span class="n">registry</span><span class="p">.</
span><span class="n">info</span><span class="p">.</span><span
class="n">ServiceInstanceData</span><span class="p">@</span>4<span
class="n">e9af21b</span><span class="p">,</span> <span
class="n">registrationTimeUTC</span><span class="p">=</span>1397574073203<span
class="p">,</span> <span class="n">serviceType</span><span
class="p">=</span><span class="n">DYNAMIC</span><span class="p">,</span> <span
class="n">uriSpec</span><span class="p">=</span><span class="n">org</span><span
class="p">.</span><span class="n">apache</span><span class="p">.</span><span
class="n">curator</span><span class="p">.</span><span class="n">x</span><span
class="p">.</span><span class="n">discovery</span><span class="p">.</span><span
class="n">UriSpec</span><span class="p">@</span><span
class="n">ef8dacf0</span><span class="p">}</span>
Modified:
websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
==============================================================================
---
websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
(original)
+++
websites/staging/slider/trunk/content/design/registry/p2p_service_registries.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="p2p-service-registries-for-apache-slider">P2P Service Registries for
Apache Slider</h1>
+<h1 id="p2p-service-registries-for-apache-slider">P2P Service Registries for
Apache Slider<a class="headerlink"
href="#p2p-service-registries-for-apache-slider" title="Permanent
link">¶</a></h1>
<p>Alongside the centralized service registries, there's much prior work on
P2P discovery systems, especially for mobile and consumer devices.</p>
<p>They perform some multicast- or distributed hash table-based lookup,
@@ -212,7 +223,7 @@ clouds. There's multiple anecdotes of
registry for in-EC2 applications. At the very least, this service and its
equivalents in other cloud providers could be used to bootstrap ZK client
bindings in cloud environments. </p>
-<h2 id="service-location-protocol">Service Location Protocol</h2>
+<h2 id="service-location-protocol">Service Location Protocol<a
class="headerlink" href="#service-location-protocol" title="Permanent
link">¶</a></h2>
<p>Service Location Protocol is a protocol for discovery services that came out
of Sun, Novell and others -it is still available for printer discovery and
suchlike</p>
@@ -234,7 +245,7 @@ mentions in Windows Server technical not
<p>[<a href="http://www.ietf.org/rfc/rfc2610.txt";>RFC2610</a>] <em>DHCP
Options for Service Location Protocol, IETF, 1999</em></p>
</li>
</ul>
-<h2 id="zeroconf"><a href="http://www.zeroconf.org/";>Zeroconf</a></h2>
+<h2 id="zeroconf"><a href="http://www.zeroconf.org/";>Zeroconf</a><a
class="headerlink" href="#zeroconf" title="Permanent link">¶</a></h2>
<p>The multicast discovery service implemented in Apple's Bonjour system
--multicasting DNS lookups to all peers in the subnet.</p>
<p>This allows for URLs and hostnames to be dynamically positioned, with
@@ -243,13 +254,13 @@ DNS domain searches allowing for enumera
subnet is is O(DNS-queries-across-subnet), hence implicitly
<code>O(devices)*O(device-activity)</code>. </p>
<p>The special domains <code>_tcp.</code>, <code>_udp.</code> and their
subdomains can also be
served up via a normal DNS server.</p>
-<h2 id="jiniapache-river"><a
href="http://river.apache.org/doc/specs/html/lookup-spec.html";>Jini/Apache
River</a></h2>
+<h2 id="jiniapache-river"><a
href="http://river.apache.org/doc/specs/html/lookup-spec.html";>Jini/Apache
River</a><a class="headerlink" href="#jiniapache-river" title="Permanent
link">¶</a></h2>
<p>Attribute-driven service enumeration, which drives the, Java-client-only
model of downloading client-side code. There's no requirement for the remote
services to be in Java, only that drivers are.</p>
-<h2 id="serf"><a href="http://www.serfdom.io/";>Serf</a></h2>
+<h2 id="serf"><a href="http://www.serfdom.io/";>Serf</a><a class="headerlink"
href="#serf" title="Permanent link">¶</a></h2>
<p>This is a library that implements the <a
href="http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf";>SWIM
protocol</a> to propagate information around a cluster. Apparently works in
virtualized clusters too. It's already been used in a Flume-on-Hoya
provider.</p>
-<h2 id="anubis"><a
href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/";>Anubis</a></h2>
+<h2 id="anubis"><a
href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/";>Anubis</a><a
class="headerlink" href="#anubis" title="Permanent link">¶</a></h2>
<p>An HP Labs-built <a
href="http://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/anubis/doc/HPL-2005-72.pdf?format=raw";>High
Availability tuple-space</a> in SmartFrog; used in production in some of HP's
telco products. An agent publishes facts into the T-Space, and within one
heartbeat all other agents have it. One heart-beat later, unless there's been a
change in the membership, the publisher knows the others have it. One heartbeat
later the agents know the publisher knows it, etc.</p>
<p>Strengths: </p>
<ul>
@@ -285,7 +296,7 @@ mutual trust of HTTPS certificates to jo
<p>It proved good for workload sharing -tasks can be published to it, any
agent can say "I'm working on it" and take up the work. If the process
fails, the task becomes available again. We used this for distributed
scheduling in a rendering farm.</p>
-<h2 id="carmen"><a
href="http://www.hpl.hp.com/techreports/2002/HPL-2002-257";>Carmen</a></h2>
+<h2 id="carmen"><a
href="http://www.hpl.hp.com/techreports/2002/HPL-2002-257";>Carmen</a><a
class="headerlink" href="#carmen" title="Permanent link">¶</a></h2>
<p>This was another HP Labs project, related to the Cooltown "ubiquitous
computing" work, which was a decade too early to be relevant. It was
also positioned by management as a B2B platform, so ended up competing
Modified: websites/staging/slider/trunk/content/design/registry/references.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/references.html
(original)
+++ websites/staging/slider/trunk/content/design/registry/references.html Mon
Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,10 +196,10 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="service-registry-references">Service Registry References</h1>
+<h1 id="service-registry-references">Service Registry References<a
class="headerlink" href="#service-registry-references" title="Permanent
link">¶</a></h1>
<p>Service registration and discovery is a problem in distributed computing
that has been explored for over thirty years, with
[Birrell81]'s <em>Grapevine</em> system the first known implementation -though
of </p>
-<h1 id="papers">Papers</h1>
+<h1 id="papers">Papers<a class="headerlink" href="#papers" title="Permanent
link">¶</a></h1>
<ul>
<li>
<p><strong>[Birrell81]</strong> Birrell, A. et al, <a
href="http://research.microsoft.com/apps/pubs/default.aspx?id=63661";><em>Grapevine:
An exercise in distributed computing</em></a>. Comm. ACM 25, 4 (Apr 1982),
pp260-274.
Modified:
websites/staging/slider/trunk/content/design/registry/registry-model.html
==============================================================================
--- websites/staging/slider/trunk/content/design/registry/registry-model.html
(original)
+++ websites/staging/slider/trunk/content/design/registry/registry-model.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,7 +196,7 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry">Apache Slider: Service Registry</h1>
+<h1 id="apache-slider-service-registry">Apache Slider: Service Registry<a
class="headerlink" href="#apache-slider-service-registry" title="Permanent
link">¶</a></h1>
<p>The service registry model is designed to support dynamically
deployed Slider applications, <em>and</em> statically deployed versions
of the same application -provided the latter also registers itself,
Modified:
websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
==============================================================================
---
websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
(original)
+++
websites/staging/slider/trunk/content/design/registry/service_registry_end_to_end_scenario.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,8 +196,8 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1 id="apache-slider-service-registry-end-to-end-scenarios">Apache Slider:
Service Registry End-to-End Scenarios</h1>
-<h2 id="am-startup">AM startup</h2>
+<h1 id="apache-slider-service-registry-end-to-end-scenarios">Apache Slider:
Service Registry End-to-End Scenarios<a class="headerlink"
href="#apache-slider-service-registry-end-to-end-scenarios" title="Permanent
link">¶</a></h1>
+<h2 id="am-startup">AM startup<a class="headerlink" href="#am-startup"
title="Permanent link">¶</a></h2>
<ol>
<li>
<p>AM starts, reads in configuration, creates provider</p>
@@ -216,7 +227,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM triggers review of application state, requesting/releasing nodes as
appropriate</p>
</li>
</ol>
-<h2 id="agent-startup-standalone">Agent Startup: standalone</h2>
+<h2 id="agent-startup-standalone">Agent Startup: standalone<a
class="headerlink" href="#agent-startup-standalone" title="Permanent
link">¶</a></h2>
<ol>
<li>
<p>Container is issued to AM</p>
@@ -237,7 +248,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM gives agent next state command.</p>
</li>
</ol>
-<h2 id="am-gets-state-from-agent">AM gets state from agent:</h2>
+<h2 id="am-gets-state-from-agent">AM gets state from agent:<a
class="headerlink" href="#am-gets-state-from-agent" title="Permanent
link">¶</a></h2>
<ol>
<li>
<p>Agent heartbeats in</p>
@@ -255,7 +266,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Agent returns all config state, including: hostnames, allocated ports,
generated values (e.g. database connection strings, URLs) - as two-level
(allows agent to define which config options are relevant to which document)</p>
</li>
</ol>
-<h2 id="am-saves-state-for-serving">AM saves state for serving</h2>
+<h2 id="am-saves-state-for-serving">AM saves state for serving<a
class="headerlink" href="#am-saves-state-for-serving" title="Permanent
link">¶</a></h2>
<ol>
<li>
<p>AM saves state in RAM (assumptions: small, will rebuild on restart)</p>
@@ -267,7 +278,7 @@ Latest release: <strong>0.80.0-incubatin
<p>AM fields HTTP GET requests on content</p>
</li>
</ol>
-<h2 id="am-serves-content">AM Serves content</h2>
+<h2 id="am-serves-content">AM Serves content<a class="headerlink"
href="#am-serves-content" title="Permanent link">¶</a></h2>
<p>A simple REST service serves up content on paths published to the service
registry. It is also possible to enumerate documents published by GET
operations on parent paths.</p>
<ol>
<li>
@@ -280,7 +291,7 @@ Latest release: <strong>0.80.0-incubatin
<p>response is streamed with headers of : <code>content-type</code>,
<code>content-length</code>, do not cache in proxy, expires,<em> (with expiry
date chosen as ??)</em></p>
</li>
</ol>
-<h1 id="slider-client">Slider Client</h1>
+<h1 id="slider-client">Slider Client<a class="headerlink"
href="#slider-client" title="Permanent link">¶</a></h1>
<p>Currently slider client enumerates the YARN registry looking for slider
instances -including any instances of the same application running before
launching a cluster. </p>
<p>This </p>
<ul>
@@ -288,7 +299,7 @@ Latest release: <strong>0.80.0-incubatin
<li>has scale limitations <code>O(apps-in-YARN-cluster)</code> +
<code>O(completed-apps-in-RM-memory)</code></li>
<li>only retrieves configuration information from slider-deployed application
instances. <em>We do not need to restrict ourselves here.</em></li>
</ul>
-<h2 id="slider-client-lists-applications">Slider Client lists applications</h2>
+<h2 id="slider-client-lists-applications">Slider Client lists applications<a
class="headerlink" href="#slider-client-lists-applications" title="Permanent
link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">registry</span> <span class="o">--</span><span class="n">list</span>
<span class="p">[</span><span class="o">--</span><span
class="n">servicetype</span> <span class="o"><</span><span
class="n">application</span><span class="o">-</span><span
class="n">type</span><span class="o">></span><span class="p">]</span>
</pre></div>
@@ -304,7 +315,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Client enumerates registered services and lists them</p>
</li>
</ol>
-<h2
id="slider-client-lists-content-published-by-an-application-instance">Slider
Client lists content published by an application instance</h2>
+<h2
id="slider-client-lists-content-published-by-an-application-instance">Slider
Client lists content published by an application instance<a class="headerlink"
href="#slider-client-lists-content-published-by-an-application-instance"
title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">registry</span> <span class="o"><</span><span
class="n">instance</span><span class="o">></span> <span
class="o">--</span><span class="n">listconf</span> <span
class="p">[</span><span class="o">--</span><span class="n">servicetype</span>
<span class="o"><</span><span class="n">application</span><span
class="o">-</span><span class="n">type</span><span class="o">></span><span
class="p">]</span>
</pre></div>
@@ -326,7 +337,7 @@ Latest release: <strong>0.80.0-incubatin
<p>Client displays list of content</p>
</li>
</ol>
-<h2
id="slider-client-retrieves-content-published-by-an-application-instance">Slider
Client retrieves content published by an application instance</h2>
+<h2
id="slider-client-retrieves-content-published-by-an-application-instance">Slider
Client retrieves content published by an application instance<a
class="headerlink"
href="#slider-client-retrieves-content-published-by-an-application-instance"
title="Permanent link">¶</a></h2>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">registry</span> <span class="o"><</span><span
class="n">instance</span><span class="o">></span> <span
class="o">--</span><span class="n">getconf</span> <span
class="o"><</span><span class="n">document</span><span class="o">></span>
<span class="p">[</span><span class="o">--</span><span class="n">format</span>
<span class="p">(</span><span class="n">xml</span><span class="o">|</span><span
class="k">properties</span><span class="o">|</span><span
class="n">text</span><span class="o">|</span><span class="n">html</span><span
class="o">|</span><span class="n">csv</span><span class="o">|</span><span
class="n">yaml</span><span class="o">|</span><span class="n">json</span><span
class="p">,...)</span> <span class="p">[</span><span class="o">--</span><span
class="n">dest</span> <span class="o"><</span><span
class="n">file</span><span class="o">></span><span class="p">]</span> <span
class="p">[</span><
span class="o">--</span><span class="n">servicetype</span> <span
class="o"><</span><span class="n">application</span><span
class="o">-</span><span class="n">type</span><span class="o">></span><span
class="p">]</span>
</pre></div>
@@ -358,9 +369,9 @@ Latest release: <strong>0.80.0-incubatin
a file underneath.</p>
</li>
</ol>
-<h2
id="slider-client-retrieves-content-set-published-by-an-application-instance">Slider
Client retrieves content set published by an application instance</h2>
+<h2
id="slider-client-retrieves-content-set-published-by-an-application-instance">Slider
Client retrieves content set published by an application instance<a
class="headerlink"
href="#slider-client-retrieves-content-set-published-by-an-application-instance"
title="Permanent link">¶</a></h2>
<p>Here a set of documents published is retrieved in the desired format of an
application.</p>
-<h2 id="slider-client-retrieves-document-and-applies-template-to-it">Slider
Client retrieves document and applies template to it</h2>
+<h2 id="slider-client-retrieves-document-and-applies-template-to-it">Slider
Client retrieves document and applies template to it<a class="headerlink"
href="#slider-client-retrieves-document-and-applies-template-to-it"
title="Permanent link">¶</a></h2>
<p>Here a set of documents published is retrieved in the desired format of an
application.</p>
<div class="codehilite"><pre><span class="n">slider</span> <span
class="n">registry</span> <span class="o"><</span><span
class="n">instance</span><span class="o">></span> <span
class="o">--</span><span class="n">source</span> <span
class="o"><</span><span class="n">document</span><span class="o">></span>
<span class="p">[</span><span class="o">--</span><span
class="n">template</span> <span class="o"><</span><span
class="n">path</span><span class="o">-</span><span class="n">to</span><span
class="o">-</span><span class="n">template</span><span
class="o">></span><span class="p">]</span> <span class="p">[</span><span
class="o">--</span><span class="n">outfile</span> <span
class="o"><</span><span class="n">file</span><span
class="o">></span><span class="p">]</span> <span class="p">[</span><span
class="o">--</span><span class="n">servicetype</span> <span
class="o"><</span><span class="n">application</span><span
class="o">-</span><span class="n">type</span><span
class="o">></span><span class="p">]</span>
</pre></div>
Modified:
websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
==============================================================================
---
websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
(original)
+++
websites/staging/slider/trunk/content/design/registry/the_YARN_application_registration_and_binding_problem.html
Mon Nov 2 17:24:47 2015
@@ -155,7 +155,7 @@
<div style="text-align: center">
<h1><a href="/index.html">Apache Slider (incubating)</a></h1>
<hr>
-Latest release: <strong>0.80.0-incubating</strong><br>
+Latest release: <strong>0.81.1-incubating</strong><br>
<br>
<a id="download-button-sidebar" class="btn btn-success btn-block"
href="/downloads/" role="button">Download</a>
</div>
@@ -168,7 +168,18 @@ Latest release: <strong>0.80.0-incubatin
<h1 class="title"></h1>
- <!---
+ <style type="text/css">
+/* The following code is added by mdx_elementid.py
+ It was originally lifted from http://subversion.apache.org/style/site.css */
+/*
+ * Hide class="elementid-permalink", except when an enclosing heading
+ * has the :hover property.
+ */
+.headerlink, .elementid-permalink {
+ visibility: hidden;
+}
+h2:hover > .headerlink, h3:hover > .headerlink, h1:hover > .headerlink,
h6:hover > .headerlink, h4:hover > .headerlink, h5:hover > .headerlink,
dt:hover > .elementid-permalink { visibility: visible }</style>
+<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -185,9 +196,9 @@ Latest release: <strong>0.80.0-incubatin
limitations under the License.
-->
-<h1
id="apache-slider-yarn-application-registration-and-binding-the-problem">Apache
Slider YARN Application Registration and Binding: the Problem</h1>
-<h2 id="march-2014">March 2014</h2>
-<h1
id="how-to-bind-client-applications-to-services-dynamically-placed-applications">How
to bind client applications to services dynamically placed applications?</h1>
+<h1
id="apache-slider-yarn-application-registration-and-binding-the-problem">Apache
Slider YARN Application Registration and Binding: the Problem<a
class="headerlink"
href="#apache-slider-yarn-application-registration-and-binding-the-problem"
title="Permanent link">¶</a></h1>
+<h2 id="march-2014">March 2014<a class="headerlink" href="#march-2014"
title="Permanent link">¶</a></h2>
+<h1
id="how-to-bind-client-applications-to-services-dynamically-placed-applications">How
to bind client applications to services dynamically placed applications?<a
class="headerlink"
href="#how-to-bind-client-applications-to-services-dynamically-placed-applications"
title="Permanent link">¶</a></h1>
<p>There are some constraints here</p>
<ol>
<li>
@@ -210,7 +221,7 @@ applications? This is an issue which has
for Hadoop clusters -but if we are looking at registration and binding
of YARN applications, there should be no reason why
static applications cannot be discovered and bonded to using the same
mechanisms. </p>
-<h1 id="other-constraints">Other constraints:</h1>
+<h1 id="other-constraints">Other constraints:<a class="headerlink"
href="#other-constraints" title="Permanent link">¶</a></h1>
<ol>
<li>
<p>Reduce the amount of change needed in existing applications to a minimum
@@ -227,12 +238,12 @@ static applications cannot be discovered
outside of Slider. Rationale: want a mechanism that works with pure-YARN
apps</p>
</li>
</ol>
-<h2 id="possible-solutions">Possible Solutions:</h2>
-<h3 id="zk">ZK</h3>
+<h2 id="possible-solutions">Possible Solutions:<a class="headerlink"
href="#possible-solutions" title="Permanent link">¶</a></h2>
+<h3 id="zk">ZK<a class="headerlink" href="#zk" title="Permanent
link">¶</a></h3>
<p>Client applications use ZK to find services (addresses #1, #2 and #3).
Requires location code in the client.</p>
<p>HBase and Accumulo do this as part of a failover-ready design.</p>
-<h3 id="dns">DNS</h3>
+<h3 id="dns">DNS<a class="headerlink" href="#dns" title="Permanent
link">¶</a></h3>
<p>Client apps use DNS to find services, with custom DNS server for a
subdomain representing YARN services. Addresses #1; with a shortened TTL and
no DNS address caching, #3. #2 addressed only if other DNS entries are used to
@@ -243,7 +254,7 @@ over time. It does require the clients t
irrespective of the published TTL). It generates a load on the DNS servers
that is <code>O(clients/TTL)</code></p>
<p>Google Chubby offers a DNS service to handle this. ZK does not -yet.</p>
-<h3 id="floating-ip-addresses">Floating IP Addresses</h3>
+<h3 id="floating-ip-addresses">Floating IP Addresses<a class="headerlink"
href="#floating-ip-addresses" title="Permanent link">¶</a></h3>
<p>If the clients know/cache IP addresses of services, these addresses could be
floated across service instances. Linux HA has floating IP address support,
while Docker containers can make use of them, especially if an integrated DHCP
@@ -251,7 +262,7 @@ server handles the assignment of IP addr
<p>ARP caching is the inevitable problem here, but it is still less brittle
than
relying on applications to know not to cache IP addresses -and nor does it
place so much on DNS servers as short-TTL DNS entries.</p>
-<h3 id="ldap">LDAP</h3>
+<h3 id="ldap">LDAP<a class="headerlink" href="#ldap" title="Permanent
link">¶</a></h3>
<p>Enterprise Directory services are used to publish/locate services. Requires
lookup into the directory on binding (#1, #2), re-lookup on failure (#3).
LDAP permissions can prevent untrusted applications registering.</p>
@@ -265,7 +276,7 @@ libraries. Spring-LDAP is focused on con
-not LDAP-driven application config.</p>
</li>
</ul>
-<h3 id="registration-web-service">Registration Web Service</h3>
+<h3 id="registration-web-service">Registration Web Service<a
class="headerlink" href="#registration-web-service" title="Permanent
link">¶</a></h3>
<p>Custom web service registration services used. </p>
<ul>
<li>
@@ -276,7 +287,7 @@ libraries. Spring-LDAP is focused on con
<p>Requires new client-side code anyway.</p>
</li>
</ul>
-<h3 id="zookeeper-url-schema">Zookeeper URL Schema</h3>
+<h3 id="zookeeper-url-schema">Zookeeper URL Schema<a class="headerlink"
href="#zookeeper-url-schema" title="Permanent link">¶</a></h3>
<p>Offer our own <code>zk://</code> URL; java & .NET implementations
(others?) to resolve, browser plugins. </p>
<ul>
<li>
@@ -287,7 +298,7 @@ libraries. Spring-LDAP is focused on con
will not work with tools such as CURL or web browsers</p>
</li>
</ul>
-<h3 id="am-side-config-generation">AM-side config generation</h3>
+<h3 id="am-side-config-generation">AM-side config generation<a
class="headerlink" href="#am-side-config-generation" title="Permanent
link">¶</a></h3>
<p>App-side config generation-YARN applications to generate client-side
configuration files for launch-time information (#1, #2).
The AM can dynamically create these, and as the storage load is all in
@@ -311,7 +322,7 @@ the entire cluster, and push them back.<
<p>Needs something similar for statically deployed applications.</p>
</li>
</ul>
-<h3 id="client-side-config-generation">Client-side config generation</h3>
+<h3 id="client-side-config-generation">Client-side config generation<a
class="headerlink" href="#client-side-config-generation" title="Permanent
link">¶</a></h3>
<p>YARN app to publish attributes as key-val pairs, client-side code to read
and
generate configs from (#1, #2). Example configuration generators could
create: Hadoop-client XML, Spring, tomcat, guice configs, something for
.NET.</p>
@@ -356,7 +367,7 @@ to log all slf/log4j to stderr, so that
<p>client side code per app to generate specific binding information</p>
</li>
</ul>
-<h3 id="load-balancer-app-yarn-app">Load-balancer app Yarn App</h3>
+<h3 id="load-balancer-app-yarn-app">Load-balancer app Yarn App<a
class="headerlink" href="#load-balancer-app-yarn-app" title="Permanent
link">¶</a></h3>
<p>Spread requests round a set of registered handlers, e.g web servers. Support
plugins for session binding/sharding. </p>
<p>Some web servers can do this already; a custom YARN app could use grizzy
