Repository: incubator-apex-site Updated Branches: refs/heads/asf-site 3a5c1de3a -> 2ea69c2b8
http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/operators/kafkaInputOperator/index.html ---------------------------------------------------------------------- diff --git a/content/docs/malhar/operators/kafkaInputOperator/index.html b/content/docs/malhar/operators/kafkaInputOperator/index.html deleted file mode 100644 index ba8ba3e..0000000 --- a/content/docs/malhar/operators/kafkaInputOperator/index.html +++ /dev/null @@ -1,455 +0,0 @@ -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - - - <title>Kafka Input - Apache Apex Malhar Documentation</title> - - - <link rel="shortcut icon" href="../../favicon.ico"> - - - - <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> - - <link rel="stylesheet" href="../../css/theme.css" type="text/css" /> - <link rel="stylesheet" href="../../css/theme_extra.css" type="text/css" /> - <link rel="stylesheet" href="../../css/highlight.css"> - - - <script> - // Current page data - var mkdocs_page_name = "Kafka Input"; - var mkdocs_page_input_path = "operators/kafkaInputOperator.md"; - var mkdocs_page_url = "/operators/kafkaInputOperator/"; - </script> - - <script src="../../js/jquery-2.1.1.min.js"></script> - <script src="../../js/modernizr-2.8.3.min.js"></script> - <script type="text/javascript" src="../../js/highlight.pack.js"></script> - <script src="../../js/theme.js"></script> - - -</head> - -<body class="wy-body-for-nav" role="document"> - - <div class="wy-grid-for-nav"> - - - <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> - <div class="wy-side-nav-search"> - <a href="../.." class="icon icon-home"> Apache Apex Malhar Documentation</a> - <div role="search"> - <form id ="rtd-search-form" class="wy-form" action="../../search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - </form> -</div> - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - <ul class="current"> - - <li> - <li class="toctree-l1 "> - <a class="" href="../..">Apache Apex Malhar</a> - - </li> -<li> - - <li> - <ul class="subnav"> - <li><span>Operators</span></li> - - - - <li class="toctree-l1 current"> - <a class="current" href="./">Kafka Input</a> - - <ul> - - <li class="toctree-l3"><a href="#kafka-input-operator">KAFKA INPUT OPERATOR</a></li> - - <li><a class="toctree-l4" href="#introduction-about-kafka-input-operator">Introduction: About Kafka Input Operator</a></li> - - <li><a class="toctree-l4" href="#why-is-it-needed">Why is it needed ?</a></li> - - <li><a class="toctree-l4" href="#abstractkafkainputoperator">AbstractKafkaInputOperator</a></li> - - <li><a class="toctree-l4" href="#kafkaconsumer">KafkaConsumer</a></li> - - <li><a class="toctree-l4" href="#pre-requisites">Pre-requisites</a></li> - - <li><a class="toctree-l4" href="#offsetmanager">OffsetManager</a></li> - - <li><a class="toctree-l4" href="#partitioning">Partitioning</a></li> - - <li><a class="toctree-l4" href="#abstractsingleportkafkainputoperator">AbstractSinglePortKafkaInputOperator</a></li> - - <li><a class="toctree-l4" href="#concrete-classes">Concrete Classes</a></li> - - <li><a class="toctree-l4" href="#application-example">Application Example</a></li> - - - </ul> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="../file_splitter/">File Splitter</a> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="../block_reader/">Block Reader</a> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="../file_output/">File Output</a> - - </li> - - - </ul> -<li> - - </ul> - </div> - - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="../..">Apache Apex Malhar Documentation</a> - </nav> - - - <div class="wy-nav-content"> - <div class="rst-content"> - <div role="navigation" aria-label="breadcrumbs navigation"> - <ul class="wy-breadcrumbs"> - <li><a href="../..">Docs</a> »</li> - - - - <li>Operators »</li> - - - - <li>Kafka Input</li> - <li class="wy-breadcrumbs-aside"> - - </li> - </ul> - <hr/> -</div> - <div role="main"> - <div class="section"> - - <h1 id="kafka-input-operator">KAFKA INPUT OPERATOR</h1> -<h3 id="introduction-about-kafka-input-operator">Introduction: About Kafka Input Operator</h3> -<p>This is an input operator that consumes data from Kafka messaging system for further processing in Apex. Kafka Input Operator is an fault-tolerant and scalable Malhar Operator.</p> -<h3 id="why-is-it-needed">Why is it needed ?</h3> -<p>Kafka is a pull-based and distributed publish subscribe messaging system, topics are partitioned and replicated across -nodes. Kafka input operator is needed when you want to read data from multiple -partitions of a Kafka topic in parallel in an Apex application.</p> -<h3 id="abstractkafkainputoperator">AbstractKafkaInputOperator</h3> -<p>This is the abstract implementation that serves as base class for consuming messages from Kafka messaging system. This class doesnât have any ports.</p> -<p><img alt="AbstractKafkaInput.png" src="../images/kafkainput/image00.png" /></p> -<h4 id="configuration-parameters">Configuration Parameters</h4> -<p><table> -<col width="25%" /> -<col width="75%" /> -<tbody> -<tr class="odd"> -<td align="left"><p>Parameter</p></td> -<td align="left"><p>Description</p></td> -</tr> -<tr class="even"> -<td align="left"><p>maxTuplesPerWindow</p></td> -<td align="left"><p>Controls the maximum number of messages emitted in each streaming window from this operator. Minimum value is 1. Default value = MAX_VALUE </p></td> -</tr> -<tr class="odd"> -<td align="left"><p>idempotentStorageManager</p></td> -<td align="left"><p>This is an instance of IdempotentStorageManager. Idempotency ensures that the operator will process the same set of messages in a window before and after a failure. For example, let's say the operator completed window 10 and failed somewhere between window 11. If the operator gets restored at window 10 then it will process the same messages again in window 10 which it did in the previous run before the failure. Idempotency is important but comes with higher cost because at the end of each window the operator needs to persist some state with respect to that window. Default Value = com.datatorrent.lib.io.IdempotentStorageManager.<br>NoopIdempotentStorageManager</p></td> -</tr> -<tr class="even"> -<td align="left"><p>strategy</p></td> -<td align="left"><p>Operator supports two types of partitioning strategies, ONE_TO_ONE and ONE_TO_MANY.</p> -<p>ONE_TO_ONE: If this is enabled, the AppMaster creates one input operator instance per Kafka topic partition. So the number of Kafka topic partitions equals the number of operator instances.</p> -<p>ONE_TO_MANY: The AppMaster creates K = min(initialPartitionCount, N) Kafka input operator instances where N is the number of Kafka topic partitions. If K is less than N, the remaining topic partitions are assigned to the K operator instances in round-robin fashion. If K is less than initialPartitionCount, the AppMaster creates one input operator instance per Kafka topic partition. For example, if initialPartitionCount = 5 and number of Kafka partitions(N) = 2 then AppMaster creates 2 Kafka input operator instances. -Default Value = ONE_TO_ONE</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>msgRateUpperBound</p></td> -<td align="left"><p>Maximum messages upper bound. Operator repartitions when the <em>msgProcessedPS</em> exceeds this bound. <em>msgProcessedPS</em> is the average number of messages processed per second by this operator.</p></td> -</tr> -<tr class="even"> -<td align="left"><p>byteRateUpperBound</p></td> -<td align="left"><p>Maximum bytes upper bound. Operator repartitions when the <em>bytesPS</em> exceeds this bound. <em>bytesPS</em> is the average number of bytes processed per second by this operator.</p> -<p></p></td> -</tr> -<tr class="odd"> -<td align="left"><p>offsetManager</p></td> -<td align="left"><p>This is an optional parameter that is useful when the application restarts or start at specific offsets (offsets are explained below)</p></td> -</tr> -<tr class="even"> -<td align="left"><p>repartitionInterval</p></td> -<td align="left"><p>Interval specified in milliseconds. This value specifies the minimum time required between two repartition actions. Default Value = 30 Seconds</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>repartitionCheckInterval</p></td> -<td align="left"><p>Interval specified in milliseconds. This value specifies the minimum interval between two offset updates. Default Value = 5 Seconds</p></td> -</tr> -<tr class="even"> -<td align="left"><p>initialPartitionCount</p></td> -<td align="left"><p>When the ONE_TO_MANY partition strategy is enabled, this value indicates the number of Kafka input operator instances. Default Value = 1</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>consumer</p></td> -<td align="left"><p>This is an instance of com.datatorrent.contrib.kafka.KafkaConsumer. Default Value = Instance of SimpleKafkaConsumer.</p></td> -</tr> -</tbody> -</table></p> -<h4 id="abstract-methods">Abstract Methods</h4> -<p>void emitTuple(Message message): Abstract method that emits tuples -extracted from Kafka message.</p> -<h3 id="kafkaconsumer">KafkaConsumer</h3> -<p>This is an abstract implementation of Kafka consumer. It sends the fetch -requests to the leading brokers of Kafka partitions. For each request, -it receives the set of messages and stores them into the buffer which is -ArrayBlockingQueue. SimpleKafkaConsumer which extends -KafkaConsumer and serves the functionality of Simple Consumer API and -HighLevelKafkaConsumer which extends KafkaConsumer and  serves the -functionality of High Level Consumer API.</p> -<h3 id="pre-requisites">Pre-requisites</h3> -<p>This operator referred the Kafka Consumer API of version -0.8.1.1. So, this operator will work with any 0.8.x and 0.7.x version of Apache Kafka.</p> -<h4 id="configuration-parameters_1">Configuration Parameters</h4> -<table> -<col width="15%" /> -<col width="15%" /> -<col width="15%" /> -<col width="55%" /> -<tbody> -<tr class="odd"> -<td align="left"><p>Parameter</p></td> -<td align="left"><p>Type</p></td> -<td align="left"><p>Default</p></td> -<td align="left"><p>Description</p></td> -</tr> -<tr class="even"> -<td align="left"><p>zookeeper</p></td> -<td align="left"><p>String</p></td> -<td align="left"><p></p></td> -<td align="left"><p>Specifies the zookeeper quorum of Kafka clusters that you want to consume messages from. zookeeper  is a string in the form of hostname1:port1,hostname2:port2,hostname3:port3  where hostname1,hostname2,hostname3 are hosts and port1,port2,port3 are ports of zookeeper server.  If the topic name is the same across the Kafka clusters and want to consume data from these clusters, then configure the zookeeper as follows: c1::hs1:p1,hs2:p2,hs3:p3;c2::hs4:p4,hs5:p5,c3::hs6:p6</p> -<p>where</p> -<p>c1,c2,c3 indicates the cluster names, hs1,hs2,hs3,hs4,hs5,hs6 are zookeeper hosts and p1,p2,p3,p4,p5,p6 are corresponding ports. Here, cluster name is optional in case of single cluster</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>cacheSize</p></td> -<td align="left"><p>int</p></td> -<td align="left"><p>1024</p></td> -<td align="left"><p>Maximum of buffered messages hold in memory.</p></td> -</tr> -<tr class="even"> -<td align="left"><p>topic</p></td> -<td align="left"><p>String</p></td> -<td align="left"><p>default_topic</p></td> -<td align="left"><p>Indicates the name of the topic.</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>initialOffset</p></td> -<td align="left"><p>String</p></td> -<td align="left"><p>latest</p></td> -<td align="left"><p>Indicates the type of offset i.e, âearliest or latestâ. If initialOffset is âlatestâ, then the operator consumes messages from latest point of Kafka queue. If initialOffset is âearliestâ, then the operator consumes messages starting from message queue. This can be overridden by OffsetManager.</p></td> -</tr> -</tbody> -</table> - -<h4 id="abstract-methods_1">Abstract Methods</h4> -<ol> -<li>void commitOffset(): Commit the offsets at checkpoint.</li> -<li>Map <KafkaPartition, Long> getCurrentOffsets(): Return the current - offset status.</li> -<li>resetPartitionsAndOffset(Set <KafkaPartition> partitionIds, - Map <KafkaPartition, Long> startOffset): Reset the partitions with - parittionIds and offsets with startOffset.</li> -</ol> -<h4 id="configuration-parameters-for-simplekafkaconsumer">Configuration Parameters for SimpleKafkaConsumer</h4> -<table> -<col width="25%" /> -<col width="15%" /> -<col width="15%" /> -<col width="45%" /> -<tbody> -<tr class="odd"> -<td align="left"><p>Parameter</p></td> -<td align="left"><p>Type</p></td> -<td align="left"><p>Default</p></td> -<td align="left"><p>Description</p></td> -</tr> -<tr class="even"> -<td align="left"><p>bufferSize</p></td> -<td align="left"><p>int</p></td> -<td align="left"><p>1 MB</p></td> -<td align="left"><p>Specifies the maximum total size of messages for each fetch request.</p></td> -</tr> -<tr class="odd"> -<td align="left"><p>metadataRefreshInterval</p></td> -<td align="left"><p>int</p></td> -<td align="left"><p>30 Seconds</p></td> -<td align="left"><p>Interval in between refresh the metadata change(broker change) in milliseconds. Enabling metadata refresh guarantees an automatic reconnect when a new broker is elected as the host. A value of -1 disables this feature.</p></td> -</tr> -<tr class="even"> -<td align="left"><p>metadataRefreshRetryLimit</p></td> -<td align="left"><p>int</p></td> -<td align="left"><p>-1</p></td> -<td align="left"><p>Specifies the maximum brokers' metadata refresh retry limit. -1 means unlimited retry.</p></td> -</tr> -</tbody> -</table> - -<h3 id="offsetmanager">OffsetManager</h3> -<p>This is an interface for offset management and is useful when consuming data -from specified offsets. Updates the offsets for all the Kafka partitions -periodically. Below is the code snippet:        </p> -<pre><code class="java">public interface OffsetManager -{ - public Map<KafkaPartition, Long> loadInitialOffsets(); - public void updateOffsets(Map<KafkaPartition, Long> offsetsOfPartitions); -} -</code></pre> - -<h4 id="abstract-methods_2">Abstract Methods</h4> -<p>Map <KafkaPartition, Long> loadInitialOffsets(): Specifies the initial offset for consuming messages; called at the activation stage.</p> -<p>updateOffsets(Map <KafkaPartition, Long> offsetsOfPartitions):  This -method is called at every repartitionCheckInterval to update offsets.</p> -<h3 id="partitioning">Partitioning</h3> -<p>The logical instance of the KafkaInputOperator acts as the Partitioner -as well as a StatsListener. This is because the -AbstractKafkaInputOperator implements both the -com.datatorrent.api.Partitioner and com.datatorrent.api.StatsListener -interfaces and provides an implementation of definePartitions(...) and -processStats(...) which makes it auto-scalable.</p> -<h4 id="response-processstatsbatchedoperatorstats-stats">Response processStats(BatchedOperatorStats stats)</h4> -<p>The application master invokes this method on the logical instance with -the stats (tuplesProcessedPS, bytesPS, etc.) of each partition. -Re-partitioning happens based on whether any new Kafka partitions added for -the topic or bytesPS and msgPS cross their respective upper bounds.</p> -<h4 id="definepartitions">DefinePartitions</h4> -<p>Based on the repartitionRequired field of the Response object which is -returned by processStats(...) method, the application master invokes -definePartitions(...) on the logical instance which is also the -partitioner instance. Dynamic partition can be disabled by setting the -parameter repartitionInterval value to a negative value.</p> -<h3 id="abstractsingleportkafkainputoperator">AbstractSinglePortKafkaInputOperator</h3> -<p>This class extends AbstractKafkaInputOperator and having single output -port, will emit the messages through this port.</p> -<h4 id="ports">Ports</h4> -<p>outputPort <T>: Tuples extracted from Kafka messages are emitted through -this port.</p> -<h4 id="abstract-methods_3">Abstract Methods</h4> -<p>T getTuple(Message msg) : Converts the Kafka message to tuple.</p> -<h3 id="concrete-classes">Concrete Classes</h3> -<ol> -<li> -<p>KafkaSinglePortStringInputOperator : -This class extends AbstractSinglePortKafkaInputOperator and getTuple() method extracts string from Kafka message.</p> -</li> -<li> -<p>KafkaSinglePortByteArrayInputOperator: -This class extends AbstractSinglePortKafkaInputOperator and getTuple() method extracts byte array from Kafka message.</p> -</li> -</ol> -<h3 id="application-example">Application Example</h3> -<p>This section builds an Apex application using Kafka input operator. -Below is the code snippet:</p> -<pre><code class="java">@ApplicationAnnotation(name = "KafkaApp") -public class ExampleKafkaApplication implements StreamingApplication -{ -@Override -public void populateDAG(DAG dag, Configuration entries) -{ - KafkaSinglePortByteArrayInputOperator input = dag.addOperator("MessageReader", new KafkaSinglePortByteArrayInputOperator()); - - ConsoleOutputOperator output = dag.addOperator("Output", new ConsoleOutputOperator()); - - dag.addStream("MessageData", input.outputPort, output.input); -} -} -</code></pre> - -<p>Below is the configuration for âtestâ Kafka topic name and -âlocalhost:2181â is the zookeeper forum:</p> -<pre><code class="xml"><property> -<name>dt.operator.MessageReader.prop.topic</name> -<value>test</value> -</property> - -<property> -<name>dt.operator.KafkaInputOperator.prop.zookeeper</nam> -<value>localhost:2181</value> -</property> -</code></pre> - - </div> - </div> - <footer> - - <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - - <a href="../file_splitter/" class="btn btn-neutral float-right" title="File Splitter">Next <span class="icon icon-circle-arrow-right"></span></a> - - - <a href="../.." class="btn btn-neutral" title="Apache Apex Malhar"><span class="icon icon-circle-arrow-left"></span> Previous</a> - - </div> - - - <hr/> - - <div role="contentinfo"> - <!-- Copyright etc --> - - </div> - - Built with <a href="http://www.mkdocs.org";>MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. -</footer> - - </div> - </div> - - </section> - - </div> - -<div class="rst-versions" role="note" style="cursor: pointer"> - <span class="rst-current-version" data-toggle="rst-current-version"> - - - <span><a href="../.." style="color: #fcfcfc;">« Previous</a></span> - - - <span style="margin-left: 15px"><a href="../file_splitter/" style="color: #fcfcfc">Next »</a></span> - - </span> -</div> - -</body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/search.html ---------------------------------------------------------------------- diff --git a/content/docs/malhar/search.html b/content/docs/malhar/search.html deleted file mode 100644 index 96dc007..0000000 --- a/content/docs/malhar/search.html +++ /dev/null @@ -1,171 +0,0 @@ -<!DOCTYPE html> -<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> -<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - - - - <title>Apache Apex Malhar Documentation</title> - - - <link rel="shortcut icon" href="favicon.ico"> - - - - <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> - - <link rel="stylesheet" href="./css/theme.css" type="text/css" /> - <link rel="stylesheet" href="./css/theme_extra.css" type="text/css" /> - <link rel="stylesheet" href="./css/highlight.css"> - - - <script src="./js/jquery-2.1.1.min.js"></script> - <script src="./js/modernizr-2.8.3.min.js"></script> - <script type="text/javascript" src="./js/highlight.pack.js"></script> - <script src="./js/theme.js"></script> - <script>var base_url = '.';</script> - <script data-main="./mkdocs/js/search.js" src="./mkdocs/js/require.js"></script> - - - -</head> - -<body class="wy-body-for-nav" role="document"> - - <div class="wy-grid-for-nav"> - - - <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> - <div class="wy-side-nav-search"> - <a href="." class="icon icon-home"> Apache Apex Malhar Documentation</a> - <div role="search"> - <form id ="rtd-search-form" class="wy-form" action="./search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - </form> -</div> - </div> - - <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> - <ul class="current"> - - <li> - <li class="toctree-l1 "> - <a class="" href=".">Apache Apex Malhar</a> - - </li> -<li> - - <li> - <ul class="subnav"> - <li><span>Operators</span></li> - - - - <li class="toctree-l1 "> - <a class="" href="operators/kafkaInputOperator/">Kafka Input</a> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="operators/file_splitter/">File Splitter</a> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="operators/block_reader/">Block Reader</a> - - </li> - - - - <li class="toctree-l1 "> - <a class="" href="operators/file_output/">File Output</a> - - </li> - - - </ul> -<li> - - </ul> - </div> - - </nav> - - <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - - - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href=".">Apache Apex Malhar Documentation</a> - </nav> - - - <div class="wy-nav-content"> - <div class="rst-content"> - <div role="navigation" aria-label="breadcrumbs navigation"> - <ul class="wy-breadcrumbs"> - <li><a href=".">Docs</a> »</li> - - - <li class="wy-breadcrumbs-aside"> - - </li> - </ul> - <hr/> -</div> - <div role="main"> - <div class="section"> - - - <h1 id="search">Search Results</h1> - - <form id="content_search" action="search.html"> - <span role="status" aria-live="polite" class="ui-helper-hidden-accessible"></span> - <input name="q" id="mkdocs-search-query" type="text" class="search_input search-query ui-autocomplete-input" placeholder="Search the Docs" autocomplete="off" autofocus> - </form> - - <div id="mkdocs-search-results"> - Searching... - </div> - - - </div> - </div> - <footer> - - - <hr/> - - <div role="contentinfo"> - <!-- Copyright etc --> - - </div> - - Built with <a href="http://www.mkdocs.org";>MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. -</footer> - - </div> - </div> - - </section> - - </div> - -<div class="rst-versions" role="note" style="cursor: pointer"> - <span class="rst-current-version" data-toggle="rst-current-version"> - - - - </span> -</div> - -</body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/searchbox.html ---------------------------------------------------------------------- diff --git a/content/docs/malhar/searchbox.html b/content/docs/malhar/searchbox.html deleted file mode 100644 index 177fcb3..0000000 --- a/content/docs/malhar/searchbox.html +++ /dev/null @@ -1,5 +0,0 @@ -<div role="search"> - <form id ="rtd-search-form" class="wy-form" action="{{ base_url }}/search.html" method="get"> - <input type="text" name="q" placeholder="Search docs" /> - </form> -</div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/sitemap.xml ---------------------------------------------------------------------- diff --git a/content/docs/malhar/sitemap.xml b/content/docs/malhar/sitemap.xml deleted file mode 100644 index 1455cd3..0000000 --- a/content/docs/malhar/sitemap.xml +++ /dev/null @@ -1,40 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9";> - - - <url> - <loc>/</loc> - <lastmod>2016-03-11</lastmod> - <changefreq>daily</changefreq> - </url> - - - - - <url> - <loc>/operators/kafkaInputOperator/</loc> - <lastmod>2016-03-11</lastmod> - <changefreq>daily</changefreq> - </url> - - <url> - <loc>/operators/file_splitter/</loc> - <lastmod>2016-03-11</lastmod> - <changefreq>daily</changefreq> - </url> - - <url> - <loc>/operators/block_reader/</loc> - <lastmod>2016-03-11</lastmod> - <changefreq>daily</changefreq> - </url> - - <url> - <loc>/operators/file_output/</loc> - <lastmod>2016-03-11</lastmod> - <changefreq>daily</changefreq> - </url> - - - -</urlset> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/toc.html ---------------------------------------------------------------------- diff --git a/content/docs/malhar/toc.html b/content/docs/malhar/toc.html deleted file mode 100644 index 6cd2fc9..0000000 --- a/content/docs/malhar/toc.html +++ /dev/null @@ -1,23 +0,0 @@ -{% if nav_item.children %} - <ul class="subnav"> - <li><span>{{ nav_item.title }}</span></li> - - {% for nav_item in nav_item.children %} - {% include 'toc.html' %} - {% endfor %} - </ul> -{% else %} - <li class="toctree-l1 {% if nav_item.active%}current{%endif%}"> - <a class="{% if nav_item.active%}current{%endif%}" href="{{ nav_item.url }}">{{ nav_item.title }}</a> - {% if nav_item == current_page %} - <ul> - {% for toc_item in toc %} - <li class="toctree-l3"><a href="{{ toc_item.url }}">{{ toc_item.title }}</a></li> - {% for toc_item in toc_item.children %} - <li><a class="toctree-l4" href="{{ toc_item.url }}">{{ toc_item.title }}</a></li> - {% endfor %} - {% endfor %} - </ul> - {% endif %} - </li> -{% endif %} http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/content/docs/malhar/versions.html ---------------------------------------------------------------------- diff --git a/content/docs/malhar/versions.html b/content/docs/malhar/versions.html deleted file mode 100644 index d12d197..0000000 --- a/content/docs/malhar/versions.html +++ /dev/null @@ -1,15 +0,0 @@ -<div class="rst-versions" role="note" style="cursor: pointer"> - <span class="rst-current-version" data-toggle="rst-current-version"> - {% if repo_name == 'GitHub' %} - <a href="{{ repo_url }}" class="icon icon-github" style="float: left; color: #fcfcfc"> GitHub</a> - {% elif repo_name == 'Bitbucket' %} - <a href="{{ repo_url }}" class="icon icon-bitbucket" style="float: left; color: #fcfcfc"> BitBucket</a> - {% endif %} - {% if previous_page %} - <span><a href="{{ previous_page.url }}" style="color: #fcfcfc;">« Previous</a></span> - {% endif %} - {% if next_page %} - <span style="margin-left: 15px"><a href="{{ next_page.url }}" style="color: #fcfcfc">Next »</a></span> - {% endif %} - </span> -</div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/apex_development_setup/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/apex_development_setup/index.html b/docs/apex-3.3/apex_development_setup/index.html index 60350b2..c3029c0 100644 --- a/docs/apex-3.3/apex_development_setup/index.html +++ b/docs/apex-3.3/apex_development_setup/index.html @@ -135,6 +135,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/application_development/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/application_development/index.html b/docs/apex-3.3/application_development/index.html index 5297074..d0bc30b 100644 --- a/docs/apex-3.3/application_development/index.html +++ b/docs/apex-3.3/application_development/index.html @@ -201,6 +201,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/application_packages/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/application_packages/index.html b/docs/apex-3.3/application_packages/index.html index 41e9a29..4ca9434 100644 --- a/docs/apex-3.3/application_packages/index.html +++ b/docs/apex-3.3/application_packages/index.html @@ -147,6 +147,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/autometrics/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/autometrics/index.html b/docs/apex-3.3/autometrics/index.html index 6fd3fd8..d5ffea8 100644 --- a/docs/apex-3.3/autometrics/index.html +++ b/docs/apex-3.3/autometrics/index.html @@ -146,6 +146,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/compatibility/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/compatibility/index.html b/docs/apex-3.3/compatibility/index.html new file mode 100644 index 0000000..fc88b80 --- /dev/null +++ b/docs/apex-3.3/compatibility/index.html @@ -0,0 +1,280 @@ +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + + + <title>Compatibility - Apache Apex Documentation</title> + + + <link rel="shortcut icon" href="../favicon.ico"> + + + + <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> + + <link rel="stylesheet" href="../css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../css/theme_extra.css" type="text/css" /> + <link rel="stylesheet" href="../css/highlight.css"> + + + <script> + // Current page data + var mkdocs_page_name = "Compatibility"; + var mkdocs_page_input_path = "compatibility.md"; + var mkdocs_page_url = "/compatibility/"; + </script> + + <script src="../js/jquery-2.1.1.min.js"></script> + <script src="../js/modernizr-2.8.3.min.js"></script> + <script type="text/javascript" src="../js/highlight.pack.js"></script> + <script src="../js/theme.js"></script> + + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> + <div class="wy-side-nav-search"> + <a href=".." class="icon icon-home"> Apache Apex Documentation</a> + <div role="search"> + <form id ="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + </form> +</div> + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + <ul class="current"> + + <li> + <li class="toctree-l1 "> + <a class="" href="..">Apache Apex</a> + + </li> +<li> + + <li> + <ul class="subnav"> + <li><span>Development</span></li> + + + + <li class="toctree-l1 "> + <a class="" href="../apex_development_setup/">Development Setup</a> + + </li> + + + + <li class="toctree-l1 "> + <a class="" href="../application_development/">Applications</a> + + </li> + + + + <li class="toctree-l1 "> + <a class="" href="../application_packages/">Packages</a> + + </li> + + + + <li class="toctree-l1 "> + <a class="" href="../operator_development/">Operators</a> + + </li> + + + + <li class="toctree-l1 "> + <a class="" href="../autometrics/">AutoMetric API</a> + + </li> + + + </ul> +<li> + + <li> + <ul class="subnav"> + <li><span>Operations</span></li> + + + + <li class="toctree-l1 "> + <a class="" href="../dtcli/">dtCli</a> + + </li> + + + </ul> +<li> + + <li> + <li class="toctree-l1 current"> + <a class="current" href="./">Compatibility</a> + + <ul> + + <li class="toctree-l3"><a href="#apache-apex-compatibility">Apache Apex Compatibility</a></li> + + <li><a class="toctree-l4" href="#purpose">Purpose</a></li> + + <li><a class="toctree-l4" href="#compatibility-types">Compatibility types</a></li> + + + </ul> + + </li> +<li> + + </ul> + </div> + + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="..">Apache Apex Documentation</a> + </nav> + + + <div class="wy-nav-content"> + <div class="rst-content"> + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="..">Docs</a> »</li> + + + + <li>Compatibility</li> + <li class="wy-breadcrumbs-aside"> + + </li> + </ul> + <hr/> +</div> + <div role="main"> + <div class="section"> + + <h1 id="apache-apex-compatibility">Apache Apex Compatibility</h1> +<h2 id="purpose">Purpose</h2> +<p>This document captures the compatibility goals of the Apache Apex project. The different types of compatibility between Apex releases that affect contributors, downstream projects, and end-users are enumerated. For each type of compatibility we:</p> +<ul> +<li>describe the impact on downstream projects or end-users</li> +<li>where applicable, call out the policy adopted when incompatible changes are permitted.</li> +</ul> +<p>Apache Apex follows <a href="http://semver.org/";>semantic versioning</a>. Depending on the compatibility type, there may be different tools or mechanisms to ensure compatibility, for example by comparing artifacts during the build process.</p> +<p>The type of change will inform the required target version number. Given a version number MAJOR.MINOR.PATCH, increment the:</p> +<ul> +<li>MAJOR version when you make incompatible API changes,</li> +<li>MINOR version when you add functionality in a backward-compatible manner, and</li> +<li>PATCH version when you make backward-compatible bug fixes.</li> +</ul> +<p>Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.</p> +<p>The overall goal is to avoid backward incompatible changes and major release upgrades. Accordingly we attempt to release new features with minor versions that are incremental to the prior release and offer our users a frictionless upgrade path. When planning contributions, please consider compatibility and release road map upfront. Specifically, certain changes that conflict with the versioning may need to be documented in JIRA and deferred until a future major release. </p> +<h2 id="compatibility-types">Compatibility types</h2> +<h3 id="java-api">Java API</h3> +<p>Public API compatibility is required to ensure end-user programs and downstream projects continue to work without modification. +The public API consists of:</p> +<ul> +<li>apex-core: all interfaces and classes in <code>api</code> and <code>common</code> modules</li> +<li>apex-malhar: all interfaces and classes in all modules except <code>demos</code>, <code>samples</code>, <code>benchmark</code> </li> +</ul> +<p>Interfaces and classes that are part of the public API and are annotated with <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html";>interface stability</a> are treated according to the rules defined by the annotation. </p> +<p>Policy</p> +<p>Changes to the public API must follow semantic versioning. +Public APIs must be deprecated for at least one minor release prior to their removal in a major release. +The <a href="https://github.com/siom79/japicmp";>japicmp Maven plugin</a> is used to enforce compatibility as part of the Travis pre-commit builds.</p> +<h3 id="semantic-compatibility">Semantic compatibility</h3> +<p>The behavior of APIs needs to remain consistent over versions, though changes for correctness may result in changes in behavior. Tests and javadocs specify the behavior. Over time, test suites should be expanded to verify compliance with the specification, effectively creating a formal specification for the subset of behaviors that can be easily tested.</p> +<p>Policy</p> +<p>The behavior of existing API cannot be modified as it would break existing user code. There are exceptional circumstances that may justify such changes, in which cases they should be discussed on the mailing list before implementation. Examples are bug fixes related to security issues, data corruption/consistency or to correct an unintended change from previous release that violated semantic compatibility. Such changes should be accompanied by test coverage for the exact behavior.</p> +<h3 id="rest-api">REST API</h3> +<p>REST API compatibility corresponds to both the URLs and request/response content over the wire. REST APIs are specifically meant for stable use by clients across releases, even major releases. </p> +<p>Policy</p> +<p>The REST API is separately versioned. This is to allow for co-existence of old and new API should there be a need for backward incompatible changes in the future.</p> +<h3 id="command-line-interface-cli">Command Line Interface (CLI)</h3> +<p>The CLI may be used either directly via the system shell or via shell scripts. Changing the path, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users.</p> +<p>Policy</p> +<p>CLI commands are to be deprecated (warning when used) in a prior minor release before they are removed or incompatibly modified in a subsequent major release.</p> +<h3 id="configuration-files">Configuration Files</h3> +<p>Configuration files are used for engine or application settings. Changes to keys and default values directly affect users and are hard to diagnose (compared to a compile error, for example).</p> +<p>Policy</p> +<p>Name, location, format, keys of configuration files should be deprecated in a prior minor release and can only be changed in major release. Best effort should be made to support the deprecated behavior for one more major release (not guaranteed). It is also desirable to provide the user with a migration tool.</p> +<h3 id="internal-wire-compatibility">Internal Wire compatibility</h3> +<p>Apex containers internally use RPC communication and netlet for the data flow. The protocols are private and user components are not exposed to it. Apex is a YARN application and automatically deployed. There is currently no situation where containers of different Apex engine versions need to be interoperable. Should such a scenario become relevant in the future, wire compatibility needs to be specified.</p> +<p>Policy</p> +<p>N/A</p> +<h3 id="internal-file-formats">Internal File formats</h3> +<p>Apex engine stores data in the file system for recovery and the data is typically obtained from serialization (from Kryo, Java etc.). Changes to internal classes may affect the ability to relaunch an application with upgraded engine code from previous state. This is currently not supported. In the future, the serialization mechanism should guarantee backward compatibility.</p> +<p>Policy</p> +<p>Currently no compatibility guarantee. User to cold-restart application on engine upgrade.</p> +<h3 id="java-classpath">Java Classpath</h3> +<p>Apex applications should not bundle Hadoop dependencies or Apex engine dependencies but use the dependencies provided in the target environment to avoid conflicts. The Apex application archetype can be used to generate a compliant project. </p> +<p>Policy</p> +<p>Apex engine dependencies can change as per semantic versioning. Following above guidelines automatically maintains the backward compatibility based on semantic versioning of Apex.</p> +<h3 id="maven-build-artifacts">Maven Build Artifacts</h3> +<p>Downstream projects reference the Apex engine dependencies and Malhar operator libraries for application development etc. Changes to the packaging (which classes are in which jar), the groupId, artifactId and which artifacts are deployed to Maven central impact upgrades.</p> +<p>Policy</p> +<p>The artifacts that contain the classes that form the public API as specified above cannot change in patch releases and should stay compatible within a major release. Patch releases can change dependencies, but only at the patch level and following semantic versioning.</p> +<h3 id="hardwaresoftware-requirements">Hardware/Software Requirements</h3> +<p>Apex depends on Apache Hadoop. The community intends to support all major Hadoop distros and current versions. Apex currently supports Hadoop 2.2.0 and higher and Java 7 and higher. Apex is written in Java and has been tested on Linux based Hadoop clusters. There are no additional restrictions on the hardware architecture. </p> +<p>To keep up with the latest advances in hardware, operating systems, JVMs, Hadoop and other software, new Apex releases may require higher versions. Upgrading Apex may require upgrading other dependent software components.</p> +<p>Policy</p> +<p>The JVM and Hadoop minimum version requirements are not expected to change outside major releases.</p> + + </div> + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + + <a href="../dtcli/" class="btn btn-neutral" title="dtCli"><span class="icon icon-circle-arrow-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <!-- Copyright etc --> + + </div> + + Built with <a href="http://www.mkdocs.org";>MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. +</footer> + + </div> + </div> + + </section> + + </div> + +<div class="rst-versions" role="note" style="cursor: pointer"> + <span class="rst-current-version" data-toggle="rst-current-version"> + + + <span><a href="../dtcli/" style="color: #fcfcfc;">« Previous</a></span> + + + </span> +</div> + +</body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/dtcli/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/dtcli/index.html b/docs/apex-3.3/dtcli/index.html index fc33b6f..bb63fab 100644 --- a/docs/apex-3.3/dtcli/index.html +++ b/docs/apex-3.3/dtcli/index.html @@ -131,6 +131,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> @@ -419,6 +426,8 @@ they must be part of the jar files that were deployed at application launch time <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + <a href="../compatibility/" class="btn btn-neutral float-right" title="Compatibility">Next <span class="icon icon-circle-arrow-right"></span></a> + <a href="../autometrics/" class="btn btn-neutral" title="AutoMetric API"><span class="icon icon-circle-arrow-left"></span> Previous</a> @@ -449,6 +458,8 @@ they must be part of the jar files that were deployed at application launch time <span><a href="../autometrics/" style="color: #fcfcfc;">« Previous</a></span> + <span style="margin-left: 15px"><a href="../compatibility/" style="color: #fcfcfc">Next »</a></span> + </span> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/index.html b/docs/apex-3.3/index.html index 95952ca..766851f 100644 --- a/docs/apex-3.3/index.html +++ b/docs/apex-3.3/index.html @@ -127,6 +127,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> @@ -218,5 +225,5 @@ <!-- MkDocs version : 0.15.3 -Build Date UTC : 2016-03-10 00:39:42.856605 +Build Date UTC : 2016-03-19 02:17:55.330407 --> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/mkdocs/search_index.json ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/mkdocs/search_index.json b/docs/apex-3.3/mkdocs/search_index.json index 34dd42f..cc08c8a 100644 --- a/docs/apex-3.3/mkdocs/search_index.json +++ b/docs/apex-3.3/mkdocs/search_index.json @@ -829,6 +829,76 @@ "location": "/dtcli/#examples", "text": "An example of defining a custom macro. The macro updates a running application by inserting a new operator. It takes three parameters and executes a logical plan changes. dt begin-macro add-console-output\nmacro begin-logical-plan-change\nmacro create-operator $1 com.datatorrent.lib.io.ConsoleOutputOperator\nmacro create-stream stream_$1 $2 $3 $1 in\nmacro submit Then execute the add-console-output macro like this dt add-console-output xyz opername portname This macro then expands to run the following command begin-logical-plan-change\ncreate-operator xyz com.datatorrent.lib.io.ConsoleOutputOperator\ncreate-stream stream_xyz opername portname xyz in\nsubmit Note : To perform runtime logical plan changes, like ability to add new operators,\nthey must be part of the jar files that were deployed at application launch time.", "title": "Examples" + }, + { + "location": "/compatibility/", + "text": "Apache Apex Compatibility\n\n\nPurpose\n\n\nThis document captures the compatibility goals of the Apache Apex project. The different types of compatibility between Apex releases that affect contributors, downstream projects, and end-users are enumerated. For each type of compatibility we:\n\n\n\n\ndescribe the impact on downstream projects or end-users\n\n\nwhere applicable, call out the policy adopted when incompatible changes are permitted.\n\n\n\n\nApache Apex follows \nsemantic versioning\n. Depending on the compatibility type, there may be different tools or mechanisms to ensure compatibility, for example by comparing artifacts during the build process.\n\n\nThe type of change will inform the required target version number. Given a version number MAJOR.MINOR.PATCH, increment the:\n\n\n\n\nMAJOR version when you make incompatible API changes,\n\n\nMINOR version when you add functionality in a backward-compatible manner, and\n\n\nPATCH version when you make b ackward-compatible bug fixes.\n\n\n\n\nAdditional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.\n\n\nThe overall goal is to avoid backward incompatible changes and major release upgrades. Accordingly we attempt to release new features with minor versions that are incremental to the prior release and offer our users a frictionless upgrade path. When planning contributions, please consider compatibility and release road map upfront. Specifically, certain changes that conflict with the versioning may need to be documented in JIRA and deferred until a future major release. \n\n\nCompatibility types\n\n\nJava API\n\n\nPublic API compatibility is required to ensure end-user programs and downstream projects continue to work without modification.\nThe public API consists of:\n\n\n\n\napex-core: all interfaces and classes in \napi\n and \ncommon\n modules\n\n\napex-malhar: all interfaces and classes in all modules except \ndemos\n, \ns amples\n, \nbenchmark\n \n\n\n\n\nInterfaces and classes that are part of the public API and are annotated with \ninterface stability\n are treated according to the rules defined by the annotation. \n\n\nPolicy\n\n\nChanges to the public API must follow semantic versioning. \nPublic APIs must be deprecated for at least one minor release prior to their removal in a major release.\nThe \njapicmp Maven plugin\n is used to enforce compatibility as part of the Travis pre-commit builds.\n\n\nSemantic compatibility\n\n\nThe behavior of APIs needs to remain consistent over versions, though changes for correctness may result in changes in behavior. Tests and javadocs specify the behavior. Over time, test suites should be expanded to verify compliance with the specification, effectively creating a formal specification for the subset of behaviors that can be easily tested.\n\n\nPolicy\n\n\nThe behavior of existing API cannot be modified as it would break existing user code. There are exceptio nal circumstances that may justify such changes, in which cases they should be discussed on the mailing list before implementation. Examples are bug fixes related to security issues, data corruption/consistency or to correct an unintended change from previous release that violated semantic compatibility. Such changes should be accompanied by test coverage for the exact behavior.\n\n\nREST API\n\n\nREST API compatibility corresponds to both the URLs and request/response content over the wire. REST APIs are specifically meant for stable use by clients across releases, even major releases. \n\n\nPolicy\n\n\nThe REST API is separately versioned. This is to allow for co-existence of old and new API should there be a need for backward incompatible changes in the future.\n\n\nCommand Line Interface (CLI)\n\n\nThe CLI may be used either directly via the system shell or via shell scripts. Changing the path, removing or renaming command line options, the order of arguments, or the command ret urn code and output break compatibility and may adversely affect users.\n\n\nPolicy\n\n\nCLI commands are to be deprecated (warning when used) in a prior minor release before they are removed or incompatibly modified in a subsequent major release.\n\n\nConfiguration Files\n\n\nConfiguration files are used for engine or application settings. Changes to keys and default values directly affect users and are hard to diagnose (compared to a compile error, for example).\n\n\nPolicy\n\n\nName, location, format, keys of configuration files should be deprecated in a prior minor release and can only be changed in major release. Best effort should be made to support the deprecated behavior for one more major release (not guaranteed). It is also desirable to provide the user with a migration tool.\n\n\nInternal Wire compatibility\n\n\nApex containers internally use RPC communication and netlet for the data flow. The protocols are private and user components are not exposed to it. Apex is a YARN application and automatically deployed. There is currently no situation where containers of different Apex engine versions need to be interoperable. Should such a scenario become relevant in the future, wire compatibility needs to be specified.\n\n\nPolicy\n\n\nN/A\n\n\nInternal File formats\n\n\nApex engine stores data in the file system for recovery and the data is typically obtained from serialization (from Kryo, Java etc.). Changes to internal classes may affect the ability to relaunch an application with upgraded engine code from previous state. This is currently not supported. In the future, the serialization mechanism should guarantee backward compatibility.\n\n\nPolicy\n\n\nCurrently no compatibility guarantee. User to cold-restart application on engine upgrade.\n\n\nJava Classpath\n\n\nApex applications should not bundle Hadoop dependencies or Apex engine dependencies but use the dependencies provided in the target environment to avoid conflicts. The Apex application arche type can be used to generate a compliant project. \n\n\nPolicy\n\n\nApex engine dependencies can change as per semantic versioning. Following above guidelines automatically maintains the backward compatibility based on semantic versioning of Apex.\n\n\nMaven Build Artifacts\n\n\nDownstream projects reference the Apex engine dependencies and Malhar operator libraries for application development etc. Changes to the packaging (which classes are in which jar), the groupId, artifactId and which artifacts are deployed to Maven central impact upgrades.\n\n\nPolicy\n\n\nThe artifacts that contain the classes that form the public API as specified above cannot change in patch releases and should stay compatible within a major release. Patch releases can change dependencies, but only at the patch level and following semantic versioning.\n\n\nHardware/Software Requirements\n\n\nApex depends on Apache Hadoop. The community intends to support all major Hadoop distros and current versions. Apex c urrently supports Hadoop 2.2.0 and higher and Java 7 and higher. Apex is written in Java and has been tested on Linux based Hadoop clusters. There are no additional restrictions on the hardware architecture. \n\n\nTo keep up with the latest advances in hardware, operating systems, JVMs, Hadoop and other software, new Apex releases may require higher versions. Upgrading Apex may require upgrading other dependent software components.\n\n\nPolicy\n\n\nThe JVM and Hadoop minimum version requirements are not expected to change outside major releases.", + "title": "Compatibility" + }, + { + "location": "/compatibility/#apache-apex-compatibility", + "text": "", + "title": "Apache Apex Compatibility" + }, + { + "location": "/compatibility/#purpose", + "text": "This document captures the compatibility goals of the Apache Apex project. The different types of compatibility between Apex releases that affect contributors, downstream projects, and end-users are enumerated. For each type of compatibility we: describe the impact on downstream projects or end-users where applicable, call out the policy adopted when incompatible changes are permitted. Apache Apex follows semantic versioning . Depending on the compatibility type, there may be different tools or mechanisms to ensure compatibility, for example by comparing artifacts during the build process. The type of change will inform the required target version number. Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backward-compatible manner, and PATCH version when you make backward-compatible bug fixes. Additional labels for pre-release and build metadat a are available as extensions to the MAJOR.MINOR.PATCH format. The overall goal is to avoid backward incompatible changes and major release upgrades. Accordingly we attempt to release new features with minor versions that are incremental to the prior release and offer our users a frictionless upgrade path. When planning contributions, please consider compatibility and release road map upfront. Specifically, certain changes that conflict with the versioning may need to be documented in JIRA and deferred until a future major release.", + "title": "Purpose" + }, + { + "location": "/compatibility/#compatibility-types", + "text": "", + "title": "Compatibility types" + }, + { + "location": "/compatibility/#java-api", + "text": "Public API compatibility is required to ensure end-user programs and downstream projects continue to work without modification.\nThe public API consists of: apex-core: all interfaces and classes in api and common modules apex-malhar: all interfaces and classes in all modules except demos , samples , benchmark Interfaces and classes that are part of the public API and are annotated with interface stability are treated according to the rules defined by the annotation. Policy Changes to the public API must follow semantic versioning. \nPublic APIs must be deprecated for at least one minor release prior to their removal in a major release.\nThe japicmp Maven plugin is used to enforce compatibility as part of the Travis pre-commit builds.", + "title": "Java API" + }, + { + "location": "/compatibility/#semantic-compatibility", + "text": "The behavior of APIs needs to remain consistent over versions, though changes for correctness may result in changes in behavior. Tests and javadocs specify the behavior. Over time, test suites should be expanded to verify compliance with the specification, effectively creating a formal specification for the subset of behaviors that can be easily tested. Policy The behavior of existing API cannot be modified as it would break existing user code. There are exceptional circumstances that may justify such changes, in which cases they should be discussed on the mailing list before implementation. Examples are bug fixes related to security issues, data corruption/consistency or to correct an unintended change from previous release that violated semantic compatibility. Such changes should be accompanied by test coverage for the exact behavior.", + "title": "Semantic compatibility" + }, + { + "location": "/compatibility/#rest-api", + "text": "REST API compatibility corresponds to both the URLs and request/response content over the wire. REST APIs are specifically meant for stable use by clients across releases, even major releases. Policy The REST API is separately versioned. This is to allow for co-existence of old and new API should there be a need for backward incompatible changes in the future.", + "title": "REST API" + }, + { + "location": "/compatibility/#command-line-interface-cli", + "text": "The CLI may be used either directly via the system shell or via shell scripts. Changing the path, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users. Policy CLI commands are to be deprecated (warning when used) in a prior minor release before they are removed or incompatibly modified in a subsequent major release.", + "title": "Command Line Interface (CLI)" + }, + { + "location": "/compatibility/#configuration-files", + "text": "Configuration files are used for engine or application settings. Changes to keys and default values directly affect users and are hard to diagnose (compared to a compile error, for example). Policy Name, location, format, keys of configuration files should be deprecated in a prior minor release and can only be changed in major release. Best effort should be made to support the deprecated behavior for one more major release (not guaranteed). It is also desirable to provide the user with a migration tool.", + "title": "Configuration Files" + }, + { + "location": "/compatibility/#internal-wire-compatibility", + "text": "Apex containers internally use RPC communication and netlet for the data flow. The protocols are private and user components are not exposed to it. Apex is a YARN application and automatically deployed. There is currently no situation where containers of different Apex engine versions need to be interoperable. Should such a scenario become relevant in the future, wire compatibility needs to be specified. Policy N/A", + "title": "Internal Wire compatibility" + }, + { + "location": "/compatibility/#internal-file-formats", + "text": "Apex engine stores data in the file system for recovery and the data is typically obtained from serialization (from Kryo, Java etc.). Changes to internal classes may affect the ability to relaunch an application with upgraded engine code from previous state. This is currently not supported. In the future, the serialization mechanism should guarantee backward compatibility. Policy Currently no compatibility guarantee. User to cold-restart application on engine upgrade.", + "title": "Internal File formats" + }, + { + "location": "/compatibility/#java-classpath", + "text": "Apex applications should not bundle Hadoop dependencies or Apex engine dependencies but use the dependencies provided in the target environment to avoid conflicts. The Apex application archetype can be used to generate a compliant project. Policy Apex engine dependencies can change as per semantic versioning. Following above guidelines automatically maintains the backward compatibility based on semantic versioning of Apex.", + "title": "Java Classpath" + }, + { + "location": "/compatibility/#maven-build-artifacts", + "text": "Downstream projects reference the Apex engine dependencies and Malhar operator libraries for application development etc. Changes to the packaging (which classes are in which jar), the groupId, artifactId and which artifacts are deployed to Maven central impact upgrades. Policy The artifacts that contain the classes that form the public API as specified above cannot change in patch releases and should stay compatible within a major release. Patch releases can change dependencies, but only at the patch level and following semantic versioning.", + "title": "Maven Build Artifacts" + }, + { + "location": "/compatibility/#hardwaresoftware-requirements", + "text": "Apex depends on Apache Hadoop. The community intends to support all major Hadoop distros and current versions. Apex currently supports Hadoop 2.2.0 and higher and Java 7 and higher. Apex is written in Java and has been tested on Linux based Hadoop clusters. There are no additional restrictions on the hardware architecture. To keep up with the latest advances in hardware, operating systems, JVMs, Hadoop and other software, new Apex releases may require higher versions. Upgrading Apex may require upgrading other dependent software components. Policy The JVM and Hadoop minimum version requirements are not expected to change outside major releases.", + "title": "Hardware/Software Requirements" } ] } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/operator_development/index.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/operator_development/index.html b/docs/apex-3.3/operator_development/index.html index deef122..cae1749 100644 --- a/docs/apex-3.3/operator_development/index.html +++ b/docs/apex-3.3/operator_development/index.html @@ -179,6 +179,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="../compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/search.html ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/search.html b/docs/apex-3.3/search.html index aaad72b..6fa6c09 100644 --- a/docs/apex-3.3/search.html +++ b/docs/apex-3.3/search.html @@ -116,6 +116,13 @@ </ul> <li> + <li> + <li class="toctree-l1 "> + <a class="" href="compatibility/">Compatibility</a> + + </li> +<li> + </ul> </div> http://git-wip-us.apache.org/repos/asf/incubator-apex-site/blob/2ea69c2b/docs/apex-3.3/sitemap.xml ---------------------------------------------------------------------- diff --git a/docs/apex-3.3/sitemap.xml b/docs/apex-3.3/sitemap.xml index edd34f3..262a94b 100644 --- a/docs/apex-3.3/sitemap.xml +++ b/docs/apex-3.3/sitemap.xml @@ -4,7 +4,7 @@ <url> <loc>/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> @@ -13,31 +13,31 @@ <url> <loc>/apex_development_setup/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> <url> <loc>/application_development/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> <url> <loc>/application_packages/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> <url> <loc>/operator_development/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> <url> <loc>/autometrics/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> @@ -47,10 +47,18 @@ <url> <loc>/dtcli/</loc> - <lastmod>2016-03-09</lastmod> + <lastmod>2016-03-18</lastmod> <changefreq>daily</changefreq> </url> + + <url> + <loc>/compatibility/</loc> + <lastmod>2016-03-18</lastmod> + <changefreq>daily</changefreq> + </url> + + </urlset> \ No newline at end of file
