http://git-wip-us.apache.org/repos/asf/storm-site/blob/ff14ea94/content/releases/0.10.1/Hooks.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Hooks.html b/content/releases/0.10.1/Hooks.html deleted file mode 100644 index 24009b3..0000000 --- a/content/releases/0.10.1/Hooks.html +++ /dev/null @@ -1,256 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Hooks</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 0.10.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.0/index.html">1.2.0</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - <li><a href="/releases/1.1.1/index.html">1.1.1</a></li> - - - - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - - - - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> - - - - - - - - - - <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> - - - - <li><a href="/releases/0.10.1/index.html">0.10.1</a></li> - - - - - - - - <li><a href="/releases/0.9.7/index.html">0.9.7</a></li> - - - - <li><a href="/releases/0.9.6/index.html">0.9.6</a></li> - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/02/15/storm120-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Hooks</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<p>Storm provides hooks with which you can insert custom code to run on any number of events within Storm. You create a hook by extending the <a href="javadocs/backtype/storm/hooks/BaseTaskHook.html">BaseTaskHook</a> class and overriding the appropriate method for the event you want to catch. There are two ways to register your hook:</p> - -<ol> -<li>In the open method of your spout or prepare method of your bolt using the <a href="javadocs/backtype/storm/task/TopologyContext.html#addTaskHook">TopologyContext</a> method.</li> -<li>Through the Storm configuration using the <a href="javadocs/backtype/storm/Config.html#TOPOLOGY_AUTO_TASK_HOOKS">"topology.auto.task.hooks"</a> config. These hooks are automatically registered in every spout or bolt, and are useful for doing things like integrating with a custom monitoring system.</li> -</ol> - - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> -
http://git-wip-us.apache.org/repos/asf/storm-site/blob/ff14ea94/content/releases/0.10.1/Implementation-docs.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Implementation-docs.html b/content/releases/0.10.1/Implementation-docs.html deleted file mode 100644 index f03b5a0..0000000 --- a/content/releases/0.10.1/Implementation-docs.html +++ /dev/null @@ -1,272 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Storm Internal Implementation</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 0.10.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.0/index.html">1.2.0</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - <li><a href="/releases/1.1.1/index.html">1.1.1</a></li> - - - - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - - - - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> - - - - - - - - - - <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> - - - - <li><a href="/releases/0.10.1/index.html">0.10.1</a></li> - - - - - - - - <li><a href="/releases/0.9.7/index.html">0.9.7</a></li> - - - - <li><a href="/releases/0.9.6/index.html">0.9.6</a></li> - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/02/15/storm120-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Storm Internal Implementation</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<p>This section of the wiki is dedicated to explaining how Storm is implemented. You should have a good grasp of how to use Storm before reading these sections. </p> - -<ul> -<li><a href="Structure-of-the-codebase.html">Structure of the codebase</a></li> -<li><a href="Lifecycle-of-a-topology.html">Lifecycle of a topology</a></li> -<li><a href="Message-passing-implementation.html">Message passing implementation</a></li> -<li><a href="Metrics.html">Metrics</a></li> -<li>How transactional topologies work - -<ul> -<li>subtopology for TransactionalSpout</li> -<li>how state is stored in ZK</li> -<li>subtleties around what to do when emitting batches out of order</li> -</ul></li> -<li>Unit testing - -<ul> -<li>time simulation</li> -<li>complete-topology</li> -<li>tracker clusters</li> -</ul></li> -</ul> - - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> - http://git-wip-us.apache.org/repos/asf/storm-site/blob/ff14ea94/content/releases/0.10.1/Installing-native-dependencies.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Installing-native-dependencies.html b/content/releases/0.10.1/Installing-native-dependencies.html deleted file mode 100644 index e75f98b..0000000 --- a/content/releases/0.10.1/Installing-native-dependencies.html +++ /dev/null @@ -1,281 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Apache Storm</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 0.10.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.0/index.html">1.2.0</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - <li><a href="/releases/1.1.1/index.html">1.1.1</a></li> - - - - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - - - - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> - - - - - - - - - - <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> - - - - <li><a href="/releases/0.10.1/index.html">0.10.1</a></li> - - - - - - - - <li><a href="/releases/0.9.7/index.html">0.9.7</a></li> - - - - <li><a href="/releases/0.9.6/index.html">0.9.6</a></li> - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/02/15/storm120-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title"></h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<p>The native dependencies are only needed on actual Storm clusters. When running Storm in local mode, Storm uses a pure Java messaging system so that you don't need to install native dependencies on your development machine.</p> - -<p>Installing ZeroMQ and JZMQ is usually straightforward. Sometimes, however, people run into issues with autoconf and get strange errors. If you run into any issues, please email the <a href="http://groups.google.com/group/storm-user";>Storm mailing list</a> or come get help in the #storm-user room on freenode. </p> - -<p>Storm has been tested with ZeroMQ 2.1.7, and this is the recommended ZeroMQ release that you install. You can download a ZeroMQ release <a href="http://download.zeromq.org/";>here</a>. Installing ZeroMQ should look something like this:</p> -<div class="highlight"><pre><code class="language-" data-lang="">wget http://download.zeromq.org/zeromq-2.1.7.tar.gz -tar -xzf zeromq-2.1.7.tar.gz -cd zeromq-2.1.7 -./configure -make -sudo make install -</code></pre></div> -<p>JZMQ is the Java bindings for ZeroMQ. JZMQ doesn't have any releases (we're working with them on that), so there is risk of a regression if you always install from the master branch. To prevent a regression from happening, you should instead install from <a href="http://github.com/nathanmarz/jzmq";>this fork</a> which is tested to work with Storm. Installing JZMQ should look something like this:</p> -<div class="highlight"><pre><code class="language-" data-lang="">#install jzmq -git clone https://github.com/nathanmarz/jzmq.git -cd jzmq -./autogen.sh -./configure -make -sudo make install -</code></pre></div> -<p>To get the JZMQ build to work, you may need to do one or all of the following:</p> - -<ol> -<li>Set JAVA_HOME environment variable appropriately</li> -<li>Install Java dev package (more info <a href="http://codeslinger.posterous.com/getting-zeromq-and-jzmq-running-on-mac-os-x";>here</a> for Mac OSX users)</li> -<li>Upgrade autoconf on your machine</li> -<li>Follow the instructions in <a href="http://blog.pmorelli.com/getting-zeromq-and-jzmq-running-on-mac-os-x";>this blog post</a></li> -</ol> - -<p>If you run into any errors when running <code>./configure</code>, <a href="http://stackoverflow.com/questions/3522248/how-do-i-compile-jzmq-for-zeromq-on-osx";>this thread</a> may provide a solution.</p> - - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> - http://git-wip-us.apache.org/repos/asf/storm-site/blob/ff14ea94/content/releases/0.10.1/Kestrel-and-Storm.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Kestrel-and-Storm.html b/content/releases/0.10.1/Kestrel-and-Storm.html deleted file mode 100644 index f4745c4..0000000 --- a/content/releases/0.10.1/Kestrel-and-Storm.html +++ /dev/null @@ -1,440 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Storm and Kestrel</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 0.10.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.0/index.html">1.2.0</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - <li><a href="/releases/1.1.1/index.html">1.1.1</a></li> - - - - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - - - - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> - - - - - - - - - - <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> - - - - <li><a href="/releases/0.10.1/index.html">0.10.1</a></li> - - - - - - - - <li><a href="/releases/0.9.7/index.html">0.9.7</a></li> - - - - <li><a href="/releases/0.9.6/index.html">0.9.6</a></li> - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/02/15/storm120-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Storm and Kestrel</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<p>This page explains how to use to Storm to consume items from a Kestrel cluster.</p> - -<h2 id="preliminaries">Preliminaries</h2> - -<h3 id="storm">Storm</h3> - -<p>This tutorial uses examples from the <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a> project and the <a href="http://github.com/apache/storm/blob/0.10.1/examples/storm-starter";>storm-starter</a> project. It's recommended that you clone those projects and follow along with the examples. Read <a href="Setting-up-development-environment.html">Setting up development environment</a> and <a href="Creating-a-new-Storm-project.html">Creating a new Storm project</a> to get your machine set up.</p> - -<h3 id="kestrel">Kestrel</h3> - -<p>It assumes you are able to run locally a Kestrel server as described <a href="https://github.com/nathanmarz/storm-kestrel";>here</a>.</p> - -<h2 id="kestrel-server-and-queue">Kestrel Server and Queue</h2> - -<p>A single kestrel server has a set of queues. A Kestrel queue is a very simple message queue that runs on the JVM and uses the memcache protocol (with some extensions) to talk to clients. For details, look at the implementation of the <a href="https://github.com/nathanmarz/storm-kestrel/blob/master/src/jvm/backtype/storm/spout/KestrelThriftClient.java";>KestrelThriftClient</a> class provided in <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a> project.</p> - -<p>Each queue is strictly ordered following the FIFO (first in, first out) principle. To keep up with performance items are cached in system memory; though, only the first 128MB is kept in memory. When stopping the server, the queue state is stored in a journal file.</p> - -<p>Further, details can be found <a href="https://github.com/nathanmarz/kestrel/blob/master/docs/guide.md";>here</a>.</p> - -<p>Kestrel is: -* fast -* small -* durable -* reliable</p> - -<p>For instance, Twitter uses Kestrel as the backbone of its messaging infrastructure as described <a href="http://bhavin.directi.com/notes-on-kestrel-the-open-source-twitter-queue/";>here</a>.</p> - -<h2 id="add-items-to-kestrel">Add items to Kestrel</h2> - -<p>At first, we need to have a program that can add items to a Kestrel queue. The following method takes benefit of the KestrelClient implementation in <a href="https://github.com/nathanmarz/storm-kestrel";>storm-kestrel</a>. It adds sentences into a Kestrel queue randomly chosen out of an array that holds five possible sentences.</p> -<div class="highlight"><pre><code class="language-" data-lang=""> private static void queueSentenceItems(KestrelClient kestrelClient, String queueName) - throws ParseError, IOException { - - String[] sentences = new String[] { - "the cow jumped over the moon", - "an apple a day keeps the doctor away", - "four score and seven years ago", - "snow white and the seven dwarfs", - "i am at two with nature"}; - - Random _rand = new Random(); - - for(int i=1; i<=10; i++){ - - String sentence = sentences[_rand.nextInt(sentences.length)]; - - String val = "ID " + i + " " + sentence; - - boolean queueSucess = kestrelClient.queue(queueName, val); - - System.out.println("queueSucess=" +queueSucess+ " [" + val +"]"); - } - } -</code></pre></div> -<h2 id="remove-items-from-kestrel">Remove items from Kestrel</h2> - -<p>This method dequeues items from a queue without removing them. -``` - private static void dequeueItems(KestrelClient kestrelClient, String queueName) throws IOException, ParseError - { - for(int i=1; i<=12; i++){</p> -<div class="highlight"><pre><code class="language-" data-lang=""> Item item = kestrelClient.dequeue(queueName); - - if(item==null){ - System.out.println("The queue (" + queueName + ") contains no items."); - } - else - { - byte[] data = item._data; - - String receivedVal = new String(data); - - System.out.println("receivedItem=" + receivedVal); - } - } -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -This method dequeues items from a queue and then removes them. -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">private static void dequeueAndRemoveItems(KestrelClient kestrelClient, String queueName) -throws IOException, ParseError - { - for(int i=1; i<=12; i++){ - - Item item = kestrelClient.dequeue(queueName); - - - if(item==null){ - System.out.println("The queue (" + queueName + ") contains no items."); - } - else - { - int itemID = item._id; - - - byte[] data = item._data; - - String receivedVal = new String(data); - - kestrelClient.ack(queueName, itemID); - - System.out.println("receivedItem=" + receivedVal); - } - } -} -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -## Add Items continuously to Kestrel - -This is our final program to run in order to add continuously sentence items to a queue called **sentence_queue** of a locally running Kestrel server. - -In order to stop it type a closing bracket char ']' in console and hit 'Enter'. - -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">import java.io.IOException; -import java.io.InputStream; -import java.util.Random; - -import backtype.storm.spout.KestrelClient; -import backtype.storm.spout.KestrelClient.Item; -import backtype.storm.spout.KestrelClient.ParseError; - -public class AddSentenceItemsToKestrel { - - /** - * @param args - */ - public static void main(String[] args) { - - InputStream is = System.in; - - char closing_bracket = ']'; - - int val = closing_bracket; - - boolean aux = true; - - try { - - KestrelClient kestrelClient = null; - String queueName = "sentence_queue"; - - while(aux){ - - kestrelClient = new KestrelClient("localhost",22133); - - queueSentenceItems(kestrelClient, queueName); - - kestrelClient.close(); - - Thread.sleep(1000); - - if(is.available()>0){ - if(val==is.read()) - aux=false; - } - } - } catch (IOException e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } - catch (ParseError e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } catch (InterruptedException e) { - // TODO Auto-generated catch block - e.printStackTrace(); - } - - System.out.println("end"); - - } -} -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">## Using KestrelSpout - -This topology reads sentences off of a Kestrel queue using KestrelSpout, splits the sentences into its constituent words (Bolt: SplitSentence), and then emits for each word the number of times it has seen that word before (Bolt: WordCount). How data is processed is described in detail in [Guaranteeing message processing](Guaranteeing-message-processing.html). - -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang="">TopologyBuilder builder = new TopologyBuilder(); -builder.setSpout("sentences", new KestrelSpout("localhost",22133,"sentence_queue",new StringScheme())); -builder.setBolt("split", new SplitSentence(), 10) - .shuffleGrouping("sentences"); -builder.setBolt("count", new WordCount(), 20) - .fieldsGrouping("split", new Fields("word")); -</code></pre></div><div class="highlight"><pre><code class="language-" data-lang=""> -## Execution - -At first, start your local kestrel server in production or development mode. - -Than, wait about 5 seconds in order to avoid a ConnectException. - -Now execute the program to add items to the queue and launch the Storm topology. The order in which you launch the programs is of no importance. - -If you run the topology with TOPOLOGY_DEBUG you should see tuples being emitted in the topology. -</code></pre></div> - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> - http://git-wip-us.apache.org/repos/asf/storm-site/blob/ff14ea94/content/releases/0.10.1/Lifecycle-of-a-topology.html ---------------------------------------------------------------------- diff --git a/content/releases/0.10.1/Lifecycle-of-a-topology.html b/content/releases/0.10.1/Lifecycle-of-a-topology.html deleted file mode 100644 index 004074e..0000000 --- a/content/releases/0.10.1/Lifecycle-of-a-topology.html +++ /dev/null @@ -1,367 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - - <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> - <link rel="icon" href="/favicon.ico" type="image/x-icon"> - - <title>Lifecycle of a Storm Topology</title> - - <!-- Bootstrap core CSS --> - <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> - <!-- Bootstrap theme --> - <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> - - <!-- Custom styles for this template --> - <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> - <link href="/css/style.css" rel="stylesheet"> - <link href="/assets/css/owl.theme.css" rel="stylesheet"> - <link href="/assets/css/owl.carousel.css" rel="stylesheet"> - <script type="text/javascript" src="/assets/js/jquery.min.js"></script> - <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> - <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> - <script type="text/javascript" src="/assets/js/storm.js"></script> - <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> - <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> - - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> - <![endif]--> - </head> - - - <body> - <header> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-5"> - <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> - </div> - <div class="col-md-5"> - - <h1>Version: 0.10.1</h1> - - </div> - <div class="col-md-2"> - <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> - </div> - </div> - </div> -</header> -<!--Header End--> -<!--Navigation Begin--> -<div class="navbar" role="banner"> - <div class="container-fluid"> - <div class="navbar-header"> - <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - </div> - <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> - <ul class="nav navbar-nav"> - <li><a href="/index.html" id="home">Home</a></li> - <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> - <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> - <ul class="dropdown-menu"> - - - <li><a href="/releases/2.0.0-SNAPSHOT/index.html">2.0.0-SNAPSHOT</a></li> - - - - <li><a href="/releases/1.2.0/index.html">1.2.0</a></li> - - - - <li><a href="/releases/1.1.2/index.html">1.1.2</a></li> - - - - <li><a href="/releases/1.1.1/index.html">1.1.1</a></li> - - - - <li><a href="/releases/1.1.0/index.html">1.1.0</a></li> - - - - - - <li><a href="/releases/1.0.6/index.html">1.0.6</a></li> - - - - <li><a href="/releases/1.0.4/index.html">1.0.4</a></li> - - - - - - <li><a href="/releases/1.0.3/index.html">1.0.3</a></li> - - - - - - - - - - <li><a href="/releases/0.10.2/index.html">0.10.2</a></li> - - - - <li><a href="/releases/0.10.1/index.html">0.10.1</a></li> - - - - - - - - <li><a href="/releases/0.9.7/index.html">0.9.7</a></li> - - - - <li><a href="/releases/0.9.6/index.html">0.9.6</a></li> - - - - - - - - - - - - - </ul> - </li> - <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> - <li><a href="/contribute/People.html">People</a></li> - <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> - </ul> - </li> - <li><a href="/2018/02/15/storm120-released.html" id="news">News</a></li> - </ul> - </nav> - </div> -</div> - - - - <div class="container-fluid"> - <h1 class="page-title">Lifecycle of a Storm Topology</h1> - <div class="row"> - <div class="col-md-12"> - <!-- Documentation --> - -<p class="post-meta"></p> - -<p>(<strong>NOTE</strong>: this page is based on the 0.7.1 code; many things have changed since then, including a split between tasks and executors, and a reorganization of the code under <code>storm-core/src</code> rather than <code>src/</code>.)</p> - -<p>This page explains in detail the lifecycle of a topology from running the "storm jar" command to uploading the topology to Nimbus to the supervisors starting/stopping workers to workers and tasks setting themselves up. It also explains how Nimbus monitors topologies and how topologies are shutdown when they are killed.</p> - -<p>First a couple of important notes about topologies:</p> - -<ol> -<li>The actual topology that runs is different than the topology the user specifies. The actual topology has implicit streams and an implicit "acker" bolt added to manage the acking framework (used to guarantee data processing). The implicit topology is created via the <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/common.clj#L188";>system-topology!</a> function.</li> -<li><code>system-topology!</code> is used in two places: - -<ul> -<li>when Nimbus is creating tasks for the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L316";>code</a></li> -<li>in the worker so it knows where it needs to route messages to <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L90";>code</a></li> -</ul></li> -</ol> - -<h2 id="starting-a-topology">Starting a topology</h2> - -<ul> -<li>"storm jar" command executes your class with the specified arguments. The only special thing that "storm jar" does is set the "storm.jar" environment variable for use by <code>StormSubmitter</code> later. <a href="https://github.com/apache/storm/blob/0.7.1/bin/storm#L101";>code</a></li> -<li><p>When your code uses <code>StormSubmitter.submitTopology</code>, <code>StormSubmitter</code> takes the following actions:</p> - -<ul> -<li>First, <code>StormSubmitter</code> uploads the jar if it hasn't been uploaded before. <a href="https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/StormSubmitter.java#L83";>code</a></li> -<li>Jar uploading is done via Nimbus's Thrift interface <a href="https://github.com/apache/storm/blob/0.7.1/src/storm.thrift#L200";>code</a></li> -<li><code>beginFileUpload</code> returns a path in Nimbus's inbox</li> -<li>15 kilobytes are uploaded at a time through <code>uploadChunk</code></li> -<li><code>finishFileUpload</code> is called when it's finished uploading</li> -<li>Here is Nimbus's implementation of those Thrift methods: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L694";>code</a></li> -<li>Second, <code>StormSubmitter</code> calls <code>submitTopology</code> on the Nimbus thrift interface <a href="https://github.com/apache/storm/blob/0.7.1/src/jvm/backtype/storm/StormSubmitter.java#L60";>code</a></li> -<li>The topology config is serialized using JSON (JSON is used so that writing DSL's in any language is as easy as possible)</li> -<li>Notice that the Thrift <code>submitTopology</code> call takes in the Nimbus inbox path where the jar was uploaded</li> -</ul></li> -<li><p>Nimbus receives the topology submission. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L639";>code</a></p></li> -<li><p>Nimbus normalizes the topology configuration. The main purpose of normalization is to ensure that every single task will have the same serialization registrations, which is critical for getting serialization working correctly. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L557";>code</a></p></li> -<li><p>Nimbus sets up the static state for the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L661";>code</a></p> - -<ul> -<li>Jars and configs are kept on local filesystem because they're too big for Zookeeper. The jar and configs are copied into the path {nimbus local dir}/stormdist/{topology id}</li> -<li><code>setup-storm-static</code> writes task -> component mapping into ZK</li> -<li><code>setup-heartbeats</code> creates a ZK "directory" in which tasks can heartbeat</li> -</ul></li> -<li><p>Nimbus calls <code>mk-assignment</code> to assign tasks to machines <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L458";>code</a></p> - -<ul> -<li>Assignment record definition is here: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/common.clj#L25";>code</a></li> -<li>Assignment contains: - -<ul> -<li><code>master-code-dir</code>: used by supervisors to download the correct jars/configs for the topology from Nimbus</li> -<li><code>task->node+port</code>: Map from a task id to the worker that task should be running on. (A worker is identified by a node/port pair)</li> -<li><code>node->host</code>: A map from node id to hostname. This is used so workers know which machines to connect to to communicate with other workers. Node ids are used to identify supervisors so that multiple supervisors can be run on one machine. One place this is done is with Mesos integration.</li> -<li><code>task->start-time-secs</code>: Contains a map from task id to the timestamp at which Nimbus launched that task. This is used by Nimbus when monitoring topologies, as tasks are given a longer timeout to heartbeat when they're first launched (the launch timeout is configured by "nimbus.task.launch.secs" config)</li> -</ul></li> -</ul></li> -<li><p>Once topologies are assigned, they're initially in a deactivated mode. <code>start-storm</code> writes data into Zookeeper so that the cluster knows the topology is active and can start emitting tuples from spouts. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L504";>code</a></p></li> -<li><p>TODO cluster state diagram (show all nodes and what's kept everywhere)</p></li> -<li><p>Supervisor runs two functions in the background:</p> - -<ul> -<li><code>synchronize-supervisor</code>: This is called whenever assignments in Zookeeper change and also every 10 seconds. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L241";>code</a> - -<ul> -<li>Downloads code from Nimbus for topologies assigned to this machine for which it doesn't have the code yet. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L258";>code</a></li> -<li>Writes into local filesystem what this node is supposed to be running. It writes a map from port -> LocalAssignment. LocalAssignment contains a topology id as well as the list of task ids for that worker. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L13";>code</a></li> -</ul></li> -<li><code>sync-processes</code>: Reads from the LFS what <code>synchronize-supervisor</code> wrote and compares that to what's actually running on the machine. It then starts/stops worker processes as necessary to synchronize. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/supervisor.clj#L177";>code</a></li> -</ul></li> -<li><p>Worker processes start up through the <code>mk-worker</code> function <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L67";>code</a></p> - -<ul> -<li>Worker connects to other workers and starts a thread to monitor for changes. So if a worker gets reassigned, the worker will automatically reconnect to the other worker's new location. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L123";>code</a></li> -<li>Monitors whether a topology is active or not and stores that state in the <code>storm-active-atom</code> variable. This variable is used by tasks to determine whether or not to call <code>nextTuple</code> on the spouts. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L155";>code</a></li> -<li>The worker launches the actual tasks as threads within it <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/worker.clj#L178";>code</a></li> -</ul></li> -<li><p>Tasks are set up through the <code>mk-task</code> function <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L160";>code</a></p> - -<ul> -<li>Tasks set up routing function which takes in a stream and an output tuple and returns a list of task ids to send the tuple to <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L207";>code</a> (there's also a 3-arity version used for direct streams)</li> -<li>Tasks set up the spout-specific or bolt-specific code with <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/task.clj#L241";>code</a></li> -</ul></li> -</ul> - -<h2 id="topology-monitoring">Topology Monitoring</h2> - -<ul> -<li>Nimbus monitors the topology during its lifetime - -<ul> -<li>Schedules recurring task on the timer thread to check the topologies <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L623";>code</a></li> -<li>Nimbus's behavior is represented as a finite state machine <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L98";>code</a></li> -<li>The "monitor" event is called on a topology every "nimbus.monitor.freq.secs", which calls <code>reassign-topology</code> through <code>reassign-transition</code> <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L497";>code</a></li> -<li><code>reassign-topology</code> calls <code>mk-assignments</code>, the same function used to assign the topology the first time. <code>mk-assignments</code> is also capable of incrementally updating a topology - -<ul> -<li><code>mk-assignments</code> checks heartbeats and reassigns workers as necessary</li> -<li>Any reassignments change the state in ZK, which will trigger supervisors to synchronize and start/stop workers</li> -</ul></li> -</ul></li> -</ul> - -<h2 id="killing-a-topology">Killing a topology</h2> - -<ul> -<li>"storm kill" command runs this code which just calls the Nimbus Thrift interface to kill the topology: <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/command/kill_topology.clj";>code</a></li> -<li>Nimbus receives the kill command <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L671";>code</a></li> -<li>Nimbus applies the "kill" transition to the topology <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L676";>code</a></li> -<li>The kill transition function changes the status of the topology to "killed" and schedules the "remove" event to run "wait time seconds" in the future. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L63";>code</a> - -<ul> -<li>The wait time defaults to the topology message timeout but can be overridden with the -w flag in the "storm kill" command</li> -<li>This causes the topology to be deactivated for the wait time before its actually shut down. This gives the topology a chance to finish processing what it's currently processing before shutting down the workers</li> -<li>Changing the status during the kill transition ensures that the kill protocol is fault-tolerant to Nimbus crashing. On startup, if the status of the topology is "killed", Nimbus schedules the remove event to run "wait time seconds" in the future <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L111";>code</a></li> -</ul></li> -<li>Removing a topology cleans out the assignment and static information from ZK <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L116";>code</a></li> -<li>A separate cleanup thread runs the <code>do-cleanup</code> function which will clean up the heartbeat dir and the jars/configs stored locally. <a href="https://github.com/apache/storm/blob/0.7.1/src/clj/backtype/storm/daemon/nimbus.clj#L577";>code</a></li> -</ul> - - - - </div> - </div> - </div> -<footer> - <div class="container-fluid"> - <div class="row"> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Meetups</h5> - <ul class="latest-news"> - - <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> - - <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> - - <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/";>NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> - - <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing";>Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> - - <li><a href="http://www.meetup.com/Boston-Storm-Users/";>Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> - - <li><a href="http://www.meetup.com/storm-london";>London Storm User Group</a> <span class="small">(London, UK)</span></li> - - <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>About Storm</h5> - <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>First Look</h5> - <ul class="footer-list"> - <li><a href="/releases/current/Rationale.html">Rationale</a></li> - <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> - <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> - <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> - </ul> - </div> - </div> - <div class="col-md-3"> - <div class="footer-widget"> - <h5>Documentation</h5> - <ul class="footer-list"> - <li><a href="/releases/current/index.html">Index</a></li> - <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> - <li><a href="/releases/current/FAQ.html">FAQ</a></li> - </ul> - </div> - </div> - </div> - <hr/> - <div class="row"> - <div class="col-md-12"> - <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. - <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. - <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> - </div> - </div> - </div> -</footer> -<!--Footer End--> -<!-- Scroll to top --> -<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> - -</body> - -</html> -
