[GEARPUMP-213] build docs with MkDocs build docs with [MkDocs](http://www.mkdocs.org/) instead of Jekyll. MkDocs has a "readthedocs" theme which supports two-level table of contents sidebar. It's easier for users to navigate through the documentation. Here is a snapshot.
 Author: manuzhang <[email protected]> Closes #88 from manuzhang/new_doc. Project: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/commit/761e04c6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/tree/761e04c6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-gearpump/diff/761e04c6 Branch: refs/heads/master Commit: 761e04c6ff0a48d4ded101ff0826b8ade01dd99e Parents: a034e74 Author: manuzhang <[email protected]> Authored: Thu Sep 29 20:06:03 2016 +0800 Committer: manuzhang <[email protected]> Committed: Thu Sep 29 20:06:03 2016 +0800 ---------------------------------------------------------------------- .gitignore | 2 +- docs/README.md | 37 +- docs/_config.yml | 21 - docs/_layouts/404.html | 159 --- docs/_layouts/global.html | 177 --- docs/_plugins/kramdown_pygments.rb | 93 -- docs/basic-concepts.md | 53 - docs/build_doc.sh | 47 +- docs/coding-style.md | 5 - docs/commandline.md | 91 -- docs/css/api-docs.css | 35 - docs/css/api-javadocs.css | 52 - docs/css/bootstrap-3.3.5.min.css | 5 - docs/css/main.css | 127 -- docs/css/pygments-default.css | 76 -- docs/deployment-configuration.md | 92 -- docs/deployment-docker.md | 10 - docs/deployment-ha.md | 80 -- docs/deployment-local.md | 41 - docs/deployment-msg-delivery.md | 66 -- docs/deployment-resource-isolation.md | 120 -- docs/deployment-security.md | 89 -- docs/deployment-standalone.md | 66 -- docs/deployment-ui-authentication.md | 374 ------ docs/deployment-yarn.md | 140 --- docs/dev-connectors.md | 249 ---- docs/dev-custom-serializer.md | 142 --- docs/dev-ide-setup.md | 33 - docs/dev-non-streaming-example.md | 140 --- docs/dev-rest-api.md | 1103 ------------------ docs/dev-storm.md | 222 ---- docs/dev-write-1st-app.md | 411 ------- docs/docs/api/java.md | 1 + docs/docs/api/scala.md | 1 + .../docs/deployment/deployment-configuration.md | 84 ++ docs/docs/deployment/deployment-docker.md | 5 + docs/docs/deployment/deployment-ha.md | 75 ++ docs/docs/deployment/deployment-local.md | 34 + docs/docs/deployment/deployment-msg-delivery.md | 60 + .../deployment/deployment-resource-isolation.md | 112 ++ docs/docs/deployment/deployment-security.md | 80 ++ docs/docs/deployment/deployment-standalone.md | 59 + .../deployment/deployment-ui-authentication.md | 290 +++++ docs/docs/deployment/deployment-yarn.md | 135 +++ .../deployment/get-gearpump-distribution.md | 83 ++ docs/docs/deployment/hardware-requirement.md | 30 + docs/docs/dev/dev-connectors.md | 237 ++++ docs/docs/dev/dev-custom-serializer.md | 137 +++ docs/docs/dev/dev-ide-setup.md | 29 + docs/docs/dev/dev-non-streaming-example.md | 133 +++ docs/docs/dev/dev-rest-api.md | 1083 +++++++++++++++++ docs/docs/dev/dev-storm.md | 214 ++++ docs/docs/dev/dev-write-1st-app.md | 370 ++++++ docs/docs/img/actor_hierarchy.png | Bin 0 -> 109855 bytes docs/docs/img/checkpoint_equation.png | Bin 0 -> 1663 bytes docs/docs/img/checkpoint_interval_equation.png | Bin 0 -> 16637 bytes docs/docs/img/checkpointing.png | Bin 0 -> 104471 bytes docs/docs/img/checkpointing_interval.png | Bin 0 -> 44284 bytes docs/docs/img/clock.png | Bin 0 -> 33147 bytes docs/docs/img/dag.png | Bin 0 -> 18263 bytes docs/docs/img/dashboard.gif | Bin 0 -> 152314 bytes docs/docs/img/dashboard.png | Bin 0 -> 29434 bytes docs/docs/img/dashboard_3.png | Bin 0 -> 42684 bytes docs/docs/img/download.jpg | Bin 0 -> 3999 bytes docs/docs/img/dynamic.png | Bin 0 -> 40091 bytes docs/docs/img/exact.png | Bin 0 -> 104471 bytes docs/docs/img/failures.png | Bin 0 -> 95078 bytes docs/docs/img/flow_control.png | Bin 0 -> 61928 bytes docs/docs/img/flowcontrol.png | Bin 0 -> 61928 bytes docs/docs/img/ha.png | Bin 0 -> 47152 bytes docs/docs/img/kafka_wordcount.png | Bin 0 -> 14520 bytes docs/docs/img/layout.png | Bin 0 -> 126947 bytes docs/docs/img/logo.png | Bin 0 -> 2053 bytes docs/docs/img/logo.svg | 71 ++ docs/docs/img/logo2.png | Bin 0 -> 7970 bytes docs/docs/img/messageLoss.png | Bin 0 -> 37166 bytes docs/docs/img/netty_transport.png | Bin 0 -> 65010 bytes docs/docs/img/replay.png | Bin 0 -> 37255 bytes docs/docs/img/shuffle.png | Bin 0 -> 23550 bytes docs/docs/img/storm_gearpump_cluster.png | Bin 0 -> 30930 bytes docs/docs/img/storm_gearpump_dag.png | Bin 0 -> 54000 bytes docs/docs/img/submit.png | Bin 0 -> 32954 bytes docs/docs/img/submit2.png | Bin 0 -> 51933 bytes docs/docs/img/through_vs_message_size.png | Bin 0 -> 20965 bytes docs/docs/index.md | 35 + docs/docs/introduction/basic-concepts.md | 46 + docs/docs/introduction/commandline.md | 84 ++ docs/docs/introduction/features.md | 67 ++ docs/docs/introduction/gearpump-internals.md | 228 ++++ docs/docs/introduction/message-delivery.md | 47 + docs/docs/introduction/performance-report.md | 34 + .../introduction/submit-your-1st-application.md | 39 + docs/faq.md | 39 - docs/features.md | 74 -- docs/fonts/glyphicons-halflings-regular.eot | Bin 20127 -> 0 bytes docs/fonts/glyphicons-halflings-regular.svg | 288 ----- docs/fonts/glyphicons-halflings-regular.ttf | Bin 45404 -> 0 bytes docs/fonts/glyphicons-halflings-regular.woff | Bin 23424 -> 0 bytes docs/fonts/glyphicons-halflings-regular.woff2 | Bin 18028 -> 0 bytes docs/gearpump-internals.md | 234 ---- docs/get-gearpump-distribution.md | 90 -- docs/hardware-requirement.md | 37 - docs/img/actor_hierarchy.png | Bin 109855 -> 0 bytes docs/img/checkpoint_equation.png | Bin 1663 -> 0 bytes docs/img/checkpoint_interval_equation.png | Bin 16637 -> 0 bytes docs/img/checkpointing.png | Bin 104471 -> 0 bytes docs/img/checkpointing_interval.png | Bin 44284 -> 0 bytes docs/img/clock.png | Bin 33147 -> 0 bytes docs/img/dag.png | Bin 18263 -> 0 bytes docs/img/dashboard.gif | Bin 152314 -> 0 bytes docs/img/dashboard.png | Bin 29434 -> 0 bytes docs/img/dashboard_3.png | Bin 42684 -> 0 bytes docs/img/download.jpg | Bin 3999 -> 0 bytes docs/img/dynamic.png | Bin 40091 -> 0 bytes docs/img/exact.png | Bin 104471 -> 0 bytes docs/img/failures.png | Bin 95078 -> 0 bytes docs/img/flow_control.png | Bin 61928 -> 0 bytes docs/img/flowcontrol.png | Bin 61928 -> 0 bytes docs/img/ha.png | Bin 47152 -> 0 bytes docs/img/kafka_wordcount.png | Bin 14520 -> 0 bytes docs/img/layout.png | Bin 126947 -> 0 bytes docs/img/logo.png | Bin 2053 -> 0 bytes docs/img/logo.svg | 71 -- docs/img/logo2.png | Bin 7970 -> 0 bytes docs/img/messageLoss.png | Bin 37166 -> 0 bytes docs/img/netty_transport.png | Bin 65010 -> 0 bytes docs/img/replay.png | Bin 37255 -> 0 bytes docs/img/shuffle.png | Bin 23550 -> 0 bytes docs/img/storm_gearpump_cluster.png | Bin 30930 -> 0 bytes docs/img/storm_gearpump_dag.png | Bin 54000 -> 0 bytes docs/img/submit.png | Bin 32954 -> 0 bytes docs/img/submit2.png | Bin 51933 -> 0 bytes docs/img/through_vs_message_size.png | Bin 20965 -> 0 bytes docs/index.md | 37 - docs/js/api-docs.js | 43 - docs/js/api-javadocs.js | 60 - docs/js/main.js | 108 -- docs/js/vendor/anchor-1.1.1.min.js | 6 - docs/js/vendor/bootstrap-3.3.5.min.js | 7 - docs/js/vendor/jquery-2.1.4.min.js | 5 - .../vendor/modernizr-2.6.1-respond-1.1.0.min.js | 11 - docs/maven-setting.md | 166 --- docs/message-delivery.md | 52 - docs/mkdocs.yml | 50 + docs/performance-report.md | 41 - docs/submit-your-1st-application.md | 46 - docs/version.yml | 6 + 147 files changed, 3997 insertions(+), 5735 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/.gitignore ---------------------------------------------------------------------- diff --git a/.gitignore b/.gitignore index 37071de..6c0954e 100644 --- a/.gitignore +++ b/.gitignore @@ -5,7 +5,7 @@ logs/ # Scala *.class *.log -_site/ +site/ .cache/ .history/ .lib/ http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/README.md ---------------------------------------------------------------------- diff --git a/docs/README.md b/docs/README.md index e820f45..9cd6293 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,21 +10,15 @@ You need to install ruby and ruby-dev first. On Ubuntu, you ca run command like sudo apt-get install ruby sudo apt-get install ruby-dev sudo apt-get install python-setuptools + sudo apt-get install pip We use Markdown to write and Jekyll to translate the documentation to static HTML. You can install all needed software via: - sudo gem install jekyll - sudo gem install kramdown + sudo pip install mkdocs sudo gem install html-proofer - sudo gem install pygments.rb - sudo easy_install Pygments + sudo gem install mustache -For Mac OSX you may need to do `sudo gem install -n /usr/local/bin jekyll` if you see the following error: -``` -ERROR: While executing gem ... (Errno::EPERM) - Operation not permitted - /usr/bin/listen -``` If you are using Mac OSX 10.11+ (El Capitan), you will need to execute following command: ``` @@ -45,9 +39,6 @@ module Library LIBCURL = '/usr/lib/libcurl.dylib' ``` -Kramdown is needed for Markdown processing and the Python based Pygments is used for syntax -highlighting. - # How to Build Command `./build_doc.sh` can be used to create a full document folder under site/. @@ -55,22 +46,8 @@ Command `./build_doc.sh` can be used to create a full document folder under site # How to contribute The documentation pages are written in -[Markdown](http://daringfireball.net/projects/markdown/syntax). It is possible to use the -[GitHub flavored syntax](http://github.github.com/github-flavored-markdown) and intermix plain html. - -In addition to Markdown, every page contains a Jekyll front matter, which specifies the title of the -page and the layout to use. The title is used as the top-level heading for the page. - - --- - title: "Title of the Page" - --- - -Furthermore, you can access variables found in `docs/_config.yml` as follows: - - {{ site.NAME }} +[Markdown](http://daringfireball.net/projects/markdown/syntax). -This will be replaced with the value of the variable called `NAME` when generating -the docs. All documents are structured with headings. From these heading, a page outline is automatically generated for each page. @@ -89,7 +66,7 @@ how a headline looks. # How to Test -Command `jekyll build` can be used to make a test build. +Command `mkdocs build` can be used to make a test build. -Command `jekyll serve --watch` can be used for debug purpose. Jekyll will start a web server at -`localhost:4000` and watch the docs directory for updates. Use this mode to experiment commits and check changes locally. +Command `mkdocs serve` can be used for debug purpose. Mkdocs will start a web server at +`localhost:8000`. Use this mode to experiment commits and check changes locally. http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/_config.yml ---------------------------------------------------------------------- diff --git a/docs/_config.yml b/docs/_config.yml deleted file mode 100644 index a8f4a2e..0000000 --- a/docs/_config.yml +++ /dev/null @@ -1,21 +0,0 @@ -highlighter: pygments -markdown: KramdownPygments - -# For some reason kramdown seems to behave differently on different -# OS/packages wrt encoding. So we hard code this config. -kramdown: - entity_output: numeric - input: GFM - -include: - - _static - - _modules - -exclude: - - README.md - - build_doc.sh - -# These allow the documentation to be updated with newer releases of Gearpump -GEARPUMP_VERSION: 0.8.2-SNAPSHOT -SCALA_BINARY_VERSION: "2.11" -SCALA_VERSION: "2.11.8" http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/_layouts/404.html ---------------------------------------------------------------------- diff --git a/docs/_layouts/404.html b/docs/_layouts/404.html deleted file mode 100644 index 1bfbe21..0000000 --- a/docs/_layouts/404.html +++ /dev/null @@ -1,159 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <meta charset="utf-8"> - <title>Page Not Found :(</title> - <style> - ::-moz-selection { - background: #b3d4fc; - text-shadow: none; - } - - ::selection { - background: #b3d4fc; - text-shadow: none; - } - - html { - padding: 30px 10px; - font-size: 20px; - line-height: 1.4; - color: #737373; - background: #f0f0f0; - -webkit-text-size-adjust: 100%; - -ms-text-size-adjust: 100%; - } - - html, - input { - font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; - } - - body { - max-width: 500px; - _width: 500px; - padding: 30px 20px 50px; - border: 1px solid #b3b3b3; - border-radius: 4px; - margin: 0 auto; - box-shadow: 0 1px 10px #a7a7a7, inset 0 1px 0 #fff; - background: #fcfcfc; - } - - h1 { - margin: 0 10px; - font-size: 50px; - text-align: center; - } - - h1 span { - color: #bbb; - } - - h3 { - margin: 1.5em 0 0.5em; - } - - p { - margin: 1em 0; - } - - ul { - padding: 0 0 0 40px; - margin: 1em 0; - } - - .container { - max-width: 380px; - _width: 380px; - margin: 0 auto; - } - - /* google search */ - - #goog-fixurl ul { - list-style: none; - padding: 0; - margin: 0; - } - - #goog-fixurl form { - margin: 0; - } - - #goog-wm-qt, - #goog-wm-sb { - border: 1px solid #bbb; - font-size: 16px; - line-height: normal; - vertical-align: top; - color: #444; - border-radius: 2px; - } - - #goog-wm-qt { - width: 220px; - height: 20px; - padding: 5px; - margin: 5px 10px 0 0; - box-shadow: inset 0 1px 1px #ccc; - } - - #goog-wm-sb { - display: inline-block; - height: 32px; - padding: 0 10px; - margin: 5px 0 0; - white-space: nowrap; - cursor: pointer; - background-color: #f5f5f5; - background-image: -webkit-linear-gradient(rgba(255, 255, 255, 0), #f1f1f1); - background-image: -moz-linear-gradient(rgba(255, 255, 255, 0), #f1f1f1); - background-image: -ms-linear-gradient(rgba(255, 255, 255, 0), #f1f1f1); - background-image: -o-linear-gradient(rgba(255, 255, 255, 0), #f1f1f1); - -webkit-appearance: none; - -moz-appearance: none; - appearance: none; - *overflow: visible; - *display: inline; - *zoom: 1; - } - - #goog-wm-sb:hover, - #goog-wm-sb:focus { - border-color: #aaa; - box-shadow: 0 1px 1px rgba(0, 0, 0, 0.1); - background-color: #f8f8f8; - } - - #goog-wm-qt:hover, - #goog-wm-qt:focus { - border-color: #105cb6; - outline: 0; - color: #222; - } - - input::-moz-focus-inner { - padding: 0; - border: 0; - } - </style> -</head> -<body> -<div class="container"> - <h1>Not found <span>:(</span></h1> - - <p>Sorry, but the page you were trying to view does not exist.</p> - - <p>It looks like this was the result of either:</p> - <ul> - <li>a mistyped address</li> - <li>an out-of-date link</li> - </ul> - <script> - var GOOG_FIXURL_LANG = (navigator.language || '').slice(0, 2), GOOG_FIXURL_SITE = location.host; - </script> - <script src="http://linkhelp.clients.google.com/tbproxy/lh/wm/fixurl.js";></script> -</div> -</body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/_layouts/global.html ---------------------------------------------------------------------- diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html deleted file mode 100644 index c49e42f..0000000 --- a/docs/_layouts/global.html +++ /dev/null @@ -1,177 +0,0 @@ -<!DOCTYPE html> -<!--[if lt IE 7]> -<html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> -<!--[if IE 7]> -<html class="no-js lt-ie9 lt-ie8"> <![endif]--> -<!--[if IE 8]> -<html class="no-js lt-ie9"> <![endif]--> -<!--[if gt IE 8]><!--> -<html class="no-js"> <!--<![endif]--> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> - <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1"/> - <title>{{ page.title }} - Gearpump {{site.GEARPUMP_VERSION}} Documentation</title> - {% if page.description %} - <meta name="description" - content="{{page.description | replace: 'GEARPUMP_VERSION', site.GEARPUMP_VERSION}}"> - {% endif %} - - {% if page.redirect %} - <meta http-equiv="refresh" content="0; url={{page.redirect}}"> - <link rel="canonical" href="{{page.redirect}}"/> - {% endif %} - - <link rel="stylesheet" href="css/bootstrap-3.3.5.min.css"> - <style> - body { - padding-top: 60px; - padding-bottom: 40px; - } - </style> - <link rel="stylesheet" href="css/main.css"> - <link rel="stylesheet" href="css/pygments-default.css"> - <script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script> -</head> -<body> -<!--[if lt IE 7]> -<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/";>Upgrade - your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true";>install - Google Chrome Frame</a> to better experience this site.</p> -<![endif]--> - -<div class="navbar navbar-inverse navbar-fixed-top" id="topbar"> - <div class="container"> - <div class="navbar-header"> - <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" - data-target="#navbar" aria-expanded="false" aria-controls="navbar"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a class="navbar-brand" href="http://gearpump.apache.org";>Gearpump - <span class="label label-primary" style="font-size: .6em">{{site.GEARPUMP_VERSION}}</span> - </a> - </div> - <div id="navbar" class="collapse navbar-collapse"> - <ul class="nav navbar-nav"> - <li><a href="index.html">Overview</a></li> - - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Introduction<b - class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="submit-your-1st-application.html">Submit Your 1st Application</a></li> - <li><a href="commandline.html">Client Command Line</a></li> - <li class="divider"></li> - <li><a href="basic-concepts.html">Basic Concepts</a></li> - <li><a href="features.html">Technical Highlights</a></li> - <li><a href="message-delivery.html">Reliable Message Delivery</a></li> - <li><a href="performance-report.html">Performance</a></li> - <li><a href="gearpump-internals.html">Gearpump Internals</a></li> - </ul> - </li> - - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a> - <ul class="dropdown-menu"> - <li class="dropdown-header">Deployment</li> - <li><a href="deployment-local.html">Local Mode</a> - <li> - <li><a href="deployment-standalone.html">Standalone Mode</a></li> - <li><a href="deployment-yarn.html">YARN Mode</a></li> - <li><a href="deployment-docker.html">Docker Mode</a> - <li> - <li class="divider"></li> - <li><a href="deployment-ui-authentication.html">UI Authentication</a></li> - <li><a href="deployment-ha.html">High Availability</a></li> - <li><a href="deployment-msg-delivery.html">Reliable Message Delivery</a></li> - <li><a href="deployment-configuration.html">Configuration</a></li> - <li><a href="deployment-resource-isolation.html">Resource Isolation</a></li> - <li class="divider"></li> - <li><a href="deployment-security.html">YARN Security Guide</a></li> - </ul> - </li> - - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guide<b - class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="dev-write-1st-app.html">Write Your 1st App</a></li> - <li><a href="dev-custom-serializer.html">Customized Message Passing</a></li> - <li class="divider"></li> - <li><a href="api/scala/index.html">Scala API</a></li> - <li><a href="api/java/index.html">Java API</a></li> - <li><a href="dev-rest-api.html">RESTful API</a></li> - <li class="divider"></li> - <li><a href="dev-connectors.html">Gearpump Connectors</a></li> - <li class="divider"></li> - <li><a href="dev-storm.html">Storm Compatibility</a></li> - <!-- - <li><a href="dev-samoa.html">Samoa Compatibility</a></li> - <li class="divider"></li> - <li><a href="dev-iot.html">Gearpump with IoT</a></li> - --> - </ul> - </li> - - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a> - <ul class="dropdown-menu"> - <li><a href="faq.html">FAQ</a></li> - </ul> - </li> - </ul> - </div> - </div> -</div> - -<div class="container" id="content"> - {% if page.displayTitle %} - <h1 class="title">{{ page.displayTitle }}</h1> - {% else %} - <h1 class="title">{{ page.title }}</h1> - {% endif %} - - {{ content }} - -</div> -<!-- /container --> - -<script src="js/vendor/jquery-2.1.4.min.js"></script> -<script src="js/vendor/bootstrap-3.3.5.min.js"></script> -<script src="js/vendor/anchor-1.1.1.min.js"></script> -<script src="js/main.js"></script> - -<!-- MathJax Section --> -<script type="text/x-mathjax-config"> - MathJax.Hub.Config({ - TeX: { equationNumbers: { autoNumber: "AMS" } } - }); - -</script> -<script> - // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS. - // We could use "//cdn.mathjax...", but that won't support "file://". - (function (d, script) { - script = d.createElement('script'); - script.type = 'text/javascript'; - script.async = true; - script.onload = function () { - MathJax.Hub.Config({ - tex2jax: { - inlineMath: [["$", "$"], ["\\\\(", "\\\\)"]], - displayMath: [["$$", "$$"], ["\\[", "\\]"]], - processEscapes: true, - skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] - } - }); - }; - script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') + - 'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'; - d.getElementsByTagName('head')[0].appendChild(script); - }(document)); -</script> -</body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/_plugins/kramdown_pygments.rb ---------------------------------------------------------------------- diff --git a/docs/_plugins/kramdown_pygments.rb b/docs/_plugins/kramdown_pygments.rb deleted file mode 100644 index 3f5067d..0000000 --- a/docs/_plugins/kramdown_pygments.rb +++ /dev/null @@ -1,93 +0,0 @@ -# We define the an additional option for the kramdown parser to look for -module Kramdown - module Options - define(:kramdown_default_lang, Symbol, nil, <<EOF) -Sets the default language for highlighting code blocks - -If no language is set for a code block, the default language is used -instead. The value has to be one of the languages supported by pygments -or nil if no default language should be used. - -Default: nil -Used by: PygmentsHtml converter -EOF - end -end - -# This class is a plugin for kramdown, to make it use pygments instead of coderay -# It has nothing to do with Jekyll, it is simply used by the custom converter below -module Kramdown - module Converter - class PygmentsHtml < Html - - begin - require 'pygments' - rescue LoadError - STDERR.puts 'You are missing a library required for syntax highlighting. Please run:' - STDERR.puts ' $ [sudo] gem install pygments' - raise FatalException.new("Missing dependency: Pygments") - end - - def convert_codeblock(el, indent) - attr = el.attr.dup - lang = extract_code_language!(attr) || @options[:kramdown_default_lang] - code = pygmentize(el.value, lang) - code_attr = {} - code_attr['class'] = "language-#{lang}" if lang - "#{' '*indent}<div class=\"highlight\"><pre#{html_attributes(attr)}><code#{html_attributes(code_attr)}>#{code}</code></pre></div>\n" - end - - def convert_codespan(el, indent) - attr = el.attr.dup - lang = extract_code_language!(attr) || @options[:kramdown_default_lang] - code = pygmentize(el.value, lang) - if lang - attr['class'] = "highlight" - if attr.has_key?('class') - attr['class'] += " language-#{lang}" - else - attr['class'] = "language-#{lang}" - end - end - "<code#{html_attributes(attr)}>#{code}</code>" - end - - def pygmentize(code, lang) - if lang - Pygments.highlight(code, - :lexer => lang, - :options => { :startinline => true, :encoding => 'utf-8', :nowrap => true }) - else - escape_html(code) - end - end - end - end -end - -# This class is the actual custom Jekyll converter. -class Jekyll::Converters::Markdown::KramdownPygments - - def initialize(config) - require 'kramdown' - @config = config - rescue LoadError - STDERR.puts 'You are missing a library required for Markdown. Please run:' - STDERR.puts ' $ [sudo] gem install kramdown' - raise FatalException.new("Missing dependency: kramdown") - end - - def convert(content) - html = Kramdown::Document.new(content, { - :auto_ids => @config['kramdown']['auto_ids'], - :footnote_nr => @config['kramdown']['footnote_nr'], - :entity_output => @config['kramdown']['entity_output'], - :toc_levels => @config['kramdown']['toc_levels'], - :smart_quotes => @config['kramdown']['smart_quotes'], - :kramdown_default_lang => @config['kramdown']['default_lang'], - :input => @config['kramdown']['input'], - :hard_wrap => @config['kramdown']['hard_wrap'] - }).to_pygments_html - return html; - end -end http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/basic-concepts.md ---------------------------------------------------------------------- diff --git a/docs/basic-concepts.md b/docs/basic-concepts.md deleted file mode 100644 index e9c4b86..0000000 --- a/docs/basic-concepts.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -layout: global -displayTitle: Gearpump Basic Concepts -title: Gearpump Basic Concepts -description: Gearpump Basic Concepts ---- - -### System timestamp and Application timestamp - -System timestamp denotes the time of backend cluster system. Application timestamp denotes the time at which message is generated. For example, for IoT edge device, the timestamp at which field sensor device creates a message is type of application timestamp, while the timestamp at which that message get received by the backend is type of system time. - -### Master, and Worker - -Gearpump follow master slave architecture. Every cluster contains one or more Master node, and several worker nodes. Worker node is responsible to manage local resources on single machine, and Master node is responsible to manage global resources of the whole cluster. - - - -### Application - -Application is what we want to parallel and run on the cluster. There are different application types, for example MapReduce application and streaming application are different application types. Gearpump natively supports Streaming Application types, it also contains several templates to help user to create custom application types, like distributedShell. - -### AppMaster and Executor - -In runtime, every application instance is represented by a single AppMaster and a list of Executors. AppMaster represents the command and controls center of the Application instance. It communicates with user, master, worker, and executor to get the job done. Each executor is a parallel unit for distributed application. Typically AppMaster and Executor will be started as JVM processes on worker nodes. - -### Application Submission Flow - -When user submits an application to Master, Master will first find an available worker to start the AppMaster. After AppMaster is started, AppMaster will request Master for more resources (worker) to start executors. The Executor now is only an empty container. After the executors are started, the AppMaster will then distribute real computation tasks to the executor and run them in parallel way. - -To submit an application, a Gearpump client specifies a computation defined within a DAG and submits this to an active master. The SubmitApplication message is sent to the Master who then forwards this to an AppManager. - - -Figure: User Submit Application - -The AppManager locates an available worker and launches an AppMaster in a sub-process JVM of the worker. The AppMaster will then negotiate with the Master for Resource allocation in order to distribute the DAG as defined within the Application. The allocated workers will then launch Executors (new JVMs). - - -Figure: Launch Executors and Tasks - -### Streaming Topology, Processor, and Task - -For streaming application type, each application contains a topology, which is a DAG (directed acyclic graph) to describe the data flow. Each node in the DAG is a processor. For example, for word count it contains two processors, Split and Sum. The Split processor splits a line to a list of words, and then the Sum processor summarize the frequency of each word. -An application is a DAG of processors. Each processor handles messages. - - -Figure: Processor DAG - -### Streaming Task and Partitioner - -For streaming application type, Task is the minimum unit of parallelism. In runtime, each Processor is paralleled to a list of tasks, with different tasks running in different executor. You can define Partitioner to denote the data shuffling rule between upstream processor tasks and downstream processor tasks. - - -Figure: Task Data Shuffling http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/build_doc.sh ---------------------------------------------------------------------- diff --git a/docs/build_doc.sh b/docs/build_doc.sh index 6b81f20..da73dc6 100755 --- a/docs/build_doc.sh +++ b/docs/build_doc.sh @@ -19,7 +19,20 @@ copy_dir() { echo "Making directory $destDir" mkdir -p $destDir echo "copy from $srcDir to $destDir..." - cp -r $srcDir/. $destDir + cp -r $srcDir/* $destDir +} + +render_files() { + for file in $2; do + if [ -d "$file" ];then + for child in "$file"/*; do + render_files $1 "$child" + done + elif [ -f "$file" ]; then + mustache $1 $file > tmp.md + mv tmp.md $file + fi + done } if [ $# -ne 2 ]; then @@ -30,23 +43,23 @@ fi export SCALA_VERSION=$1 export BUILD_API=$2 -# generate _site documents -jekyll build + +# render file templates +echo "Rendering file templates using mustache..." +TEMP_DIR="tmp" +rm -rf $TEMP_DIR +copy_dir docs $TEMP_DIR +render_files version.yml "$TEMP_DIR/introduction $TEMP_DIR/dev $TEMP_DIR/deployment $TEMP_DIR/api $TEMP_DIR/index.md" + +# generate site documents +mkdocs build --clean # check html link validity -# htmlproof has been renamed to htmlproofer in Dec 2015 -HTMLPROOF="htmlproof" -HTML_IGNORE_HREF='--href-ignore "#/"' -if ! `which $HTMLPROOF >/dev/null`; then - HTMLPROOF="htmlproofer" - HTML_IGNORE_HREF="--allow-hash-href" -fi -echo "Checking generated HTMLs using $HTMLPROOF..." +echo "Checking generated HTMLs using htmlproofer..." -$HTMLPROOF _site \ +htmlproofer site \ --disable-external \ - $HTML_IGNORE_HREF \ - --url-ignore \#,api/scala/index.html,api/java/index.html,/download.html + --file-ignore site/base.html,site/breadcrumbs.html,site/versions.html,site/toc.html,site/footer.html # generate API doc if [ "$BUILD_API" = 1 ]; then @@ -59,11 +72,11 @@ if [ "$BUILD_API" = 1 ]; then cd $CURDIR echo "Removing old docs" - rm -rf _site/api + rm -rf site/api #copy unified ScalaDoc - copy_dir "../target/scala-$SCALA_VERSION/unidoc" "_site/api/scala" + copy_dir "../target/scala-$SCALA_VERSION/unidoc" "site/api/scala" #copy unified java doc - copy_dir "../target/javaunidoc" "_site/api/java" + copy_dir "../target/javaunidoc" "site/api/java" fi http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/coding-style.md ---------------------------------------------------------------------- diff --git a/docs/coding-style.md b/docs/coding-style.md index d5bee39..402f211 100644 --- a/docs/coding-style.md +++ b/docs/coding-style.md @@ -1,8 +1,3 @@ ---- -layout: global -title: Gearpump Coding Style ---- - ## Scala coding style For Scala code, Gearpump follows the standard [Scala coding style](http://docs.scala-lang.org/style/). http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/commandline.md ---------------------------------------------------------------------- diff --git a/docs/commandline.md b/docs/commandline.md deleted file mode 100644 index 7e264f9..0000000 --- a/docs/commandline.md +++ /dev/null @@ -1,91 +0,0 @@ ---- -layout: global -displayTitle: Gearpump Client Commandline -title: Gearpump Client Commandline -description: Gearpump Client Commandline ---- - -The commands can be found at: "bin" folder of Gearpump binary. - -**NOTE:** on MS Windows platform, please use window shell gear.bat script instead. bash script doesn't work well in cygwin/mingw. - -### Creating an uber-jar - -If you use Maven you can have a look [here](https://maven.apache.org/plugins/maven-shade-plugin/) whereas SBT users may find [this](https://github.com/sbt/sbt-assembly) useful. - -### Submit an new application - -You can use the command `gear` under the bin directory to submit, query and terminate an application: - -```bash -gear app [-namePrefix <application name prefix>] [-executors <number of executors to launch>] [-conf <custom gearpump config file>] -jar xx.jar MainClass <arg1> <arg2> ... -``` - -### List all running applications -To list all running applications: - -```bash -gear info [-conf <custom gearpump config file>] -``` - -### Kill a running application -To kill an application: - -```bash -gear kill -appid <application id> [-conf <custom gearpump config file>] -``` - -### Submit a storm application to Gearpump Cluster -For example, to submit a storm application jar: - -```bash -storm -verbose -config storm.yaml -jar storm-starter-${STORM_VERSION}.jar storm.starter.ExclamationTopology exclamation -``` - -[Storm Compatibility Guide](dev-storm.html) - -### Start Gearpump Cluster on YARN -To start a Gearpump Cluster on YARN, you can: - -```bash -yarnclient launch -package /usr/lib/gearpump/gearpump-{{ site.SCALA_BINARY_VERSION }}-{{ site.GEARPUMP_VERSION }}.zip -``` -/usr/lib/gearpump/gearpump-{{ site.SCALA_BINARY_VERSION }}-{{ site.GEARPUMP_VERSION }}.zip should be available on HDFS. - -Please check [YARN Deployment Guide](deployment-yarn.html) for more information. - -### Start a local cluster -Masters and workers will be started in one machine: - -```bash -local -``` - -Check [Deployment Guide for Local Cluster](deployment-local.html) for more information. - -### Start master daemons - -```bash -master -ip <Ip address> -port <port where this master is hooking> -``` - -Please check [Deployment for Standalone mode](deployment-standalone.html) for more information. - -### Start worker daemons - -```bash -worker -``` - -Please check [Deployment for Standalone mode](deployment-standalone.html) for more information. - -### Start UI server - -To start UI server, you can: - -```bash -services [-master <host:port>] -``` - -The default username and password is "admin:admin", you can check -[UI Authentication](deployment-ui-authentication.html) to find how to manage users. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/css/api-docs.css ---------------------------------------------------------------------- diff --git a/docs/css/api-docs.css b/docs/css/api-docs.css deleted file mode 100644 index 7cf222a..0000000 --- a/docs/css/api-docs.css +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* Dynamically injected style for the API docs */ - -.developer { - background-color: #44751E; -} - -.experimental { - background-color: #257080; -} - -.alphaComponent { - background-color: #bb0000; -} - -.badge { - font-family: Arial, san-serif; - float: right; -} http://git-wip-us.apache.org/repos/asf/incubator-gearpump/blob/761e04c6/docs/css/api-javadocs.css ---------------------------------------------------------------------- diff --git a/docs/css/api-javadocs.css b/docs/css/api-javadocs.css deleted file mode 100644 index e48ebcf..0000000 --- a/docs/css/api-javadocs.css +++ /dev/null @@ -1,52 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* Dynamically injected style for the API docs */ - -.badge { - font-family: Arial, san-serif; - float: right; - margin: 4px; - /* The following declarations are taken from the ScalaDoc template.css */ - display: inline-block; - padding: 2px 4px; - font-size: 11.844px; - font-weight: bold; - line-height: 14px; - color: #ffffff; - text-shadow: 0 -1px 0 rgba(0, 0, 0, 0.25); - white-space: nowrap; - vertical-align: baseline; - background-color: #999999; - padding-right: 9px; - padding-left: 9px; - -webkit-border-radius: 9px; - -moz-border-radius: 9px; - border-radius: 9px; -} - -.developer { - background-color: #44751E; -} - -.experimental { - background-color: #257080; -} - -.alphaComponent { - background-color: #bb0000; -}
