Added 0.6.0 incubating release docs and updates trunk to latest
Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/commit/5058c744 Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/tree/5058c744 Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/diff/5058c744 Branch: refs/heads/asf-site Commit: 5058c74485ceaa859327eb272064826ead265bb2 Parents: 4c8c01f Author: Suma Shivaprasad <[email protected]> Authored: Thu Dec 31 11:11:08 2015 +0530 Committer: Suma Shivaprasad <[email protected]> Committed: Thu Dec 31 11:11:08 2015 +0530 ---------------------------------------------------------------------- 0.6.0-incubating/Architecture.html | 246 ++++++ 0.6.0-incubating/Bridge-Hive.html | 301 ++++++++ 0.6.0-incubating/Configuration.html | 356 +++++++++ 0.6.0-incubating/HighAvailability.html | 305 ++++++++ 0.6.0-incubating/InstallationSteps.html | 403 ++++++++++ 0.6.0-incubating/Notification-Entity.html | 251 ++++++ 0.6.0-incubating/QuickStart.html | 242 ++++++ 0.6.0-incubating/Repository.html | 223 ++++++ 0.6.0-incubating/Search.html | 311 ++++++++ 0.6.0-incubating/TypeSystem.html | 246 ++++++ 0.6.0-incubating/api/apple-touch-icon.png | Bin 0 -> 640 bytes 0.6.0-incubating/api/application.wadl | 443 +++++++++++ 0.6.0-incubating/api/crossdomain.xml | 25 + 0.6.0-incubating/api/css/home.gif | Bin 0 -> 128 bytes 0.6.0-incubating/api/css/prettify.css | 1 + 0.6.0-incubating/api/css/style.css | 759 +++++++++++++++++++ 0.6.0-incubating/api/favicon.ico | Bin 0 -> 1150 bytes 0.6.0-incubating/api/index.html | 123 +++ 0.6.0-incubating/api/js/libs/dd_belatedpng.js | 13 + .../api/js/libs/jquery-1.5.1.min.js | 16 + .../api/js/libs/modernizr-1.7.min.js | 2 + .../api/js/libs/prettify/lang-apollo.js | 2 + .../api/js/libs/prettify/lang-clj.js | 18 + .../api/js/libs/prettify/lang-css.js | 2 + .../api/js/libs/prettify/lang-go.js | 1 + .../api/js/libs/prettify/lang-hs.js | 2 + .../api/js/libs/prettify/lang-lisp.js | 3 + .../api/js/libs/prettify/lang-lua.js | 2 + .../api/js/libs/prettify/lang-ml.js | 2 + 0.6.0-incubating/api/js/libs/prettify/lang-n.js | 4 + .../api/js/libs/prettify/lang-proto.js | 1 + .../api/js/libs/prettify/lang-scala.js | 2 + .../api/js/libs/prettify/lang-sql.js | 2 + .../api/js/libs/prettify/lang-tex.js | 1 + .../api/js/libs/prettify/lang-vb.js | 2 + .../api/js/libs/prettify/lang-vhdl.js | 3 + .../api/js/libs/prettify/lang-wiki.js | 2 + .../api/js/libs/prettify/lang-xq.js | 3 + .../api/js/libs/prettify/lang-yaml.js | 2 + .../api/js/libs/prettify/prettify.js | 28 + 0.6.0-incubating/api/js/libs/xbreadcrumbs.js | 93 +++ 0.6.0-incubating/api/model.html | 111 +++ .../api/resource_AdminResource.html | 157 ++++ .../api/resource_EntityResource.html | 475 ++++++++++++ .../api/resource_HiveLineageResource.html | 230 ++++++ .../api/resource_MetadataDiscoveryResource.html | 269 +++++++ .../api/resource_RexsterGraphResource.html | 353 +++++++++ .../api/resource_TypesResource.html | 245 ++++++ 0.6.0-incubating/api/rest.html | 112 +++ 0.6.0-incubating/api/robots.txt | 5 + .../css/apache-maven-fluido-1.3.0.min.css | 9 + 0.6.0-incubating/css/print.css | 23 + 0.6.0-incubating/css/site.css | 1 + .../images/accessories-text-editor.png | Bin 0 -> 746 bytes 0.6.0-incubating/images/add.gif | Bin 0 -> 397 bytes .../images/apache-incubator-logo.png | Bin 0 -> 4234 bytes .../images/apache-maven-project-2.png | Bin 0 -> 33442 bytes .../images/application-certificate.png | Bin 0 -> 923 bytes 0.6.0-incubating/images/atlas-logo.png | Bin 0 -> 3115 bytes 0.6.0-incubating/images/contact-new.png | Bin 0 -> 736 bytes 0.6.0-incubating/images/document-properties.png | Bin 0 -> 577 bytes 0.6.0-incubating/images/drive-harddisk.png | Bin 0 -> 700 bytes 0.6.0-incubating/images/fix.gif | Bin 0 -> 366 bytes 0.6.0-incubating/images/icon_error_sml.gif | Bin 0 -> 633 bytes 0.6.0-incubating/images/icon_help_sml.gif | Bin 0 -> 1072 bytes 0.6.0-incubating/images/icon_info_sml.gif | Bin 0 -> 638 bytes 0.6.0-incubating/images/icon_success_sml.gif | Bin 0 -> 604 bytes 0.6.0-incubating/images/icon_warning_sml.gif | Bin 0 -> 625 bytes 0.6.0-incubating/images/image-x-generic.png | Bin 0 -> 662 bytes .../images/internet-web-browser.png | Bin 0 -> 1017 bytes .../images/logos/build-by-maven-black.png | Bin 0 -> 2294 bytes .../images/logos/build-by-maven-white.png | Bin 0 -> 2260 bytes 0.6.0-incubating/images/logos/maven-feather.png | Bin 0 -> 3330 bytes 0.6.0-incubating/images/network-server.png | Bin 0 -> 536 bytes 0.6.0-incubating/images/package-x-generic.png | Bin 0 -> 717 bytes .../images/profiles/pre-release.png | Bin 0 -> 32607 bytes 0.6.0-incubating/images/profiles/retired.png | Bin 0 -> 22003 bytes 0.6.0-incubating/images/profiles/sandbox.png | Bin 0 -> 33010 bytes 0.6.0-incubating/images/remove.gif | Bin 0 -> 607 bytes 0.6.0-incubating/images/rss.png | Bin 0 -> 474 bytes 0.6.0-incubating/images/twiki/architecture.png | Bin 0 -> 58775 bytes 0.6.0-incubating/images/twiki/data-types.png | Bin 0 -> 413738 bytes .../images/twiki/guide-class-diagram.png | Bin 0 -> 40375 bytes .../images/twiki/guide-instance-graph.png | Bin 0 -> 179941 bytes 0.6.0-incubating/images/twiki/notification.png | Bin 0 -> 137448 bytes .../images/twiki/types-instance.png | Bin 0 -> 445893 bytes 0.6.0-incubating/images/update.gif | Bin 0 -> 1090 bytes 0.6.0-incubating/images/window-new.png | Bin 0 -> 583 bytes .../img/glyphicons-halflings-white.png | Bin 0 -> 8777 bytes 0.6.0-incubating/img/glyphicons-halflings.png | Bin 0 -> 12799 bytes 0.6.0-incubating/index.html | 285 +++++++ 0.6.0-incubating/issue-tracking.html | 227 ++++++ .../js/apache-maven-fluido-1.3.0.min.js | 21 + 0.6.0-incubating/license.html | 429 +++++++++++ 0.6.0-incubating/mail-lists.html | 241 ++++++ 0.6.0-incubating/project-info.html | 246 ++++++ 0.6.0-incubating/security.html | 317 ++++++++ 0.6.0-incubating/source-repository.html | 240 ++++++ 0.6.0-incubating/team-list.html | 433 +++++++++++ Architecture.html | 19 +- Bridge-Hive.html | 25 +- Configuration.html | 49 +- HighAvailability.html | 305 ++++++++ InstallationSteps.html | 42 +- Notification-Entity.html | 8 +- QuickStart.html | 8 +- Repository.html | 8 +- Search.html | 8 +- Security.html | 11 +- TypeSystem.html | 8 +- api/application.wadl | 72 +- api/resource_EntityResource.html | 129 +++- index.html | 11 +- issue-tracking.html | 8 +- license.html | 8 +- mail-lists.html | 8 +- project-info.html | 8 +- source-repository.html | 8 +- team-list.html | 8 +- 119 files changed, 9453 insertions(+), 168 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/5058c744/0.6.0-incubating/Architecture.html ---------------------------------------------------------------------- diff --git a/0.6.0-incubating/Architecture.html b/0.6.0-incubating/Architecture.html new file mode 100644 index 0000000..aadd792 --- /dev/null +++ b/0.6.0-incubating/Architecture.html @@ -0,0 +1,246 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-12-30 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Architecture</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Architecture</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Architecture"></a>Architecture</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3></div> +<div class="section"> +<h3><a name="Atlas_High_Level_Architecture_-_Overview"></a>Atlas High Level Architecture - Overview</h3> +<p><img src="images/twiki/architecture.png" alt="" /></p> +<p>Architecturally, Atlas has the following components:</p> +<p></p> +<ul> +<li><b>A Web service</b>: This exposes RESTful APIs and a Web user interface to create, update and query metadata.</li> +<li><b>Metadata store</b>: Metadata is modeled using a graph model, implemented using the Graph database Titan. Titan has options for a variety of backing stores for persisting the graph, including an embedded Berkeley DB, Apache HBase and Apache Cassandra. The choice of the backing store determines the level of service availability.</li> +<li><b>Index store</b>: For powering full text searches on metadata, Atlas also indexes the metadata, again via Titan. The backing store for the full text search is a search backend like ElasticSearch or Apache Solr.</li> +<li><b>Bridges / Hooks</b>: To add metadata to Atlas, libraries called ‘hooks’ are enabled in various systems like Apache Hive, Apache Falcon and Apache Sqoop which capture metadata events in the respective systems and propagate those events to Atlas. The Atlas server consumes these events and updates its stores.</li> +<li><b>Metadata notification events</b>: Any updates to metadata in Atlas, either via the Hooks or the API are propagated from Atlas to downstream systems via events. Systems like Apache Ranger consume these events and allow administrators to act on them, for e.g. to configure policies for Access control.</li> +<li><b>Notification Server</b>: Atlas uses Apache Kafka as a notification server for communication between hooks and downstream consumers of metadata notification events. Events are written by the hooks and Atlas to different Kafka topics. Kafka enables a loosely coupled integration between these disparate systems.</li></ul></div> +<div class="section"> +<h3><a name="Bridges"></a>Bridges</h3> +<p>External components like hive/sqoop/storm/falcon should model their taxonomy using typesystem and register the types with Atlas. For every entity created in this external component, the corresponding entity should be registered in Atlas as well. This is typically done in a hook which runs in the external component and is called for every entity operation. Hook generally processes the entity asynchronously using a thread pool to avoid adding latency to the main operation. The hook can then build the entity and register the entity using Atlas REST APIs. Howerver, any failure in APIs because of network issue etc can in result entity not registered in Atlas and hence inconsistent metadata.</p> +<p>Atlas exposes notification interface and can be used for reliable entity registration by hook as well. The hook can send notification message containing the list of entities to be registered. Atlas service contains hook consumer that listens to these messages and registers the entities.</p> +<p>Available bridges are:</p> +<ul> +<li><a href="./Bridge-Hive.html">Hive Bridge</a></li></ul></div> +<div class="section"> +<h3><a name="Notification"></a>Notification</h3> +<p>Notification is used for reliable entity registration from hooks and for entity/type change notifications. Atlas, by default, provides Kafka integration, but its possible to provide other implementations as well. Atlas service starts embedded Kafka server by default.</p> +<p>Atlas also provides <a href="./NotificationHookConsumer.html">NotificationHookConsumer</a> that runs in Atlas Service and listens to messages from hook and registers the entities in Atlas. <img src="images/twiki/notification.png" alt="" /></p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/5058c744/0.6.0-incubating/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/0.6.0-incubating/Bridge-Hive.html b/0.6.0-incubating/Bridge-Hive.html new file mode 100644 index 0000000..8067435 --- /dev/null +++ b/0.6.0-incubating/Bridge-Hive.html @@ -0,0 +1,301 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-12-30 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Hive Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Hive Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Hive_Atlas_Bridge"></a>Hive Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Hive_Model"></a>Hive Model</h3> +<p>The default hive modelling is available in org.apache.atlas.hive.model.HiveDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +hive_object_type(EnumType) - values [GLOBAL, DATABASE, TABLE, PARTITION, COLUMN] +hive_resource_type(EnumType) - values [JAR, FILE, ARCHIVE] +hive_principal_type(EnumType) - values [USER, ROLE, GROUP] +hive_db(ClassType) - super types [Referenceable] - attributes [name, clusterName, description, locationUri, parameters, ownerName, ownerType] +hive_order(StructType) - attributes [col, order] +hive_resourceuri(StructType) - attributes [resourceType, uri] +hive_serde(StructType) - attributes [name, serializationLib, parameters] +hive_type(ClassType) - super types [] - attributes [name, type1, type2, fields] +hive_storagedesc(ClassType) - super types [Referenceable] - attributes [cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories] +hive_role(ClassType) - super types [] - attributes [roleName, createTime, ownerName] +hive_column(ClassType) - super types [Referenceable] - attributes [name, type, comment] +hive_table(ClassType) - super types [DataSet] - attributes [tableName, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, parameters, viewOriginalText, viewExpandedText, tableType, temporary] +hive_partition(ClassType) - super types [Referenceable] - attributes [values, table, createTime, lastAccessTime, sd, columns, parameters] +hive_process(ClassType) - super types [Process] - attributes [startTime, endTime, userName, operationType, queryText, queryPlan, queryId, queryGraph] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName and tableName should be in lower case. clusterName is explained below.</p> +<ul> +<li>hive_db - attribute qualifiedName - <dbName>@<clusterName></li> +<li>hive_table - attribute name - <dbName>.<tableName>@<clusterName></li> +<li>hive_column - attribute qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> +<li>hive_partition - attribute qualifiedName - <dbName>.<tableName>.<partitionValues('-' separated)>@<clusterName></li> +<li>hive_process - attribute name - <queryString> - trimmed query string in lower case</li></ul></div> +<div class="section"> +<h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3> +<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the hive metadata into Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. import-hive.sh command can be used to facilitate this. Set the following configuration in <atlas-conf>/client.properties and set environment variable $HIVE_CONF_DIR to the hive conf directory:</p> +<div class="source"> +<pre> + <property> + <name>atlas.cluster.name</name> + <value>primary</value> + </property> + +</pre></div> +<p>Usage: <atlas package>/bin/import-hive.sh. The logs are in <atlas package>/logs/import-hive.log</p></div> +<div class="section"> +<h3><a name="Hive_Hook"></a>Hive Hook</h3> +<p>Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities. Follow these instructions in your hive set-up to add hive hook for Atlas:</p> +<ul> +<li>Set-up atlas hook in hive-site.xml of your hive configuration:</li></ul> +<div class="source"> +<pre> + <property> + <name>hive.exec.post.hooks</name> + <value>org.apache.atlas.hive.hook.HiveHook</value> + </property> + +</pre></div> +<div class="source"> +<pre> + <property> + <name>atlas.cluster.name</name> + <value>primary</value> + </property> + +</pre></div> +<p></p> +<ul> +<li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li> +<li>Copy <atlas-conf>/client.properties and <atlas-conf>/application.properties to the hive conf directory.</li></ul> +<p>The following properties in <atlas-conf>/client.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.hive.minThreads - core number of threads. default 5</li> +<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li> +<li>Only the following hive operations are captured by hive hook currently - create database, create table, create view, CTAS, load, import, export, query, alter table rename and alter view rename</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/5058c744/0.6.0-incubating/Configuration.html ---------------------------------------------------------------------- diff --git a/0.6.0-incubating/Configuration.html b/0.6.0-incubating/Configuration.html new file mode 100644 index 0000000..535bae4 --- /dev/null +++ b/0.6.0-incubating/Configuration.html @@ -0,0 +1,356 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-12-30 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Configuring Apache Atlas - Application Properties</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Configuring Apache Atlas - Application Properties</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Configuring_Apache_Atlas_-_Application_Properties"></a>Configuring Apache Atlas - Application Properties</h2> +<p>All configuration in Atlas uses java properties style configuration. The main configuration file is application.properties which is in the <b>conf</b> dir at the deployed location. It consists of the following sections:</p></div> +<div class="section"> +<h3><a name="Graph_Configs"></a>Graph Configs</h3></div> +<div class="section"> +<h4><a name="Graph_persistence_engine"></a>Graph persistence engine</h4> +<p>This section sets up the graph db - titan - to use a persistence engine. Please refer to <a href="http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html";>link</a> for more details. The example below uses BerkeleyDBJE.</p> +<div class="source"> +<pre> +atlas.graph.storage.backend=berkeleyje +atlas.graph.storage.directory=data/berkley + +</pre></div></div> +<div class="section"> +<h5><a name="Graph_persistence_engine_-_Hbase"></a>Graph persistence engine - Hbase</h5> +<p>Basic configuration</p> +<div class="source"> +<pre> +atlas.graph.storage.backend=hbase +#For standalone mode , specify localhost +#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2 +atlas.graph.storage.hostname=<ZooKeeper Quorum> + +</pre></div> +<p>HBASE_CONF_DIR environment variable needs to be set to point to the Hbase client configuration directory which is added to classpath when Atlas starts up. hbase-site.xml needs to have the following properties set according to the cluster setup</p> +<div class="source"> +<pre> +#Set below to /hbase-secure if the Hbase server is setup in secure mode +zookeeper.znode.parent=/hbase-unsecure + +</pre></div> +<p>Advanced configuration</p> +<p># If you are planning to use any of the configs mentioned below, they need to be prefixed with "atlas.graph." to take effect in ATLAS Refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_hbase";>http://s3.thinkaurelius.com/docs/titan/0.5.4/titan-config-ref.html#_storage_hbase</a></p> +<p>Permissions</p> +<p>When Atlas is configured with HBase as the storage backend the graph db (titan) needs sufficient user permissions to be able to create and access an HBase table. In a secure cluster it may be necessary to grant permissions to the 'atlas' user for the 'titan' table.</p> +<p>With Ranger, a policy can be configured for 'titan'.</p> +<p>Without Ranger, HBase shell can be used to set the permissions.</p> +<div class="source"> +<pre> + su hbase + kinit -k -t <hbase keytab> <hbase principal> + echo "grant 'atlas', 'RWXCA', 'titan'" | hbase shell + +</pre></div></div> +<div class="section"> +<h4><a name="Graph_Search_Index"></a>Graph Search Index</h4> +<p>This section sets up the graph db - titan - to use an search indexing system. The example configuration below sets up to use an embedded Elastic search indexing system.</p> +<div class="source"> +<pre> +atlas.graph.index.search.backend=elasticsearch +atlas.graph.index.search.directory=data/es +atlas.graph.index.search.elasticsearch.client-only=false +atlas.graph.index.search.elasticsearch.local-mode=true +atlas.graph.index.search.elasticsearch.create.sleep=2000 + +</pre></div></div> +<div class="section"> +<h5><a name="Graph_Search_Index_-_Solr"></a>Graph Search Index - Solr</h5> +<p>Please note that Solr installation in Cloud mode is a prerequisite before configuring Solr as the search indexing backend. Refer <a href="./InstallationSteps.html">InstallationSteps</a> section for Solr installation/configuration.</p> +<div class="source"> +<pre> + atlas.graph.index.search.backend=solr5 + atlas.graph.index.search.solr.mode=cloud + atlas.graph.index.search.solr.zookeeper-url=<the ZK quorum setup for solr as comma separated value> eg: 10.1.6.4:2181,10.1.6.5:2181 + +</pre></div></div> +<div class="section"> +<h4><a name="Choosing_between_Persistence_and_Indexing_Backends"></a>Choosing between Persistence and Indexing Backends</h4> +<p>Refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html</a> and <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html</a> for choosing between the persistence backends. BerkeleyDB is suitable for smaller data sets in the range of upto 10 million vertices with ACID gurantees. HBase on the other hand doesnt provide ACID guarantees but is able to scale for larger graphs. HBase also provides HA inherently.</p></div> +<div class="section"> +<h4><a name="Choosing_between_Persistence_Backends"></a>Choosing between Persistence Backends</h4> +<p>Refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/bdb.html</a> and <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/hbase.html</a> for choosing between the persistence backends. BerkeleyDB is suitable for smaller data sets in the range of upto 10 million vertices with ACID gurantees. HBase on the other hand doesnt provide ACID guarantees but is able to scale for larger graphs. HBase also provides HA inherently.</p></div> +<div class="section"> +<h4><a name="Choosing_between_Indexing_Backends"></a>Choosing between Indexing Backends</h4> +<p>Refer <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/elasticsearch.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/elasticsearch.html</a> and <a class="externalLink" href="http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.html";>http://s3.thinkaurelius.com/docs/titan/0.5.4/solr.html</a> for chossing between <a href="./ElasticSarch.html">ElasticSarch</a> and Solr. Solr in cloud mode is the recommended setup.</p></div> +<div class="section"> +<h4><a name="Switching_Persistence_Backend"></a>Switching Persistence Backend</h4> +<p>For switching the storage backend from BerkeleyDB to HBase and vice versa, refer the documentation for "Graph Persistence Engine" described above and restart ATLAS. The data in the indexing backend needs to be cleared else there will be discrepancies between the storage and indexing backend which could result in errors during the search. <a href="./ElasticSearch.html">ElasticSearch</a> runs by default in embedded mode and the data could easily be cleared by deleting the ATLAS_HOME/data/es directory. For Solr, the collections which were created during ATLAS Installation - vertex_index, edge_index, fulltext_index could be deleted which will cleanup the indexes</p></div> +<div class="section"> +<h4><a name="Switching_Index_Backend"></a>Switching Index Backend</h4> +<p>Switching the Index backend requires clearing the persistence backend data. Otherwise there will be discrepancies between the persistence and index backends since switching the indexing backend means index data will be lost. This leads to "Fulltext" queries not working on the existing data For clearing the data for BerkeleyDB, delete the ATLAS_HOME/data/berkeley directory For clearing the data for HBase, in Hbase shell, run 'disable titan' and 'drop titan'</p></div> +<div class="section"> +<h3><a name="Lineage_Configs"></a>Lineage Configs</h3> +<p>The higher layer services like lineage, schema, etc. are driven by the type system and this section encodes the specific types for the hive data model.</p> +<p># This models reflects the base super types for Data and Process</p> +<div class="source"> +<pre> +atlas.lineage.hive.table.type.name=DataSet +atlas.lineage.hive.process.type.name=Process +atlas.lineage.hive.process.inputs.name=inputs +atlas.lineage.hive.process.outputs.name=outputs + +## Schema +atlas.lineage.hive.table.schema.query=hive_table where name=?, columns + +</pre></div></div> +<div class="section"> +<h3><a name="Notification_Configs"></a>Notification Configs</h3> +<p>Refer <a class="externalLink" href="http://kafka.apache.org/documentation.html#configuration";>http://kafka.apache.org/documentation.html#configuration</a> for Kafka configuration. All Kafka configs should be prefixed with 'atlas.kafka.'</p> +<div class="source"> +<pre> +atlas.notification.embedded=true +atlas.kafka.data=${sys:atlas.home}/data/kafka +atlas.kafka.zookeeper.connect=localhost:9026 +atlas.kafka.bootstrap.servers=localhost:9027 +atlas.kafka.zookeeper.session.timeout.ms=400 +atlas.kafka.zookeeper.sync.time.ms=20 +atlas.kafka.auto.commit.interval.ms=1000 +atlas.kafka.hook.group.id=atlas + +</pre></div> +<p>Note that Kafka group ids are specified for a specific topic. The Kafka group id configuration for entity notifications is 'atlas.kafka.entities.group.id'</p> +<div class="source"> +<pre> +atlas.kafka.entities.group.id=<consumer id> + +</pre></div></div> +<div class="section"> +<h3><a name="Client_Configs"></a>Client Configs</h3> +<div class="source"> +<pre> +atlas.client.readTimeoutMSecs=60000 +atlas.client.connectTimeoutMSecs=60000 +atlas.rest.address=<http/https>://<atlas-fqdn>:<atlas port> - default http://localhost:21000 + +</pre></div></div> +<div class="section"> +<h3><a name="Security_Properties"></a>Security Properties</h3></div> +<div class="section"> +<h4><a name="SSL_config"></a>SSL config</h4> +<p>The following property is used to toggle the SSL feature.</p> +<div class="source"> +<pre> +atlas.enableTLS=false + +</pre></div></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/5058c744/0.6.0-incubating/HighAvailability.html ---------------------------------------------------------------------- diff --git a/0.6.0-incubating/HighAvailability.html b/0.6.0-incubating/HighAvailability.html new file mode 100644 index 0000000..bca44aa --- /dev/null +++ b/0.6.0-incubating/HighAvailability.html @@ -0,0 +1,305 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2015-12-30 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20151230" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Fault Tolerance and High Availability Options</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Fault Tolerance and High Availability Options</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2015-12-30</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.6-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Fault_Tolerance_and_High_Availability_Options"></a>Fault Tolerance and High Availability Options</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3> +<p>Apache Atlas uses and interacts with a variety of systems to provide metadata management and data lineage to data administrators. By choosing and configuring these dependencies appropriately, it is possible to achieve a good degree of service availability with Atlas. This document describes the state of high availability support in Atlas, including its capabilities and current limitations, and also the configuration required for achieving a this level of high availability.</p> +<p><a href="./Architecture.html">The architecture page</a> in the wiki gives an overview of the various components that make up Atlas. The options mentioned below for various components derive context from the above page, and would be worthwhile to review before proceeding to read this page.</p></div> +<div class="section"> +<h3><a name="Atlas_Web_Service"></a>Atlas Web Service</h3> +<p>Currently, the Atlas Web service has a limitation that it can only have one active instance at a time. Therefore, in case of errors to the host running the service, a new Atlas web service instance should be brought up and pointed to from the clients. In future versions of the system, we plan to provide full High Availability of the service, thereby enabling hot failover. To minimize service loss, we recommend the following:</p> +<p></p> +<ul> +<li>An extra physical host with the Atlas system software and configuration is available to be brought up on demand.</li> +<li>It would be convenient to have the web service fronted by a proxy solution like <a class="externalLink" href="https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.2";>HAProxy</a> which can be used to provide both the monitoring and transparent switching of the backend instance clients talk to. +<ul> +<li>An example HAProxy configuration of this form will allow a transparent failover to a backup server:</li></ul></li></ul> +<div class="source"> +<pre> + listen atlas + bind <proxy hostname>:<proxy port> + balance roundrobin + server inst1 <atlas server hostname>:<port> check + server inst2 <atlas backup server hostname>:<port> check backup + +</pre></div> +<p></p> +<ul> +<li>The stores that hold Atlas data can be configured to be highly available as described below.</li></ul></div> +<div class="section"> +<h3><a name="Metadata_Store"></a>Metadata Store</h3> +<p>As described above, Atlas uses Titan to store the metadata it manages. By default, Titan uses BerkeleyDB as an embedded backing store. However, this option would result in loss of data if the node running the Atlas server fails. In order to provide HA for the metadata store, we recommend that Atlas be configured to use HBase as the backing store for Titan. Doing this implies that you could benefit from the HA guarantees HBase provides. In order to configure Atlas to use HBase in HA mode, do the following:</p> +<p></p> +<ul> +<li>Choose an existing HBase cluster that is set up in HA mode to configure in Atlas (OR) Set up a new HBase cluster in <a class="externalLink" href="http://hbase.apache.org/book.html#quickstart_fully_distributed";>HA mode</a>. +<ul> +<li>If setting up HBase for Atlas, please following instructions listed for setting up HBase in the <a href="./InstallationSteps.html">Installation Steps</a>.</li></ul></li> +<li>We recommend using more than one HBase masters (at least 2) in the cluster on different physical hosts that use Zookeeper for coordination to provide redundancy and high availability of HBase. +<ul> +<li>Refer to the <a href="./Configuration.html">Configuration page</a> for the options to configure in atlas.properties to setup Atlas with HBase.</li></ul></li></ul></div> +<div class="section"> +<h3><a name="Index_Store"></a>Index Store</h3> +<p>As described above, Atlas indexes metadata through Titan to support full text search queries. In order to provide HA for the index store, we recommend that Atlas be configured to use Solr as the backing index store for Titan. In order to configure Atlas to use Solr in HA mode, do the following:</p> +<p></p> +<ul> +<li>Choose an existing SolrCloud cluster setup in HA mode to configure in Atlas (OR) Set up a new <a class="externalLink" href="https://cwiki.apache.org/confluence/display/solr/SolrCloud";>SolrCloud cluster</a>. +<ul> +<li>Ensure Solr is brought up on at least 2 physical hosts for redundancy, and each host runs a Solr node.</li> +<li>We recommend the number of replicas to be set to at least 2 for redundancy.</li></ul></li> +<li>Create the SolrCloud collections required by Atlas, as described in <a href="./InstallationSteps.html">Installation Steps</a></li> +<li>Refer to the <a href="./Configuration.html">Configuration page</a> for the options to configure in atlas.properties to setup Atlas with Solr.</li></ul></div> +<div class="section"> +<h3><a name="Notification_Server"></a>Notification Server</h3> +<p>Metadata notification events from Hooks are sent to Atlas by writing them to a Kafka topic called <b>ATLAS_HOOK</b>. Similarly, events from Atlas to other integrating components like Ranger, are written to a Kafka topic called <b>ATLAS_ENTITIES</b>. Since Kafka persists these messages, the events will not be lost even if the consumers are down as the events are being sent. In addition, we recommend Kafka is also setup for fault tolerance so that it has higher availability guarantees. In order to configure Atlas to use Kafka in HA mode, do the following:</p> +<p></p> +<ul> +<li>Choose an existing Kafka cluster set up in HA mode to configure in Atlas (OR) Set up a new Kafka cluster.</li> +<li>We recommend that there are more than one Kafka brokers in the cluster on different physical hosts that use Zookeeper for coordination to provide redundancy and high availability of Kafka. +<ul> +<li>Setup at least 2 physical hosts for redundancy, each hosting a Kafka broker.</li></ul></li> +<li>Set up Kafka topics for Atlas usage: +<ul> +<li>The number of partitions for the ATLAS topics should be set to 1 (numPartitions)</li> +<li>Decide number of replicas for Kafka topic: Set this to at least 2 for redundancy.</li> +<li>Run the following commands:</li></ul></li></ul> +<div class="source"> +<pre> + $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper <list of zookeeper host:port entries> --topic ATLAS_HOOK --replication-factor <numReplicas> --partitions 1 + $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper <list of zookeeper host:port entries> --topic ATLAS_ENTITIES --replication-factor <numReplicas> --partitions 1 + Here KAFKA_HOME points to the Kafka installation directory. + +</pre></div> +<p></p> +<ul> +<li>In application.properties, set the following configuration:</li></ul> +<div class="source"> +<pre> + atlas.notification.embedded=false + atlas.kafka.zookeeper.connect=<comma separated list of servers forming Zookeeper quorum used by Kafka> + atlas.kafka.bootstrap.servers=<comma separated list of Kafka broker endpoints in host:port form> - Give at least 2 for redundancy. + +</pre></div></div> +<div class="section"> +<h3><a name="Known_Issues"></a>Known Issues</h3> +<p></p> +<ul> +<li><a class="externalLink" href="https://issues.apache.org/jira/browse/ATLAS-338";>ATLAS-338</a>: ATLAS-338: Metadata events generated from a Hive CLI (as opposed to Beeline or any client going <a href="./HiveServer.html">HiveServer</a>2) would be lost if Atlas server is down.</li> +<li>If the HBase region servers hosting the Atlas ‘titan’ HTable are down, Atlas would not be able to store or retrieve metadata from HBase until they are brought back online.</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
