updating site for 0.7 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/commit/60041d8d Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/tree/60041d8d Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/diff/60041d8d Branch: refs/heads/asf-site Commit: 60041d8d3fa14c4ad6ed19b99dfb0459739d9ae6 Parents: 05ef63f Author: Shwetha GS <[email protected]> Authored: Sun Jul 10 00:12:01 2016 +0530 Committer: Shwetha GS <[email protected]> Committed: Sun Jul 10 00:12:01 2016 +0530 ---------------------------------------------------------------------- 0.7.0-incubating/Architecture.html | 261 +++++++ .../Authentication-Authorization.html | 382 ++++++++++ 0.7.0-incubating/Bridge-Falcon.html | 279 +++++++ 0.7.0-incubating/Bridge-Hive.html | 325 ++++++++ 0.7.0-incubating/Bridge-Sqoop.html | 262 +++++++ 0.7.0-incubating/Configuration.html | 459 +++++++++++ 0.7.0-incubating/HighAvailability.html | 405 ++++++++++ 0.7.0-incubating/InstallationSteps.html | 556 ++++++++++++++ 0.7.0-incubating/Notification-Entity.html | 263 +++++++ 0.7.0-incubating/QuickStart.html | 254 +++++++ 0.7.0-incubating/Repository.html | 235 ++++++ 0.7.0-incubating/Search.html | 341 +++++++++ 0.7.0-incubating/StormAtlasHook.html | 304 ++++++++ 0.7.0-incubating/TypeSystem.html | 258 +++++++ 0.7.0-incubating/api/apple-touch-icon.png | Bin 0 -> 640 bytes 0.7.0-incubating/api/application.wadl | 674 ++++++++++++++++ 0.7.0-incubating/api/atlas-webapp-php.zip | Bin 0 -> 2066 bytes 0.7.0-incubating/api/atlas-webapp.rb | 246 ++++++ 0.7.0-incubating/api/crossdomain.xml | 25 + 0.7.0-incubating/api/css/home.gif | Bin 0 -> 128 bytes 0.7.0-incubating/api/css/prettify.css | 1 + 0.7.0-incubating/api/css/style.css | 759 +++++++++++++++++++ 0.7.0-incubating/api/downloads.html | 163 ++++ 0.7.0-incubating/api/el_ns0_errorBean.html | 132 ++++ 0.7.0-incubating/api/el_ns0_results.html | 131 ++++ 0.7.0-incubating/api/favicon.ico | Bin 0 -> 1150 bytes 0.7.0-incubating/api/index.html | 157 ++++ 0.7.0-incubating/api/js/libs/dd_belatedpng.js | 13 + .../api/js/libs/jquery-1.5.1.min.js | 16 + .../api/js/libs/modernizr-1.7.min.js | 2 + .../api/js/libs/prettify/lang-apollo.js | 2 + .../api/js/libs/prettify/lang-clj.js | 18 + .../api/js/libs/prettify/lang-css.js | 2 + .../api/js/libs/prettify/lang-go.js | 1 + .../api/js/libs/prettify/lang-hs.js | 2 + .../api/js/libs/prettify/lang-lisp.js | 3 + .../api/js/libs/prettify/lang-lua.js | 2 + .../api/js/libs/prettify/lang-ml.js | 2 + 0.7.0-incubating/api/js/libs/prettify/lang-n.js | 4 + .../api/js/libs/prettify/lang-proto.js | 1 + .../api/js/libs/prettify/lang-scala.js | 2 + .../api/js/libs/prettify/lang-sql.js | 2 + .../api/js/libs/prettify/lang-tex.js | 1 + .../api/js/libs/prettify/lang-vb.js | 2 + .../api/js/libs/prettify/lang-vhdl.js | 3 + .../api/js/libs/prettify/lang-wiki.js | 2 + .../api/js/libs/prettify/lang-xq.js | 3 + .../api/js/libs/prettify/lang-yaml.js | 2 + .../api/js/libs/prettify/prettify.js | 28 + 0.7.0-incubating/api/js/libs/xbreadcrumbs.js | 93 +++ 0.7.0-incubating/api/model.html | 140 ++++ 0.7.0-incubating/api/ns0.html | 131 ++++ 0.7.0-incubating/api/ns0.xsd | 25 + 0.7.0-incubating/api/ns0_errorBean.html | 189 +++++ 0.7.0-incubating/api/ns0_results.html | 173 +++++ .../api/resource_AdminResource.html | 199 +++++ .../api/resource_DataSetLineageResource.html | 234 ++++++ .../api/resource_EntityResource.html | 599 +++++++++++++++ .../api/resource_EntityService.html | 376 +++++++++ .../api/resource_LineageResource.html | 234 ++++++ .../api/resource_MetadataDiscoveryResource.html | 273 +++++++ .../api/resource_TaxonomyService.html | 758 ++++++++++++++++++ .../api/resource_TypesResource.html | 249 ++++++ 0.7.0-incubating/api/rest.html | 120 +++ 0.7.0-incubating/api/robots.txt | 5 + .../css/apache-maven-fluido-1.3.0.min.css | 9 + 0.7.0-incubating/css/print.css | 23 + 0.7.0-incubating/css/site.css | 1 + .../images/accessories-text-editor.png | Bin 0 -> 746 bytes 0.7.0-incubating/images/add.gif | Bin 0 -> 397 bytes .../images/apache-incubator-logo.png | Bin 0 -> 4234 bytes .../images/apache-maven-project-2.png | Bin 0 -> 33442 bytes .../images/application-certificate.png | Bin 0 -> 923 bytes 0.7.0-incubating/images/atlas-logo.png | Bin 0 -> 3115 bytes 0.7.0-incubating/images/contact-new.png | Bin 0 -> 736 bytes 0.7.0-incubating/images/document-properties.png | Bin 0 -> 577 bytes 0.7.0-incubating/images/drive-harddisk.png | Bin 0 -> 700 bytes 0.7.0-incubating/images/fix.gif | Bin 0 -> 366 bytes 0.7.0-incubating/images/icon_error_sml.gif | Bin 0 -> 633 bytes 0.7.0-incubating/images/icon_help_sml.gif | Bin 0 -> 1072 bytes 0.7.0-incubating/images/icon_info_sml.gif | Bin 0 -> 638 bytes 0.7.0-incubating/images/icon_success_sml.gif | Bin 0 -> 604 bytes 0.7.0-incubating/images/icon_warning_sml.gif | Bin 0 -> 625 bytes 0.7.0-incubating/images/image-x-generic.png | Bin 0 -> 662 bytes .../images/internet-web-browser.png | Bin 0 -> 1017 bytes .../images/logos/build-by-maven-black.png | Bin 0 -> 2294 bytes .../images/logos/build-by-maven-white.png | Bin 0 -> 2260 bytes 0.7.0-incubating/images/logos/maven-feather.png | Bin 0 -> 3330 bytes 0.7.0-incubating/images/network-server.png | Bin 0 -> 536 bytes 0.7.0-incubating/images/package-x-generic.png | Bin 0 -> 717 bytes .../images/profiles/pre-release.png | Bin 0 -> 32607 bytes 0.7.0-incubating/images/profiles/retired.png | Bin 0 -> 22003 bytes 0.7.0-incubating/images/profiles/sandbox.png | Bin 0 -> 33010 bytes 0.7.0-incubating/images/remove.gif | Bin 0 -> 607 bytes 0.7.0-incubating/images/rss.png | Bin 0 -> 474 bytes 0.7.0-incubating/images/twiki/architecture.png | Bin 0 -> 58775 bytes 0.7.0-incubating/images/twiki/data-types.png | Bin 0 -> 413738 bytes .../images/twiki/guide-class-diagram.png | Bin 0 -> 40375 bytes .../images/twiki/guide-instance-graph.png | Bin 0 -> 179941 bytes 0.7.0-incubating/images/twiki/notification.png | Bin 0 -> 137448 bytes .../images/twiki/types-instance.png | Bin 0 -> 445893 bytes 0.7.0-incubating/images/update.gif | Bin 0 -> 1090 bytes 0.7.0-incubating/images/window-new.png | Bin 0 -> 583 bytes .../img/glyphicons-halflings-white.png | Bin 0 -> 8777 bytes 0.7.0-incubating/img/glyphicons-halflings.png | Bin 0 -> 12799 bytes 0.7.0-incubating/index.html | 304 ++++++++ 0.7.0-incubating/issue-tracking.html | 239 ++++++ .../js/apache-maven-fluido-1.3.0.min.js | 21 + 0.7.0-incubating/license.html | 441 +++++++++++ 0.7.0-incubating/mail-lists.html | 253 +++++++ 0.7.0-incubating/project-info.html | 258 +++++++ 0.7.0-incubating/security.html | 501 ++++++++++++ 0.7.0-incubating/source-repository.html | 252 ++++++ 0.7.0-incubating/team-list.html | 463 +++++++++++ Architecture.html | 16 +- Authentication-Authorization.html | 16 +- Bridge-Falcon.html | 16 +- Bridge-Hive.html | 16 +- Bridge-Sqoop.html | 16 +- Configuration.html | 16 +- HighAvailability.html | 16 +- InstallationSteps.html | 16 +- Notification-Entity.html | 16 +- QuickStart.html | 16 +- Repository.html | 16 +- Search.html | 16 +- Security.html | 16 +- StormAtlasHook.html | 16 +- TypeSystem.html | 16 +- api/atlas-webapp-php.zip | Bin 0 -> 2066 bytes api/atlas-webapp.rb | 246 ++++++ api/downloads.html | 163 ++++ api/el_ns0_errorBean.html | 132 ++++ api/el_ns0_results.html | 131 ++++ api/index.html | 32 + api/model.html | 29 + api/ns0.html | 131 ++++ api/ns0.xsd | 25 + api/ns0_errorBean.html | 189 +++++ api/ns0_results.html | 173 +++++ api/rest.html | 6 + index.html | 14 +- issue-tracking.html | 16 +- license.html | 16 +- mail-lists.html | 16 +- project-info.html | 16 +- source-repository.html | 16 +- team-list.html | 16 +- 148 files changed, 15046 insertions(+), 109 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/60041d8d/0.7.0-incubating/Architecture.html ---------------------------------------------------------------------- diff --git a/0.7.0-incubating/Architecture.html b/0.7.0-incubating/Architecture.html new file mode 100644 index 0000000..6c62d81 --- /dev/null +++ b/0.7.0-incubating/Architecture.html @@ -0,0 +1,261 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-07-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160709" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Architecture</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Architecture</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-07-09</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Architecture"></a>Architecture</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3></div> +<div class="section"> +<h3><a name="Atlas_High_Level_Architecture_-_Overview"></a>Atlas High Level Architecture - Overview</h3> +<p><img src="images/twiki/architecture.png" alt="" /></p> +<p>Architecturally, Atlas has the following components:</p> +<p></p> +<ul> +<li><b>A Web service</b>: This exposes RESTful APIs and a Web user interface to create, update and query metadata.</li> +<li><b>Metadata store</b>: Metadata is modeled using a graph model, implemented using the Graph database Titan. Titan has options for a variety of backing stores for persisting the graph, including an embedded Berkeley DB, Apache HBase and Apache Cassandra. The choice of the backing store determines the level of service availability.</li> +<li><b>Index store</b>: For powering full text searches on metadata, Atlas also indexes the metadata, again via Titan. The backing store for the full text search is a search backend like ElasticSearch or Apache Solr.</li> +<li><b>Bridges / Hooks</b>: To add metadata to Atlas, libraries called ‘hooks’ are enabled in various systems like Apache Hive, Apache Falcon and Apache Sqoop which capture metadata events in the respective systems and propagate those events to Atlas. The Atlas server consumes these events and updates its stores.</li> +<li><b>Metadata notification events</b>: Any updates to metadata in Atlas, either via the Hooks or the API are propagated from Atlas to downstream systems via events. Systems like Apache Ranger consume these events and allow administrators to act on them, for e.g. to configure policies for Access control.</li> +<li><b>Notification Server</b>: Atlas uses Apache Kafka as a notification server for communication between hooks and downstream consumers of metadata notification events. Events are written by the hooks and Atlas to different Kafka topics. Kafka enables a loosely coupled integration between these disparate systems.</li></ul></div> +<div class="section"> +<h3><a name="Bridges"></a>Bridges</h3> +<p>External components like hive/sqoop/storm/falcon should model their taxonomy using typesystem and register the types with Atlas. For every entity created in this external component, the corresponding entity should be registered in Atlas as well. This is typically done in a hook which runs in the external component and is called for every entity operation. Hook generally processes the entity asynchronously using a thread pool to avoid adding latency to the main operation. The hook can then build the entity and register the entity using Atlas REST APIs. Howerver, any failure in APIs because of network issue etc can result in entity not registered in Atlas and hence inconsistent metadata.</p> +<p>Atlas exposes notification interface and can be used for reliable entity registration by hook as well. The hook can send notification message containing the list of entities to be registered. Atlas service contains hook consumer that listens to these messages and registers the entities.</p> +<p>Available bridges are:</p> +<ul> +<li><a href="./Bridge-Hive.html">Hive Bridge</a></li> +<li><a href="./Bridge-Sqoop.html">Sqoop Bridge</a></li> +<li><a href="./Bridge-Falcon.html">Falcon Bridge</a></li> +<li><a href="./StormAtlasHook.html">Storm Bridge</a></li></ul></div> +<div class="section"> +<h3><a name="Notification"></a>Notification</h3> +<p>Notification is used for reliable entity registration from hooks and for entity/type change notifications. Atlas, by default, provides Kafka integration, but its possible to provide other implementations as well. Atlas service starts embedded Kafka server by default.</p> +<p>Atlas also provides NotificationHookConsumer that runs in Atlas Service and listens to messages from hook and registers the entities in Atlas. <img src="images/twiki/notification.png" alt="" /></p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/60041d8d/0.7.0-incubating/Authentication-Authorization.html ---------------------------------------------------------------------- diff --git a/0.7.0-incubating/Authentication-Authorization.html b/0.7.0-incubating/Authentication-Authorization.html new file mode 100644 index 0000000..9800a2f --- /dev/null +++ b/0.7.0-incubating/Authentication-Authorization.html @@ -0,0 +1,382 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-07-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160709" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Authentication & Authorization in Apache Atlas.</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Authentication & Authorization in Apache Atlas.</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-07-09</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h3><a name="Authentication__Authorization_in_Apache_Atlas."></a>Authentication & Authorization in Apache Atlas.</h3></div> +<div class="section"> +<h4><a name="Authentication"></a>Authentication</h4> +<p>Atlas supports following authentication methods</p> +<p></p> +<ul> +<li><b>File</b></li> +<li><b>Kerberos</b></li> +<li><b>LDAP</b></li></ul> +<p>Following properties should be set true to enable the authentication of that type in <tt>atlas-application.properties</tt> file.</p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos=true|false +atlas.authentication.method.ldap=true|false +atlas.authentication.method.file=true|false + +</pre></div> +<p>If two or more authentication methods are set to true, then the authentication falls back to the latter method if the earlier one fails. For example if Kerberos authentication is set to true and ldap authentication is also set to true then, if for a request without kerberos principal and keytab LDAP authentication will be used as a fallback scenario.</p></div> +<div class="section"> +<h5><a name="FILE_method."></a>FILE method.</h5> +<p>File authentication requires users' login details in users credentials file in the format specified below and the file path should set to property <tt>atlas.authentication.method.file.filename</tt> in <tt>atlas-application.properties</tt>.</p> +<div class="source"> +<pre> +atlas.authentication.method.file=true +atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties + +</pre></div> +<p>The users credentials file should have below format</p> +<div class="source"> +<pre> +username=group::sha256-password + +</pre></div> +<p>For e.g.</p> +<div class="source"> +<pre> +admin=ADMIN::e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a + +</pre></div> +<p>Users group can be either <b>ADMIN</b>, <b>DATA_STEWARD</b> OR <b>DATA_SCIENTIST</b></p> +<p><b>Note</b>:-password is encoded with sha256 encoding method and can be generated using unix tool.</p> +<p>For e.g.</p> +<div class="source"> +<pre> +echo -n "Password" | sha256sum +e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a - + +</pre></div></div> +<div class="section"> +<h5><a name="Kerberos_Method."></a>Kerberos Method.</h5> +<p>To enable the authentication in Kerberos mode in Atlas, set the property <tt>atlas.authentication.method.kerberos</tt> to true in <tt>atlas-application.properties</tt></p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos = true + +</pre></div> +<p>Also following properties should be set.</p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos.principal=<principal>/<fqdn>@EXAMPLE.COM +atlas.authentication.method.kerberos.keytab = /<key tab filepath>.keytab +atlas.authentication.method.kerberos.name.rules = RULE:[2:$1@$0]([email protected])s/.*/atlas/ + +</pre></div></div> +<div class="section"> +<h5><a name="LDAP_Method."></a>LDAP Method.</h5> +<p>To enable the authentication in LDAP mode in Atlas, set the property <tt>atlas.authentication.method.ldap</tt> to true and also set Ldap type to property <tt>atlas.authentication.method.ldap.type</tt> to LDAP or AD in <tt>atlas-application.properties</tt>. Use AD if connecting to Active Directory.</p> +<div class="source"> +<pre> +atlas.authentication.method.ldap=true +atlas.authentication.method.ldap.type=ldap|ad + +</pre></div> +<p>For LDAP or AD the following configuration needs to be set in atlas application properties.</p> +<p><b>Active Directory</b></p> +<div class="source"> +<pre> +atlas.authentication.method.ldap.ad.domain= example.com +atlas.authentication.method.ldap.ad.url=ldap://<AD server ip>:389 +atlas.authentication.method.ldap.ad.base.dn=DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.dn=CN=Administrator,CN=Users,DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.password=<password> +atlas.authentication.method.ldap.ad.referral=ignore +atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0}) +atlas.authentication.method.ldap.ad.default.role=ROLE_USER + +</pre></div> +<p><b>LDAP Directroy</b></p> +<div class="source"> +<pre> +atlas.authentication.method.ldap.url=ldap://<Ldap server ip>:389 +atlas.authentication.method.ldap.userDNpattern=uid={0],ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupSearchBase=dc=example,dc=com +atlas.authentication.method.ldap.groupSearchFilter=(member=cn={0},ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupRoleAttribute=cn +atlas.authentication.method.ldap.base.dn=dc=example,dc=com +atlas.authentication.method.ldap.bind.dn=cn=Manager,dc=example,dc=com +atlas.authentication.method.ldap.bind.password=<password> +atlas.authentication.method.ldap.referral=ignore +atlas.authentication.method.ldap.user.searchfilter=(uid={0}) +atlas.authentication.method.ldap.default.role=ROLE_USER + +</pre></div></div> +<div class="section"> +<h4><a name="Authorization"></a>Authorization</h4></div> +<div class="section"> +<h5><a name="Atlas_Authorization_Methods_SimpleRanger"></a>Atlas Authorization Methods [Simple/Ranger]</h5> +<p>To set authorization in atlas, update the <tt>atlas.authorizer.impl</tt> properties in <tt>atlas-application.properties</tt></p> +<ul> +<li><b>Simple</b></li> +<li><b>Ranger</b></li></ul> +<div class="source"> +<pre> +atlas.authorizer.impl=simple | ranger | <Qualified Authorizer Class Name> + +</pre></div></div> +<div class="section"> +<h5><a name="Simple_Authorizer."></a>Simple Authorizer.</h5> +<p>In Simple Authorizer the policy store file is configured locally. The path of policy store file is set in <tt>atlas.auth.policy.file</tt> property of <tt>atlas-application.properties</tt></p> +<div class="source"> +<pre> +atlas.auth.policy.file={{conf_dir}}/policy-store.txt + +</pre></div> +<p>The policy store file format is as follows:</p> +<div class="source"> +<pre> +Policy_Name;;User_Name:Operations_Allowed;;Group_Name:Operations_Allowed;;Resource_Type:Resource_Name + +</pre></div> +<p>eg. of admin policy:</p> +<div class="source"> +<pre> +adminPolicy;;admin:rwud;;ROLE_ADMIN:rwud;;type:*,entity:*,operation:*,taxonomy:*,term:* + +</pre></div> +<p>Note : The User_Name, Group_Name and Operations_Allowed are comma(,) separated lists.</p> +<p>Authorizer Resource Types:</p> +<ul> +<li>Operation</li> +<li>Type</li> +<li>Entity</li> +<li>Taxonomy</li> +<li>Term</li> +<li>Unknown</li></ul> +<p>Operations_Allowed are r = read, w = write, u = update, d = delete</p></div> +<div class="section"> +<h5><a name="Ranger_Authorizer."></a>Ranger Authorizer.</h5> +<p>Ranger Authorizer is enabled by activating Atlas-Ranger plugin from Ambari.</p> +<p>For more details visit the <a class="externalLink" href="http://ranger.apache.org/";>Apache-Ranger documentation</a>.</p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/60041d8d/0.7.0-incubating/Bridge-Falcon.html ---------------------------------------------------------------------- diff --git a/0.7.0-incubating/Bridge-Falcon.html b/0.7.0-incubating/Bridge-Falcon.html new file mode 100644 index 0000000..bc0dbb8 --- /dev/null +++ b/0.7.0-incubating/Bridge-Falcon.html @@ -0,0 +1,279 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-07-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160709" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Falcon Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Falcon Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-07-09</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Falcon_Model"></a>Falcon Model</h3> +<p>The default falcon modelling is available in org.apache.atlas.falcon.model.FalconDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +falcon_cluster(ClassType) - super types [Infrastructure] - attributes [timestamp, colo, owner, tags] +falcon_feed(ClassType) - super types [DataSet] - attributes [timestamp, stored-in, owner, groups, tags] +falcon_feed_creation(ClassType) - super types [Process] - attributes [timestamp, stored-in, owner] +falcon_feed_replication(ClassType) - super types [Process] - attributes [timestamp, owner] +falcon_process(ClassType) - super types [Process] - attributes [timestamp, runs-on, owner, tags, pipelines, workflow-properties] + +</pre></div> +<p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p> +<p>The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p> +<ul> +<li>falcon_process - <process name>@<cluster name></li> +<li>falcon_cluster - <cluster name></li> +<li>falcon_feed - <feed name>@<cluster name></li> +<li>falcon_feed_creation - <feed name></li> +<li>falcon_feed_replication - <feed name></li></ul></div> +<div class="section"> +<h3><a name="Falcon_Hook"></a>Falcon Hook</h3> +<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model defined in org.apache.atlas.falcon.model.FalconDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.</p> +<ul> +<li>Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li> +<li>Link falcon hook jars in falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> +<li>In <falcon_conf>/falcon-env.sh, set an environment variable as follows:</li></ul> +<div class="source"> +<pre> + export FALCON_SERVER_OPTS="$FALCON_SERVER_OPTS -Datlas.conf=<atlas-conf>" + +</pre></div> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> +<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/60041d8d/0.7.0-incubating/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/0.7.0-incubating/Bridge-Hive.html b/0.7.0-incubating/Bridge-Hive.html new file mode 100644 index 0000000..0197417 --- /dev/null +++ b/0.7.0-incubating/Bridge-Hive.html @@ -0,0 +1,325 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-07-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160709" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Hive Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Hive Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-07-09</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Hive_Atlas_Bridge"></a>Hive Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Hive_Model"></a>Hive Model</h3> +<p>The default hive modelling is available in org.apache.atlas.hive.model.HiveDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +hive_db(ClassType) - super types [Referenceable] - attributes [name, clusterName, description, locationUri, parameters, ownerName, ownerType] +hive_storagedesc(ClassType) - super types [Referenceable] - attributes [cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories] +hive_column(ClassType) - super types [Referenceable] - attributes [name, type, comment, table] +hive_table(ClassType) - super types [DataSet] - attributes [name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary] +hive_process(ClassType) - super types [Process] - attributes [name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId] +hive_principal_type(EnumType) - values [USER, ROLE, GROUP] +hive_order(StructType) - attributes [col, order] +hive_serde(StructType) - attributes [name, serializationLib, parameters] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.</p> +<ul> +<li>hive_db - attribute qualifiedName - <dbName>@<clusterName></li> +<li>hive_table - attribute qualifiedName - <dbName>.<tableName>@<clusterName></li> +<li>hive_column - attribute qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> +<li>hive_process - attribute name - <queryString> - trimmed query string in lower case</li></ul></div> +<div class="section"> +<h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3> +<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator.</p> +<p>import-hive.sh command can be used to facilitate this. The script needs Hadoop and Hive classpath jars.</p> +<p>* For Hadoop jars, please make sure that the environment variable HADOOP_CLASSPATH is set. Another way is to set HADOOP_HOME to point to root directory of your Hadoop installation</p> +<p>* Similarly, for Hive jars, set HIVE_HOME to the root of Hive installation</p> +<p>* Set environment variable HIVE_CONF_DIR to Hive configuration directory</p> +<div class="source"> +<pre> + Usage: <atlas package>/bin/import-hive.sh + +</pre></div> +<p>The logs are in <atlas package>/logs/import-hive.log</p> +<p>If you you are importing metadata in a kerberized cluster you need to run the command like this:</p> +<div class="source"> +<pre> +<atlas package>/bin/import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=[krb5.conf location] -Djava.security.auth.login.config=[jaas.conf location] + +</pre></div> +<p></p> +<ul> +<li>krb5.conf is typically found at /etc/krb5.conf</li> +<li>for details about jaas.conf and a suggested location see the <a href="./Security.html">atlas security documentation</a></li></ul></div> +<div class="section"> +<h3><a name="Hive_Hook"></a>Hive Hook</h3> +<p>Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities. Follow these instructions in your hive set-up to add hive hook for Atlas:</p> +<ul> +<li>Set-up atlas hook in hive-site.xml of your hive configuration:</li></ul> +<div class="source"> +<pre> + <property> + <name>hive.exec.post.hooks</name> + <value>org.apache.atlas.hive.hook.HiveHook</value> + </property> + +</pre></div> +<div class="source"> +<pre> + <property> + <name>atlas.cluster.name</name> + <value>primary</value> + </property> + +</pre></div> +<p></p> +<ul> +<li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li> +<li>Copy <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.</li> +<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.hive.minThreads - core number of threads. default 5</li> +<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li> +<li>The following hive operations are captured by hive hook currently +<ul> +<li>create database</li> +<li>create table/view, create table as select</li> +<li>load, import, export</li> +<li>DMLs (insert)</li> +<li>alter database</li> +<li>alter table (skewed table information, stored as, protection is not supported)</li> +<li>alter view</li></ul></li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/60041d8d/0.7.0-incubating/Bridge-Sqoop.html ---------------------------------------------------------------------- diff --git a/0.7.0-incubating/Bridge-Sqoop.html b/0.7.0-incubating/Bridge-Sqoop.html new file mode 100644 index 0000000..cb2a224 --- /dev/null +++ b/0.7.0-incubating/Bridge-Sqoop.html @@ -0,0 +1,262 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2016-07-09 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20160709" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Sqoop Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Sqoop Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2016-07-09</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.7-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Sqoop_Model"></a>Sqoop Model</h3> +<p>The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] +sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] +sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] +sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source</p></div> +<div class="section"> +<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> +<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:</p> +<p></p> +<ul> +<li>Sqoop Job publisher class. Currently only one publishing class is supported</li></ul><property> <name>sqoop.job.data.publish.class</name> <value>org.apache.atlas.sqoop.hook.SqoopHook</value> </property> +<ul> +<li>Atlas cluster name</li></ul><property> <name>atlas.cluster.name</name> <value><clustername></value> </property> +<ul> +<li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li> +<li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2016 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
