updated site for 0.8 release
Project: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/commit/ff453c82 Tree: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/tree/ff453c82 Diff: http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/diff/ff453c82 Branch: refs/heads/asf-site Commit: ff453c8256711cbcdf60eab40af5c27ef27665c8 Parents: d3bcb17 Author: Madhan Neethiraj <[email protected]> Authored: Thu Mar 16 22:32:33 2017 -0700 Committer: Madhan Neethiraj <[email protected]> Committed: Thu Mar 16 22:32:33 2017 -0700 ---------------------------------------------------------------------- 0.8.0-incubating/Architecture.html | 280 +++++++++ .../Authentication-Authorization.html | 394 +++++++++++++ 0.8.0-incubating/Bridge-Falcon.html | 291 ++++++++++ 0.8.0-incubating/Bridge-Hive.html | 374 ++++++++++++ 0.8.0-incubating/Bridge-Sqoop.html | 274 +++++++++ 0.8.0-incubating/Configuration.html | 508 +++++++++++++++++ 0.8.0-incubating/EclipseSetup.html | 336 +++++++++++ 0.8.0-incubating/HighAvailability.html | 417 ++++++++++++++ 0.8.0-incubating/InstallationSteps.html | 571 +++++++++++++++++++ 0.8.0-incubating/Notification-Entity.html | 275 +++++++++ 0.8.0-incubating/QuickStart.html | 266 +++++++++ 0.8.0-incubating/Repository.html | 247 ++++++++ 0.8.0-incubating/Search.html | 368 ++++++++++++ 0.8.0-incubating/StormAtlasHook.html | 316 ++++++++++ 0.8.0-incubating/TypeSystem.html | 406 +++++++++++++ .../css/apache-maven-fluido-1.3.0.min.css | 9 + 0.8.0-incubating/css/print.css | 23 + 0.8.0-incubating/css/site.css | 1 + .../images/accessories-text-editor.png | Bin 0 -> 746 bytes 0.8.0-incubating/images/add.gif | Bin 0 -> 397 bytes .../images/apache-incubator-logo.png | Bin 0 -> 4234 bytes .../images/apache-maven-project-2.png | Bin 0 -> 33442 bytes .../images/application-certificate.png | Bin 0 -> 923 bytes 0.8.0-incubating/images/atlas-logo.png | Bin 0 -> 3115 bytes 0.8.0-incubating/images/column_lineage_ex1.png | Bin 0 -> 34057 bytes 0.8.0-incubating/images/contact-new.png | Bin 0 -> 736 bytes 0.8.0-incubating/images/document-properties.png | Bin 0 -> 577 bytes 0.8.0-incubating/images/drive-harddisk.png | Bin 0 -> 700 bytes 0.8.0-incubating/images/fix.gif | Bin 0 -> 366 bytes 0.8.0-incubating/images/icon_error_sml.gif | Bin 0 -> 633 bytes 0.8.0-incubating/images/icon_help_sml.gif | Bin 0 -> 1072 bytes 0.8.0-incubating/images/icon_info_sml.gif | Bin 0 -> 638 bytes 0.8.0-incubating/images/icon_success_sml.gif | Bin 0 -> 604 bytes 0.8.0-incubating/images/icon_warning_sml.gif | Bin 0 -> 625 bytes 0.8.0-incubating/images/image-x-generic.png | Bin 0 -> 662 bytes .../images/internet-web-browser.png | Bin 0 -> 1017 bytes .../images/logos/build-by-maven-black.png | Bin 0 -> 2294 bytes .../images/logos/build-by-maven-white.png | Bin 0 -> 2260 bytes 0.8.0-incubating/images/logos/maven-feather.png | Bin 0 -> 3330 bytes 0.8.0-incubating/images/network-server.png | Bin 0 -> 536 bytes 0.8.0-incubating/images/package-x-generic.png | Bin 0 -> 717 bytes .../images/profiles/pre-release.png | Bin 0 -> 32607 bytes 0.8.0-incubating/images/profiles/retired.png | Bin 0 -> 22003 bytes 0.8.0-incubating/images/profiles/sandbox.png | Bin 0 -> 33010 bytes 0.8.0-incubating/images/remove.gif | Bin 0 -> 607 bytes 0.8.0-incubating/images/rss.png | Bin 0 -> 474 bytes 0.8.0-incubating/images/twiki/architecture.png | Bin 0 -> 315403 bytes .../images/twiki/guide-class-diagram.png | Bin 0 -> 40375 bytes .../images/twiki/guide-instance-graph.png | Bin 0 -> 179941 bytes 0.8.0-incubating/images/update.gif | Bin 0 -> 1090 bytes 0.8.0-incubating/images/window-new.png | Bin 0 -> 583 bytes .../img/glyphicons-halflings-white.png | Bin 0 -> 8777 bytes 0.8.0-incubating/img/glyphicons-halflings.png | Bin 0 -> 12799 bytes 0.8.0-incubating/index.html | 322 +++++++++++ 0.8.0-incubating/issue-tracking.html | 251 ++++++++ .../js/apache-maven-fluido-1.3.0.min.js | 21 + 0.8.0-incubating/license.html | 453 +++++++++++++++ 0.8.0-incubating/mail-lists.html | 265 +++++++++ 0.8.0-incubating/project-info.html | 270 +++++++++ 0.8.0-incubating/security.html | 513 +++++++++++++++++ 0.8.0-incubating/source-repository.html | 264 +++++++++ 0.8.0-incubating/team-list.html | 547 ++++++++++++++++++ Architecture.html | 16 +- Authentication-Authorization.html | 16 +- Bridge-Falcon.html | 18 +- Bridge-Hive.html | 61 +- Bridge-Sqoop.html | 16 +- Configuration.html | 39 +- EclipseSetup.html | 16 +- HighAvailability.html | 16 +- InstallationSteps.html | 17 +- Notification-Entity.html | 16 +- QuickStart.html | 16 +- Repository.html | 16 +- Search.html | 45 +- Security.html | 16 +- StormAtlasHook.html | 16 +- TypeSystem.html | 16 +- images/column_lineage_ex1.png | Bin 0 -> 34057 bytes index.html | 16 +- issue-tracking.html | 16 +- license.html | 16 +- mail-lists.html | 16 +- project-info.html | 16 +- source-repository.html | 16 +- team-list.html | 39 +- 86 files changed, 8621 insertions(+), 132 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/ff453c82/0.8.0-incubating/Architecture.html ---------------------------------------------------------------------- diff --git a/0.8.0-incubating/Architecture.html b/0.8.0-incubating/Architecture.html new file mode 100644 index 0000000..d189b65 --- /dev/null +++ b/0.8.0-incubating/Architecture.html @@ -0,0 +1,280 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-03-16 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170316" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Architecture</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Architecture</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Architecture"></a>Architecture</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3></div> +<div class="section"> +<h3><a name="Atlas_High_Level_Architecture_-_Overview"></a>Atlas High Level Architecture - Overview</h3> +<p><img src="images/twiki/architecture.png" alt="" /></p> +<p>The components of Atlas can be grouped under the following major categories:</p></div> +<div class="section"> +<h4><a name="Core"></a>Core</h4> +<p>This category contains the components that implement the core of Atlas functionality, including:</p> +<p><b>Type System</b>: Atlas allows users to define a model for the metadata objects they want to manage. The model is composed of definitions called ‘types’. Instances of ‘types’ called ‘entities’ represent the actual metadata objects that are managed. The Type System is a component that allows users to define and manage the types and entities. All metadata objects managed by Atlas out of the box (like Hive tables, for e.g.) are modelled using types and represented as entities. To store new types of metadata in Atlas, one needs to understand the concepts of the type system component.</p> +<p>One key point to note is that the generic nature of the modelling in Atlas allows data stewards and integrators to define both technical metadata and business metadata. It is also possible to define rich relationships between the two using features of Atlas.</p> +<p><b>Ingest / Export</b>: The Ingest component allows metadata to be added to Atlas. Similarly, the Export component exposes metadata changes detected by Atlas to be raised as events. Consumers can consume these change events to react to metadata changes in real time.</p> +<p><b>Graph Engine</b>: Internally, Atlas represents metadata objects it manages using a Graph model. It does this to achieve great flexibility and rich relations between the metadata objects. The Graph Engine is a component that is responsible for translating between types and entities of the Type System, and the underlying Graph model. In addition to managing the Graph objects, The Graph Engine also creates the appropriate indices for the metadata objects so that they can be searched for efficiently.</p> +<p><b>Titan</b>: Currently, Atlas uses the Titan Graph Database to store the metadata objects. Titan is used as a library within Atlas. Titan uses two stores: The Metadata store is configured to !HBase by default and the Index store is configured to Solr. It is also possible to use the Metadata store as BerkeleyDB and Index store as ElasticSearch by building with corresponding profiles. The Metadata store is used for storing the metadata objects proper, and the Index store is used for storing indices of the Metadata properties, that allows efficient search.</p></div> +<div class="section"> +<h4><a name="Integration"></a>Integration</h4> +<p>Users can manage metadata in Atlas using two methods:</p> +<p><b>API</b>: All functionality of Atlas is exposed to end users via a REST API that allows types and entities to be created, updated and deleted. It is also the primary mechanism to query and discover the types and entities managed by Atlas.</p> +<p><b>Messaging</b>: In addition to the API, users can choose to integrate with Atlas using a messaging interface that is based on Kafka. This is useful both for communicating metadata objects to Atlas, and also to consume metadata change events from Atlas using which applications can be built. The messaging interface is particularly useful if one wishes to use a more loosely coupled integration with Atlas that could allow for better scalability, reliability etc. Atlas uses Apache Kafka as a notification server for communication between hooks and downstream consumers of metadata notification events. Events are written by the hooks and Atlas to different Kafka topics.</p></div> +<div class="section"> +<h4><a name="Metadata_sources"></a>Metadata sources</h4> +<p>Atlas supports integration with many sources of metadata out of the box. More integrations will be added in future as well. Currently, Atlas supports ingesting and managing metadata from the following sources:</p> +<p></p> +<ul> +<li><a href="./Bridge-Hive.html">Hive</a></li> +<li><a href="./Bridge-Sqoop.html">Sqoop</a></li> +<li><a href="./Bridge-Falcon.html">Falcon</a></li> +<li><a href="./StormAtlasHook.html">Storm</a></li></ul> +<p>The integration implies two things: There are metadata models that Atlas defines natively to represent objects of these components. There are components Atlas provides to ingest metadata objects from these components (in real time or in batch mode in some cases).</p></div> +<div class="section"> +<h4><a name="Applications"></a>Applications</h4> +<p>The metadata managed by Atlas is consumed by a variety of applications for satisfying many governance use cases.</p> +<p><b>Atlas Admin UI</b>: This component is a web based application that allows data stewards and scientists to discover and annotate metadata. Of primary importance here is a search interface and SQL like query language that can be used to query the metadata types and objects managed by Atlas. The Admin UI uses the REST API of Atlas for building its functionality.</p> +<p><b>Tag Based Policies</b>: <a class="externalLink" href="http://ranger.apache.org/";>Apache Ranger</a> is an advanced security management solution for the Hadoop ecosystem having wide integration with a variety of Hadoop components. By integrating with Atlas, Ranger allows security administrators to define metadata driven security policies for effective governance. Ranger is a consumer to the metadata change events notified by Atlas.</p> +<p><b>Business Taxonomy</b>: The metadata objects ingested into Atlas from the Metadata sources are primarily a form of technical metadata. To enhance the discoverability and governance capabilities, Atlas comes with a Business Taxonomy interface that allows users to first, define a hierarchical set of business terms that represent their business domain and associate them to the metadata entities Atlas manages. Business Taxonomy is a web application that is part of the Atlas Admin UI currently and integrates with Atlas using the REST API.</p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2017 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/ff453c82/0.8.0-incubating/Authentication-Authorization.html ---------------------------------------------------------------------- diff --git a/0.8.0-incubating/Authentication-Authorization.html b/0.8.0-incubating/Authentication-Authorization.html new file mode 100644 index 0000000..199f2f4 --- /dev/null +++ b/0.8.0-incubating/Authentication-Authorization.html @@ -0,0 +1,394 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-03-16 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170316" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Authentication & Authorization in Apache Atlas.</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Authentication & Authorization in Apache Atlas.</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h3><a name="Authentication__Authorization_in_Apache_Atlas."></a>Authentication & Authorization in Apache Atlas.</h3></div> +<div class="section"> +<h4><a name="Authentication"></a>Authentication</h4> +<p>Atlas supports following authentication methods</p> +<p></p> +<ul> +<li><b>File</b></li> +<li><b>Kerberos</b></li> +<li><b>LDAP</b></li></ul> +<p>Following properties should be set true to enable the authentication of that type in <tt>atlas-application.properties</tt> file.</p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos=true|false +atlas.authentication.method.ldap=true|false +atlas.authentication.method.file=true|false + +</pre></div> +<p>If two or more authentication methods are set to true, then the authentication falls back to the latter method if the earlier one fails. For example if Kerberos authentication is set to true and ldap authentication is also set to true then, if for a request without kerberos principal and keytab LDAP authentication will be used as a fallback scenario.</p></div> +<div class="section"> +<h5><a name="FILE_method."></a>FILE method.</h5> +<p>File authentication requires users' login details in users credentials file in the format specified below and the file path should set to property <tt>atlas.authentication.method.file.filename</tt> in <tt>atlas-application.properties</tt>.</p> +<div class="source"> +<pre> +atlas.authentication.method.file=true +atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties + +</pre></div> +<p>The users credentials file should have below format</p> +<div class="source"> +<pre> +username=group::sha256-password + +</pre></div> +<p>For e.g.</p> +<div class="source"> +<pre> +admin=ADMIN::e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a + +</pre></div> +<p>Users group can be either <b>ADMIN</b>, <b>DATA_STEWARD</b> OR <b>DATA_SCIENTIST</b></p> +<p><b>Note</b>:-password is encoded with sha256 encoding method and can be generated using unix tool.</p> +<p>For e.g.</p> +<div class="source"> +<pre> +echo -n "Password" | sha256sum +e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a - + +</pre></div></div> +<div class="section"> +<h5><a name="Kerberos_Method."></a>Kerberos Method.</h5> +<p>To enable the authentication in Kerberos mode in Atlas, set the property <tt>atlas.authentication.method.kerberos</tt> to true in <tt>atlas-application.properties</tt></p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos = true + +</pre></div> +<p>Also following properties should be set.</p> +<div class="source"> +<pre> +atlas.authentication.method.kerberos.principal=<principal>/<fqdn>@EXAMPLE.COM +atlas.authentication.method.kerberos.keytab = /<key tab filepath>.keytab +atlas.authentication.method.kerberos.name.rules = RULE:[2:$1@$0]([email protected])s/.*/atlas/ + +</pre></div></div> +<div class="section"> +<h5><a name="LDAP_Method."></a>LDAP Method.</h5> +<p>To enable the authentication in LDAP mode in Atlas, set the property <tt>atlas.authentication.method.ldap</tt> to true and also set Ldap type to property <tt>atlas.authentication.method.ldap.type</tt> to LDAP or AD in <tt>atlas-application.properties</tt>. Use AD if connecting to Active Directory.</p> +<div class="source"> +<pre> +atlas.authentication.method.ldap=true +atlas.authentication.method.ldap.type=ldap|ad + +</pre></div> +<p>For LDAP or AD the following configuration needs to be set in atlas application properties.</p> +<p><b>Active Directory</b></p> +<div class="source"> +<pre> +atlas.authentication.method.ldap.ad.domain= example.com +atlas.authentication.method.ldap.ad.url=ldap://<AD server ip>:389 +atlas.authentication.method.ldap.ad.base.dn=DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.dn=CN=Administrator,CN=Users,DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.password=<password> +atlas.authentication.method.ldap.ad.referral=ignore +atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0}) +atlas.authentication.method.ldap.ad.default.role=ROLE_USER + +</pre></div> +<p><b>LDAP Directroy</b></p> +<div class="source"> +<pre> +atlas.authentication.method.ldap.url=ldap://<Ldap server ip>:389 +atlas.authentication.method.ldap.userDNpattern=uid={0],ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupSearchBase=dc=example,dc=com +atlas.authentication.method.ldap.groupSearchFilter=(member=cn={0},ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupRoleAttribute=cn +atlas.authentication.method.ldap.base.dn=dc=example,dc=com +atlas.authentication.method.ldap.bind.dn=cn=Manager,dc=example,dc=com +atlas.authentication.method.ldap.bind.password=<password> +atlas.authentication.method.ldap.referral=ignore +atlas.authentication.method.ldap.user.searchfilter=(uid={0}) +atlas.authentication.method.ldap.default.role=ROLE_USER + +</pre></div></div> +<div class="section"> +<h4><a name="Authorization"></a>Authorization</h4></div> +<div class="section"> +<h5><a name="Atlas_Authorization_Methods_SimpleRanger"></a>Atlas Authorization Methods [Simple/Ranger]</h5> +<p>To set authorization in atlas, update the <tt>atlas.authorizer.impl</tt> properties in <tt>atlas-application.properties</tt></p> +<ul> +<li><b>Simple</b></li> +<li><b>Ranger</b></li></ul> +<div class="source"> +<pre> +atlas.authorizer.impl=simple | ranger | <Qualified Authorizer Class Name> + +</pre></div></div> +<div class="section"> +<h5><a name="Simple_Authorizer."></a>Simple Authorizer.</h5> +<p>In Simple Authorizer the policy store file is configured locally. The path of policy store file is set in <tt>atlas.auth.policy.file</tt> property of <tt>atlas-application.properties</tt></p> +<div class="source"> +<pre> +atlas.auth.policy.file={{conf_dir}}/policy-store.txt + +</pre></div> +<p>The policy store file format is as follows:</p> +<div class="source"> +<pre> +Policy_Name;;User_Name:Operations_Allowed;;Group_Name:Operations_Allowed;;Resource_Type:Resource_Name + +</pre></div> +<p>eg. of admin policy:</p> +<div class="source"> +<pre> +adminPolicy;;admin:rwud;;ROLE_ADMIN:rwud;;type:*,entity:*,operation:*,taxonomy:*,term:* + +</pre></div> +<p>Note : The User_Name, Group_Name and Operations_Allowed are comma(,) separated lists.</p> +<p>Authorizer Resource Types:</p> +<ul> +<li>Operation</li> +<li>Type</li> +<li>Entity</li> +<li>Taxonomy</li> +<li>Term</li> +<li>Unknown</li></ul> +<p>Operations_Allowed are r = read, w = write, u = update, d = delete</p></div> +<div class="section"> +<h5><a name="Ranger_Authorizer."></a>Ranger Authorizer.</h5> +<p>Ranger Authorizer is enabled by activating Atlas-Ranger plugin from Ambari.</p> +<p>For more details visit the <a class="externalLink" href="http://ranger.apache.org/";>Apache-Ranger documentation</a>.</p></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2017 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/ff453c82/0.8.0-incubating/Bridge-Falcon.html ---------------------------------------------------------------------- diff --git a/0.8.0-incubating/Bridge-Falcon.html b/0.8.0-incubating/Bridge-Falcon.html new file mode 100644 index 0000000..af81c20 --- /dev/null +++ b/0.8.0-incubating/Bridge-Falcon.html @@ -0,0 +1,291 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-03-16 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170316" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Falcon Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Falcon Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Falcon_Model"></a>Falcon Model</h3> +<p>The default falcon modelling is available in org.apache.atlas.falcon.model.FalconDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +falcon_cluster(ClassType) - super types [Infrastructure] - attributes [timestamp, colo, owner, tags] +falcon_feed(ClassType) - super types [DataSet] - attributes [timestamp, stored-in, owner, groups, tags] +falcon_feed_creation(ClassType) - super types [Process] - attributes [timestamp, stored-in, owner] +falcon_feed_replication(ClassType) - super types [Process] - attributes [timestamp, owner] +falcon_process(ClassType) - super types [Process] - attributes [timestamp, runs-on, owner, tags, pipelines, workflow-properties] + +</pre></div> +<p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p> +<p>The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p> +<ul> +<li>falcon_process - <process name>@<cluster name></li> +<li>falcon_cluster - <cluster name></li> +<li>falcon_feed - <feed name>@<cluster name></li> +<li>falcon_feed_creation - <feed name></li> +<li>falcon_feed_replication - <feed name></li></ul></div> +<div class="section"> +<h3><a name="Falcon_Hook"></a>Falcon Hook</h3> +<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model defined in org.apache.atlas.falcon.model.FalconDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.</p> +<ul> +<li>Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li> +<li>Link falcon hook jars in falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> +<li>In <falcon_conf>/falcon-env.sh, set an environment variable as follows:</li></ul> +<div class="source"> +<pre> + export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS" + +</pre></div> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> +<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2017 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/ff453c82/0.8.0-incubating/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/0.8.0-incubating/Bridge-Hive.html b/0.8.0-incubating/Bridge-Hive.html new file mode 100644 index 0000000..e549c72 --- /dev/null +++ b/0.8.0-incubating/Bridge-Hive.html @@ -0,0 +1,374 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-03-16 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170316" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Hive Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Hive Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Hive_Atlas_Bridge"></a>Hive Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Hive_Model"></a>Hive Model</h3> +<p>The default hive modelling is available in org.apache.atlas.hive.model.HiveDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +hive_db(ClassType) - super types [Referenceable] - attributes [name, clusterName, description, locationUri, parameters, ownerName, ownerType] +hive_storagedesc(ClassType) - super types [Referenceable] - attributes [cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories] +hive_column(ClassType) - super types [Referenceable] - attributes [name, type, comment, table] +hive_table(ClassType) - super types [DataSet] - attributes [name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary] +hive_process(ClassType) - super types [Process] - attributes [name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId] +hive_principal_type(EnumType) - values [USER, ROLE, GROUP] +hive_order(StructType) - attributes [col, order] +hive_serde(StructType) - attributes [name, serializationLib, parameters] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.</p> +<ul> +<li>hive_db - attribute qualifiedName - <dbName>@<clusterName></li> +<li>hive_table - attribute qualifiedName - <dbName>.<tableName>@<clusterName></li> +<li>hive_column - attribute qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> +<li>hive_process - attribute name - <queryString> - trimmed query string in lower case</li></ul></div> +<div class="section"> +<h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3> +<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. import-hive.sh command can be used to facilitate this. The script needs Hadoop and Hive classpath jars. * For Hadoop jars, please make sure that the environment variable HADOOP_CLASSPATH is set. Another way is to set HADOOP_HOME to point to root directory of your Hadoop installation * Similarly, for Hive jars, set HIVE_HOME to the root of Hive installation * Set environment variable HIVE_CONF_DIR to Hive configuration directory * Copy <atlas-conf>/atlas-application.properties to the hive conf directory</p> +<div class="source"> +<pre> + Usage: <atlas package>/hook-bin/import-hive.sh + +</pre></div> +<p>The logs are in <atlas package>/logs/import-hive.log</p> +<p>If you you are importing metadata in a kerberized cluster you need to run the command like this:</p> +<div class="source"> +<pre> +<atlas package>/hook-bin/import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=[krb5.conf location] -Djava.security.auth.login.config=[jaas.conf location] + +</pre></div> +<p></p> +<ul> +<li>krb5.conf is typically found at /etc/krb5.conf</li> +<li>for details about jaas.conf and a suggested location see the <a href="./Security.html">atlas security documentation</a></li></ul></div> +<div class="section"> +<h3><a name="Hive_Hook"></a>Hive Hook</h3> +<p>Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities. Follow these instructions in your hive set-up to add hive hook for Atlas:</p> +<ul> +<li>Set-up atlas hook in hive-site.xml of your hive configuration:</li></ul> +<div class="source"> +<pre> + <property> + <name>hive.exec.post.hooks</name> + <value>org.apache.atlas.hive.hook.HiveHook</value> + </property> + +</pre></div> +<div class="source"> +<pre> + <property> + <name>atlas.cluster.name</name> + <value>primary</value> + </property> + +</pre></div> +<p></p> +<ul> +<li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li> +<li>Copy <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.</li> +<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.hive.minThreads - core number of threads. default 5</li> +<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Column_Level_Lineage"></a>Column Level Lineage</h3> +<p>Starting from 0.8-incubating version of Atlas, Column level lineage is captured in Atlas. Below are the details</p></div> +<div class="section"> +<h4><a name="Model"></a>Model</h4> +<p></p> +<ul> +<li>ColumnLineageProcess type is a subclass of Process</li></ul> +<p></p> +<ul> +<li>This relates an output Column to a set of input Columns or the Input Table</li></ul> +<p></p> +<ul> +<li>The Lineage also captures the kind of Dependency: currently the values are SIMPLE, EXPRESSION, SCRIPT +<ul> +<li>A SIMPLE dependency means the output column has the same value as the input</li> +<li>An EXPRESSION dependency means the output column is transformed by some expression in the runtime(for e.g. a Hive SQL expression) on the Input Columns.</li> +<li>SCRIPT means that the output column is transformed by a user provided script.</li></ul></li></ul> +<p></p> +<ul> +<li>In case of EXPRESSION dependency the expression attribute contains the expression in string form</li></ul> +<p></p> +<ul> +<li>Since Process links input and output DataSets, we make Column a subclass of DataSet</li></ul></div> +<div class="section"> +<h4><a name="Examples"></a>Examples</h4> +<p>For a simple CTAS below:</p> +<div class="source"> +<pre> +create table t2 as select id, name from T1 + +</pre></div> +<p>The lineage is captured as</p> +<p><img src="images/column_lineage_ex1.png" alt="" /></p></div> +<div class="section"> +<h4><a name="Extracting_Lineage_from_Hive_commands"></a>Extracting Lineage from Hive commands</h4> +<p>* The HiveHook maps the LineageInfo in the HookContext to Column lineage instances</p> +<p>* The LineageInfo in Hive provides column-level lineage for the final FileSinkOperator, linking them to the input columns in the Hive Query</p></div> +<div class="section"> +<h4><a name="NOTE"></a>NOTE</h4> +<p>Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112";>HIVE-13112</a> is applied to Hive source</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li> +<li>The following hive operations are captured by hive hook currently +<ul> +<li>create database</li> +<li>create table/view, create table as select</li> +<li>load, import, export</li> +<li>DMLs (insert)</li> +<li>alter database</li> +<li>alter table (skewed table information, stored as, protection is not supported)</li> +<li>alter view</li></ul></li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2017 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/incubator-atlas-website/blob/ff453c82/0.8.0-incubating/Bridge-Sqoop.html ---------------------------------------------------------------------- diff --git a/0.8.0-incubating/Bridge-Sqoop.html b/0.8.0-incubating/Bridge-Sqoop.html new file mode 100644 index 0000000..2e89b0d --- /dev/null +++ b/0.8.0-incubating/Bridge-Sqoop.html @@ -0,0 +1,274 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-03-16 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170316" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Sqoop Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + + + <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarEnabled"> + + + + + + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + + + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="index.html" title="About">About</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> +</li> + + <li> <a href="https://git-wip-us.apache.org/repos/asf/incubator-atlas.git"; title="Git">Git</a> +</li> + + <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> +</li> + + <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> +</li> + + <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="project-info.html" title="Summary">Summary</a> +</li> + + <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> +</li> + + <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> +</li> + + <li> <a href="team-list.html" title="Team">Team</a> +</li> + + <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> +</li> + + <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> +</li> + + <li> <a href="license.html" title="License">License</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/dyn/closer.cgi/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> +</li> + + <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> +</li> + + <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> +</li> + + <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> +</li> + + <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> +</li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + + <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> +</li> + + <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> +</li> + + <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> +</li> + + <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> +</li> + </ul> + </li> + </ul> + + <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > + + <input value="http://atlas.incubator.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> + + + + + + <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.incubator.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + + <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> + + <ul class="nav pull-right"><li style="margin-top: 10px;"> + + <div class="g-plusone" data-href="http://atlas.incubator.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + + </li></ul> + + + </div> + + </div> + </div> + </div> + + <div class="container"> + <div id="banner"> + <div class="pull-left"> + <a href=".." id="bannerLeft"> + <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> + </a> + </div> + <div class="pull-right"> <a href="http://incubator.apache.org"; id="bannerRight"> + <img src="images/apache-incubator-logo.png" alt="Apache Incubator"/> + </a> + </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="index.html" title="Atlas"> + Atlas</a> + </li> + <li class="divider ">/</li> + <li class="">Sqoop Atlas Bridge</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-03-16</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.8-incubating</li> + + </ul> + </div> + + + + <div id="bodyColumn" > + + <div class="section"> +<h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Sqoop_Model"></a>Sqoop Model</h3> +<p>The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:</p> +<div class="source"> +<pre> +sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] +sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] +sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] +sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName] + +</pre></div> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source</p></div> +<div class="section"> +<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> +<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:</p> +<p></p> +<ul> +<li>Sqoop Job publisher class. Currently only one publishing class is supported</li></ul><property> <name>sqoop.job.data.publish.class</name> <value>org.apache.atlas.sqoop.hook.SqoopHook</value> </property> +<ul> +<li>Atlas cluster name</li></ul><property> <name>atlas.cluster.name</name> <value><clustername></value> </property> +<ul> +<li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li> +<li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Limitations"></a>Limitations</h3> +<p></p> +<ul> +<li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> + </div> + </div> + + <hr/> + + <footer> + <div class="container"> + <div class="row span12">Copyright © 2015-2017 + <a href="http://www.apache.org";>Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + <p id="poweredBy" class="pull-right"> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> + </a> + </p> + + </div> + </footer> + </body> +</html>
