http://git-wip-us.apache.org/repos/asf/atlas-website/blob/c3e5130b/Bridge-Falcon.html ---------------------------------------------------------------------- diff --git a/Bridge-Falcon.html b/Bridge-Falcon.html index 4bb8a0e..e4daf41 100644 --- a/Bridge-Falcon.html +++ b/Bridge-Falcon.html @@ -1,288 +1,187 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-08-09 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Falcon.twiki at 2018-01-25 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170809" /> + <meta name="Date-Revision-yyyymmdd" content="20180125" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Falcon Atlas Bridge</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="./css/site.css" /> <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarEnabled"> - - - - - + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> <div id="topbar" class="navbar navbar-fixed-top "> <div class="navbar-inner"> - <div class="container" style="width: 68%;"><div class="nav-collapse"> - - - <ul class="nav"> - <li class="dropdown"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="index.html" title="About">About</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> -</li> - - <li> <a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> -</li> - - <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> -</li> - </ul> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="project-info.html" title="Summary">Summary</a> -</li> - - <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> -</li> - - <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> -</li> - - <li> <a href="team-list.html" title="Team">Team</a> -</li> - - <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> -</li> - - <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - </ul> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> </li> - </ul> - - <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > - + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> <input class="search-query" name="q" id="query" type="text" /> </form> -<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> - - - - - - <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; scrolling="no" frameborder="0" - style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> - - <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> - - <ul class="nav pull-right"><li style="margin-top: 10px;"> - + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> - - </li></ul> - - - </div> - + </li></ul> + </div> </div> </div> </div> - - <div class="container"> - <div id="banner"> - <div class="pull-left"> - <a href=".." id="bannerLeft"> - <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> - </a> - </div> - <div class="pull-right"> </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="index.html" title="Atlas"> - Atlas</a> - </li> - <li class="divider ">/</li> - <li class="">Falcon Atlas Bridge</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2017-08-09</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.9-SNAPSHOT</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Falcon Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-01-25</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-alpha</li> + </ul> </div> - - - <div id="bodyColumn" > - - <div class="section"> +<div class="section"> <h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div> <div class="section"> <h3><a name="Falcon_Model"></a>Falcon Model</h3> -<p>The default falcon modelling is available in org.apache.atlas.falcon.model.FalconDataModelGenerator. It defines the following types:</p> -<div class="source"> -<pre> -falcon_cluster(ClassType) - super types [Infrastructure] - attributes [timestamp, colo, owner, tags] -falcon_feed(ClassType) - super types [DataSet] - attributes [timestamp, stored-in, owner, groups, tags] -falcon_feed_creation(ClassType) - super types [Process] - attributes [timestamp, stored-in, owner] -falcon_feed_replication(ClassType) - super types [Process] - attributes [timestamp, owner] -falcon_process(ClassType) - super types [Process] - attributes [timestamp, runs-on, owner, tags, pipelines, workflow-properties] - -</pre></div> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>falcon_cluster +<ul> +<li>super-types: Infrastructure</li> +<li>attributes: timestamp, colo, owner, tags</li></ul></li> +<li>falcon_feed +<ul> +<li>super-types: DataSet</li> +<li>attributes: timestamp, stored-in, owner, groups, tags</li></ul></li> +<li>falcon_feed_creation +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, stored-in, owner</li></ul></li> +<li>falcon_feed_replication +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, owner</li></ul></li> +<li>falcon_process +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, runs-on, owner, tags, pipelines, workflow-properties</li></ul></li></ul></li></ul> <p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p> <p>The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p> <ul> -<li>falcon_process - <process name>@<cluster name></li> -<li>falcon_cluster - <cluster name></li> -<li>falcon_feed - <feed name>@<cluster name></li> -<li>falcon_feed_creation - <feed name></li> -<li>falcon_feed_replication - <feed name></li></ul></div> +<li>falcon_process.qualifiedName - <process name>@<cluster name></li> +<li>falcon_cluster.qualifiedName - <cluster name></li> +<li>falcon_feed.qualifiedName - <feed name>@<cluster name></li> +<li>falcon_feed_creation.qualifiedName - <feed name></li> +<li>falcon_feed_replication.qualifiedName - <feed name></li></ul></div> <div class="section"> <h3><a name="Falcon_Hook"></a>Falcon Hook</h3> -<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model defined in org.apache.atlas.falcon.model.FalconDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities.</p> +<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model detailed above. Follow the instructions below to setup Atlas hook in Falcon:</p> <ul> <li>Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li> -<li>Link falcon hook jars in falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> +<li>Link Atlas hook jars in Falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> <li>In <falcon_conf>/falcon-env.sh, set an environment variable as follows:</li></ul> -<div class="source"> -<pre> +<div class="source"><pre class="prettyprint"> export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS" - </pre></div> <p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> <ul> -<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> -<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> -<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> -<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> +<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> <li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li> -<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> +<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> <p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> <div class="section"> -<h3><a name="Limitations"></a>Limitations</h3> +<h3><a name="NOTES"></a>NOTES</h3> <p></p> <ul> <li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div> - </div> - </div> - + </div> + </div> <hr/> - <footer> - <div class="container"> - <div class="row span12">Copyright © 2015-2017 - <a href="http://www.apache.org";>Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> </div> - - - <p id="poweredBy" class="pull-right"> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </p> - - </div> </footer> </body> </html>
http://git-wip-us.apache.org/repos/asf/atlas-website/blob/c3e5130b/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/Bridge-Hive.html b/Bridge-Hive.html index e43aed9..95cde52 100644 --- a/Bridge-Hive.html +++ b/Bridge-Hive.html @@ -1,281 +1,188 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-08-09 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Hive.twiki at 2018-01-25 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170809" /> + <meta name="Date-Revision-yyyymmdd" content="20180125" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Hive Atlas Bridge</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="./css/site.css" /> <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarEnabled"> - - - - - + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> <div id="topbar" class="navbar navbar-fixed-top "> <div class="navbar-inner"> - <div class="container" style="width: 68%;"><div class="nav-collapse"> - - - <ul class="nav"> - <li class="dropdown"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="index.html" title="About">About</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> -</li> - - <li> <a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> -</li> - - <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> -</li> - </ul> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="project-info.html" title="Summary">Summary</a> -</li> - - <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> -</li> - - <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> -</li> - - <li> <a href="team-list.html" title="Team">Team</a> -</li> - - <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> -</li> - - <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - </ul> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> </li> - </ul> - - <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > - + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> <input class="search-query" name="q" id="query" type="text" /> </form> -<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> - - - - - - <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; scrolling="no" frameborder="0" - style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> - - <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> - - <ul class="nav pull-right"><li style="margin-top: 10px;"> - + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> - - </li></ul> - - - </div> - + </li></ul> + </div> </div> </div> </div> - - <div class="container"> - <div id="banner"> - <div class="pull-left"> - <a href=".." id="bannerLeft"> - <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> - </a> - </div> - <div class="pull-right"> </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="index.html" title="Atlas"> - Atlas</a> - </li> - <li class="divider ">/</li> - <li class="">Hive Atlas Bridge</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2017-08-09</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.9-SNAPSHOT</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Hive Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-01-25</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-alpha</li> + </ul> </div> - - - <div id="bodyColumn" > - - <div class="section"> +<div class="section"> <h2><a name="Hive_Atlas_Bridge"></a>Hive Atlas Bridge</h2></div> <div class="section"> <h3><a name="Hive_Model"></a>Hive Model</h3> -<p>The default hive modelling is available in org.apache.atlas.hive.model.HiveDataModelGenerator. It defines the following types:</p> -<div class="source"> -<pre> -hive_db(ClassType) - super types [Referenceable] - attributes [name, clusterName, description, locationUri, parameters, ownerName, ownerType] -hive_storagedesc(ClassType) - super types [Referenceable] - attributes [cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories] -hive_column(ClassType) - super types [Referenceable] - attributes [name, type, comment, table] -hive_table(ClassType) - super types [DataSet] - attributes [name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary] -hive_process(ClassType) - super types [Process] - attributes [name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId] -hive_principal_type(EnumType) - values [USER, ROLE, GROUP] -hive_order(StructType) - attributes [col, order] -hive_serde(StructType) - attributes [name, serializationLib, parameters] - -</pre></div> -<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.</p> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>hive_db <ul> -<li>hive_db - attribute qualifiedName - <dbName>@<clusterName></li> -<li>hive_table - attribute qualifiedName - <dbName>.<tableName>@<clusterName></li> -<li>hive_column - attribute qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> -<li>hive_process - attribute name - <queryString> - trimmed query string in lower case</li></ul></div> +<li>super-types: Referenceable</li> +<li>attributes: name, clusterName, description, locationUri, parameters, ownerName, ownerType</li></ul></li> +<li>hive_storagedesc +<ul> +<li>super-types: Referenceable</li> +<li>attributes: cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories</li></ul></li> +<li>hive_column +<ul> +<li>super-types: Referenceable</li> +<li>attributes: name, type, comment, table</li></ul></li> +<li>hive_table +<ul> +<li>super-types: DataSet</li> +<li>attributes: name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary</li></ul></li> +<li>hive_process +<ul> +<li>super-types: Process</li> +<li>attributes: name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId</li></ul></li> +<li>hive_column_lineage +<ul> +<li>super-types: Process</li> +<li>attributes: query, depenendencyType, expression</li></ul></li></ul></li></ul> +<p></p> +<ul> +<li>Enum types: +<ul> +<li>hive_principal_type +<ul> +<li>values: USER, ROLE, GROUP</li></ul></li></ul></li></ul> +<p></p> +<ul> +<li>Struct types: +<ul> +<li>hive_order +<ul> +<li>attributes: col, order</li></ul></li> +<li>hive_serde +<ul> +<li>attributes: name, serializationLib, parameters</li></ul></li></ul></li></ul> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.</p> +<ul> +<li>hive_db.qualifiedName - <dbName>@<clusterName></li> +<li>hive_table.qualifiedName - <dbName>.<tableName>@<clusterName></li> +<li>hive_column.qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> +<li>hive_process.queryString - trimmed query string in lower case</li></ul></div> <div class="section"> <h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3> -<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. import-hive.sh command can be used to facilitate this. The script needs Hadoop and Hive classpath jars. * For Hadoop jars, please make sure that the environment variable HADOOP_CLASSPATH is set. Another way is to set HADOOP_HOME to point to root directory of your Hadoop installation * Similarly, for Hive jars, set HIVE_HOME to the root of Hive installation * Set environment variable HIVE_CONF_DIR to Hive configuration directory * Copy <atlas-conf>/atlas-application.properties to the hive conf directory</p> -<div class="source"> -<pre> +<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined above. import-hive.sh command can be used to facilitate this.</p> +<div class="source"><pre class="prettyprint"> Usage: <atlas package>/hook-bin/import-hive.sh - </pre></div> -<p>The logs are in <atlas package>/logs/import-hive.log</p> -<p>If you you are importing metadata in a kerberized cluster you need to run the command like this:</p> -<div class="source"> -<pre> -<atlas package>/hook-bin/import-hive.sh -Dsun.security.jgss.debug=true -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=[krb5.conf location] -Djava.security.auth.login.config=[jaas.conf location] - -</pre></div> -<p></p> -<ul> -<li>krb5.conf is typically found at /etc/krb5.conf</li> -<li>for details about jaas.conf and a suggested location see the <a href="./Security.html">atlas security documentation</a></li></ul></div> +<p>The logs are in <atlas package>/logs/import-hive.log</p></div> <div class="section"> <h3><a name="Hive_Hook"></a>Hive Hook</h3> -<p>Hive supports listeners on hive command execution using hive hooks. This is used to add/update/remove entities in Atlas using the model defined in org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the request to a thread pool executor to avoid blocking the command execution. The thread submits the entities as message to the notification server and atlas server reads these messages and registers the entities. Follow these instructions in your hive set-up to add hive hook for Atlas:</p> +<p>Atlas Hive hook registers with Hive to listen for create/update/delete operations and updates the metadata in Atlas, via Kafka notifications, for the changes in Hive. Follow the instructions below to setup Atlas hook in Hive:</p> <ul> -<li>Set-up atlas hook in hive-site.xml of your hive configuration:</li></ul> -<div class="source"> -<pre> +<li>Set-up Atlas hook in hive-site.xml by adding the following:</li></ul> +<div class="source"><pre class="prettyprint"> <property> <name>hive.exec.post.hooks</name> <value>org.apache.atlas.hive.hook.HiveHook</value> </property> - -</pre></div> -<div class="source"> -<pre> - <property> - <name>atlas.cluster.name</name> - <value>primary</value> - </property> - </pre></div> <p></p> <ul> @@ -283,12 +190,12 @@ hive_serde(StructType) - attributes [name, serializationLib, parameters] <li>Copy <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul> <p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> <ul> -<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.</li> -<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> -<li>atlas.hook.hive.minThreads - core number of threads. default 5</li> -<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.</li> +<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.hive.minThreads - core number of threads. default 1</li> +<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> <li>atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10</li> -<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> +<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> <p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> <div class="section"> <h3><a name="Column_Level_Lineage"></a>Column Level Lineage</h3> @@ -297,30 +204,28 @@ hive_serde(StructType) - attributes [name, serializationLib, parameters] <h4><a name="Model"></a>Model</h4> <p></p> <ul> -<li>ColumnLineageProcess type is a subclass of Process</li></ul> +<li>ColumnLineageProcess type is a subtype of Process</li></ul> <p></p> <ul> <li>This relates an output Column to a set of input Columns or the Input Table</li></ul> <p></p> <ul> -<li>The Lineage also captures the kind of Dependency: currently the values are SIMPLE, EXPRESSION, SCRIPT +<li>The lineage also captures the kind of dependency, as listed below: <ul> -<li>A SIMPLE dependency means the output column has the same value as the input</li> -<li>An EXPRESSION dependency means the output column is transformed by some expression in the runtime(for e.g. a Hive SQL expression) on the Input Columns.</li> -<li>SCRIPT means that the output column is transformed by a user provided script.</li></ul></li></ul> +<li>SIMPLE: output column has the same value as the input</li> +<li>EXPRESSION: output column is transformed by some expression at runtime (for e.g. a Hive SQL expression) on the Input Columns.</li> +<li>SCRIPT: output column is transformed by a user provided script.</li></ul></li></ul> <p></p> <ul> <li>In case of EXPRESSION dependency the expression attribute contains the expression in string form</li></ul> <p></p> <ul> -<li>Since Process links input and output DataSets, we make Column a subclass of DataSet</li></ul></div> +<li>Since Process links input and output DataSets, Column is a subtype of DataSet</li></ul></div> <div class="section"> <h4><a name="Examples"></a>Examples</h4> <p>For a simple CTAS below:</p> -<div class="source"> -<pre> +<div class="source"><pre class="prettyprint"> create table t2 as select id, name from T1 - </pre></div> <p>The lineage is captured as</p> <p><img src="images/column_lineage_ex1.png" alt="" /></p></div> @@ -329,12 +234,10 @@ create table t2 as select id, name from T1 <p>* The HiveHook maps the LineageInfo in the HookContext to Column lineage instances</p> <p>* The LineageInfo in Hive provides column-level lineage for the final FileSinkOperator, linking them to the input columns in the Hive Query</p></div> <div class="section"> -<h4><a name="NOTE"></a>NOTE</h4> -<p>Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112";>HIVE-13112</a> is applied to Hive source</p></div> -<div class="section"> -<h3><a name="Limitations"></a>Limitations</h3> +<h3><a name="NOTES"></a>NOTES</h3> <p></p> <ul> +<li>Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112";>HIVE-13112</a> is applied to Hive source</li> <li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li> <li>The following hive operations are captured by hive hook currently <ul> @@ -345,27 +248,17 @@ create table t2 as select id, name from T1 <li>alter database</li> <li>alter table (skewed table information, stored as, protection is not supported)</li> <li>alter view</li></ul></li></ul></div> - </div> - </div> - + </div> + </div> <hr/> - <footer> - <div class="container"> - <div class="row span12">Copyright © 2015-2017 - <a href="http://www.apache.org";>Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> </div> - - - <p id="poweredBy" class="pull-right"> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </p> - - </div> </footer> </body> </html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/c3e5130b/Bridge-Sqoop.html ---------------------------------------------------------------------- diff --git a/Bridge-Sqoop.html b/Bridge-Sqoop.html index 79b5eee..6f91d6c 100644 --- a/Bridge-Sqoop.html +++ b/Bridge-Sqoop.html @@ -1,271 +1,180 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-08-09 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Sqoop.twiki at 2018-01-25 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170809" /> + <meta name="Date-Revision-yyyymmdd" content="20180125" /> <meta http-equiv="Content-Language" content="en" /> <title>Apache Atlas – Sqoop Atlas Bridge</title> - <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="./css/site.css" /> <link rel="stylesheet" href="./css/print.css" media="print" /> - - - <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarEnabled"> - - - - - + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> <div id="topbar" class="navbar navbar-fixed-top "> <div class="navbar-inner"> - <div class="container" style="width: 68%;"><div class="nav-collapse"> - - - <ul class="nav"> - <li class="dropdown"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="index.html" title="About">About</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a> -</li> - - <li> <a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a> -</li> - - <li> <a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a> -</li> - - <li> <a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a> -</li> - - <li> <a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a> -</li> - </ul> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="project-info.html" title="Summary">Summary</a> -</li> - - <li> <a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a> -</li> - - <li> <a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a> -</li> - - <li> <a href="team-list.html" title="Team">Team</a> -</li> - - <li> <a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a> -</li> - - <li> <a href="source-repository.html" title="Source Repository">Source Repository</a> -</li> - - <li> <a href="license.html" title="License">License</a> -</li> - </ul> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/1.0.0-alpha/"; title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a> -</li> - - <li> <a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a> -</li> - - <li> <a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a> -</li> - - <li> <a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a> -</li> - - <li> <a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a> -</li> - </ul> + <li><a href="1.0.0-alpha/index.html" title="1.0.0-alpha">1.0.0-alpha</a></li> + <li><a href="0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> </li> - <li class="dropdown"> + <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> <ul class="dropdown-menu"> - - <li> <a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a> -</li> - - <li> <a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a> -</li> - - <li> <a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a> -</li> - - <li> <a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a> -</li> - </ul> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> </li> - </ul> - - <form id="search-form" action="http://www.google.com/search"; method="get" class="navbar-search pull-right" > - + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> <input class="search-query" name="q" id="query" type="text" /> </form> -<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=search-form";></script> - - - - - - <iframe src="http://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; scrolling="no" frameborder="0" - style="border:none; width:80px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> - - <script type="text/javascript" src="https://apis.google.com/js/plusone.js";></script> - - <ul class="nav pull-right"><li style="margin-top: 10px;"> - + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> - - </li></ul> - - - </div> - + </li></ul> + </div> </div> </div> </div> - - <div class="container"> - <div id="banner"> - <div class="pull-left"> - <a href=".." id="bannerLeft"> - <img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/> - </a> - </div> - <div class="pull-right"> </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="index.html" title="Atlas"> - Atlas</a> - </li> - <li class="divider ">/</li> - <li class="">Sqoop Atlas Bridge</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2017-08-09</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.9-SNAPSHOT</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Sqoop Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-01-25</li> + <li id="projectVersion" class="pull-right">Version: 1.0.0-alpha</li> + </ul> </div> - - - <div id="bodyColumn" > - - <div class="section"> +<div class="section"> <h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> <div class="section"> <h3><a name="Sqoop_Model"></a>Sqoop Model</h3> -<p>The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:</p> -<div class="source"> -<pre> -sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] -sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] -sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] -sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName] - -</pre></div> -<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source</p></div> -<div class="section"> -<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> -<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:</p> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>sqoop_process +<ul> +<li>super-types: Process</li> +<li>attributes: name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName</li></ul></li> +<li>sqoop_dbdatastore +<ul> +<li>super-types: DataSet</li> +<li>attributes: name, dbStoreType, storeUse, storeUri, source, description, ownerName</li></ul></li></ul></li></ul> <p></p> <ul> -<li>Sqoop Job publisher class. Currently only one publishing class is supported</li></ul><property> <name>sqoop.job.data.publish.class</name> <value>org.apache.atlas.sqoop.hook.SqoopHook</value> </property> +<li>Enum types: <ul> -<li>Atlas cluster name</li></ul><property> <name>atlas.cluster.name</name> <value><clustername></value> </property> +<li>sqoop_operation_type +<ul> +<li>values: IMPORT, EXPORT, EVAL</li></ul></li> +<li>sqoop_dbstore_usage +<ul> +<li>values: TABLE, QUERY, PROCEDURE, OTHER</li></ul></li></ul></li></ul> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well:</p> +<ul> +<li>sqoop_process.qualifiedName - dbStoreType-storeUri-endTime</li> +<li>sqoop_dbdatastore.qualifiedName - dbStoreType-storeUri-source</li></ul></div> +<div class="section"> +<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> +<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in SqoopHook. This is used to add entities in Atlas using the model detailed above.</p> +<p>Follow the instructions below to setup Atlas hook in Hive:</p> +<p>Add the following properties to to enable Atlas hook in Sqoop:</p> +<ul> +<li>Set-up Atlas hook in <sqoop-conf>/sqoop-site.xml by adding the following:</li></ul> +<div class="source"><pre class="prettyprint"> + <property> + <name>sqoop.job.data.publish.class</name> + <value>org.apache.atlas.sqoop.hook.SqoopHook</value> + </property> +</pre></div> +<p></p> <ul> <li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li> <li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> <p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> <div class="section"> -<h3><a name="Limitations"></a>Limitations</h3> +<h3><a name="NOTES"></a>NOTES</h3> <p></p> <ul> <li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> - </div> - </div> - + </div> + </div> <hr/> - <footer> - <div class="container"> - <div class="row span12">Copyright © 2015-2017 - <a href="http://www.apache.org";>Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> </div> - - - <p id="poweredBy" class="pull-right"> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /> - </a> - </p> - - </div> </footer> </body> </html>
