ATLAS-2913 Update Atlas Website with Release Details 0.8.3
Project: http://git-wip-us.apache.org/repos/asf/atlas-website/repo Commit: http://git-wip-us.apache.org/repos/asf/atlas-website/commit/2d4dc1a0 Tree: http://git-wip-us.apache.org/repos/asf/atlas-website/tree/2d4dc1a0 Diff: http://git-wip-us.apache.org/repos/asf/atlas-website/diff/2d4dc1a0 Branch: refs/heads/asf-site Commit: 2d4dc1a0714dd1ab2b75fe1509e675c5a6ba8074 Parents: d115dad Author: Abhishek <abhishek.kada...@gmail.com> Authored: Thu Nov 1 18:15:55 2018 +0530 Committer: nixonrodrigues <ni...@apache.org> Committed: Thu Nov 1 20:01:41 2018 +0530 ---------------------------------------------------------------------- 0.8.3/Architecture.html | 166 + 0.8.3/Authentication-Authorization.html | 266 + 0.8.3/Bridge-Falcon.html | 188 + 0.8.3/Bridge-Hive.html | 265 + 0.8.3/Bridge-Sqoop.html | 181 + 0.8.3/Configuration.html | 313 + 0.8.3/Downloads.html | 333 + 0.8.3/EclipseSetup.html | 218 + 0.8.3/Export-API.html | 296 + 0.8.3/Export-HDFS-API.html | 156 + 0.8.3/HighAvailability.html | 296 + 0.8.3/Import-API-Options.html | 259 + 0.8.3/Import-API.html | 285 + 0.8.3/Import-Export-API.html | 160 + 0.8.3/InstallationSteps.html | 394 + 0.8.3/Notification-Entity.html | 158 + 0.8.3/QuickStart.html | 151 + 0.8.3/Search.html | 417 + 0.8.3/StormAtlasHook.html | 197 + 0.8.3/TypeSystem.html | 276 + 0.8.3/api/v2/apidocs/application.wadl | 920 + 0.8.3/api/v2/apidocs/css/style.css | 115 + 0.8.3/api/v2/apidocs/data.html | 446 + 0.8.3/api/v2/apidocs/index.html | 576 + .../api/v2/apidocs/json_AtlasAttributeDef.html | 245 + .../v2/apidocs/json_AtlasBaseModelObject.html | 136 + 0.8.3/api/v2/apidocs/json_AtlasBaseTypeDef.html | 319 + .../v2/apidocs/json_AtlasClassification.html | 142 + .../v2/apidocs/json_AtlasClassificationDef.html | 368 + .../v2/apidocs/json_AtlasClassifications.html | 167 + .../api/v2/apidocs/json_AtlasConstraintDef.html | 143 + .../apidocs/json_AtlasEntitiesWithExtInfo.html | 249 + 0.8.3/api/v2/apidocs/json_AtlasEntity.html | 234 + 0.8.3/api/v2/apidocs/json_AtlasEntityDef.html | 368 + .../api/v2/apidocs/json_AtlasEntityExtInfo.html | 187 + .../api/v2/apidocs/json_AtlasEntityHeader.html | 204 + .../v2/apidocs/json_AtlasEntityWithExtInfo.html | 223 + 0.8.3/api/v2/apidocs/json_AtlasEnumDef.html | 310 + .../v2/apidocs/json_AtlasEnumElementDef.html | 150 + .../v2/apidocs/json_AtlasFullTextResult.html | 163 + 0.8.3/api/v2/apidocs/json_AtlasLineageInfo.html | 225 + 0.8.3/api/v2/apidocs/json_AtlasQueryType.html | 138 + .../api/v2/apidocs/json_AtlasSearchResult.html | 361 + 0.8.3/api/v2/apidocs/json_AtlasStruct.html | 145 + 0.8.3/api/v2/apidocs/json_AtlasStructDef.html | 348 + .../api/v2/apidocs/json_AtlasTypeDefHeader.html | 150 + 0.8.3/api/v2/apidocs/json_AtlasTypesDef.html | 592 + .../v2/apidocs/json_AtlasUserSavedSearch.html | 207 + .../v2/apidocs/json_AttributeSearchResult.html | 140 + 0.8.3/api/v2/apidocs/json_Cardinality.html | 126 + .../json_ClassificationAssociateRequest.html | 146 + 0.8.3/api/v2/apidocs/json_Condition.html | 122 + 0.8.3/api/v2/apidocs/json_Currency.html | 194 + 0.8.3/api/v2/apidocs/json_DateFormat.html | 599 + .../v2/apidocs/json_EntityMutationResponse.html | 561 + 0.8.3/api/v2/apidocs/json_EntityOperation.html | 130 + 0.8.3/api/v2/apidocs/json_FilterCriteria.html | 182 + 0.8.3/api/v2/apidocs/json_Format.html | 125 + 0.8.3/api/v2/apidocs/json_LineageDirection.html | 126 + 0.8.3/api/v2/apidocs/json_LineageRelation.html | 140 + 0.8.3/api/v2/apidocs/json_Locale.html | 357 + 0.8.3/api/v2/apidocs/json_NumberFormat.html | 609 + 0.8.3/api/v2/apidocs/json_Operator.html | 176 + 0.8.3/api/v2/apidocs/json_PList.html | 180 + 0.8.3/api/v2/apidocs/json_RoundingMode.html | 146 + 0.8.3/api/v2/apidocs/json_SavedSearchType.html | 122 + 0.8.3/api/v2/apidocs/json_SearchParameters.html | 252 + 0.8.3/api/v2/apidocs/json_SortType.html | 126 + 0.8.3/api/v2/apidocs/json_Status.html | 122 + 0.8.3/api/v2/apidocs/json_TimeZone.html | 191 + 0.8.3/api/v2/apidocs/json_TypeCategory.html | 150 + 0.8.3/api/v2/apidocs/ns0.xsd | 128 + .../api/v2/apidocs/resource_DiscoveryREST.html | 2763 ++ 0.8.3/api/v2/apidocs/resource_EntityREST.html | 4689 ++++ 0.8.3/api/v2/apidocs/resource_LineageREST.html | 272 + 0.8.3/api/v2/apidocs/resource_TypesREST.html | 5240 ++++ 0.8.3/api/v2/apidocs/resources.html | 218 + 0.8.3/api/v2/apidocs/syntax_json.html | 391 + 0.8.3/api/v2/apidocs/syntax_xml.html | 144 + 0.8.3/api/v2/apidocs/ui/css/print.css | 1367 + 0.8.3/api/v2/apidocs/ui/css/reset.css | 125 + 0.8.3/api/v2/apidocs/ui/css/screen.css | 1494 + 0.8.3/api/v2/apidocs/ui/css/style.css | 250 + 0.8.3/api/v2/apidocs/ui/css/typography.css | 14 + .../api/v2/apidocs/ui/fonts/DroidSans-Bold.ttf | Bin 0 -> 42480 bytes 0.8.3/api/v2/apidocs/ui/fonts/DroidSans.ttf | Bin 0 -> 41028 bytes .../ui/fonts/droid-sans-v6-latin-700.eot | Bin 0 -> 22922 bytes .../ui/fonts/droid-sans-v6-latin-700.svg | 411 + .../ui/fonts/droid-sans-v6-latin-700.ttf | Bin 0 -> 40513 bytes .../ui/fonts/droid-sans-v6-latin-700.woff | Bin 0 -> 25992 bytes .../ui/fonts/droid-sans-v6-latin-700.woff2 | Bin 0 -> 11480 bytes .../ui/fonts/droid-sans-v6-latin-regular.eot | Bin 0 -> 22008 bytes .../ui/fonts/droid-sans-v6-latin-regular.svg | 403 + .../ui/fonts/droid-sans-v6-latin-regular.ttf | Bin 0 -> 39069 bytes .../ui/fonts/droid-sans-v6-latin-regular.woff | Bin 0 -> 24868 bytes .../ui/fonts/droid-sans-v6-latin-regular.woff2 | Bin 0 -> 11304 bytes 0.8.3/api/v2/apidocs/ui/images/collapse.gif | Bin 0 -> 69 bytes 0.8.3/api/v2/apidocs/ui/images/expand.gif | Bin 0 -> 73 bytes .../api/v2/apidocs/ui/images/explorer_icons.png | Bin 0 -> 5115 bytes .../api/v2/apidocs/ui/images/favicon-16x16.png | Bin 0 -> 445 bytes .../api/v2/apidocs/ui/images/favicon-32x32.png | Bin 0 -> 1141 bytes 0.8.3/api/v2/apidocs/ui/images/favicon.ico | Bin 0 -> 5430 bytes 0.8.3/api/v2/apidocs/ui/images/logo_small.png | Bin 0 -> 455 bytes .../api/v2/apidocs/ui/images/pet_store_api.png | Bin 0 -> 631 bytes 0.8.3/api/v2/apidocs/ui/images/throbber.gif | Bin 0 -> 9257 bytes 0.8.3/api/v2/apidocs/ui/images/wordnik_api.png | Bin 0 -> 670 bytes 0.8.3/api/v2/apidocs/ui/index.html | 118 + 0.8.3/api/v2/apidocs/ui/lang/ca.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/el.js | 56 + 0.8.3/api/v2/apidocs/ui/lang/en.js | 56 + 0.8.3/api/v2/apidocs/ui/lang/es.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/fr.js | 54 + 0.8.3/api/v2/apidocs/ui/lang/geo.js | 56 + 0.8.3/api/v2/apidocs/ui/lang/it.js | 52 + 0.8.3/api/v2/apidocs/ui/lang/ja.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/ko-kr.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/pl.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/pt.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/ru.js | 56 + 0.8.3/api/v2/apidocs/ui/lang/tr.js | 53 + 0.8.3/api/v2/apidocs/ui/lang/translator.js | 39 + 0.8.3/api/v2/apidocs/ui/lang/zh-cn.js | 53 + 0.8.3/api/v2/apidocs/ui/lib/backbone-min.js | 15 + 0.8.3/api/v2/apidocs/ui/lib/es5-shim.js | 2065 ++ 0.8.3/api/v2/apidocs/ui/lib/handlebars-2.0.0.js | 28 + 0.8.3/api/v2/apidocs/ui/lib/handlebars-4.0.5.js | 4608 +++ .../api/v2/apidocs/ui/lib/highlight.7.3.pack.js | 1 + .../v2/apidocs/ui/lib/highlight.9.1.0.pack.js | 2 + .../ui/lib/highlight.9.1.0.pack_extended.js | 34 + 0.8.3/api/v2/apidocs/ui/lib/jquery-1.8.0.min.js | 2 + .../api/v2/apidocs/ui/lib/jquery.ba-bbq.min.js | 18 + .../api/v2/apidocs/ui/lib/jquery.slideto.min.js | 1 + .../api/v2/apidocs/ui/lib/jquery.wiggle.min.js | 8 + 0.8.3/api/v2/apidocs/ui/lib/js-yaml.min.js | 3 + 0.8.3/api/v2/apidocs/ui/lib/jsoneditor.min.js | 11 + 0.8.3/api/v2/apidocs/ui/lib/lodash.min.js | 102 + 0.8.3/api/v2/apidocs/ui/lib/marked.js | 1272 + .../apidocs/ui/lib/object-assign-pollyfill.js | 23 + .../api/v2/apidocs/ui/lib/sanitize-html.min.js | 6 + 0.8.3/api/v2/apidocs/ui/lib/swagger-oauth.js | 347 + 0.8.3/api/v2/apidocs/ui/lib/underscore-min.js | 6 + 0.8.3/api/v2/apidocs/ui/lib/underscore-min.map | 1 + 0.8.3/api/v2/apidocs/ui/o2c.html | 20 + 0.8.3/api/v2/apidocs/ui/swagger-ui.js | 24993 +++++++++++++++++ 0.8.3/api/v2/apidocs/ui/swagger-ui.min.js | 14 + 0.8.3/api/v2/apidocs/ui/swagger.json | 16593 +++++++++++ 0.8.3/api/v2/apidocs/xml_ns0_PList.html | 242 + .../apidocs/xml_ns0_atlasBaseModelObject.html | 151 + .../v2/apidocs/xml_ns0_atlasClassification.html | 159 + 0.8.3/api/v2/apidocs/xml_ns0_atlasStruct.html | 168 + 0.8.3/api/v2/apidocs/xml_ns0_searchFilter.html | 242 + 0.8.3/api/v2/apidocs/xml_ns0_sortType.html | 130 + 0.8.3/css/apache-maven-fluido-1.7.min.css | 17 + 0.8.3/css/print.css | 23 + 0.8.3/css/site.css | 1 + 0.8.3/fonts/glyphicons-halflings-regular.eot | Bin 0 -> 35283 bytes 0.8.3/fonts/glyphicons-halflings-regular.svg | 229 + 0.8.3/fonts/glyphicons-halflings-regular.ttf | Bin 0 -> 55016 bytes 0.8.3/fonts/glyphicons-halflings-regular.woff | Bin 0 -> 41793 bytes 0.8.3/images/accessories-text-editor.png | Bin 0 -> 746 bytes 0.8.3/images/add.gif | Bin 0 -> 397 bytes 0.8.3/images/apache-incubator-logo.png | Bin 0 -> 4234 bytes 0.8.3/images/apache-maven-project-2.png | Bin 0 -> 33442 bytes 0.8.3/images/application-certificate.png | Bin 0 -> 923 bytes 0.8.3/images/atlas-logo.png | Bin 0 -> 3115 bytes 0.8.3/images/column_lineage_ex1.png | Bin 0 -> 34057 bytes 0.8.3/images/contact-new.png | Bin 0 -> 736 bytes 0.8.3/images/document-properties.png | Bin 0 -> 577 bytes 0.8.3/images/drive-harddisk.png | Bin 0 -> 700 bytes 0.8.3/images/fix.gif | Bin 0 -> 366 bytes 0.8.3/images/icon_error_sml.gif | Bin 0 -> 633 bytes 0.8.3/images/icon_help_sml.gif | Bin 0 -> 1072 bytes 0.8.3/images/icon_info_sml.gif | Bin 0 -> 638 bytes 0.8.3/images/icon_success_sml.gif | Bin 0 -> 604 bytes 0.8.3/images/icon_warning_sml.gif | Bin 0 -> 625 bytes 0.8.3/images/image-x-generic.png | Bin 0 -> 662 bytes 0.8.3/images/internet-web-browser.png | Bin 0 -> 1017 bytes 0.8.3/images/logos/build-by-maven-black.png | Bin 0 -> 2294 bytes 0.8.3/images/logos/build-by-maven-white.png | Bin 0 -> 2260 bytes 0.8.3/images/logos/maven-feather.png | Bin 0 -> 3330 bytes 0.8.3/images/network-server.png | Bin 0 -> 536 bytes 0.8.3/images/package-x-generic.png | Bin 0 -> 717 bytes 0.8.3/images/profiles/pre-release.png | Bin 0 -> 32607 bytes 0.8.3/images/profiles/retired.png | Bin 0 -> 22003 bytes 0.8.3/images/profiles/sandbox.png | Bin 0 -> 33010 bytes 0.8.3/images/remove.gif | Bin 0 -> 607 bytes 0.8.3/images/rss.png | Bin 0 -> 474 bytes 0.8.3/images/twiki/architecture.png | Bin 0 -> 315403 bytes 0.8.3/images/twiki/guide-class-diagram.png | Bin 0 -> 40375 bytes 0.8.3/images/twiki/guide-instance-graph.png | Bin 0 -> 179941 bytes 0.8.3/images/update.gif | Bin 0 -> 1090 bytes 0.8.3/images/window-new.png | Bin 0 -> 583 bytes 0.8.3/img/glyphicons-halflings-white.png | Bin 0 -> 8777 bytes 0.8.3/img/glyphicons-halflings.png | Bin 0 -> 12799 bytes 0.8.3/index.html | 219 + 0.8.3/issue-tracking.html | 135 + 0.8.3/js/apache-maven-fluido-1.7.min.js | 25 + 0.8.3/license.html | 339 + 0.8.3/mail-lists.html | 156 + 0.8.3/project-info.html | 155 + 0.8.3/security.html | 386 + 0.8.3/source-repository.html | 146 + 0.8.3/team-list.html | 468 + Downloads.html | 10 +- api/apple-touch-icon.png | Bin 0 -> 640 bytes api/application.wadl | 756 + api/atlas-webapp-php.zip | Bin 0 -> 2066 bytes api/atlas-webapp.rb | 246 + api/crossdomain.xml | 25 + api/css/home.gif | Bin 0 -> 128 bytes api/css/prettify.css | 1 + api/css/style.css | 759 + api/downloads.html | 163 + api/el_ns0_errorBean.html | 132 + api/el_ns0_results.html | 131 + api/favicon.ico | Bin 0 -> 1150 bytes api/index.html | 157 + api/js/libs/dd_belatedpng.js | 13 + api/js/libs/jquery-1.5.1.min.js | 16 + api/js/libs/modernizr-1.7.min.js | 2 + api/js/libs/prettify/lang-apollo.js | 2 + api/js/libs/prettify/lang-clj.js | 18 + api/js/libs/prettify/lang-css.js | 2 + api/js/libs/prettify/lang-go.js | 1 + api/js/libs/prettify/lang-hs.js | 2 + api/js/libs/prettify/lang-lisp.js | 3 + api/js/libs/prettify/lang-lua.js | 2 + api/js/libs/prettify/lang-ml.js | 2 + api/js/libs/prettify/lang-n.js | 4 + api/js/libs/prettify/lang-proto.js | 1 + api/js/libs/prettify/lang-scala.js | 2 + api/js/libs/prettify/lang-sql.js | 2 + api/js/libs/prettify/lang-tex.js | 1 + api/js/libs/prettify/lang-vb.js | 2 + api/js/libs/prettify/lang-vhdl.js | 3 + api/js/libs/prettify/lang-wiki.js | 2 + api/js/libs/prettify/lang-xq.js | 3 + api/js/libs/prettify/lang-yaml.js | 2 + api/js/libs/prettify/prettify.js | 28 + api/js/libs/xbreadcrumbs.js | 93 + api/model.html | 140 + api/ns0.html | 131 + api/ns0.xsd | 25 + api/ns0_errorBean.html | 189 + api/ns0_results.html | 173 + api/resource_AdminResource.html | 199 + api/resource_DataSetLineageResource.html | 234 + api/resource_EntityResource.html | 685 + api/resource_EntityService.html | 376 + api/resource_LineageResource.html | 234 + api/resource_MetadataDiscoveryResource.html | 324 + api/resource_TaxonomyService.html | 758 + api/resource_TypesResource.html | 267 + api/rest.html | 120 + index.html | 2 + 255 files changed, 96654 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Architecture.html ---------------------------------------------------------------------- diff --git a/0.8.3/Architecture.html b/0.8.3/Architecture.html new file mode 100644 index 0000000..4d20c62 --- /dev/null +++ b/0.8.3/Architecture.html @@ -0,0 +1,166 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Architecture.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Architecture</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Architecture</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Architecture"></a>Architecture</h2></div> +<div class="section"> +<h3><a name="Introduction"></a>Introduction</h3></div> +<div class="section"> +<h3><a name="Atlas_High_Level_Architecture_-_Overview"></a>Atlas High Level Architecture - Overview</h3> +<p><img src="images/twiki/architecture.png" alt="" /></p> +<p>The components of Atlas can be grouped under the following major categories:</p></div> +<div class="section"> +<h4><a name="Core"></a>Core</h4> +<p>This category contains the components that implement the core of Atlas functionality, including:</p> +<p><b>Type System</b>: Atlas allows users to define a model for the metadata objects they want to manage. The model is composed of definitions called ‘types’. Instances of ‘types’ called ‘entities’ represent the actual metadata objects that are managed. The Type System is a component that allows users to define and manage the types and entities. All metadata objects managed by Atlas out of the box (like Hive tables, for e.g.) are modelled using types and represented as entities. To store new types of metadata in Atlas, one needs to understand the concepts of the type system component.</p> +<p>One key point to note is that the generic nature of the modelling in Atlas allows data stewards and integrators to define both technical metadata and business metadata. It is also possible to define rich relationships between the two using features of Atlas.</p> +<p><b>Ingest / Export</b>: The Ingest component allows metadata to be added to Atlas. Similarly, the Export component exposes metadata changes detected by Atlas to be raised as events. Consumers can consume these change events to react to metadata changes in real time.</p> +<p><b>Graph Engine</b>: Internally, Atlas represents metadata objects it manages using a Graph model. It does this to achieve great flexibility and rich relations between the metadata objects. The Graph Engine is a component that is responsible for translating between types and entities of the Type System, and the underlying Graph model. In addition to managing the Graph objects, The Graph Engine also creates the appropriate indices for the metadata objects so that they can be searched for efficiently.</p> +<p><b>Titan</b>: Currently, Atlas uses the Titan Graph Database to store the metadata objects. Titan is used as a library within Atlas. Titan uses two stores: The Metadata store is configured to !HBase by default and the Index store is configured to Solr. It is also possible to use the Metadata store as BerkeleyDB and Index store as ElasticSearch by building with corresponding profiles. The Metadata store is used for storing the metadata objects proper, and the Index store is used for storing indices of the Metadata properties, that allows efficient search.</p></div> +<div class="section"> +<h4><a name="Integration"></a>Integration</h4> +<p>Users can manage metadata in Atlas using two methods:</p> +<p><b>API</b>: All functionality of Atlas is exposed to end users via a REST API that allows types and entities to be created, updated and deleted. It is also the primary mechanism to query and discover the types and entities managed by Atlas.</p> +<p><b>Messaging</b>: In addition to the API, users can choose to integrate with Atlas using a messaging interface that is based on Kafka. This is useful both for communicating metadata objects to Atlas, and also to consume metadata change events from Atlas using which applications can be built. The messaging interface is particularly useful if one wishes to use a more loosely coupled integration with Atlas that could allow for better scalability, reliability etc. Atlas uses Apache Kafka as a notification server for communication between hooks and downstream consumers of metadata notification events. Events are written by the hooks and Atlas to different Kafka topics.</p></div> +<div class="section"> +<h4><a name="Metadata_sources"></a>Metadata sources</h4> +<p>Atlas supports integration with many sources of metadata out of the box. More integrations will be added in future as well. Currently, Atlas supports ingesting and managing metadata from the following sources:</p> +<p></p> +<ul> +<li><a href="./Bridge-Hive.html">Hive</a></li> +<li><a href="./Bridge-Sqoop.html">Sqoop</a></li> +<li><a href="./Bridge-Falcon.html">Falcon</a></li> +<li><a href="./StormAtlasHook.html">Storm</a></li> +<li><a href="./StormAtlasHook.html">Storm</a></li> +<li>HBase - <i>documentation work-in-progress</i></li></ul> +<p>The integration implies two things: There are metadata models that Atlas defines natively to represent objects of these components. There are components Atlas provides to ingest metadata objects from these components (in real time or in batch mode in some cases).</p></div> +<div class="section"> +<h4><a name="Applications"></a>Applications</h4> +<p>The metadata managed by Atlas is consumed by a variety of applications for satisfying many governance use cases.</p> +<p><b>Atlas Admin UI</b>: This component is a web based application that allows data stewards and scientists to discover and annotate metadata. Of primary importance here is a search interface and SQL like query language that can be used to query the metadata types and objects managed by Atlas. The Admin UI uses the REST API of Atlas for building its functionality.</p> +<p><b>Tag Based Policies</b>: <a class="externalLink" href="http://ranger.apache.org/";>Apache Ranger</a> is an advanced security management solution for the Hadoop ecosystem having wide integration with a variety of Hadoop components. By integrating with Atlas, Ranger allows security administrators to define metadata driven security policies for effective governance. Ranger is a consumer to the metadata change events notified by Atlas.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Authentication-Authorization.html ---------------------------------------------------------------------- diff --git a/0.8.3/Authentication-Authorization.html b/0.8.3/Authentication-Authorization.html new file mode 100644 index 0000000..39fa061 --- /dev/null +++ b/0.8.3/Authentication-Authorization.html @@ -0,0 +1,266 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Authentication-Authorization.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Authentication & Authorization in Apache Atlas.</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Authentication & Authorization in Apache Atlas.</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h3><a name="Authentication_.26_Authorization_in_Apache_Atlas."></a>Authentication & Authorization in Apache Atlas.</h3></div> +<div class="section"> +<h4><a name="Authentication"></a>Authentication</h4> +<p>Atlas supports following authentication methods</p> +<p></p> +<ul> +<li><b>File</b></li> +<li><b>Kerberos</b></li> +<li><b>LDAP</b></li></ul> +<p>Following properties should be set true to enable the authentication of that type in <tt>atlas-application.properties</tt> file.</p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.kerberos=true|false +atlas.authentication.method.ldap=true|false +atlas.authentication.method.file=true|false + +</pre></div> +<p>If two or more authentication methods are set to true, then the authentication falls back to the latter method if the earlier one fails. For example if Kerberos authentication is set to true and ldap authentication is also set to true then, if for a request without kerberos principal and keytab LDAP authentication will be used as a fallback scenario.</p></div> +<div class="section"> +<h5><a name="FILE_method."></a>FILE method.</h5> +<p>File authentication requires users' login details in users credentials file in the format specified below and the file path should set to property <tt>atlas.authentication.method.file.filename</tt> in <tt>atlas-application.properties</tt>.</p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.file=true +atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties + +</pre></div> +<p>The users credentials file should have below format</p> +<div class="source"><pre class="prettyprint"> +username=group::sha256-password + +</pre></div> +<p>For e.g.</p> +<div class="source"><pre class="prettyprint"> +admin=ADMIN::e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a + +</pre></div> +<p>Users group can be either <b>ADMIN</b>, <b>DATA_STEWARD</b> OR <b>DATA_SCIENTIST</b></p> +<p><b>Note</b>:-password is encoded with sha256 encoding method and can be generated using unix tool.</p> +<p>For e.g.</p> +<div class="source"><pre class="prettyprint"> +echo -n "Password" | sha256sum +e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a - + +</pre></div></div> +<div class="section"> +<h5><a name="Kerberos_Method."></a>Kerberos Method.</h5> +<p>To enable the authentication in Kerberos mode in Atlas, set the property <tt>atlas.authentication.method.kerberos</tt> to true in <tt>atlas-application.properties</tt></p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.kerberos = true + +</pre></div> +<p>Also following properties should be set.</p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.kerberos.principal=<principal>/<fqdn>@EXAMPLE.COM +atlas.authentication.method.kerberos.keytab = /<key tab filepath>.keytab +atlas.authentication.method.kerberos.name.rules = RULE:[2:$1@$0](at...@example.com)s/.*/atlas/ +atlas.authentication.method.kerberos.token.validity = 3600 [ in Seconds (optional)] + +</pre></div></div> +<div class="section"> +<h5><a name="LDAP_Method."></a>LDAP Method.</h5> +<p>To enable the authentication in LDAP mode in Atlas, set the property <tt>atlas.authentication.method.ldap</tt> to true and also set Ldap type to property <tt>atlas.authentication.method.ldap.type</tt> to LDAP or AD in <tt>atlas-application.properties</tt>. Use AD if connecting to Active Directory.</p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.ldap=true +atlas.authentication.method.ldap.type=ldap|ad + +</pre></div> +<p>For LDAP or AD the following configuration needs to be set in atlas application properties.</p> +<p><b>Active Directory</b></p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.ldap.ad.domain= example.com +atlas.authentication.method.ldap.ad.url=ldap://<AD server ip>:389 +atlas.authentication.method.ldap.ad.base.dn=DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.dn=CN=Administrator,CN=Users,DC=example,DC=com +atlas.authentication.method.ldap.ad.bind.password=<password> +atlas.authentication.method.ldap.ad.referral=ignore +atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0}) +atlas.authentication.method.ldap.ad.default.role=ROLE_USER + +</pre></div> +<p><b>LDAP Directroy</b></p> +<div class="source"><pre class="prettyprint"> +atlas.authentication.method.ldap.url=ldap://<Ldap server ip>:389 +atlas.authentication.method.ldap.userDNpattern=uid={0],ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupSearchBase=dc=example,dc=com +atlas.authentication.method.ldap.groupSearchFilter=(member=cn={0},ou=users,dc=example,dc=com +atlas.authentication.method.ldap.groupRoleAttribute=cn +atlas.authentication.method.ldap.base.dn=dc=example,dc=com +atlas.authentication.method.ldap.bind.dn=cn=Manager,dc=example,dc=com +atlas.authentication.method.ldap.bind.password=<password> +atlas.authentication.method.ldap.referral=ignore +atlas.authentication.method.ldap.user.searchfilter=(uid={0}) +atlas.authentication.method.ldap.default.role=ROLE_USER + +</pre></div></div> +<div class="section"> +<h4><a name="Authorization"></a>Authorization</h4></div> +<div class="section"> +<h5><a name="Atlas_Authorization_Methods_.5BSimple.2FRanger.5D"></a>Atlas Authorization Methods [Simple/Ranger]</h5> +<p>To set authorization in atlas, update the <tt>atlas.authorizer.impl</tt> properties in <tt>atlas-application.properties</tt></p> +<ul> +<li><b>Simple</b></li> +<li><b>Ranger</b></li></ul> +<div class="source"><pre class="prettyprint"> +atlas.authorizer.impl=simple | ranger | <Qualified Authorizer Class Name> + +</pre></div></div> +<div class="section"> +<h5><a name="Simple_Authorizer."></a>Simple Authorizer.</h5> +<p>In Simple Authorizer the policy store file is configured locally. The path of policy store file is set in <tt>atlas.auth.policy.file</tt> property of <tt>atlas-application.properties</tt></p> +<div class="source"><pre class="prettyprint"> +atlas.auth.policy.file={{conf_dir}}/policy-store.txt + +</pre></div> +<p>The policy store file format is as follows:</p> +<div class="source"><pre class="prettyprint"> +Policy_Name;;User_Name:Operations_Allowed;;Group_Name:Operations_Allowed;;Resource_Type:Resource_Name + +</pre></div> +<p>eg. of admin policy:</p> +<div class="source"><pre class="prettyprint"> +adminPolicy;;admin:rwud;;ROLE_ADMIN:rwud;;type:*,entity:*,operation:*,taxonomy:*,term:* + +</pre></div> +<p>Note : The User_Name, Group_Name and Operations_Allowed are comma(,) separated lists.</p> +<p>Authorizer Resource Types:</p> +<ul> +<li>Operation</li> +<li>Type</li> +<li>Entity</li> +<li>Taxonomy</li> +<li>Term</li> +<li>Unknown</li></ul> +<p>Operations_Allowed are r = read, w = write, u = update, d = delete</p></div> +<div class="section"> +<h5><a name="Ranger_Authorizer."></a>Ranger Authorizer.</h5> +<p>Ranger Authorizer is enabled by activating Atlas-Ranger plugin from Ambari.</p> +<p>For more details visit the <a class="externalLink" href="http://ranger.apache.org/";>Apache-Ranger documentation</a>.</p></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Bridge-Falcon.html ---------------------------------------------------------------------- diff --git a/0.8.3/Bridge-Falcon.html b/0.8.3/Bridge-Falcon.html new file mode 100644 index 0000000..50e3e17 --- /dev/null +++ b/0.8.3/Bridge-Falcon.html @@ -0,0 +1,188 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Falcon.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Falcon Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Falcon Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Falcon_Atlas_Bridge"></a>Falcon Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Falcon_Model"></a>Falcon Model</h3> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>falcon_cluster +<ul> +<li>super-types: Infrastructure</li> +<li>attributes: timestamp, colo, owner, tags</li></ul></li> +<li>falcon_feed +<ul> +<li>super-types: DataSet</li> +<li>attributes: timestamp, stored-in, owner, groups, tags</li></ul></li> +<li>falcon_feed_creation +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, stored-in, owner</li></ul></li> +<li>falcon_feed_replication +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, owner</li></ul></li> +<li>falcon_process +<ul> +<li>super-types: Process</li> +<li>attributes: timestamp, runs-on, owner, tags, pipelines, workflow-properties</li></ul></li></ul></li></ul> +<p>One falcon_process entity is created for every cluster that the falcon process is defined for.</p> +<p>The entities are created and de-duped using unique qualifiedName attribute. They provide namespace and can be used for querying/lineage as well. The unique attributes are:</p> +<ul> +<li>falcon_process.qualifiedName - <process name>@<cluster name></li> +<li>falcon_cluster.qualifiedName - <cluster name></li> +<li>falcon_feed.qualifiedName - <feed name>@<cluster name></li> +<li>falcon_feed_creation.qualifiedName - <feed name></li> +<li>falcon_feed_replication.qualifiedName - <feed name></li></ul></div> +<div class="section"> +<h3><a name="Falcon_Hook"></a>Falcon Hook</h3> +<p>Falcon supports listeners on falcon entity submission. This is used to add entities in Atlas using the model detailed above. Follow the instructions below to setup Atlas hook in Falcon:</p> +<ul> +<li>Add 'org.apache.atlas.falcon.service.AtlasService' to application.services in <falcon-conf>/startup.properties</li> +<li>Link Atlas hook jars in Falcon classpath - 'ln -s <atlas-home>/hook/falcon/* <falcon-home>/server/webapp/falcon/WEB-INF/lib/'</li> +<li>In <falcon_conf>/falcon-env.sh, set an environment variable as follows:</li></ul> +<div class="source"><pre class="prettyprint"> + export FALCON_SERVER_OPTS="<atlas_home>/hook/falcon/*:$FALCON_SERVER_OPTS" +</pre></div> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.falcon.synchronous - boolean, true to run the hook synchronously. default false</li> +<li>atlas.hook.falcon.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.falcon.minThreads - core number of threads. default 5</li> +<li>atlas.hook.falcon.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.falcon.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.falcon.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="NOTES"></a>NOTES</h3> +<p></p> +<ul> +<li>In falcon cluster entity, cluster name used should be uniform across components like hive, falcon, sqoop etc. If used with ambari, ambari cluster name should be used for cluster entity</li></ul></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Bridge-Hive.html ---------------------------------------------------------------------- diff --git a/0.8.3/Bridge-Hive.html b/0.8.3/Bridge-Hive.html new file mode 100644 index 0000000..b673365 --- /dev/null +++ b/0.8.3/Bridge-Hive.html @@ -0,0 +1,265 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Hive.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Hive Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Hive Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Hive_Atlas_Bridge"></a>Hive Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Hive_Model"></a>Hive Model</h3> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>hive_db +<ul> +<li>super-types: Referenceable</li> +<li>attributes: name, clusterName, description, locationUri, parameters, ownerName, ownerType</li></ul></li> +<li>hive_storagedesc +<ul> +<li>super-types: Referenceable</li> +<li>attributes: cols, location, inputFormat, outputFormat, compressed, numBuckets, serdeInfo, bucketCols, sortCols, parameters, storedAsSubDirectories</li></ul></li> +<li>hive_column +<ul> +<li>super-types: Referenceable</li> +<li>attributes: name, type, comment, table</li></ul></li> +<li>hive_table +<ul> +<li>super-types: DataSet</li> +<li>attributes: name, db, owner, createTime, lastAccessTime, comment, retention, sd, partitionKeys, columns, aliases, parameters, viewOriginalText, viewExpandedText, tableType, temporary</li></ul></li> +<li>hive_process +<ul> +<li>super-types: Process</li> +<li>attributes: name, startTime, endTime, userName, operationType, queryText, queryPlan, queryId</li></ul></li> +<li>hive_column_lineage +<ul> +<li>super-types: Process</li> +<li>attributes: query, depenendencyType, expression</li></ul></li></ul></li></ul> +<p></p> +<ul> +<li>Enum types: +<ul> +<li>hive_principal_type +<ul> +<li>values: USER, ROLE, GROUP</li></ul></li></ul></li></ul> +<p></p> +<ul> +<li>Struct types: +<ul> +<li>hive_order +<ul> +<li>attributes: col, order</li></ul></li> +<li>hive_serde +<ul> +<li>attributes: name, serializationLib, parameters</li></ul></li></ul></li></ul> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying/lineage as well. Note that dbName, tableName and columnName should be in lower case. clusterName is explained below.</p> +<ul> +<li>hive_db.qualifiedName - <dbName>@<clusterName></li> +<li>hive_table.qualifiedName - <dbName>.<tableName>@<clusterName></li> +<li>hive_column.qualifiedName - <dbName>.<tableName>.<columnName>@<clusterName></li> +<li>hive_process.queryString - trimmed query string in lower case</li></ul></div> +<div class="section"> +<h3><a name="Importing_Hive_Metadata"></a>Importing Hive Metadata</h3> +<p>org.apache.atlas.hive.bridge.HiveMetaStoreBridge imports the Hive metadata into Atlas using the model defined above. import-hive.sh command can be used to facilitate this.</p> +<div class="source"><pre class="prettyprint"> + Usage: <atlas package>/hook-bin/import-hive.sh +</pre></div> +<p>The logs are in <atlas package>/logs/import-hive.log</p></div> +<div class="section"> +<h3><a name="Hive_Hook"></a>Hive Hook</h3> +<p>Atlas Hive hook registers with Hive to listen for create/update/delete operations and updates the metadata in Atlas, via Kafka notifications, for the changes in Hive. Follow the instructions below to setup Atlas hook in Hive:</p> +<ul> +<li>Set-up Atlas hook in hive-site.xml by adding the following:</li></ul> +<div class="source"><pre class="prettyprint"> + <property> + <name>hive.exec.post.hooks</name> + <value>org.apache.atlas.hive.hook.HiveHook</value> + </property> +</pre></div> +<p></p> +<ul> +<li>Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in hive-env.sh of your hive configuration</li> +<li>Copy <atlas-conf>/atlas-application.properties to the hive conf directory.</li></ul> +<p>The following properties in <atlas-conf>/atlas-application.properties control the thread pool and notification details:</p> +<ul> +<li>atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.</li> +<li>atlas.hook.hive.numRetries - number of retries for notification failure. default 3</li> +<li>atlas.hook.hive.minThreads - core number of threads. default 1</li> +<li>atlas.hook.hive.maxThreads - maximum number of threads. default 5</li> +<li>atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10</li> +<li>atlas.hook.hive.queueSize - queue size for the threadpool. default 10000</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="Column_Level_Lineage"></a>Column Level Lineage</h3> +<p>Starting from 0.8-incubating version of Atlas, Column level lineage is captured in Atlas. Below are the details</p></div> +<div class="section"> +<h4><a name="Model"></a>Model</h4> +<p></p> +<ul> +<li>ColumnLineageProcess type is a subtype of Process</li></ul> +<p></p> +<ul> +<li>This relates an output Column to a set of input Columns or the Input Table</li></ul> +<p></p> +<ul> +<li>The lineage also captures the kind of dependency, as listed below: +<ul> +<li>SIMPLE: output column has the same value as the input</li> +<li>EXPRESSION: output column is transformed by some expression at runtime (for e.g. a Hive SQL expression) on the Input Columns.</li> +<li>SCRIPT: output column is transformed by a user provided script.</li></ul></li></ul> +<p></p> +<ul> +<li>In case of EXPRESSION dependency the expression attribute contains the expression in string form</li></ul> +<p></p> +<ul> +<li>Since Process links input and output DataSets, Column is a subtype of DataSet</li></ul></div> +<div class="section"> +<h4><a name="Examples"></a>Examples</h4> +<p>For a simple CTAS below:</p> +<div class="source"><pre class="prettyprint"> +create table t2 as select id, name from T1 +</pre></div> +<p>The lineage is captured as</p> +<p><img src="images/column_lineage_ex1.png" alt="" /></p></div> +<div class="section"> +<h4><a name="Extracting_Lineage_from_Hive_commands"></a>Extracting Lineage from Hive commands</h4> +<p>* The HiveHook maps the LineageInfo in the HookContext to Column lineage instances</p> +<p>* The LineageInfo in Hive provides column-level lineage for the final FileSinkOperator, linking them to the input columns in the Hive Query</p></div> +<div class="section"> +<h3><a name="NOTES"></a>NOTES</h3> +<p></p> +<ul> +<li>Column level lineage works with Hive version 1.2.1 after the patch for <a href="https://issues.apache.org/jira/browse/HIVE-13112";>HIVE-13112</a> is applied to Hive source</li> +<li>Since database name, table name and column names are case insensitive in hive, the corresponding names in entities are lowercase. So, any search APIs should use lowercase while querying on the entity names</li> +<li>The following hive operations are captured by hive hook currently +<ul> +<li>create database</li> +<li>create table/view, create table as select</li> +<li>load, import, export</li> +<li>DMLs (insert)</li> +<li>alter database</li> +<li>alter table (skewed table information, stored as, protection is not supported)</li> +<li>alter view</li></ul></li></ul></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/atlas-website/blob/2d4dc1a0/0.8.3/Bridge-Sqoop.html ---------------------------------------------------------------------- diff --git a/0.8.3/Bridge-Sqoop.html b/0.8.3/Bridge-Sqoop.html new file mode 100644 index 0000000..32a3bea --- /dev/null +++ b/0.8.3/Bridge-Sqoop.html @@ -0,0 +1,181 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/twiki/Bridge-Sqoop.twiki at 2018-10-31 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20181031" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Apache Atlas – Sqoop Atlas Bridge</title> + <link rel="stylesheet" href="./css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="./css/site.css" /> + <link rel="stylesheet" href="./css/print.css" media="print" /> + <script type="text/javascript" src="./js/apache-maven-fluido-1.7.min.js"></script> + </head> + <body class="topBarEnabled"> + <div id="topbar" class="navbar navbar-fixed-top "> + <div class="navbar-inner"> + <div class="container" style="width: 68%;"><div class="nav-collapse"> + <ul class="nav"> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Atlas <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="index.html" title="About">About</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="Wiki">Wiki</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS"; title="News">News</a></li> + <li><a href="https://git-wip-us.apache.org/repos/asf/atlas.git"; title="Git">Git</a></li> + <li><a href="https://issues.apache.org/jira/browse/ATLAS"; title="Jira">Jira</a></li> + <li><a href="https://cwiki.apache.org/confluence/display/ATLAS/PoweredBy"; title="Powered by">Powered by</a></li> + <li><a href="http://blogs.apache.org/atlas/"; title="Blog">Blog</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Project Information <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="project-info.html" title="Summary">Summary</a></li> + <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> + <li><a href="http://webchat.freenode.net?channels=apacheatlas&uio=d4"; title="IRC">IRC</a></li> + <li><a href="team-list.html" title="Team">Team</a></li> + <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> + <li><a href="source-repository.html" title="Source Repository">Source Repository</a></li> + <li><a href="license.html" title="License">License</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Releases <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/dyn/closer.cgi/atlas/0.8.2/"; title="0.8.2">0.8.2</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.1/"; title="0.8.1">0.8.1</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.8.0-incubating/"; title="0.8-incubating">0.8-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.1-incubating/"; title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.7.0-incubating/"; title="0.7-incubating">0.7-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.6.0-incubating/"; title="0.6-incubating">0.6-incubating</a></li> + <li><a href="http://archive.apache.org/dist/incubator/atlas/0.5.0-incubating/"; title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="../index.html" title="latest">latest</a></li> + <li><a href="../0.8.2/index.html" title="0.8.2">0.8.2</a></li> + <li><a href="../0.8.1/index.html" title="0.8.1">0.8.1</a></li> + <li><a href="../0.8.0-incubating/index.html" title="0.8-incubating">0.8-incubating</a></li> + <li><a href="../0.7.1-incubating/index.html" title="0.7.1-incubating">0.7.1-incubating</a></li> + <li><a href="../0.7.0-incubating/index.html" title="0.7-incubating">0.7-incubating</a></li> + <li><a href="../0.6.0-incubating/index.html" title="0.6-incubating">0.6-incubating</a></li> + <li><a href="../0.5.0-incubating/index.html" title="0.5-incubating">0.5-incubating</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/foundation/how-it-works.html"; title="How Apache Works">How Apache Works</a></li> + <li><a href="http://www.apache.org/foundation/"; title="Foundation">Foundation</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; title="Sponsoring Apache">Sponsoring Apache</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; title="Thanks">Thanks</a></li> + </ul> + </li> + </ul> +<form id="search-form" action="https://www.google.com/search"; method="get" class="navbar-search pull-right" > + <input value="http://atlas.apache.org"; name="sitesearch" type="hidden"/> + <input class="search-query" name="q" id="query" type="text" /> +</form> +<script type="text/javascript">asyncJs( 'https://cse.google.com/brand?form=search-form' )</script> + <iframe src="https://www.facebook.com/plugins/like.php?href=http://atlas.apache.org/atlas-docs&send=false&layout=button_count&show-faces=false&action=like&colorscheme=dark"; + scrolling="no" frameborder="0" + style="border:none; width:100px; height:20px; margin-top: 10px;" class="pull-right" ></iframe> + <script type="text/javascript">asyncJs( 'https://apis.google.com/js/plusone.js' )</script> + <ul class="nav pull-right"><li style="margin-top: 10px;"> + <div class="g-plusone" data-href="http://atlas.apache.org/atlas-docs"; data-size="medium" width="60px" align="right" ></div> + </li></ul> + </div> + </div> + </div> + </div> + <div class="container"> + <div id="banner"> + <div class="pull-left"><a href=".." id="bannerLeft"><img src="images/atlas-logo.png" alt="Apache Atlas" width="200px" height="45px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="index.html" title="Atlas">Atlas</a><span class="divider">/</span></li> + <li class="active ">Sqoop Atlas Bridge</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-10-31</li> + <li id="projectVersion" class="pull-right">Version: 0.8.3</li> + </ul> + </div> + <div id="bodyColumn" > +<div class="section"> +<h2><a name="Sqoop_Atlas_Bridge"></a>Sqoop Atlas Bridge</h2></div> +<div class="section"> +<h3><a name="Sqoop_Model"></a>Sqoop Model</h3> +<p>The default hive model includes the following types:</p> +<ul> +<li>Entity types: +<ul> +<li>sqoop_process +<ul> +<li>super-types: Process</li> +<li>attributes: name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName</li></ul></li> +<li>sqoop_dbdatastore +<ul> +<li>super-types: DataSet</li> +<li>attributes: name, dbStoreType, storeUse, storeUri, source, description, ownerName</li></ul></li></ul></li></ul> +<p></p> +<ul> +<li>Enum types: +<ul> +<li>sqoop_operation_type +<ul> +<li>values: IMPORT, EXPORT, EVAL</li></ul></li> +<li>sqoop_dbstore_usage +<ul> +<li>values: TABLE, QUERY, PROCEDURE, OTHER</li></ul></li></ul></li></ul> +<p>The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well:</p> +<ul> +<li>sqoop_process.qualifiedName - dbStoreType-storeUri-endTime</li> +<li>sqoop_dbdatastore.qualifiedName - dbStoreType-storeUri-source</li></ul></div> +<div class="section"> +<h3><a name="Sqoop_Hook"></a>Sqoop Hook</h3> +<p>Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in SqoopHook. This is used to add entities in Atlas using the model detailed above.</p> +<p>Follow the instructions below to setup Atlas hook in Hive:</p> +<p>Add the following properties to to enable Atlas hook in Sqoop:</p> +<ul> +<li>Set-up Atlas hook in <sqoop-conf>/sqoop-site.xml by adding the following:</li></ul> +<div class="source"><pre class="prettyprint"> + <property> + <name>sqoop.job.data.publish.class</name> + <value>org.apache.atlas.sqoop.hook.SqoopHook</value> + </property> +</pre></div> +<p></p> +<ul> +<li>Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/</li> +<li>Link <atlas-home>/hook/sqoop/*.jar in sqoop lib</li></ul> +<p>Refer <a href="./Configuration.html">Configuration</a> for notification related configurations</p></div> +<div class="section"> +<h3><a name="NOTES"></a>NOTES</h3> +<p></p> +<ul> +<li>Only the following sqoop operations are captured by sqoop hook currently - hiveImport</li></ul></div> + </div> + </div> + <hr/> + <footer> + <div class="container"> + <div class="row"> +Copyright é 2018 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. + </div> + <p id="poweredBy" class="pull-right"><a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> +</p> + </div> + </footer> + </body> +</html>
