HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/881ec4d9 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/881ec4d9 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/881ec4d9 Branch: refs/heads/YARN-5734 Commit: 881ec4d97bd1db4582027aec3a4204156a4eda17 Parents: a96afae Author: Mingliang Liu <lium...@apache.org> Authored: Tue Mar 7 16:29:19 2017 -0800 Committer: Mingliang Liu <lium...@apache.org> Committed: Fri Mar 10 00:16:09 2017 -0800 ---------------------------------------------------------------------- .../src/site/markdown/index.md | 55 ++++++++------------ 1 file changed, 21 insertions(+), 34 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/881ec4d9/hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md b/hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md index 9355241..3a16253 100644 --- a/hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md +++ b/hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md @@ -14,28 +14,15 @@ # Hadoop Azure Data Lake Support -* [Introduction](#Introduction) -* [Features](#Features) -* [Limitations](#Limitations) -* [Usage](#Usage) - * [Concepts](#Concepts) - * [OAuth2 Support](#OAuth2_Support) - * [Configuring Credentials and FileSystem](#Configuring_Credentials) - * [Using Refresh Token](#Refresh_Token) - * [Using Client Keys](#Client_Credential_Token) - * [Protecting the Credentials with Credential Providers](#Credential_Provider) - * [Enabling ADL Filesystem](#Enabling_ADL) - * [Accessing `adl` URLs](#Accessing_adl_URLs) - * [User/Group Representation](#OIDtoUPNConfiguration) -* [Testing the `hadoop-azure` Module](#Testing_the_hadoop-azure_Module) - -## <a name="Introduction" />Introduction +<!-- MACRO{toc|fromDepth=1|toDepth=3} --> + +## Introduction The `hadoop-azure-datalake` module provides support for integration with the [Azure Data Lake Store](https://azure.microsoft.com/en-in/documentation/services/data-lake-store/). This support comes via the JAR file `azure-datalake-store.jar`. -## <a name="Features" />Features +## Features * Read and write data stored in an Azure Data Lake Storage account. * Reference file system paths using URLs using the `adl` scheme for Secure Webhdfs i.e. SSL @@ -46,7 +33,7 @@ This support comes via the JAR file `azure-datalake-store.jar`. * API `setOwner()`, `setAcl`, `removeAclEntries()`, `modifyAclEntries()` accepts UPN or OID (Object ID) as user and group names. -## <a name="Limitations" />Limitations +## Limitations Partial or no support for the following operations : @@ -62,9 +49,9 @@ Partial or no support for the following operations : * User and group information returned as `listStatus()` and `getFileStatus()` is in the form of the GUID associated in Azure Active Directory. -## <a name="Usage" />Usage +## Usage -### <a name="Concepts" />Concepts +### Concepts Azure Data Lake Storage access path syntax is: ``` @@ -74,7 +61,7 @@ adl://<Account Name>.azuredatalakestore.net/ For details on using the store, see [**Get started with Azure Data Lake Store using the Azure Portal**](https://azure.microsoft.com/en-in/documentation/articles/data-lake-store-get-started-portal/) -### <a name="#OAuth2_Support" />OAuth2 Support +#### OAuth2 Support Usage of Azure Data Lake Storage requires an OAuth2 bearer token to be present as part of the HTTPS header as per the OAuth2 specification. @@ -86,11 +73,11 @@ and identity management service. See [*What is ActiveDirectory*](https://azure.m Following sections describes theOAuth2 configuration in `core-site.xml`. -#### <a name="Configuring_Credentials" />Configuring Credentials & FileSystem +### Configuring Credentials and FileSystem Credentials can be configured using either a refresh token (associated with a user), or a client credential (analogous to a service principal). -#### <a name="Refresh_Token" />Using Refresh Tokens +#### Using Refresh Tokens Add the following properties to the cluster's `core-site.xml` @@ -119,9 +106,9 @@ service associated with the client id. See [*Active Directory Library For Java*] ``` -### <a name="Client_Credential_Token" />Using Client Keys +#### Using Client Keys -#### Generating the Service Principal +##### Generating the Service Principal 1. Go to [the portal](https://portal.azure.com) 2. Under "Browse", look for Active Directory and click on it. @@ -135,13 +122,13 @@ service associated with the client id. See [*Active Directory Library For Java*] - The token endpoint (select "View endpoints" at the bottom of the page and copy/paste the OAuth2 .0 Token Endpoint value) - Resource: Always https://management.core.windows.net/ , for all customers -#### Adding the service principal to your ADL Account +##### Adding the service principal to your ADL Account 1. Go to the portal again, and open your ADL account 2. Select Users under Settings 3. Add your user name you created in Step 6 above (note that it does not show up in the list, but will be found if you searched for the name) 4. Add "Owner" role -### Configure core-site.xml +##### Configure core-site.xml Add the following properties to your `core-site.xml` ```xml @@ -161,7 +148,7 @@ Add the following properties to your `core-site.xml` </property> ``` -### <a name="Credential_Provider" />Protecting the Credentials with Credential Providers +#### Protecting the Credentials with Credential Providers In many Hadoop clusters, the `core-site.xml` file is world-readable. To protect these credentials, it is recommended that you use the @@ -171,7 +158,7 @@ All ADLS credential properties can be protected by credential providers. For additional reading on the credential provider API, see [Credential Provider API](../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html). -#### Provisioning +##### Provisioning ```bash hadoop credential create dfs.adls.oauth2.refresh.token -value 123 @@ -180,7 +167,7 @@ hadoop credential create dfs.adls.oauth2.credential -value 123 -provider localjceks://file/home/foo/adls.jceks ``` -#### Configuring core-site.xml or command line property +##### Configuring core-site.xml or command line property ```xml <property> @@ -190,7 +177,7 @@ hadoop credential create dfs.adls.oauth2.credential -value 123 </property> ``` -#### Running DistCp +##### Running DistCp ```bash hadoop distcp @@ -203,7 +190,7 @@ NOTE: You may optionally add the provider path property to the `distcp` command line instead of added job specific configuration to a generic `core-site.xml`. The square brackets above illustrate this capability.` -### <a name="Accessing_adl_URLs" />Accessing adl URLs +### Accessing adl URLs After credentials are configured in `core-site.xml`, any Hadoop component may reference files in that Azure Data Lake Storage account by using URLs of the following @@ -230,7 +217,7 @@ hadoop fs -put testFile adl://yourcontainer.azuredatalakestore.net/testDir/testF hadoop fs -cat adl://yourcontainer.azuredatalakestore.net/testDir/testFile test file content ``` -### <a name="OIDtoUPNConfiguration" />User/Group Representation +### User/Group Representation The `hadoop-azure-datalake` module provides support for configuring how User/Group information is represented during @@ -254,7 +241,7 @@ Add the following properties to `core-site.xml` </description> </property> ``` -## <a name="Testing_the_hadoop-azure_Module" />Testing the azure-datalake-store Module +## Testing the azure-datalake-store Module The `hadoop-azure` module includes a full suite of unit tests. Most of the tests will run without additional configuration by running mvn test. This includes tests against mocked storage, which is an in-memory emulation of Azure Data Lake Storage. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
