This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git
The following commit(s) were added to refs/heads/gh-pages by this push:
new 9291880 DRILL-6662
9291880 is described below
commit 92918806772d5d3869a42737c3d287a8e23a6901
Author: Bridget Bevens <[email protected]>
AuthorDate: Fri Dec 21 20:01:30 2018 -0800
DRILL-6662
---
.../plugins/110-s3-storage-plugin.md | 93 ++++++++++------------
1 file changed, 41 insertions(+), 52 deletions(-)
diff --git a/_docs/connect-a-data-source/plugins/110-s3-storage-plugin.md
b/_docs/connect-a-data-source/plugins/110-s3-storage-plugin.md
index 3c7fd7a..ac693d1 100644
--- a/_docs/connect-a-data-source/plugins/110-s3-storage-plugin.md
+++ b/_docs/connect-a-data-source/plugins/110-s3-storage-plugin.md
@@ -1,6 +1,6 @@
---
title: "S3 Storage Plugin"
-date: 2018-12-18
+date: 2018-12-22
parent: "Connect a Data Source"
---
Drill works with data stored in the cloud. With a few simple steps, you can
configure the S3 storage plugin for Drill and be off to the races running
queries.
@@ -9,7 +9,6 @@ Drill has the ability to query files stored on Amazon's S3
cloud storage using t
To connect Drill to S3:
-
- Provide your AWS credentials.
- Configure the S3 storage plugin with an S3 bucket name.
@@ -20,9 +19,11 @@ For additional information, refer to the [HDFS S3
documentation](https://hadoop.
## Providing AWS Credentials
-Your environment determines where you provide your AWS credentials. You can
use either of the following methods to define your AWS credentials:
+Your environment determines where you provide your AWS credentials. You can
use the following methods to define your AWS credentials:
-- Directly in the S3 storage plugin. Note that this method is the least
secure, but sufficient for use on a single machine, such as a laptop.
+- In the S3 storage plugin configuration:
+ - [You can point to an encrypted file in an external
provider.]({{site.baseurl}}/docs/s3-storage-plugin/#using-an-external-provider-for-credentials)
(Drill 1.15 and later)
+ - [You can put your access and secret keys directly in the storage
plugin
configuration.]({{site.baseurl}}/docs/s3-storage-plugin/#adding-credentials-directly-to-the-s3-plugin)
Note that this method is the least secure, but sufficient for use on a single
machine, such as a laptop.
- In a non-Hadoop environment, you can use the Drill-specific core-site.xml
file to provide the AWS credentials.
### Defining Access Keys in the S3 Storage Plugin
@@ -31,7 +32,7 @@ Refer to [Configuring the S3 Storage
Plugin]({{site.baseurl}}/docs/s3-storage-pl
### Defining Access Keys in the Drill core-site.xml File
-To configure the access keys in Drill's core-site.xml file, navigate to the
`$DRILL_HOME/conf` or `$DRILL_SITE` directory, and rename the
core-site-example.xml file to core-site.xml. Replace the text
`ENTER_YOUR_ACESSKEY` and `ENTER_YOUR_SECRETKEY` with your AWS credentials and
also include the endpoint, as shown in the following example:
+To configure the access keys in Drill's core-site.xml file, navigate to the
`$DRILL_HOME/conf` or `$DRILL_SITE` directory, and rename the
`core-site-example.xml` file to `core-site.xml`. Replace the text
`ENTER_YOUR_ACESSKEY` and `ENTER_YOUR_SECRETKEY` with your AWS credentials and
also include the endpoint, as shown in the following example:
<configuration>
<property>
@@ -61,27 +62,46 @@ If you use IAM roles/Instance profiles, to access data in
s3, use the following
**Note:** When you rename the file, Hadoop support breaks if `$HADOOP_HOME`
was in the path because Drill pulls in the Drill core-site.xml file instead of
the Hadoop core-site.xml file. In this situation, make the changes in the
Hadoop core-site.xml file. Do not create a core-site.xml file for Drill.
-## Configuring the S3 Storage Plugin
+##Configuring the S3 Storage Plugin
+
+The Storage page in the Drill Web UI provides an S3 storage plugin that you
configure to connect Drill to the S3 distributed file system registered in
core-site.xml. If you did not define your AWS credentials in the core-site.xml
file, you can define them in the storage plugin configuration. You can define
the credentials directly in the configuration, or you can use an external
provider.
+
+To configure the S3 storage plugin, log in to the Drill Web UI at
`http://<drill-hostname>:8047`. The drill-hostname is a node on which Drill is
running. Go to the **Storage** page and click **Update** next to the S3 storage
plugin option. Edit the configuration and then click **Update** to save the
configuration.
+
+**Note:** The `"config"` block in the S3 storage plugin configuration contains
contains properties to define your AWS credentials. Do not include the
`"config"` block in your S3 storage plugin configuration if you defined your
AWS credentials in the core-site.xml file.
-The Storage page in the Drill Web UI provides an S3 storage plugin that you
configure to connect Drill to the S3 distributed file system registered in
core-site.xml. If you did not define your AWS credentials in the core-site.xml
file, you can define them in the storage plugin configuration.
+Use either of the following methods to provide your credentials:
-To configure the S3 storage plugin, log in to the Drill Web UI and then update
the S3 configuration with the bucket name, as described in the following steps:
+### Using an External Provider for Credentials
+Starting in Drill 1.15, the S3 storage plugin supports the [Hadoop Credential
Provider
API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]),
which allows you to store secret keys and other sensitive data in an encrypted
file in an external provider versus storing them in plain text in a
configuration file or storage plugin configuration.
-1\. To access the Drill Web UI, enter the following URL in the address bar of
your web browser:
+When you configure the S3 storage plugin to use an external provider, Drill
first checks the external provider for the keys. If the keys are not available
via the provider, or the provider is not configured, Drill can fall back to
using the plain text data in the `core-site.xml` file or S3 configuration,
unless the `hadoop.security.credential.clear-text-fallback` property is set to
`false`.
- http://<drill-hostname>:8047
-
- //The drill-hostname is a node on which Drill is running.
+**Configuring the S3 Plugin to use an External Provider**
+Add the bucket name, `hadoop.security.credential.provider.path` and
`fs.s3a.impl.disable.cache` properties to the S3 storage plugin configuration,
as shown in the following example:
+
+ {
+ "type":
+ "file",
+ "connection": "s3a://bucket-name/",
+ "config": {
+
"hadoop.security.credential.provider.path":"jceks://file/tmp/s3.jceks",
+ "Fs.s3a.impl.disable.cache":"true",
+ ...
+ },
+ "workspaces": {
+ ...
+ }
-2\. To configure the S3 storage plugin in Drill, complete the following steps:
+
+**Note:** The `hadoop.security.credential.provider.path` property should point
to a file that contains your encrypted passwords. The
`fs.s3a.impl.disable.cache` option must be set to true.
- a\. Click on the **Storage** page.
- b\. Find the S3 option on the page and then click **Update** next to the
option.
- c\. Configure the S3 storage plugin, specifying the bucket in the
`"connection"` property, as shown in the following example:
+###Adding Credentials Directly to the S3 Plugin
+You can add your AWS credentials directly to the S3 configuration, though this
method is the least secure, but sufficient for use on a single machine, such as
a laptop.
-**Note:** The `"config"` block in the following S3 storage plugin
configuration contains the access key and endpoint properties required if you
want to define your AWS credentials here. Do not include the `"config"` block
in your S3 storage plugin configuration if you defined your AWS credentials in
the core-site.xml file.
+Add the S3 bucket name and the `"config"` block with the properties shown in
the following example:
- {
+ {
"type": "file",
"enabled": true,
"connection": "s3a://bucket-name/",
@@ -90,42 +110,11 @@ To configure the S3 storage plugin, log in to the Drill
Web UI and then update t
"fs.s3a.secret.key": "<key>",
"fs.s3a.endpoint": "s3.us-west-1.amazonaws.com"
},
- "workspaces": {
- "root": {
- "location": "/user/robot/drill",
- "writable": true,
- "defaultInputFormat": null
+ "workspaces": {...
},
- "tmp": {
- "location": "/tmp",
- "writable": true,
- "defaultInputFormat": null
- }
- },
- "formats": {
- "psv": {
- "type": "text",
- "extensions": [
- "tbl"
- ],
- "delimiter": "|"
- },
- "csv": {
- "type": "text",
- "extensions": [
- "csv"
- ],
- "delimiter": ","
- }
- }
- }
-
-
-4-Click **Update** to save the configuration.
-5-Navigate back to the **Storage** page.
-6-On the **Storage** page, click **Enable** next to the S3 option.
-Drill should now be able to use the HDFS s3a library to access data in S3.
+
+Drill can now use the HDFS s3a library to access data in S3.
## Quering Parquet Format Files On S3
