This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 55efe5a edit s3 SP doc
55efe5a is described below
commit 55efe5a8af8e7d0a96eb7ce8fde12fe1905ab7a2
Author: Bridget Bevens <[email protected]>
AuthorDate: Thu Jun 21 16:29:54 2018 -0700
edit s3 SP doc
---
docs/s3-storage-plugin/index.html | 61 +++++++++++++++++----------------------
feed.xml | 4 +--
2 files changed, 28 insertions(+), 37 deletions(-)
diff --git a/docs/s3-storage-plugin/index.html
b/docs/s3-storage-plugin/index.html
index 30c5919..8663260 100644
--- a/docs/s3-storage-plugin/index.html
+++ b/docs/s3-storage-plugin/index.html
@@ -1230,36 +1230,37 @@
</div>
- Jun 19, 2018
+ Jun 21, 2018
<link href="/css/docpage.css" rel="stylesheet" type="text/css">
<div class="int_text" align="left">
- <p>Drill works with data stored in the cloud. With a few simple steps,
you can configure the S3 storage plugin for Drill and be off to the races
running queries.</p>
+ <p>Drill works with data stored in the cloud. With a few simple steps,
you can configure the S3 storage plugin for Drill and be off to the races
running queries. Drill has the ability to query files stored on Amazon's S3
cloud storage using the HDFS s3a library. The HDFS s3a library adds support for
files larger than 5 gigabytes (these were unsupported using the older HDFS s3n
library).</p>
-<h2 id="connecting-drill-to-s3">Connecting Drill to S3</h2>
+<p>To connect Drill to S3, you must provide your AWS credentials, and
configure the S3 storage plugin with an S3 bucket. </p>
-<p>Drill has the ability to query files stored on Amazon's S3 cloud
storage using the HDFS s3a library. The HDFS s3a library adds support for files
larger than 5 gigabytes (these were unsupported using the older HDFS s3n
library).</p>
+<p>For additional information, refer to the <a
href="https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html";>HDFS
S3 documentation</a>. </p>
-<p>To connect Drill to S3: </p>
+<p><strong>Note:</strong> Drill does not use HDFS 3.x, therefore Drill does
not support AWS temporary credentials, as described in the s3a
documentation.</p>
-<ul>
-<li>Provide your AWS credentials.<br></li>
-<li>Configure the S3 storage plugin with an S3 bucket.<br></li>
-</ul>
+<h2 id="providing-aws-credentials">Providing AWS Credentials</h2>
-<p>For additional information, you can refer to the <a
href="https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html";>HDFS
S3 documentation</a>. </p>
+<p>Your environment determines where you provide your AWS credentials. You can
define your AWS credentials one of three ways: </p>
-<p><strong>Note:</strong> Drill does not use HDFS 3.x, therefore Drill does
not support AWS temporary credentials, as described in the s3a
documentation.</p>
+<ul>
+<li>Directly in the S3 storage plugin. Note that this method is the least
secure, but sufficient for use on a single machine, such as a laptop.<br></li>
+<li>In a non-Hadoop environment, you can use the Drill-specific core-site.xml
file to provide the AWS credentials.<br></li>
+<li>In a Hadoop environment, you can use the existing S3 configuration for
Hadoop. The S3 access keys should already be defined. All you need to do is <a
href="/docs/s3-storage-plugin/#configuring-the-s3-storage-plugin">configure the
S3 storage plugin</a>.<br></li>
+</ul>
-<h3 id="providing-aws-credentials">Providing AWS Credentials</h3>
+<h3 id="defining-access-keys-in-the-s3-storage-plugin">Defining Access Keys in
the S3 Storage Plugin</h3>
-<p>Define the AWS credentials (access keys) in the core-site.xml file or as
storage plugin configuration properties in the S3 storage plugin configuration
in the Drill Web UI. However, configuring credentials in the S3 storage plugin
in not very secure, but sufficient for use on a single machine, such as a
laptop. </p>
+<p>Refer to <a
href="/docs/s3-storage-plugin/#configuring-the-s3-storage-plugin">Configuring
the S3 Storage Plugin</a>. </p>
-<p><strong>Configuring Access Keys in core-site.xml</strong></p>
+<h3 id="defining-access-keys-in-the-drill-core-site.xml-file">Defining Access
Keys in the Drill core-site.xml File</h3>
-<p>To configure the access keys in the core-site.xml file, navigate to the
<code>$DRILL_HOME/conf</code> or <code>$DRILL_SITE</code> directory, and rename
the core-site-example.xml file to core-site.xml. Replace the text
<code>ENTER_YOUR_ACESSKEY</code> and <code>ENTER_YOUR_SECRETKEY</code> with
your AWS credentials and also include the endpoint, as shown in the following
example: </p>
+<p>To configure the access keys in Drill's core-site.xml file, navigate to
the <code>$DRILL_HOME/conf</code> or <code>$DRILL_SITE</code> directory, and
rename the core-site-example.xml file to core-site.xml. Replace the text
<code>ENTER_YOUR_ACESSKEY</code> and <code>ENTER_YOUR_SECRETKEY</code> with
your AWS credentials and also include the endpoint, as shown in the following
example: </p>
<div class="highlight"><pre><code class="language-text" data-lang="text">
<configuration>
<property>
<name>fs.s3a.access.key</name>
@@ -1277,34 +1278,24 @@
</code></pre></div>
<p><strong>Note:</strong> When you rename the file, Hadoop support breaks if
<code>$HADOOP_HOME</code> was in the path because Drill pulls in the Drill
core-site.xml file instead of the Hadoop core-site.xml file. In this situation,
make the changes in the Hadoop core-site.xml file. Do not create a
core-site.xml file for Drill. </p>
-<p><strong>Configuring Access Keys in the S3 Storage Plugin</strong> </p>
-
-<p>If you choose to configure the access keys in the S3 plugin, include the
access keys and the endpoint in the S3 storage plugin configuration, as shown
in the following example: </p>
-<div class="highlight"><pre><code class="language-text" data-lang="text">
"connection": "s3a://<bucket-name>/",
- "config": {
- "fs.s3a.access.key": "<key>",
- "fs.s3a.secret.key": "<key>",
- "fs.s3a.endpoint": "s3.us-west-1.amazonaws.com"
- },
-</code></pre></div>
-<p><strong>Note:</strong> General instructions for configuring the S3 storage
plugin are provided in the next section of this document. </p>
+<h2 id="configuring-the-s3-storage-plugin">Configuring the S3 Storage
Plugin</h2>
-<h3 id="configuring-the-s3-storage-plugin">Configuring the S3 Storage
Plugin</h3>
+<p>The Storage page in the Drill Web UI provides an S3 storage plugin that you
configure to connect Drill to the S3 distributed file system registered in
core-site.xml. If you did not define your AWS credentials in the core-site.xml
file, you can define them in the storage plugin configuration. </p>
-<p>The Storage page in the Drill Web UI provides an S3 storage plugin that you
configure to connect Drill to the S3 distributed file system registered in
core-site.xml. Once configured, you can query the files stored in S3. To
configure the S3 storage plugin, log in to the Drill Web UI and then update the
S3 configuration, as described in the following steps: </p>
+<p>To configure the S3 storage plugin, log in to the Drill Web UI and then
update the S3 configuration with the bucket name, as described in the following
steps: </p>
<p>1- To access the Drill Web UI, enter the following URL in the address bar
of your web browser: </p>
<div class="highlight"><pre><code class="language-text" data-lang="text">
http://<drill-hostname>:8047
//The drill-hostname is a node on which Drill is running.
</code></pre></div>
-<p>2-To configure the S3 storage plugin, complete the following steps: </p>
+<p>2-To configure the S3 storage plugin in Drill, complete the following
steps: </p>
-<p>a. Click on the Storage page.<br>
+<p>a. Click on the <strong>Storage</strong> page.<br>
b. Find the S3 option on the page and then click <strong>Update</strong>
next to the option.<br>
- c. Configure the S3 storage plugin, specifying the bucket in the
"connection" property, as shown in the following example: </p>
+ c. Configure the S3 storage plugin, specifying the bucket in the
<code>"connection"</code> property, as shown in the following
example: </p>
-<p><strong>Note:</strong> The "config" section with the access key
and endpoint properties are included in the following configuration as an
example. Do not include the "config" section in your S3 configuration
if you used core-site.xml to store your access keys. </p>
+<p><strong>Note:</strong> The <code>"config"</code> block in the
following S3 storage plugin configuration contains the access key and endpoint
properties required if you want to define your AWS credentials in the storage
plugin configuration. Do not include the <code>"config"</code> block
in your S3 storage plugin configuration if you defined your AWS credentials in
the core-site.xml file. </p>
<div class="highlight"><pre><code class="language-text" data-lang="text"> {
"type": "file",
"enabled": true,
@@ -1345,10 +1336,10 @@
}
</code></pre></div>
<p>4-Click <strong>Update</strong> to save the configuration.<br>
-5-Navigate back to the Storage page.<br>
-6-On the Storage page, click <strong>Enable</strong> next to the S3 option.
</p>
+5-Navigate back to the <strong>Storage</strong> page.<br>
+6-On the <strong>Storage</strong> page, click <strong>Enable</strong> next to
the S3 option. </p>
-<p>Drill should now be able to use the HDFS s3a library to access data in S3.
</p>
+<p>Drill should now be able to use the HDFS s3a library to access data in
S3.</p>
<h2 id="quering-parquet-format-files-on-s3">Quering Parquet Format Files On
S3</h2>
diff --git a/feed.xml b/feed.xml
index 7cfe89a..fe30a50 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>/</link>
<atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 20 Jun 2018 18:52:18 -0700</pubDate>
- <lastBuildDate>Wed, 20 Jun 2018 18:52:18 -0700</lastBuildDate>
+ <pubDate>Thu, 21 Jun 2018 16:28:11 -0700</pubDate>
+ <lastBuildDate>Thu, 21 Jun 2018 16:28:11 -0700</lastBuildDate>
<generator>Jekyll v2.5.2</generator>
<item>
