This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git
The following commit(s) were added to refs/heads/gh-pages by this push:
new ca8ee66 Doc updates and edits for Drill 1.15
ca8ee66 is described below
commit ca8ee666973e694fc11fc55edd9e94d508fb62de
Author: Bridget Bevens <[email protected]>
AuthorDate: Thu Dec 27 22:24:55 2018 -0800
Doc updates and edits for Drill 1.15
---
.../120-configuring-the-drill-shell.md | 18 ++++--
.../configuration-options/020-start-up-options.md | 20 +++++--
...093-configuring-custom-acls-to-secure-znodes.md | 25 ++++++--
.../030-date-time-functions-and-arithmetic.md | 66 +++++++++++-----------
4 files changed, 81 insertions(+), 48 deletions(-)
diff --git a/_docs/configure-drill/120-configuring-the-drill-shell.md
b/_docs/configure-drill/120-configuring-the-drill-shell.md
index 1f5783a..41eb5e8 100644
--- a/_docs/configure-drill/120-configuring-the-drill-shell.md
+++ b/_docs/configure-drill/120-configuring-the-drill-shell.md
@@ -1,11 +1,11 @@
---
title: "Configuring the Drill Shell"
-date: 2018-12-27
+date: 2018-12-28
parent: "Configure Drill"
---
Drill uses SQLLine as the Drill shell. SQLLine is a pure-Java console-based
utility for connecting to relational databases and running SQL commands.
-Starting in Drill 1.15, Drill uses SQLLine 1.6, which you can customize
through the Drill [configuration file,
drill-sqlline-override.conf]({{site.baseurl}}/docs/configuring-the-drill-shell/#customizing-sqlline-in-the-drill-sqlline-override-conf-file).
Before installing and running Drill with SQLLine 1.6, delete the old SQLLine
history file The history file is located in the following location:
+Starting in Drill 1.15, Drill uses SQLLine 1.6, which you can customize
through the Drill [configuration file,
drill-sqlline-override.conf]({{site.baseurl}}/docs/configuring-the-drill-shell/#customizing-sqlline-in-the-drill-sqlline-override-conf-file).
Before installing and running Drill with SQLLine 1.6, delete the old SQLLine
history file located in:
- $HOME/.sqlline/history (UNIX, Linux, Mac OS)
@@ -19,9 +19,12 @@ Formatting tables takes time, which you may notice when
running a huge query usi
## Drill Shell Commands
-The following table lists the Drill shell commands that you can run from the
command line:
+The following table lists some of the Drill shell commands that you can run
from the command line. To see a complete list of shell commands issue the
`!help` command.
-| Command | Description
|
+**Note:** Some new shell commands, such as `!reset`, and key-strokes were
introduced when Drill upgraded to SQLLine 1.6 (Drill 1.15 and later); however,
not all of the commands are supported.
+
+
+| **Command** | **Description **
|
|---------------|---------------------------------------------------------------------------------------------------------------------------------------|
| !brief | Set verbose mode off.
|
| !close | Close the current connection to the database.
|
@@ -34,7 +37,10 @@ The following table lists the Drill shell commands that you
can run from the com
| !properties | Connect to the database specified in the properties file(s).
|
| !quit | Exits the Drill shell.
|
| !reconnect | Reconnect to the database.
|
-| !record | Record all output to the specified file.
|
+| !record | Record all output to the specified file.
+|
+| !reset | Reset a sqlline variable
+|
|
| !run | Run a script from the specified file.
|
| !save | Save the current variables and aliases.
|
| !script | Start saving a script to a file.
|
@@ -70,7 +76,7 @@ The following table lists some of the set command variables
that you can use wit
**Note:** Issue the !set command without a variable to see the full list of
command variables.
-| Variable Name | Valid Variable Values | Description
|
+| **Variable Name** | **Valid Variable Values** | **Description**
|
|-----------------|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| autoCommit | true/false | Enable/disable automatic
transaction commit. Should remain enabled (true). Drill performs read-only
operations primarily, and autocommit writes. Drill JDBC throws an exception if
autoCommit is disabled.
|
| autoSave | true/false | Automatically save preferences.
|
diff --git
a/_docs/configure-drill/configuration-options/020-start-up-options.md
b/_docs/configure-drill/configuration-options/020-start-up-options.md
index 78164a1..bfb2e38 100644
--- a/_docs/configure-drill/configuration-options/020-start-up-options.md
+++ b/_docs/configure-drill/configuration-options/020-start-up-options.md
@@ -1,6 +1,6 @@
---
title: "Start-Up Options"
-date: 2018-07-06 21:22:11 UTC
+date: 2018-12-28
parent: "Configuration Options"
---
The start-up options for Drill reside in a
[HOCON](https://github.com/typesafehub/config/blob/master/HOCON.md)
configuration file format, which is a hybrid between a properties file and a
JSON file. Drill start-up options consist of a group of files with a nested
relationship. At the bottom of the file hierarchy are the default files that
Drill provides, starting with `drill-default.conf`.
@@ -42,10 +42,20 @@ Run the following query to see a list of the available
start-up options:
## Configuring Start-Up Options
-You can configure start-up options for each drillbit in
`<drill_home>/conf/drill-override.conf` .
+You can configure start-up options for each Drillbit in
`<drill_home>/conf/drill-override.conf`.
The summary of start-up options, also known as boot options, lists default
values. The following descriptions provide more detail on key options that are
frequently reconfigured:
+* **store.parquet.complex.batch.num_records**
+ Introduced in Drill 1.15. Sets batch sizing (number of rows per batch) for
the Parquet reader. The default value is 4,000 rows. If rows are large and
queries return out-of-memory errors, reduce the value. You can set this option
at the system level in `<drill_home>/conf/drill-override.conf` or at the
session level using the SET command, as shown:
+
+ SET `store.parquet.complex.batch.num_records`=2000;
+ +-------+---------------------------------------------------+
+ | ok | summary |
+ +-------+---------------------------------------------------+
+ | true | store.parquet.complex.batch.num_records updated. |
+ +-------+---------------------------------------------------+
+
* **drill.exec.http.ssl_enabled**
Introduced in Drill 1.2. Enables or disables [HTTPS
support]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security/#https-support).
Settings are TRUE and FALSE, respectively. The default is FALSE.
@@ -56,10 +66,10 @@ The summary of start-up options, also known as boot
options, lists default value
Defines the amount of memory available, in terms of record batches, to hold
data on the downstream side of an operation. Drill pushes data downstream as
quickly as possible to make data immediately available. This requires Drill to
use memory to hold the data pending operations. When data on a downstream
operation is required, that data is immediately available so Drill does not
have to go over the network to process it. Providing more memory to this option
increases the speed at which [...]
* **drill.exec.spill.fs**
-Introduced in Drill 1.11. The default file system on the local machine into
which the Sort, Hash Aggregate, and Hash Join operators spill data. This is the
recommended option to use for spilling. You can configure this option so that
data spills into a distributed file system, such as hdfs. For example,
"hdfs:///". The default setting is "file:///". See [Sort-Based and Hash-Based
Memory Constrained
Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memory-constrained-operators/)
[...]
+ Introduced in Drill 1.11. The default file system on the local machine into
which the Sort, Hash Aggregate, and Hash Join operators spill data. This is the
recommended option to use for spilling. You can configure this option so that
data spills into a distributed file system, such as hdfs. For example,
"hdfs:///". The default setting is "file:///". See [Sort-Based and Hash-Based
Memory Constrained
Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memory-constrained-operators/
[...]
* **drill.exec.spill.directories**
-Introduced in Drill 1.11. The list of directories into which the Sort, Hash
Aggregate, and Hash Join operators spill data. The list must be an array with
directories separated by a comma, for example ["/fs1/drill/spill" ,
"/fs2/drill/spill" , "/fs3/drill/spill"]. This is the recommended option for
spilling to multiple directories. The default setting is ["/tmp/drill/spill"].
See [Sort-Based and Hash-Based Memory Constrained
Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memor [...]
+ Introduced in Drill 1.11. The list of directories into which the Sort, Hash
Aggregate, and Hash Join operators spill data. The list must be an array with
directories separated by a comma, for example ["/fs1/drill/spill" ,
"/fs2/drill/spill" , "/fs3/drill/spill"]. This is the recommended option for
spilling to multiple directories. The default setting is ["/tmp/drill/spill"].
See [Sort-Based and Hash-Based Memory Constrained
Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-mem [...]
* **drill.exec.zk.connect**
Provides Drill with the ZooKeeper quorum to use to connect to data sources.
Change this setting to point to the ZooKeeper quorum that you want Drill to
use. You must configure this option on each Drillbit node.
@@ -71,7 +81,7 @@ Introduced in Drill 1.11. The list of directories into which
the Sort, Hash Aggr
Introduced in Drill 1.11. Sets the maximum number of most recent profiles to
retain in memory when the `drill.exec.profiles.store.inmemory` option is
enabled. Default is 1000.
* **drill.exec.storage.action\_on\_plugins\_override\_file**
-Introduced in Drill 1.14. Determines what happens to the
[`storage-plugins-override.conf`]({{site.baseurl}}/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-override.conf-file)
file after Drill successfully updates storage plugin configurations. If you do
not want Drill to apply the configurations after restarting, set the option to
`"rename"` or `"remove"`. This option accepts the following values:
+ Introduced in Drill 1.14. Determines what happens to the
[`storage-plugins-override.conf`]({{site.baseurl}}/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-override.conf-file)
file after Drill successfully updates storage plugin configurations. If you do
not want Drill to apply the configurations after restarting, set the option to
`"rename"` or `"remove"`. This option accepts the following values:
- `"none"` - (Default) The file remains in the directory after Drill
uses the file.
- `"rename"` - The `storage-plugins-override.conf` file name is changed
to `storage-plugins-override-[current_timestamp].conf` after Drill uses the
file.
diff --git
a/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
b/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
index 965d69d..b43c2a9 100644
---
a/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
+++
b/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
@@ -1,15 +1,18 @@
---
title: "Configuring Custom ACLs to Secure znodes"
-date: 2018-12-08
+date: 2018-12-28
parent: "Securing Drill"
---
-Drill uses ZooKeeper for dynamic service discovery in a cluster; ZooKeeper
uses the Curator framework and Service Discovery recipe to discover services.
In addition to discovering services, Drill uses ZooKeeper to store certain
cluster-level configuration and query profile information in znodes. A znode is
an internal data tree in ZooKeeper that stores coordination- and
execution-related information. Each time a Drillbit starts up and establish a
new session with Zookeeper (using the Cur [...]
+Drill uses the Curator Framework to interact with ZooKeeper to discover
services in a cluster. In addition to discovering services, Drill uses
ZooKeeper to store certain cluster-level configuration and query profile
information in znodes. A znode is an internal data tree in ZooKeeper that
stores coordination- and execution-related information. If information in the
znodes is not properly secured, cluster privacy and/or security is compromised.
-ZooKeeper uses ACLs (access control lists) to control access to znodes and
secure the information they store. ACLs specify sets of ids and permissions
that are associated with the ids. The ZooKeeper ACLs are set such that only the
Drillbit process user can access (create, delete, read, write, administer) all
the ZooKeeper nodes in a Drill cluster, except for the service discovery
znodes. When a Drillbit shuts-down, the ZooKeeper session ends and the znode is
removed.
+Drill allows users to create a custom ACL (access control list) on the znodes
to secure data. ACLs specify sets of ids and permissions that are associated
with the ids. ZooKeeper uses ACLs to control access to znodes and secure the
information they store.
-Prior to Drill 1.15, ZooKeeper ACLs in secure and unsecure clusters were set
to [world:all], meaning that all users had create, delete, read, write, and
administrator access to the zknodes. Starting in Drill 1.15, ACLs in unsecure
clusters are set to [world:all]. ACLs in secure clusters are set to [authid:
all], which provides only the authenticated user that created the znode with
full access. Discovery znodes (znodes with the list of Drillbits) have an
additional ACL set to [world:read [...]
+Prior to Drill 1.15, ZooKeeper ACLs in secure and unsecure clusters were set
to [world:all], meaning that all users had create, delete, read, write, and
administrator access to the zknodes. Starting in Drill 1.15, ACLs in unsecure
clusters are set to [world:all]. ACLs in secure clusters are set to [authid:
all], which provides only the authenticated user that created the znode with
full access. Discovery znodes (znodes with the list of Drillbits) have an
additional ACL set to [world:read [...]
+- View the
[drill-override-example.conf](https://github.com/apache/drill/blob/master/distribution/src/resources/drill-override-example.conf)
file to see example ACL configurations.
+
+
##Securing znodes
Complete the following steps to create a custom ACL and secure znodes:
@@ -22,7 +25,19 @@ Complete the following steps to create a custom ACL and
secure znodes:
4. In `$DRILL_HOME/conf/drill-override.conf`, set `zk.acl_provider` to the
`ZKACLProviderTemplate` type.
5. Restart Drill.
-When you restart Drill, the ACL, as mentioned in your custom class, is applied
to the znode created when Drill starts.
+When you restart Drill, the ACL, as mentioned in your custom class, is applied
to the znode created when Drill starts.
+
+***
+**NOTE**
+Existing ACLs for persistent znodes will not be affected if a Drillbit is
restarted with a different ACL setting. ACLs are applied only at znode creation
time, and Drill does not recreate any znode that is already present. If you
want to change an ACL for existing znodes, connect to the ZooKeeper server
using zkCli and then use option a or b, as described:
+
+- a) Shutdown Drillbits, delete the persistent znodes, change the ACL settings
and then restart the Drillbit
+
+- b) Manually change the ACLs on the existing znodes to reflect the new ACL
settings, using the setAcl command in the zkCli.
+
+For either option to work, an authenticated connection between the zkCli and
ZooKeeper Server must be established.
+
+***
For additional information, refer to:
diff --git
a/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
b/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
index 177b493..708d08e 100644
---
a/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
+++
b/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
@@ -1,6 +1,6 @@
---
title: "Date/Time Functions and Arithmetic"
-date: 2018-12-27
+date: 2018-12-28
parent: "SQL Functions"
---
@@ -570,63 +570,65 @@ SELECT UNIX_TIMESTAMP('2015-05-29 08:18:53.0',
'yyyy-MM-dd HH:mm:ss.SSS') FROM (
Adds an interval of time, in the given time units, to the date expression.
###TIMESTAMPADD Syntax
-TIMESTAMPADD(*time\_unit,interval,date\_expression*)
+TIMESTAMPADD(*time\_unit,interval, keyword date\_expression*)
###TIMESTAMPADD Usage Notes
-- A date expressions is in the format YYYY-MM-DD.
-- Supports the following time units: Day, Week, Month, Quarter, Year
+- *Keyword* is the type of *date\_expression*: date, time, or timestamp
+- Supports date, time, and timestamp values in the following formats:
+ - Date format: YYYY-MM-DD
+ - Time format: HH:MI:SS
+ - Timestamp format: YYYY-MM-DD HH:MI:SS
+- Supports the following time units: Nanosecond, Microsecond, Second, Minute,
Hour, Day, Month, Year, Week, Quarter
- Drill uses the unit of time to infer the return type.
- You can include the SQL_TSI_ prefix with the any of the supported time
units, as shown:
- SELECT TIMESTAMPADD(SQL_TSI_YEAR,3,'1982-05-06');
+ SELECT TIMESTAMPADD(SQL_TSI_MINUTE,5,TIME '05:05:05');
+ +-----------+
+ | EXPR$0 |
+ +-----------+
+ | 05:10:05 |
+ +-----------+
###TIMESTAMPADD Examples
Add three years to the given date:
- SELECT TIMESTAMPADD(YEAR,3,'1982-05-06');
+ SELECT TIMESTAMPADD(YEAR,3,DATE '1982-05-06');
+------------------------+
| EXPR$0 |
+------------------------+
| 1985-05-06 00:00:00.0 |
+------------------------+
-
-Add one quarter of a year (3 months) to the given date:
-
- SELECT TIMESTAMPADD(QUARTER,1,'1982-05-06');
- +------------------------+
- | EXPR$0 |
- +------------------------+
- | 1982-08-06 00:00:00.0 |
- +------------------------+
-Add three months to the given date:
+Add one quarter of a year (3 months) to the given date:
- SELECT TIMESTAMPADD(MONTH,3,'1982-05-06');
+ SELECT TIMESTAMPADD(QUARTER,1,DATE'1982-05-06');
+------------------------+
| EXPR$0 |
+------------------------+
| 1982-08-06 00:00:00.0 |
- +------------------------+
+ +------------------------+
+
+Add 225 seconds to the given time:
-Add three weeks to the given date:
+ SELECT TIMESTAMPADD(SECOND,225,TIME '02:02:02');
+ +-----------+
+ | EXPR$0 |
+ +-----------+
+ | 02:05:47 |
+ +-----------+
- SELECT TIMESTAMPADD(WEEK,3,'1982-05-06');
- +------------------------+
- | EXPR$0 |
- +------------------------+
- | 1982-05-27 00:00:00.0 |
- +------------------------+
+Add 5555500000 microseconds to the given timestamp value:
+
+SELECT TIMESTAMPADD(MICROSECOND,5555500000, TIMESTAMP '2003-02-01 12:05:35');
++--------------------------+
+| EXPR$0 |
++--------------------------+
+| 2003-02-01 12:26:35.532 |
++--------------------------+
-Add three days to the given date:
- SELECT TIMESTAMPADD(DAY,3,'1982-05-06');
- +------------------------+
- | EXPR$0 |
- +------------------------+
- | 1982-05-09 00:00:00.0 |
- +------------------------+
##TIMESTAMPDIFF
Calculates an interval of time, in the given time units, by subtracting
*datetime\_expression1* from *datetime\_expression2* (*datetime\_expression2* −
*datetime\_expression1*).
