Repository: drill-site Updated Branches: refs/heads/asf-site b38c239af -> e38c2bba1
drill 1.13 doc updates and edits Project: http://git-wip-us.apache.org/repos/asf/drill-site/repo Commit: http://git-wip-us.apache.org/repos/asf/drill-site/commit/e38c2bba Tree: http://git-wip-us.apache.org/repos/asf/drill-site/tree/e38c2bba Diff: http://git-wip-us.apache.org/repos/asf/drill-site/diff/e38c2bba Branch: refs/heads/asf-site Commit: e38c2bba14d34a6d2425479211e8552082fdae7d Parents: b38c239 Author: Bridget Bevens <[email protected]> Authored: Mon Mar 26 11:05:14 2018 -0700 Committer: Bridget Bevens <[email protected]> Committed: Mon Mar 26 11:05:14 2018 -0700 ---------------------------------------------------------------------- docs/configuring-drill-memory/index.html | 6 +-- docs/data-type-conversion/index.html | 21 ++++---- docs/parquet-filter-pushdown/index.html | 53 ++++++++++++--------- docs/partition-pruning-introduction/index.html | 11 ++++- docs/supported-data-types/index.html | 4 +- feed.xml | 4 +- 6 files changed, 60 insertions(+), 39 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/configuring-drill-memory/index.html ---------------------------------------------------------------------- diff --git a/docs/configuring-drill-memory/index.html b/docs/configuring-drill-memory/index.html index 476d195..10d18d9 100644 --- a/docs/configuring-drill-memory/index.html +++ b/docs/configuring-drill-memory/index.html @@ -1216,7 +1216,7 @@ </div> - Mar 22, 2018 + Mar 26, 2018 <link href="/css/docpage.css" rel="stylesheet" type="text/css"> @@ -1279,7 +1279,7 @@ export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"<limit>&qu <p><strong>Example</strong> </p> -<p>If a system has 48GB of free memory and you set the following parameters in drill-env.sh: </p> +<p>If a system has 48GB of memory and you set the following parameters in drill-env.sh: </p> <div class="highlight"><pre><code class="language-text" data-lang="text"> DRILLBIT_MAX_PROC_MEM=25% DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=10G @@ -1290,7 +1290,7 @@ export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"<limit>&qu [ERROR] Unable to start Drillbit due to memory constraint violations Total Memory Requested : 19 GB Check and modify the settings or increase the maximum amount of memory permitted. </code></pre></div> -<p>If DRILLBIT_MAX_PROC_MEM is increased to 50%; the Drillbit starts up with the following warnings: </p> +<p>If <code>DRILLBIT_MAX_PROC_MEM</code> is increased to 50%; the Drillbit starts up with the following warnings: </p> <div class="highlight"><pre><code class="language-text" data-lang="text"> [WARN] 50% of the system memory (48 GB) translates to 24 GB. [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory. http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/data-type-conversion/index.html ---------------------------------------------------------------------- diff --git a/docs/data-type-conversion/index.html b/docs/data-type-conversion/index.html index ea61807..7685349 100644 --- a/docs/data-type-conversion/index.html +++ b/docs/data-type-conversion/index.html @@ -1218,7 +1218,7 @@ </div> - + Mar 26, 2018 <link href="/css/docpage.css" rel="stylesheet" type="text/css"> @@ -1377,13 +1377,18 @@ CONVERT_FROM(column, type) <h3 id="convert_to-and-convert_from-usage-notes">CONVERT_TO and CONVERT_FROM Usage Notes</h3> -<p>CONVERT_FROM and CONVERT_TO methods transform a known binary representation/encoding to a Drill internal format. Use CONVERT_TO and CONVERT_FROM instead of the CAST function for converting binary data types. CONVERT_TO/FROM functions work for data in a binary representation and are more efficient to use than CAST. </p> - -<p>Drill can optimize scans of HBase tables when you use the *_BE encoded types shown in section <a href="/docs/supported-data-types/#data-types-for-convert_to-and-convert_from-functions">"Data Types for CONVERT_TO and CONVERT_FROM Functions"</a> on big endian-encoded data. You need to use the HBase storage plugin and query data as described in <a href="/docs/querying-hbase">"Querying Hbase"</a>. To write Parquet binary data, convert SQL data <em>to</em> binary data and store the data in a Parquet table while creating a table as a selection (CTAS).</p> - -<p>CONVERT_TO also converts an SQL data type to complex types, including HBase byte arrays, JSON and Parquet arrays, and maps. CONVERT_FROM converts from complex types, including HBase arrays, JSON and Parquet arrays and maps to an SQL data type. </p> - -<p>You can use <a href="/docs/data-type-conversion/#string_binary-function">STRING_BINARY</a> and <a href="/docs/data-type-conversion/#binary_string-function">BINARY_STRING</a> custom Drill functions with CONVERT_TO and CONVERT_FROM to get meaningful results.</p> +<ul> +<li><p>CONVERT_FROM and CONVERT_TO methods transform a known binary representation/encoding to a Drill internal format. Use CONVERT_TO and CONVERT_FROM instead of the CAST function for converting binary data types. CONVERT_TO/FROM functions work for data in a binary representation and are more efficient to use than CAST. </p></li> +<li><p>Drill can optimize scans on HBase tables when you use the *_BE encoded types shown in section <a href="/docs/supported-data-types/#data-types-for-convert_to-and-convert_from-functions">"Data Types for CONVERT_TO and CONVERT_FROM Functions"</a> on big endian-encoded data. You need to use the HBase storage plugin and query data as described in <a href="/docs/querying-hbase">"Querying Hbase"</a>. To write Parquet binary data, convert SQL data <em>to</em> binary data and store the data in a Parquet table while creating a table as a selection (CTAS). </p></li> +<li><p>CONVERT_TO also converts an SQL data type to complex types, including HBase byte arrays, JSON and Parquet arrays, and maps. CONVERT_FROM converts from complex types, including HBase arrays, JSON and Parquet arrays and maps to an SQL data type. </p></li> +<li><p>You can use <a href="/docs/data-type-conversion/#string_binary-function">STRING_BINARY</a> and <a href="/docs/data-type-conversion/#binary_string-function">BINARY_STRING</a> custom Drill functions with CONVERT_TO and CONVERT_FROM to get meaningful results. </p></li> +<li><p>Drill 1.13 and later supports <a href="/docs/json-data-model/">NaN and Infinity values as numeric data types</a>. +You can use the convert_to and convert_from functions in queries on JSON data with NaN and Infinity values, as shown in the following query examples: </p> +<div class="highlight"><pre><code class="language-text" data-lang="text"> select convert_fromJSON('{"num": 55, "nan": NaN, "inf": -Infinity}'); + select convert_fromJSON(jsonColumn) from mysql.someTable; + select string_binary(convert_toJSON(convert_fromJSON(jsonColumn) from mysql.someTable; +</code></pre></div></li> +</ul> <h3 id="conversion-of-data-types-examples">Conversion of Data Types Examples</h3> http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/parquet-filter-pushdown/index.html ---------------------------------------------------------------------- diff --git a/docs/parquet-filter-pushdown/index.html b/docs/parquet-filter-pushdown/index.html index fdde36b..5f69651 100644 --- a/docs/parquet-filter-pushdown/index.html +++ b/docs/parquet-filter-pushdown/index.html @@ -1216,7 +1216,7 @@ </div> - Nov 21, 2016 + Mar 26, 2018 <link href="/css/docpage.css" rel="stylesheet" type="text/css"> @@ -1226,6 +1226,13 @@ <p>Parquet filter pushdown is enabled by default. When a query contains a filter expression, you can run the <a href="/docs/explain-commands/">EXPLAIN PLAN command</a> to see if Drill applies Parquet filter pushdown to the query. You can enable and disable this feature using the <a href="/docs/alter-system/">ALTER SYSTEM|SESSION SET</a> command with the <code>planner.store.parquet.rowgroup.filter.pushdown</code> option. </p> +<p>As of Drill 1.13, the query planner in Drill can apply project push down, filter push down, and partition pruning to star queries in common table expressions (CTEs), views, and subqueries, for example: </p> +<div class="highlight"><pre><code class="language-text" data-lang="text"> select col1 from (select * from t) +</code></pre></div> +<p>When a CTE, view, or subquery contains a star filter condition, the query planner in Drill can apply the filter and prune extraneous data, further reducing the amount of data that the scanner reads and improving performance. </p> + +<p><strong>Note:</strong> Currently, Drill only supports pushdown for simple star subselect queries without filters. See <a href="https://www.google.com/url?q=https://issues.apache.org/jira/browse/DRILL-6219&sa=D&ust=1522084453671000&usg=AFQjCNFXp-nWMRXzM466BSRFlV3F63_ZYA";>DRILL-6219</a> for more information. </p> + <h2 id="how-parquet-filter-pushdown-works">How Parquet Filter Pushdown Works</h2> <p>Drill applies Parquet filter pushdown during the query planning phase. The query planner in Drill performs Parquet filter pushdown by evaluating the filter expressions in the query. If no filter expression exists, the underlying scan operator reads all of the data in a Parquet file and then sends the data to operators downstream. When filter expressions exist, the planner applies each filter and prunes the data, reducing the amount of data that the scanner and Parquet reader must read.</p> @@ -1273,44 +1280,46 @@ <h2 id="support">Support</h2> -<p>The following table lists the supported and unsupported clauses, operators, data types, and scenarios for Parquet filter pushdown: </p> +<p>The following table lists the supported and unsupported clauses, operators, data types, function, and scenarios for Parquet filter pushdown: </p> + +<p><strong>Note:</strong> An asterisk (*) indicates support as of Drill 1.13</p> <table><thead> <tr> <th></th> -<th>Supported</th> -<th>Not Supported</th> +<th><strong>Supported</strong></th> +<th><strong>Not Supported</strong></th> </tr> </thead><tbody> <tr> -<td>Clauses</td> -<td>WHERE, HAVING (HAVING is supported if Drill can pass the filter through GROUP BY.)</td> -<td></td> +<td><strong>Clauses</strong></td> +<td>WHERE, *WITH, HAVING (HAVING is supported if Drill can pass the filter through GROUP BY.)</td> +<td>--</td> </tr> <tr> -<td>Operators</td> -<td>AND, OR, IN (An IN list is converted to OR if the number in the IN list is within a certain threshold, for example 20. If greater than the threshold, pruning cannot occur.)</td> -<td>NOT, ITEM (Drill does not push the filter past the ITEM operator, which is used for complex fields.)</td> +<td><strong>Operators</strong></td> +<td>AND, OR, NOT, *IS [NOT] NULL, *IS [NOT] TRUE</td> +<td>FALSE, IN (An IN list is converted to OR if the number in the IN list is within a certain threshold, for example 20. If greater than the threshold, pruning cannot occur.)</td> </tr> <tr> -<td>Comparison Operators</td> -<td><>, <, >, <=, >=, =</td> -<td>IS [NOT] NULL</td> +<td>Comparison Operators</td> +<td><>, <, >, <=, >=, =</td> +<td>--</td> </tr> <tr> -<td>Data Types</td> -<td>INT, BIGINT, FLOAT, DOUBLE, DATE, TIMESTAMP, TIME</td> -<td>CHAR, VARCHAR columns, Hive TIMESTAMP</td> +<td><strong>Data Types</strong></td> +<td>INT, BIGINT, FLOAT, DOUBLE, DATE, TIMESTAMP, TIME, *BOOLEAN (true</td> +<td>false)</td> </tr> <tr> -<td>Function</td> -<td>CAST is supported among these four numeric types only: int, bigint, float, double</td> -<td></td> +<td><strong>Function</strong></td> +<td>CAST is supported among the following types only: int, bigint, float, double, *date, *timestamp, and *time</td> +<td>--</td> </tr> <tr> -<td>Other</td> -<td>--</td> -<td>Joins, Files with multiple row groups, Enabled Native Hive reader</td> +<td><strong>Other</strong></td> +<td>Files with multiple row groups</td> +<td>Joins, Enabled Native Hive reader</td> </tr> </tbody></table> http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/partition-pruning-introduction/index.html ---------------------------------------------------------------------- diff --git a/docs/partition-pruning-introduction/index.html b/docs/partition-pruning-introduction/index.html index c881e68..4b8c110 100644 --- a/docs/partition-pruning-introduction/index.html +++ b/docs/partition-pruning-introduction/index.html @@ -1218,7 +1218,7 @@ </div> - Aug 15, 2016 + Mar 26, 2018 <link href="/css/docpage.css" rel="stylesheet" type="text/css"> @@ -1228,7 +1228,14 @@ <p>As of Drill 1.8, partition pruning also applies to the Parquet metadata cache. When data is partitioned in a directory hierarchy, Drill attempts to read the metadata cache file from a sub-partition, based on matching filter criteria instead of reading from the top level partition, to reduce the amount of metadata read during the query planning time. If you created a metadata cache file in a previous version of Drill, you must issue the REFRESH TABLE METADATA command to regenerate the metadata cache file before running queries for metadata cache pruning to occur. See <a href="/docs/optimizing-parquet-metadata-reading/">Optimizing Parquet Metadata Reading</a> for more information. </p> -<p>The query planner in Drill performs partition pruning by evaluating the filters. If no partition filters are present, the underlying Scan operator reads all files in all directories and then sends the data to operators, such as Filter, downstream. When partition filters are present, the query planner pushes the filters down to the Scan if possible. The Scan reads only the directories that match the partition filters, thus reducing disk I/O.</p> +<p>The query planner in Drill performs partition pruning by evaluating the filters. If no partition filters are present, the underlying Scan operator reads all files in all directories and then sends the data to operators, such as Filter, downstream. When partition filters are present, the query planner pushes the filters down to the Scan if possible. The Scan reads only the directories that match the partition filters, thus reducing disk I/O. </p> + +<p>As of Drill 1.13, the query planner in Drill can apply project push down, filter push down, and partition pruning to star queries in common table expressions (CTEs), views, and subqueries, for example: </p> +<div class="highlight"><pre><code class="language-text" data-lang="text"> select col1 from (select * from t) +</code></pre></div> +<p>When a CTE, view, or subquery contains a star filter condition, the query planner in Drill can apply the filter and prune extraneous data, further reducing the amount of data that the scanner reads and improving performance. </p> + +<p><strong>Note:</strong> Currently, Drill only supports pushdown for simple star subselect queries without filters. See <a href="https://www.google.com/url?q=https://issues.apache.org/jira/browse/DRILL-6219&sa=D&ust=1522084453671000&usg=AFQjCNFXp-nWMRXzM466BSRFlV3F63_ZYA";>DRILL-6219</a> for more information.</p> <h2 id="using-partitioned-drill-data">Using Partitioned Drill Data</h2> http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/docs/supported-data-types/index.html ---------------------------------------------------------------------- diff --git a/docs/supported-data-types/index.html b/docs/supported-data-types/index.html index af064cf..f31a164 100644 --- a/docs/supported-data-types/index.html +++ b/docs/supported-data-types/index.html @@ -1218,7 +1218,7 @@ </div> - Apr 5, 2017 + Mar 26, 2018 <link href="/css/docpage.css" rel="stylesheet" type="text/css"> @@ -1760,7 +1760,7 @@ Converts a string to TIMESTAMP.</li> <p>The <a href="/docs/data-type-conversion/#convert_to-and-convert_from">CONVERT_TO function</a> converts data to bytes from the input type. The <a href="/docs/data-type-conversion/#convert_to-and-convert_from">CONVERT_FROM function</a> converts data from bytes to the input type. For example, the following CONVERT_TO function converts an integer to bytes using big endian encoding:</p> <div class="highlight"><pre><code class="language-text" data-lang="text">CONVERT_TO(mycolumn, 'INT_BE') </code></pre></div> -<p>The following table lists the data types for use with the CONVERT_TO +<p>The following table lists the data types you can use with the CONVERT_TO and CONVERT_FROM functions:</p> <table><thead> http://git-wip-us.apache.org/repos/asf/drill-site/blob/e38c2bba/feed.xml ---------------------------------------------------------------------- diff --git a/feed.xml b/feed.xml index 6e56d6d..83cfc33 100644 --- a/feed.xml +++ b/feed.xml @@ -6,8 +6,8 @@ </description> <link>/</link> <atom:link href="/feed.xml" rel="self" type="application/rss+xml"/> - <pubDate>Thu, 22 Mar 2018 11:56:49 -0700</pubDate> - <lastBuildDate>Thu, 22 Mar 2018 11:56:49 -0700</lastBuildDate> + <pubDate>Mon, 26 Mar 2018 11:02:52 -0700</pubDate> + <lastBuildDate>Mon, 26 Mar 2018 11:02:52 -0700</lastBuildDate> <generator>Jekyll v2.5.2</generator> <item>
