This is an automated email from the ASF dual-hosted git repository.
bridgetb pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/drill-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cc74e50 edits
cc74e50 is described below
commit cc74e5055233476bfb4445b4256b74285ab2095d
Author: Bridget Bevens <[email protected]>
AuthorDate: Wed Sep 5 18:50:36 2018 -0700
edits
---
.../index.html | 22 ++++++++++------------
feed.xml | 4 ++--
2 files changed, 12 insertions(+), 14 deletions(-)
diff --git
a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
index 59cb98b..19c5885 100644
--- a/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
+++ b/docs/sort-based-and-hash-based-memory-constrained-operators/index.html
@@ -1285,11 +1285,9 @@
</ul>
<p>Drill only uses the External Sort operator to sort data. Drill uses the
Hash-Aggregate operator to aggregate data. Alternatively, Drill can sort the
data and then use the (lightweight) Streaming-Aggregate operator to aggregate
data.
-Drill uses the Hash-Join operator to join data. Alternatively, Drill can use
the Nested-Loop-Join or sort the data and then use the (lightweight)
Merge-Join. Drill typically uses Hash operators for joining and aggregation, as
they perform better than the Sort operator (Hash - O(N) vs. Sort - O(N *
log(N))). However, if the Hash operators are disabled, or the data is already
sorted, Drill uses the alternative methods previously described.</p>
+Drill uses the Hash-Join operator to join data. Alternatively, Drill can use
the Nested-Loop-Join or sort the data and then use the (lightweight)
Merge-Join. Drill typically uses Hash operators for joining and aggregation, as
they perform better than the Sort operator (Hash - O(N) vs. Sort - O(N *
log(N))). However, if you disable the Hash operators, or the data is already
sorted, Drill uses the alternative methods previously described.</p>
-<p>The memory configuration in Drill is specified as the memory limit
per-query, per-node. The allocated memory is equally divided among all
instances of the spillable operators (per query on each node). The number of
instances is the number of spillable operators in the query plan multiplied by
the maximal degree of parallelism. The maximal degree of parallelism is the
number of minor fragments required to perform the work for each instance of a
spillable operator. When an instance of a [...]
-
-<p>To see the difference in memory consumption between the operators, run a
query and then view the query profile in the Drill Web UI. Optionally, you can
disable the Hash operators, which forces Drill to use the Merge-Join and
Streaming-Aggregate operators. </p>
+<p>The memory configuration in Drill is specified as the memory limit
per-query, per-node. The allocated memory is equally divided among all
instances of the spillable operators (per query on each node). The number of
instances is the number of spillable operators in the query plan multiplied by
the maximal degree of parallelism. The maximal degree of parallelism is the
number of minor fragments required to perform the work for each instance of a
spillable operator. When an instance of a [...]
<h2 id="spill-to-disk">Spill to Disk</h2>
@@ -1305,7 +1303,7 @@ Drill uses the Hash-Join operator to join data.
Alternatively, Drill can use the
<p>Spillable operators write data to a temporary work area on disk when they
cannot process all of the data in memory. The default location of the temporary
work area is <code>/tmp/drill/spill</code> on the local file system. </p>
-<p>The <code>/tmp/drill/spill</code> directory should suffice for small
workloads or examples, however it is highly recommended that you redirect the
default spill location to a location with enough disk space to support spilling
for large workloads.</p>
+<p>The <code>/tmp/drill/spill</code> directory should suffice for small
workloads or examples; however, you should redirect the default spill location
to a location with enough disk space to support spilling for large
workloads.</p>
<p><strong>Note:</strong> Spilled data may require more space than the table
referenced in the query that is spilling the data. For example, when the
underlying table is compressed (Parquet), or when the operator received data
joined from multiple tables.</p>
@@ -1313,15 +1311,15 @@ Drill uses the Hash-Join operator to join data.
Alternatively, Drill can use the
<p><strong>Configuring Spill to Disk</strong> </p>
-<p>The <code>drill-override.conf</code> file, located in the
<code>/conf</code> directory, contains options that set the spill locations for
the Hash and Sort operators. An administrator can change the file system and
directories into which the operators spill data. Refer to the
<code>drill-override-example.conf</code> file included in the
<code>/conf</code> directory for examples. </p>
+<p>The <code>drill-override.conf</code> file, located in the
<code>/conf</code> directory, contains options that set the spill locations for
the spillable operators. An administrator can change the file system and
directories into which the operators spill data. Refer to the
<code>drill-override-example.conf</code> file included in the
<code>/conf</code> directory for examples. </p>
<p>The following list describes the spill to disk configuration options: </p>
<ul>
-<li><strong>drill.exec.spill.fs</strong><br>
-Introduced in Drill 1.11. The default file system on the local machine into
which the Sort, Hash Aggregate, and Hash Join operators spill data. You can
configure this option so that data spills into a distributed file system, such
as hdfs. For example, "hdfs:///". The default setting is
"file:///".</li>
-<li><strong>drill.exec.spill.directories</strong><br>
-Introduced in Drill 1.11. The list of directories into which the Sort, Hash
Aggregate, and Hash Join operators spill data. The list must be an array with
directories separated by a comma, for example ["/fs1/drill/spill" ,
"/fs2/drill/spill" , "/fs3/drill/spill"]. The default
setting is ["/tmp/drill/spill"].<br></li>
+<li><p><strong>drill.exec.spill.fs</strong><br>
+Introduced in Drill 1.11. The default file system on the local machine into
which the spillable operators spill data. You can configure this option so that
data spills into a distributed file system, such as hdfs. For example,
"hdfs:///". The default setting is "file:///". </p></li>
+<li><p><strong>drill.exec.spill.directories</strong><br>
+Introduced in Drill 1.11. The list of directories into which the spillable
operators spill data. The list must be an array with directories separated by a
comma, for example ["/fs1/drill/spill" , "/fs2/drill/spill"
, "/fs3/drill/spill"]. The default setting is
["/tmp/drill/spill"]. </p></li>
</ul>
<p><strong>Note:</strong> The following options were available prior to Drill
1.11, but have since been deprecated and replaced with the options described
above: </p>
@@ -1334,7 +1332,7 @@ Introduced in Drill 1.11. The list of directories into
which the Sort, Hash Aggr
<h2 id="memory-allocation">Memory Allocation</h2>
-<p>Drill evenly splits the available memory among all instances of the
spillable operators. When a query is parallelized, the number of operators is
multiplied, which reduces the amount of memory given to each instance of the
operators during a query. </p>
+<p>Drill evenly splits the available memory among all instances of the
spillable operators. When a query is parallelized, the number of operators is
multiplied, which reduces the amount of memory given to each instance of the
operators during a query. To see the difference in memory consumption between
the operators, you can run a query and then view the query profile in the Drill
Web UI. Optionally, you can disable the Hash operators, which forces Drill to
use the Merge-Join and Streami [...]
<p><strong>Memory Allocation Configuration Options</strong> </p>
@@ -1344,7 +1342,7 @@ Introduced in Drill 1.11. The list of directories into
which the Sort, Hash Aggr
<li><p><strong>planner.memory.max_query_memory_per_node</strong><br>
The <code>planner.memory.max_query_memory_per_node</code> option is the
minimum amount of memory available to Drill per query on a node. The default of
2 GB typically allows between two and three concurrent queries to run when the
JVM is configured to use 8 GB of direct memory (default). When the memory
requirement for Drill increases, the default of 2 GB is constraining. You must
increase the amount of memory for queries to complete, unless the setting for
the <code>planner.memory.perce [...]
<li><p><strong>planner.memory.percent_per_query</strong><br>
-Alternatively, the <code>planner.memory.percent_per_query</code> option sets
the memory as a percentage of the total direct memory. The default is 5%. This
value is only used when throttling is disabled. Setting the value to 0 disables
the option. You can increase or decrease the value, however you should set the
percentage well below the JVM direct memory to account for the cases where
Drill does not manage memory, such as for the less memory intensive operators.
</p>
+Alternatively, the <code>planner.memory.percent_per_query</code> option sets
the memory as a percentage of the total direct memory. The default is 5%. This
value is only used when throttling is disabled. Setting the value to 0 disables
the option. You can increase or decrease the value; however, you should set the
percentage well below the JVM direct memory to account for the cases where
Drill does not manage memory, such as for the less memory intensive operators.
</p>
<ul>
<li><p>The percentage is calculated using the following formula: </p>
diff --git a/feed.xml b/feed.xml
index 3a87690..0e23b7d 100644
--- a/feed.xml
+++ b/feed.xml
@@ -6,8 +6,8 @@
</description>
<link>/</link>
<atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Wed, 05 Sep 2018 18:33:55 -0700</pubDate>
- <lastBuildDate>Wed, 05 Sep 2018 18:33:55 -0700</lastBuildDate>
+ <pubDate>Wed, 05 Sep 2018 18:48:40 -0700</pubDate>
+ <lastBuildDate>Wed, 05 Sep 2018 18:48:40 -0700</lastBuildDate>
<generator>Jekyll v2.5.2</generator>
<item>
