Author: jhyde
Date: Thu Nov 2 23:40:52 2017
New Revision: 1814123
URL: http://svn.apache.org/viewvc?rev=1814123&view=rev
Log:
[CALCITE-2021] Document the interfaces that you can use to extend Calcite
Modified:
calcite/site/docs/adapter.html
calcite/site/docs/tutorial.html
Modified: calcite/site/docs/adapter.html
URL:
http://svn.apache.org/viewvc/calcite/site/docs/adapter.html?rev=1814123&r1=1814122&r2=1814123&view=diff
==============================================================================
--- calcite/site/docs/adapter.html (original)
+++ calcite/site/docs/adapter.html Thu Nov 2 23:40:52 2017
@@ -227,7 +227,7 @@ as implemented by Avaticaâs
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#PARSER_FACTORY">parserFactory</a></td>
- <td style="text-align: left">Parser factory. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/sql/parser/SqlParserImplFactory.html">SqlParserImplFactory</a>
and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant.</td>
+ <td style="text-align: left">Parser factory. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/sql/parser/SqlParserImplFactory.html"><tt>interface
SqlParserImplFactory</tt></a> and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant.</td>
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#QUOTING">quoting</a></td>
@@ -243,7 +243,7 @@ as implemented by Avaticaâs
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SCHEMA_FACTORY">schemaFactory</a></td>
- <td style="text-align: left">Schema factory. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/schema/SchemaFactory.html">SchemaFactory</a>
and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant. Ignored if <code
class="highlighter-rouge">model</code> is specified.</td>
+ <td style="text-align: left">Schema factory. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/schema/SchemaFactory.html"><tt>interface
SchemaFactory</tt></a> and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant. Ignored if <code
class="highlighter-rouge">model</code> is specified.</td>
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SCHEMA_TYPE">schemaType</a></td>
@@ -259,7 +259,7 @@ as implemented by Avaticaâs
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#TYPE_SYSTEM">typeSystem</a></td>
- <td style="text-align: left">Type system. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/rel/type/RelDataTypeSystem.html">RelDataTypeSystem</a>
and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant.</td>
+ <td style="text-align: left">Type system. The name of a class that
implements <a
href="/apidocs/org/apache/calcite/rel/type/RelDataTypeSystem.html"><tt>interface
RelDataTypeSystem</tt></a> and has a public default constructor or an <code
class="highlighter-rouge">INSTANCE</code> constant.</td>
</tr>
<tr>
<td style="text-align: left"><a
href="/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#UNQUOTED_CASING">unquotedCasing</a></td>
@@ -300,6 +300,401 @@ For example,</p>
<p>Note how each key in the <code class="highlighter-rouge">operand</code>
section appears with a <code class="highlighter-rouge">schema.</code> prefix in
the connect string.</p>
+<h2 id="extensibility">Extensibility</h2>
+
+<p>There are many other APIs that allow you to extend Calciteâs
capabilities.</p>
+
+<p>In this section, we briefly describe those APIs, to give you an idea what is
+possible. To fully use these APIs you will need to read other documentation
+such as the javadoc for the interfaces, and possibly seek out the tests that
+we have written for them.</p>
+
+<h3 id="functions-and-operators">Functions and operators</h3>
+
+<p>There are several ways to add operators or functions to Calcite.
+Weâll describe the simplest (and least powerful) first.</p>
+
+<p><em>User-defined functions</em> are the simplest (but least powerful).
+They are straightforward to write (you just write a Java class and register it
+in your schema) but do not offer much flexibility in the number and type of
+arguments, resolving overloaded functions, or deriving the return type.</p>
+
+<p>It you want that flexibility, you probably need to write you a
+<em>user-defined operator</em>
+(see <a href="/apidocs/org/apache/calcite/sql/SqlOperator.html"><tt>interface
SqlOperator</tt></a>).</p>
+
+<p>If your operator does not adhere to standard SQL function syntax,
+â<code class="highlighter-rouge">f(arg1, arg2, ...)</code>â, then you need
to
+<a href="#extending-the-parser">extend the parser</a>.</p>
+
+<p>There are many good examples in the tests:
+<a
href="https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/UdfTest.java";><tt>class
UdfTest</tt></a>
+tests user-defined functions and user-defined aggregate functions.</p>
+
+<h3 id="aggregate-functions">Aggregate functions</h3>
+
+<p><em>User-defined aggregate functions</em> are similar to user-defined
functions,
+but each function has several corresponding Java methods, one for each
+stage in the life-cycle of an aggregate:</p>
+
+<ul>
+ <li><code class="highlighter-rouge">init</code> creates an accumulator;</li>
+ <li><code class="highlighter-rouge">add</code> adds one rowâs value to an
accumulator;</li>
+ <li><code class="highlighter-rouge">merge</code> combines two accumulators
into one;</li>
+ <li><code class="highlighter-rouge">result</code> finalizes an accumulator
and converts it to a result.</li>
+</ul>
+
+<p>For example, the methods (in pseudo-code) for <code
class="highlighter-rouge">SUM(int)</code> are as follows:</p>
+
+<figure class="highlight"><pre><code class="language-java"
data-lang="java"><span class="n">struct</span> <span
class="n">Accumulator</span> <span class="o">{</span>
+ <span class="kd">final</span> <span class="kt">int</span> <span
class="n">sum</span><span class="o">;</span>
+<span class="o">}</span>
+<span class="n">Accumulator</span> <span class="nf">init</span><span
class="p">(</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span
class="n">Accumulator</span><span class="o">(</span><span
class="mi">0</span><span class="o">);</span>
+<span class="o">}</span>
+<span class="n">Accumulator</span> <span class="nf">add</span><span
class="p">(</span><span class="n">Accumulator</span> <span
class="n">a</span><span class="o">,</span> <span class="kt">int</span> <span
class="n">x</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span
class="n">Accumulator</span><span class="o">(</span><span
class="n">a</span><span class="o">.</span><span class="na">sum</span> <span
class="o">+</span> <span class="n">x</span><span class="o">);</span>
+<span class="o">}</span>
+<span class="n">Accumulator</span> <span class="nf">merge</span><span
class="p">(</span><span class="n">Accumulator</span> <span
class="n">a</span><span class="o">,</span> <span class="n">Accumulator</span>
<span class="n">a2</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span
class="n">Accumulator</span><span class="o">(</span><span
class="n">a</span><span class="o">.</span><span class="na">sum</span> <span
class="o">+</span> <span class="n">a2</span><span class="o">.</span><span
class="na">sum</span><span class="o">);</span>
+<span class="o">}</span>
+<span class="kt">int</span> <span class="nf">result</span><span
class="p">(</span><span class="n">Accumulator</span> <span
class="n">a</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span
class="n">Accumulator</span><span class="o">(</span><span
class="n">a</span><span class="o">.</span><span class="na">sum</span> <span
class="o">+</span> <span class="n">x</span><span class="o">);</span>
+<span class="o">}</span></code></pre></figure>
+
+<p>Here is the sequence of calls to compute the sum of two rows with column
values 4 and 7:</p>
+
+<figure class="highlight"><pre><code class="language-java"
data-lang="java"><span class="n">a</span> <span class="o">=</span> <span
class="n">init</span><span class="o">()</span> <span class="err">#</span>
<span class="n">a</span> <span class="o">=</span> <span class="o">{</span><span
class="mi">0</span><span class="o">}</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">4</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">4</span><span class="o">}</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">7</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">11</span><span class="o">}</span>
+<span class="k">return</span> <span class="nf">result</span><span
class="p">(</span><span class="n">a</span><span class="o">)</span> <span
class="err">#</span> <span class="n">returns</span> <span
class="mi">11</span></code></pre></figure>
+
+<h3 id="window-functions">Window functions</h3>
+
+<p>A window function is similar to an aggregate function but it is applied to
a set
+of rows gathered by an <code class="highlighter-rouge">OVER</code> clause
rather than by a <code class="highlighter-rouge">GROUP BY</code> clause.
+Every aggregate function can be used as a window function, but there are some
+key differences. The rows seen by a window function may be ordered, and
+window functions that rely upon order (<code
class="highlighter-rouge">RANK</code>, for example) cannot be used as
+aggregate functions.</p>
+
+<p>Another difference is that windows are <em>non-disjoint</em>: a particular
row can
+appear in more than one window. For example, 10:37 appears in both the
+9:00-10:00 hour and also the 9:15-9:45 hour.</p>
+
+<p>Window functions are computed incrementally: when the clock ticks from
+10:14 to 10:15, two rows might enter the window and three rows leave.
+For this, window functions have have an extra life-cycle operation:</p>
+
+<ul>
+ <li><code class="highlighter-rouge">remove</code> removes a value from an
accumulator.</li>
+</ul>
+
+<p>It pseudo-code for <code class="highlighter-rouge">SUM(int)</code> would
be:</p>
+
+<figure class="highlight"><pre><code class="language-java"
data-lang="java"><span class="n">Accumulator</span> <span
class="nf">remove</span><span class="p">(</span><span
class="n">Accumulator</span> <span class="n">a</span><span class="o">,</span>
<span class="kt">int</span> <span class="n">x</span><span class="o">)</span>
<span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span
class="n">Accumulator</span><span class="o">(</span><span
class="n">a</span><span class="o">.</span><span class="na">sum</span> <span
class="o">-</span> <span class="n">x</span><span class="o">);</span>
+<span class="o">}</span></code></pre></figure>
+
+<p>Here is the sequence of calls to compute the moving sum,
+over the previous 2 rows, of 4 rows with values 4, 7, 2 and 3:</p>
+
+<figure class="highlight"><pre><code class="language-java"
data-lang="java"><span class="n">a</span> <span class="o">=</span> <span
class="n">init</span><span class="o">()</span> <span class="err">#</span>
<span class="n">a</span> <span class="o">=</span> <span class="o">{</span><span
class="mi">0</span><span class="o">}</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">4</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">4</span><span class="o">}</span>
+<span class="n">emit</span> <span class="nf">result</span><span
class="p">(</span><span class="n">a</span><span class="o">)</span> <span
class="err">#</span> <span class="n">emits</span> <span class="mi">4</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">7</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">11</span><span class="o">}</span>
+<span class="n">emit</span> <span class="nf">result</span><span
class="p">(</span><span class="n">a</span><span class="o">)</span> <span
class="err">#</span> <span class="n">emits</span> <span class="mi">11</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">remove</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">4</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">7</span><span class="o">}</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">2</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">9</span><span class="o">}</span>
+<span class="n">emit</span> <span class="nf">result</span><span
class="p">(</span><span class="n">a</span><span class="o">)</span> <span
class="err">#</span> <span class="n">emits</span> <span class="mi">9</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">remove</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">7</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">2</span><span class="o">}</span>
+<span class="n">a</span> <span class="o">=</span> <span
class="n">add</span><span class="o">(</span><span class="n">a</span><span
class="o">,</span> <span class="mi">3</span><span class="o">)</span> <span
class="err">#</span> <span class="n">a</span> <span class="o">=</span> <span
class="o">{</span><span class="mi">5</span><span class="o">}</span>
+<span class="n">emit</span> <span class="nf">result</span><span
class="p">(</span><span class="n">a</span><span class="o">)</span> <span
class="err">#</span> <span class="n">emits</span> <span
class="mi">5</span></code></pre></figure>
+
+<h3 id="grouped-window-functions">Grouped window functions</h3>
+
+<p>Grouped window functions are functions that operate the <code
class="highlighter-rouge">GROUP BY</code> clause
+to gather together records into sets. The built-in grouped window functions
+are <code class="highlighter-rouge">HOP</code>, <code
class="highlighter-rouge">TUMBLE</code> and <code
class="highlighter-rouge">SESSION</code>.
+You can define additional functions by implementing
+<a
href="/apidocs/org/apache/calcite/sql/fun/SqlGroupedWindowFunction.html"><tt>interface
SqlGroupedWindowFunction</tt></a>.</p>
+
+<h3 id="table-functions-and-table-macros">Table functions and table macros</h3>
+
+<p><em>User-defined table functions</em>
+are defined in a similar way to regular âscalarâ user-defined functions,
+but are used in the <code class="highlighter-rouge">FROM</code> clause of a
query. The following query uses a table
+function called <code class="highlighter-rouge">Ramp</code>:</p>
+
+<figure class="highlight"><pre><code class="language-sql"
data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span
class="k">FROM</span> <span class="k">TABLE</span><span class="p">(</span><span
class="n">Ramp</span><span class="p">(</span><span class="mi">3</span><span
class="p">,</span> <span class="mi">4</span><span
class="p">))</span></code></pre></figure>
+
+<p><em>User-defined table macros</em> use the same SQL syntax as table
functions,
+but are defined differently. Rather than generating data, they generate an
+relational expression.
+Table macros are invoked during query preparation and the relational expression
+they produce can then be optimized.
+(Calciteâs implementation of views uses table macros.)</p>
+
+<p><a
href="https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/TableFunctionTest.java";><tt>class
TableFunctionTest</tt></a>
+tests table functions and contains several useful examples.</p>
+
+<h3 id="extending-the-parser">Extending the parser</h3>
+
+<p>Suppose you need to extend Calciteâs SQL grammar in a way that will be
+compatible with future changes to the grammar. Making a copy of the grammar
file
+<code class="highlighter-rouge">Parser.jj</code> in your project would be
foolish, because the grammar is edited
+quite frequently.</p>
+
+<p>Fortunately, <code class="highlighter-rouge">Parser.jj</code> is actually an
+<a href="http://freemarker.apache.org/";>Apache FreeMarker</a>
+template that contains variables that can be substituted.
+The parser in <code class="highlighter-rouge">calcite-core</code> instantiates
the template with default values of
+the variables, typically empty, but you can override.
+If your project would like a different parser, you can provide your
+own <code class="highlighter-rouge">config.fmpp</code> and <code
class="highlighter-rouge">parserImpls.ftl</code> files and therefore generate an
+extended parser.</p>
+
+<p>The <code class="highlighter-rouge">calcite-server</code> module, which was
created in
+[<a href="https://issues.apache.org/jira/browse/CALCITE-707";>CALCITE-707</a>]
and
+adds DDL statements such as <code class="highlighter-rouge">CREATE
TABLE</code>, is an example that you could follow.
+Also see
+<a
href="https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/sql/parser/parserextensiontesting/ExtensionSqlParserTest.java";><tt>class
ExtensionSqlParserTest</tt></a>.</p>
+
+<h3 id="customizing-sql-dialect-accepted-and-generated">Customizing SQL
dialect accepted and generated</h3>
+
+<p>To customize what SQL extensions the parser should accept, implement
+<a
href="/apidocs/org/apache/calcite/sql/validate/SqlConformance.html"><tt>interface
SqlConformance</tt></a>
+or use one of the built-in values in
+<a
href="/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html"><tt>enum
SqlConformanceEnum</tt></a>.</p>
+
+<p>To control how SQL is generated for an external database (usually via the
JDBC
+adapter), use
+<a href="/apidocs/org/apache/calcite/sql/SqlDialect.html"><tt>class
SqlDialect</tt></a>.
+The dialect also describes the engineâs capabilities, such as whether it
+supports <code class="highlighter-rouge">OFFSET</code> and <code
class="highlighter-rouge">FETCH</code> clauses.</p>
+
+<h3 id="defining-a-custom-schema">Defining a custom schema</h3>
+
+<p>To define a custom schema, you need to implement
+<a href="/apidocs/org/apache/calcite/schema/SchemaFactory.html"><tt>interface
SchemaFactory</tt></a>.</p>
+
+<p>During query preparation, Calcite will call this interface to find out
+what tables and sub-schemas your schema contains. When a table in your schema
+is referenced in a query, Calcite will ask your schema to create an instance of
+<a href="/apidocs/org/apache/calcite/schema/Table.html"><tt>interface
Table</tt></a>.</p>
+
+<p>That table will be wrapped in a
+<a
href="/apidocs/org/apache/calcite/rel/core/TableScan.html"><tt>TableScan</tt></a>
+and will undergo the query optimization process.</p>
+
+<h3 id="reflective-schema">Reflective schema</h3>
+
+<p>A reflective schema
+(<a
href="/apidocs/org/apache/calcite/adapter/java/ReflectiveSchema.html"><tt>class
ReflectiveSchema</tt></a>)
+is a way of wrapping a Java object so that it appears
+as a schema. Its collection-valued fields will appear as tables.</p>
+
+<p>It is not a schema factory but an actual schema; you have to create the
object
+and wrap it in the schema by calling APIs.</p>
+
+<p>See
+<a
href="https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/ReflectiveSchemaTest.java";><tt>class
ReflectiveSchemaTest</tt></a>.</p>
+
+<h3 id="defining-a-custom-table">Defining a custom table</h3>
+
+<p>To define a custom table, you need to implement
+<a href="/apidocs/org/apache/calcite/schema/TableFactory.html"><tt>interface
TableFactory</tt></a>.
+Whereas a schema factory a set of named tables, a table factory produces a
+single table when bound to a schema with a particular name (and optionally a
+set of extra operands).</p>
+
+<h3 id="modifying-data">Modifying data</h3>
+
+<p>If your table is to support DML operations (INSERT, UPDATE, DELETE, MERGE),
+your implementation of <code class="highlighter-rouge">interface Table</code>
must implement
+<a
href="/apidocs/org/apache/calcite/schema/ModifiableTable.html"><tt>interface
ModifiableTable</tt></a>.</p>
+
+<h3 id="streaming">Streaming</h3>
+
+<p>If your table is to support streaming queries,
+your implementation of <code class="highlighter-rouge">interface Table</code>
must implement
+<a
href="/apidocs/org/apache/calcite/schema/StreamableTable.html"><tt>interface
StreamableTable</tt></a>.</p>
+
+<p>See
+<a
href="https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/StreamTest.java";><tt>class
StreamTest</tt></a>
+for examples.</p>
+
+<h3 id="pushing-operations-down-to-your-table">Pushing operations down to your
table</h3>
+
+<p>If you wish to push processing down to your custom tableâs source system,
+consider implementing either
+<a
href="/apidocs/org/apache/calcite/schema/FilterableTable.html"><tt>interface
FilterableTable</tt></a>
+or
+<a
href="/apidocs/org/apache/calcite/schema/ProjectableFilterableTable.html"><tt>interface
ProjectableFilterableTable</tt></a>.</p>
+
+<p>If you want more control, you should write a <a
href="#planner-rule">planner rule</a>.
+This will allow you to push down expressions, to make a cost-based decision
+about whether to push down processing, and push down more complex operations
+such as join, aggregation, and sort.</p>
+
+<h3 id="type-system">Type system</h3>
+
+<p>You can customize some aspects of the type system by implementing
+<a
href="/apidocs/org/apache/calcite/rel/type/RelDataTypeSystem.html"><tt>interface
RelDataTypeSystem</tt></a>.</p>
+
+<h3 id="relational-operators">Relational operators</h3>
+
+<p>All relational operators implement
+<a href="/apidocs/org/apache/calcite/rel/RelNode.html"><tt>interface
RelNode</tt></a>
+and most extend
+<a href="/apidocs/org/apache/calcite/rel/AbstractRelNode.html"><tt>class
AbstractRelNode</tt></a>.
+The core operators (used by
+<a
href="/apidocs/org/apache/calcite/sql2rel/SqlToRelConverter.html"><tt>SqlToRelConverter</tt></a>
+and covering conventional relational algebra) are
+<a
href="/apidocs/org/apache/calcite/rel/core/TableScan.html"><tt>TableScan</tt></a>,
+<a
href="/apidocs/org/apache/calcite/rel/core/TableModify.html"><tt>TableModify</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Values.html"><tt>Values</tt></a>,
+<a
href="/apidocs/org/apache/calcite/rel/core/Project.html"><tt>Project</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Filter.html"><tt>Filter</tt></a>,
+<a
href="/apidocs/org/apache/calcite/rel/core/Aggregate.html"><tt>Aggregate</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Join.html"><tt>Join</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Sort.html"><tt>Sort</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Union.html"><tt>Union</tt></a>,
+<a
href="/apidocs/org/apache/calcite/rel/core/Intersect.html"><tt>Intersect</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Minus.html"><tt>Minus</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Window.html"><tt>Window</tt></a>
and
+<a
href="/apidocs/org/apache/calcite/rel/core/Match.html"><tt>Match</tt></a>.</p>
+
+<p>Each of these has a âpureâ logical sub-class,
+<a
href="/apidocs/org/apache/calcite/rel/logical/LogicalProject.html"><tt>LogicalProject</tt></a>
+and so forth. Any given adapter will have counterparts for the operations that
+its engine can implement efficiently; for example, the Cassandra adapter has
+<a
href="/apidocs/org/apache/calcite/rel/cassandra/CassandraProject.html"><tt>CassandraProject</tt></a>
+but there is no <code class="highlighter-rouge">CassandraJoin</code>.</p>
+
+<p>You can define your own sub-class of <code
class="highlighter-rouge">RelNode</code> to add a new operator, or
+an implementation of an existing operator in a particular engine.</p>
+
+<p>To make an operator useful and powerful, you will need
+<a href="#planner-rule">planner rules</a> to combine it with existing
operators.
+(And also provide metadata, see <a href="#statistics-and-cost">below</a>).
+This being algebra, the effects are combinatorial: you write a few
+rules, but they combine to handle an exponential number of query patterns.</p>
+
+<p>If possible, make your operator a sub-class of an existing
+operator; then you may be able to re-use or adapt its rules.
+Even better, if your operator is a logical operation that you can rewrite
+(again, via a planner rule) in terms of existing operators, you should do that.
+You will be able to re-use the rules, metadata and implementations of those
+operators with no extra work.</p>
+
+<h3 id="planner-rule">Planner rule</h3>
+
+<p>A planner rule
+(<a href="/apidocs/org/apache/calcite/plan/RelOptRule.html"><tt>class
RelOptRule</tt></a>)
+transforms a relational expression into an equivalent relational
expression.</p>
+
+<p>A planner engine has many planner rules registered and fires them
+to transform the input query into something more efficient. Planner rules are
+therefore central to the optimization process, but surprisingly each planner
+rule does not concern itself with cost. The planner engine is responsible for
+firing rules in a sequence that produces an optimal plan, but each individual
+rules only concerns itself with correctness.</p>
+
+<p>Calcite has two built-in planner engines:
+<a
href="/apidocs/org/apache/calcite/plan/volcano/VolcanoPlanner.html"><tt>class
VolcanoPlanner</tt></a>
+uses dynamic programming and is good for exhaustive search, whereas
+<a href="/apidocs/org/apache/calcite/plan/hep/HepPlanner.html"><tt>class
HepPlanner</tt></a>
+fires a sequence of rules in a more fixed order.</p>
+
+<h3 id="calling-conventions">Calling conventions</h3>
+
+<p>A calling convention is a protocol used by a particular data engine.
+For example, the Cassandra engine has a collection of relational operators,
+<code class="highlighter-rouge">CassandraProject</code>, <code
class="highlighter-rouge">CassandraFilter</code> and so forth, and these
operators can be
+connected to each other without the data having to be converted from one format
+to another.</p>
+
+<p>If data needs to be converted from one calling convention to another,
Calcite
+uses a special sub-class of relational expression called a converter
+(see <a
href="/apidocs/org/apache/calcite/rel/convert/Converter.html"><tt>class
Converter</tt></a>).
+But of course converting data has a runtime cost.</p>
+
+<p>When planning a query that uses multiple engines, Calcite âcolorsâ
regions of
+the relational expression tree according to their calling convention. The
+planner pushes operations into data sources by firing rules. If the engine does
+not support a particular operation, the rule will not fire. Sometimes an
+operation can occur in more than one place, and ultimately the best plan is
+chosen according to cost.</p>
+
+<p>A calling convention is a class that implements
+<a href="/apidocs/org/apache/calcite/plan/Convention.html"><tt>interface
Convention</tt></a>,
+an auxiliary interface (for instance
+<a
href="/apidocs/org/apache/calcite/adapter/cassandra/CassandraRel.html"><tt>interface
CassandraRel</tt></a>),
+and a set of sub-classes of
+<a href="/apidocs/org/apache/calcite/rel/RelNode.html"><tt>class
RelNode</tt></a>
+that implement that interface for the core relational operators
+(<a
href="/apidocs/org/apache/calcite/rel/core/Project.html"><tt>Project</tt></a>,
+<a href="/apidocs/org/apache/calcite/rel/core/Filter.html"><tt>Filter</tt></a>,
+<a
href="/apidocs/org/apache/calcite/rel/core/Aggregate.html"><tt>Aggregate</tt></a>,
+and so forth).</p>
+
+<h3 id="built-in-sql-implementation">Built-in SQL implementation</h3>
+
+<p>How does Calcite implement SQL, if an adapter does not implement all of the
core
+relational operators?</p>
+
+<p>The answer is a particular built-in calling convention,
+<a
href="/apidocs/org/apache/calcite/adapter/EnumerableConvention.html"><tt>EnumerableConvention</tt></a>.
+Relational expressions of enumerable convention are implemented as
âbuilt-insâ:
+Calcite generates Java code, compiles it, and executes inside its own JVM.
+Enumerable convention is less efficient than, say, a distributed engine
+running over column-oriented data files, but it can implement all core
+relational operators and all built-in SQL functions and operators. If a data
+source cannot an implement a relational operator, enumerable convention is
+a fall-back.</p>
+
+<h3 id="statistics-and-cost">Statistics and cost</h3>
+
+<p>Calcite has a metadata system that allow you to define cost functions and
+statistics about relational operators, collectively referred to as
<em>metadata</em>.
+Each kind of metadata has an interface with (usually) one method.
+For example, selectivity is defined by
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdSelectivity.html"><tt>interface
RelMdSelectivity</tt></a>
+and the method
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMetadataQuery.html#getSelectivity-org.apache.calcite.rel.RelNode-org.apache.calcite.rex.RexNode-"><tt>getSelectivity(RelNode
rel, RexNode predicate)</tt></a>.</p>
+
+<p>There are many built-in kinds of metadata, including
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdCollation.html">collation</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdColumnOrigins.html">column
origins</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdColumnUniqueness.html">column
uniqueness</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.html">distinct
row count</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdDistribution.html">distribution</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdExplainVisibility.html">explain
visibility</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdExpressionLineage.html">expression
lineage</a>,
+<a href="/apidocs/org/apache/calcite/rel/metadata/RelMdMaxRowCount.html">max
row count</a>,
+<a href="/apidocs/org/apache/calcite/rel/metadata/RelMdNodeTypes.html">node
types</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdParallelism.html">parallelism</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdPercentageOriginalRows.html">percentage
original rows</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdPopulationSize.html">population
size</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdPredicates.html">predicates</a>,
+<a href="/apidocs/org/apache/calcite/rel/metadata/RelMdRowCount.html">row
count</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdSelectivity.html">selectivity</a>,
+<a href="/apidocs/org/apache/calcite/rel/metadata/RelMdSize.html">size</a>,
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdTableReferences.html">table
references</a>,
+<a href="/apidocs/org/apache/calcite/rel/metadata/RelMdUniqueKeys.html">unique
keys</a>, and
+<a
href="/apidocs/org/apache/calcite/rel/metadata/RelMdSelectivity.html">selectivity</a>;
+you can also define your own.</p>
+
+<p>You can then supply a <em>metadata provider</em> that computes that kind of
metadata
+for particular sub-classes of <code class="highlighter-rouge">RelNode</code>.
Metadata providers can handle built-in
+and extended metadata types, and built-in and extended <code
class="highlighter-rouge">RelNode</code> types.
+While preparing a query Calcite combines all of the applicable metadata
+providers and maintains a cache so that a given piece of metadata (for example
+the selectivity of the condition <code class="highlighter-rouge">x >
10</code> in a particular <code class="highlighter-rouge">Filter</code>
operator)
+is computed only once.</p>
+
+
Modified: calcite/site/docs/tutorial.html
URL:
http://svn.apache.org/viewvc/calcite/site/docs/tutorial.html?rev=1814123&r1=1814122&r2=1814123&view=diff
==============================================================================
--- calcite/site/docs/tutorial.html (original)
+++ calcite/site/docs/tutorial.html Thu Nov 2 23:40:52 2017
@@ -757,46 +757,9 @@ initial implementations.</p>
<h2 id="further-topics">Further topics</h2>
-<h3 id="defining-a-custom-schema">Defining a custom schema</h3>
+<p>There are many other ways to extend Calcite not yet described in this
tutorial.
+The <a href="adapter.html">adapter specification</a> describes the APIs
involved.</p>
-<p>(To be written.)</p>
-
-<h3 id="modifying-data">Modifying data</h3>
-
-<p>How to enable DML operations (INSERT, UPDATE and DELETE) on your schema.</p>
-
-<p>(To be written.)</p>
-
-<h3 id="calling-conventions">Calling conventions</h3>
-
-<p>(To be written.)</p>
-
-<h3 id="statistics-and-cost">Statistics and cost</h3>
-
-<p>(To be written.)</p>
-
-<h3 id="defining-and-using-user-defined-functions">Defining and using
user-defined functions</h3>
-
-<p>(To be written.)</p>
-
-<h3 id="defining-tables-in-a-schema">Defining tables in a schema</h3>
-
-<p>(To be written.)</p>
-
-<h3 id="defining-custom-tables">Defining custom tables</h3>
-
-<p>(To be written.)</p>
-
-<h3 id="built-in-sql-implementation">Built-in SQL implementation</h3>
-
-<p>How does Calcite implement SQL, if an adapter does not implement all of the
core
-relational operators?</p>
-
-<p>(To be written.)</p>
-
-<h3 id="table-functions">Table functions</h3>
-
-<p>(To be written.)</p>
