This is an automated email from the ASF dual-hosted git repository.
chenliang613 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/carbondata-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8fbbf17 reset english website
8fbbf17 is described below
commit 8fbbf17e56b8eeebe558260138cbce0237f463a0
Author: chenliang613 <[email protected]>
AuthorDate: Tue Jul 27 22:46:16 2021 +0800
reset english website
---
content/WEB-INF/classes/application.conf | 1 +
content/bloomfilter-datamap-guide.html | 392 ++++++++++++++++++++++++
content/datamap-developer-guide.html | 266 ++++++++++++++++
content/datamap-management.html | 394 ++++++++++++++++++++++++
content/lucene-datamap-guide.html | 389 ++++++++++++++++++++++++
content/preaggregate-datamap-guide.html | 502 +++++++++++++++++++++++++++++++
content/timeseries-datamap-guide.html | 369 +++++++++++++++++++++++
7 files changed, 2313 insertions(+)
diff --git a/content/WEB-INF/classes/application.conf
b/content/WEB-INF/classes/application.conf
index 970edf4..0cc31c3 100644
--- a/content/WEB-INF/classes/application.conf
+++ b/content/WEB-INF/classes/application.conf
@@ -29,6 +29,7 @@ fileList=["configuration-parameters",
"prestosql-guide",
"scd-and-cdc-guide",
"spatial-index-guide",
+ "clean-files",
]
diff --git a/content/bloomfilter-datamap-guide.html
b/content/bloomfilter-datamap-guide.html
new file mode 100644
index 0000000..19ee42a
--- /dev/null
+++ b/content/bloomfilter-datamap-guide.html
@@ -0,0 +1,392 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-bloomfilter-datamap" class="anchor"
href="#carbondata-bloomfilter-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData
BloomFilter DataMap</h1>
+<ul>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#bloomfilter-datamap-introduction">BloomFilter Datamap
Introduction</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#data-management-with-bloomfilter-datamap">Data
Management</a></li>
+<li><a href="#useful-tips">Useful Tips</a></li>
+</ul>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Management</h4>
+<p>Creating BloomFilter DataMap</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING 'bloomfilter'
+DMPROPERTIES ('index_columns'='city, name', 'BLOOM_SIZE'='640000',
'BLOOM_FPP'='0.00001')
+</code></pre>
+<p>Dropping specified datamap</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>Showing all DataMaps on this table</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>Disable Datamap</p>
+<blockquote>
+<p>The datamap by default is enabled. To support tuning on query, we can
disable a specific datamap during query to observe whether we can gain
performance enhancement from it. This is effective only for current session.</p>
+</blockquote>
+<pre><code>// disable the datamap
+SET carbon.datamap.visible.dbName.tableName.dataMapName = false
+// enable the datamap
+SET carbon.datamap.visible.dbName.tableName.dataMapName = true
+</code></pre>
+<h2>
+<a id="bloomfilter-datamap-introduction" class="anchor"
href="#bloomfilter-datamap-introduction" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>BloomFilter DataMap
Introduction</h2>
+<p>A Bloom filter is a space-efficient probabilistic data structure that is
used to test whether an element is a member of a set.
+Carbondata introduced BloomFilter as an index datamap to enhance the
performance of querying with precise value.
+It is well suitable for queries that do precise match on high cardinality
columns(such as Name/ID).
+Internally, CarbonData maintains a BloomFilter per blocklet for each index
column to indicate that whether a value of the column is in this blocklet.
+Just like the other datamaps, BloomFilter datamap is managed along with main
tables by CarbonData.
+User can create BloomFilter datamap on specified columns with specified
BloomFilter configurations such as size and probability.</p>
+<p>For instance, main table called <strong>datamap_test</strong> which is
defined as:</p>
+<pre><code>CREATE TABLE datamap_test (
+ id string,
+ name string,
+ age int,
+ city string,
+ country string)
+STORED AS carbondata
+TBLPROPERTIES('SORT_COLUMNS'='id')
+</code></pre>
+<p>In the above example, <code>id</code> and <code>name</code> are high
cardinality columns
+and we always query on <code>id</code> and <code>name</code> with precise
value.
+since <code>id</code> is in the sort_columns and it is orderd,
+query on it will be fast because CarbonData can skip all the irrelative
blocklets.
+But queries on <code>name</code> may be bad since the blocklet minmax may not
help,
+because in each blocklet the range of the value of <code>name</code> may be
the same -- all from A* to z*.
+In this case, user can create a BloomFilter datamap on column
<code>name</code>.
+Moreover, user can also create a BloomFilter datamap on the sort_columns.
+This is useful if user has too many segments and the range of the value of
sort_columns are almost the same.</p>
+<p>User can create BloomFilter datamap using the Create DataMap DDL:</p>
+<pre><code>CREATE DATAMAP dm
+ON TABLE datamap_test
+USING 'bloomfilter'
+DMPROPERTIES ('INDEX_COLUMNS' = 'name,id', 'BLOOM_SIZE'='640000',
'BLOOM_FPP'='0.00001', 'BLOOM_COMPRESS'='true')
+</code></pre>
+<p><strong>Properties for BloomFilter DataMap</strong></p>
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Is Required</th>
+<th>Default Value</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>INDEX_COLUMNS</td>
+<td>YES</td>
+<td></td>
+<td>Carbondata will generate BloomFilter index on these columns. Queries on
these columns are usually like 'COL = VAL'.</td>
+</tr>
+<tr>
+<td>BLOOM_SIZE</td>
+<td>NO</td>
+<td>640000</td>
+<td>This value is internally used by BloomFilter as the number of expected
insertions, it will affect the size of BloomFilter index. Since each blocklet
has a BloomFilter here, so the default value is the approximate distinct index
values in a blocklet assuming that each blocklet contains 20 pages and each
page contains 32000 records. The value should be an integer.</td>
+</tr>
+<tr>
+<td>BLOOM_FPP</td>
+<td>NO</td>
+<td>0.00001</td>
+<td>This value is internally used by BloomFilter as the False-Positive
Probability, it will affect the size of bloomfilter index as well as the number
of hash functions for the BloomFilter. The value should be in the range (0, 1).
In one test scenario, a 96GB TPCH customer table with bloom_size=320000 and
bloom_fpp=0.00001 will result in 18 false positive samples.</td>
+</tr>
+<tr>
+<td>BLOOM_COMPRESS</td>
+<td>NO</td>
+<td>true</td>
+<td>Whether to compress the BloomFilter index files.</td>
+</tr>
+</tbody>
+</table>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Loading Data</h2>
+<p>When loading data to main table, BloomFilter files will be generated for
all the
+index_columns given in DMProperties which contains the blockletId and a
BloomFilter for each index column.
+These index files will be written inside a folder named with datamap name
+inside each segment folders.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Querying Data</h2>
+<p>User can verify whether a query can leverage BloomFilter datamap by
executing <code>EXPLAIN</code> command,
+which will show the transformed logical plan, and thus user can check whether
the BloomFilter datamap can skip blocklets during the scan.
+If the datamap does not prune blocklets well, you can try to increase the
value of property <code>BLOOM_SIZE</code> and decrease the value of property
<code>BLOOM_FPP</code>.</p>
+<h2>
+<a id="data-management-with-bloomfilter-datamap" class="anchor"
href="#data-management-with-bloomfilter-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Data Management With
BloomFilter DataMap</h2>
+<p>Data management with BloomFilter datamap has no difference with that on
Lucene datamap.
+You can refer to the corresponding section in <code>CarbonData Lucene
DataMap</code>.</p>
+<h2>
+<a id="useful-tips" class="anchor" href="#useful-tips"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Useful Tips</h2>
+<ul>
+<li>BloomFilter DataMap is suggested to be created on the high cardinality
columns.
+Query conditions on these columns are always simple <code>equal</code> or
<code>in</code>,
+such as 'col1=XX', 'col1 in (XX, YY)'.</li>
+<li>We can create multiple BloomFilter datamaps on one table,
+but we do recommend you to create one BloomFilter datamap that contains
multiple index columns,
+because the data loading and query performance will be better.</li>
+<li>
+<code>BLOOM_FPP</code> is only the expected number from user, the actually FPP
may be worse.
+If the BloomFilter datamap does not work well,
+you can try to increase <code>BLOOM_SIZE</code> and decrease
<code>BLOOM_FPP</code> at the same time.
+Notice that bigger <code>BLOOM_SIZE</code> will increase the size of index file
+and smaller <code>BLOOM_FPP</code> will increase runtime calculation while
performing query.</li>
+<li>'0' skipped blocklets of BloomFilter datamap in explain output indicates
that
+BloomFilter datamap does not prune better than Main datamap.
+(For example since the data is not ordered, a specific value may be contained
in many blocklets. In this case, bloom may not work better than Main DataMap.)
+If this occurs very often, it means that current BloomFilter is useless. You
can disable or drop it.
+Sometimes we cannot see any pruning result about BloomFilter datamap in the
explain output,
+this indicates that the previous datamap has pruned all the blocklets and
there is no need to continue pruning.</li>
+<li>In some scenarios, the BloomFilter datamap may not enhance the query
performance significantly
+but if it can reduce the number of spark task,
+there is still a chance that BloomFilter datamap can enhance the performance
for concurrent query.</li>
+<li>Note that BloomFilter datamap will decrease the data loading performance
and may cause slightly storage expansion (for datamap index file).</li>
+</ul>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if
(!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/datamap-developer-guide.html
b/content/datamap-developer-guide.html
new file mode 100644
index 0000000..f442fe2
--- /dev/null
+++ b/content/datamap-developer-guide.html
@@ -0,0 +1,266 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div><h1>
+<a id="datamap-developer-guide" class="anchor" href="#datamap-developer-guide"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Developer Guide</h1>
+<h3>
+<a id="introduction" class="anchor" href="#introduction"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Introduction</h3>
+<p>DataMap is a data structure that can be used to accelerate certain query of
the table. Different DataMap can be implemented by developers.
+Currently, there are two 2 types of DataMap supported:</p>
+<ol>
+<li>IndexDataMap: DataMap that leverages index to accelerate filter query</li>
+<li>MVDataMap: DataMap that leverages Materialized View to accelerate olap
style query, like SPJG query (select, predicate, join, groupby)</li>
+</ol>
+<h3>
+<a id="datamap-provider" class="anchor" href="#datamap-provider"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Provider</h3>
+<p>When user issues <code>CREATE DATAMAP dm ON TABLE main USING
'provider'</code>, the corresponding DataMapProvider implementation will be
created and initialized.
+Currently, the provider string can be:</p>
+<ol>
+<li>preaggregate: A type of MVDataMap that do pre-aggregate of single
table</li>
+<li>timeseries: A type of MVDataMap that do pre-aggregate based on time
dimension of the table</li>
+<li>class name IndexDataMapFactory implementation: Developer can implement
new type of IndexDataMap by extending IndexDataMapFactory</li>
+</ol>
+<p>When user issues <code>DROP DATAMAP dm ON TABLE main</code>, the
corresponding DataMapProvider interface will be called.</p>
+<p>Click for more details about <a
href="./datamap-management.html#datamap-management">DataMap Management</a> and
supported <a href="./datamap-management.html#overview">DSL</a>.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__docs').addClass('selected');
+
+ // Display docs subnav items
+ if (!$('.b-nav__docs').parent().hasClass('nav__item__with__subs--expanded'))
{
+ $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/datamap-management.html b/content/datamap-management.html
new file mode 100644
index 0000000..f5f9678
--- /dev/null
+++ b/content/datamap-management.html
@@ -0,0 +1,394 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-datamap-management" class="anchor"
href="#carbondata-datamap-management" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData DataMap
Management</h1>
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#automatic-refresh">Automatic Refresh</a></li>
+<li><a href="#manual-refresh">Manual Refresh</a></li>
+<li><a href="#datamap-catalog">DataMap Catalog</a></li>
+<li>
+<a href="#datamap-related-commands">DataMap Related Commands</a>
+<ul>
+<li><a href="#explain">Explain</a></li>
+<li><a href="#show-datamap">Show DataMap</a></li>
+<li><a href="#compaction-on-datamap">Compaction on DataMap</a></li>
+</ul>
+</li>
+</ul>
+<h2>
+<a id="overview" class="anchor" href="#overview" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Overview</h2>
+<p>DataMap can be created using following DDL</p>
+<pre><code> CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ [ON TABLE main_table]
+ USING "datamap_provider"
+ [WITH DEFERRED REBUILD]
+ DMPROPERTIES ('key'='value', ...)
+ AS
+ SELECT statement
+</code></pre>
+<p>Currently, there are 5 DataMap implementations in CarbonData.</p>
+<table>
+<thead>
+<tr>
+<th>DataMap Provider</th>
+<th>Description</th>
+<th>DMPROPERTIES</th>
+<th>Management</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>preaggregate</td>
+<td>single table pre-aggregate table</td>
+<td>No DMPROPERTY is required</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>timeseries</td>
+<td>time dimension rollup table</td>
+<td>event_time, xx_granularity, please refer to <a
href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
+</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>mv</td>
+<td>multi-table pre-aggregate table</td>
+<td>No DMPROPERTY is required</td>
+<td>Manual</td>
+</tr>
+<tr>
+<td>lucene</td>
+<td>lucene indexing for text column</td>
+<td>index_columns to specifying the index columns</td>
+<td>Automatic</td>
+</tr>
+<tr>
+<td>bloomfilter</td>
+<td>bloom filter for high cardinality column, geospatial column</td>
+<td>index_columns to specifying the index columns</td>
+<td>Automatic</td>
+</tr>
+</tbody>
+</table>
+<h2>
+<a id="datamap-management" class="anchor" href="#datamap-management"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Management</h2>
+<p>There are two kinds of management semantic for DataMap.</p>
+<ol>
+<li>Automatic Refresh: Create datamap without <code>WITH DEFERRED
REBUILD</code> in the statement, which is by default.</li>
+<li>Manual Refresh: Create datamap with <code>WITH DEFERRED REBUILD</code> in
the statement</li>
+</ol>
+<p><strong>CAUTION:</strong>
+If user create MV datamap without specifying <code>WITH DEFERRED
REBUILD</code>, carbondata will give a warning and treat the datamap as
deferred rebuild.</p>
+<h3>
+<a id="automatic-refresh" class="anchor" href="#automatic-refresh"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Automatic Refresh</h3>
+<p>When user creates a datamap on the main table without using <code>WITH
DEFERRED REBUILD</code> syntax, the datamap will be managed by system
automatically.
+For every data load to the main table, system will immediately trigger a load
to the datamap automatically. These two data loading (to main table and
datamap) is executed in a transactional manner, meaning that it will be either
both success or neither success.</p>
+<p>The data loading to datamap is incremental based on Segment concept,
avoiding a expensive total rebuild.</p>
+<p>If user perform following command on the main table, system will return
failure. (reject the operation)</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>,
<code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported,
and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the
pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by
throwing exception.</li>
+<li>Partition management command: `ALTER TABLE ADD/DROP PARTITION</li>
+</ol>
+<p>If user do want to perform above operations on the main table, user can
first drop the datamap, perform the operation, and re-create the datamap
again.</p>
+<p>If user drop the main table, the datamap will be dropped immediately
too.</p>
+<p>We do recommend you to use this management for index datamap.</p>
+<h3>
+<a id="manual-refresh" class="anchor" href="#manual-refresh"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Manual Refresh</h3>
+<p>When user creates a datamap specifying manual refresh semantic, the datamap
is created with status <em>disabled</em> and query will NOT use this datamap
until user can issue REBUILD DATAMAP command to build the datamap. For every
REBUILD DATAMAP command, system will trigger a full rebuild of the datamap.
After rebuild is done, system will change datamap status to <em>enabled</em>,
so that it can be used in query rewrite.</p>
+<p>For every new data loading, data update, delete, the related datamap will
be made <em>disabled</em>,
+which means that the following queries will not benefit from the datamap
before it becomes <em>enabled</em> again.</p>
+<p>If the main table is dropped by user, the related datamap will be dropped
immediately.</p>
+<p><strong>Note</strong>:</p>
+<ul>
+<li>If you are creating a datamap on external table, you need to do manual
management of the datamap.</li>
+<li>For index datamap such as BloomFilter datamap, there is no need to do
manual refresh.
+By default it is automatic refresh,
+which means its data will get refreshed immediately after the datamap is
created or the main table is loaded.
+Manual refresh on this datamap will has no impact.</li>
+</ul>
+<h2>
+<a id="datamap-catalog" class="anchor" href="#datamap-catalog"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Catalog</h2>
+<p>Currently, when user creates a datamap, system will store the datamap
metadata in a configurable <em>system</em> folder in HDFS or S3.</p>
+<p>In this <em>system</em> folder, it contains:</p>
+<ul>
+<li>DataMapSchema file. It is a json file containing schema for one datamap.
Ses DataMapSchema class. If user creates 100 datamaps (on different tables),
there will be 100 files in <em>system</em> folder.</li>
+<li>DataMapStatus file. Only one file, it is in json format, and each entry in
the file represents for one datamap. Ses DataMapStatusDetail class</li>
+</ul>
+<p>There is a DataMapCatalog interface to retrieve schema of all datamap, it
can be used in optimizer to get the metadata of datamap.</p>
+<h2>
+<a id="datamap-related-commands" class="anchor"
href="#datamap-related-commands" aria-hidden="true"><span aria-hidden="true"
class="octicon octicon-link"></span></a>DataMap Related Commands</h2>
+<h3>
+<a id="explain" class="anchor" href="#explain" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Explain</h3>
+<p>How can user know whether datamap is used in the query?</p>
+<p>User can set enable.query.statistics = true and use EXPLAIN command to
know, it will print out something like</p>
+<pre lang="text"><code>== CarbonData Profiler ==
+Hit mv DataMap: datamap1
+Scan Table: default.datamap1_table
++- filter:
++- pruning by CG DataMap
++- all blocklets: 1
+ skipped blocklets: 0
+</code></pre>
+<h3>
+<a id="show-datamap" class="anchor" href="#show-datamap"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Show DataMap</h3>
+<p>There is a SHOW DATAMAPS command, when this is issued, system will read all
datamap from <em>system</em> folder and print all information on screen. The
current information includes:</p>
+<ul>
+<li>DataMapName</li>
+<li>DataMapProviderName like mv, preaggreagte, timeseries, etc</li>
+<li>Associated Table</li>
+</ul>
+<h3>
+<a id="compaction-on-datamap" class="anchor" href="#compaction-on-datamap"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Compaction on DataMap</h3>
+<p>This feature applies for preaggregate datamap only</p>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table
will <strong>not automatically</strong> compact the pre-aggregate tables
created on the main table. User need to run Compaction command separately on
each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction
is performed on main table but not performed on pre-aggregate table, all
queries still can benefit from pre-aggregate tables. To further improve the
query performance, compaction on pre-aggregate tables can be triggered to merge
the segments and files in the pre-aggregate tables.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if
(!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/lucene-datamap-guide.html
b/content/lucene-datamap-guide.html
new file mode 100644
index 0000000..f461ca5
--- /dev/null
+++ b/content/lucene-datamap-guide.html
@@ -0,0 +1,389 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-lucene-datamap-alpha-feature" class="anchor"
href="#carbondata-lucene-datamap-alpha-feature" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Lucene
DataMap (Alpha Feature)</h1>
+<ul>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#lucene-datamap-introduction">Lucene Datamap</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#data-management-with-lucene-datamap">Data Management</a></li>
+</ul>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Management</h4>
+<p>Lucene DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING 'lucene'
+DMPROPERTIES ('index_columns'='city, name', ...)
+</code></pre>
+<p>DataMap can be dropped using following DDL:</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="lucene-datamap-introduction" class="anchor"
href="#lucene-datamap-introduction" aria-hidden="true"><span aria-hidden="true"
class="octicon octicon-link"></span></a>Lucene DataMap Introduction</h2>
+<p>Lucene is a high performance, full featured text search engine. Lucene is
integrated to carbon as
+an index datamap and managed along with main tables by CarbonData. User can
create lucene datamap
+to improve query performance on string columns which has content of more
length. So, user can
+search tokenized word or pattern of it using lucene query on text content.</p>
+<p>For instance, main table called <strong>datamap_test</strong> which is
defined as:</p>
+<pre><code>CREATE TABLE datamap_test (
+ name string,
+ age int,
+ city string,
+ country string)
+STORED AS carbondata
+</code></pre>
+<p>User can create Lucene datamap using the Create DataMap DDL:</p>
+<pre><code>CREATE DATAMAP dm
+ON TABLE datamap_test
+USING 'lucene'
+DMPROPERTIES ('INDEX_COLUMNS' = 'name, country',)
+</code></pre>
+<p><strong>DMProperties</strong></p>
+<ol>
+<li>INDEX_COLUMNS: The list of string columns on which lucene creates
indexes.</li>
+<li>FLUSH_CACHE: size of the cache to maintain in Lucene writer, if specified
then it tries to
+aggregate the unique data till the cache limit and flush to Lucene. It is best
suitable for low
+cardinality dimensions.</li>
+<li>SPLIT_BLOCKLET: when made as true then store the data in blocklet wise in
lucene , it means new
+folder will be created for each blocklet, thus, it eliminates storing
blockletid in lucene and
+also it makes lucene small chunks of data.</li>
+</ol>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Loading data</h2>
+<p>When loading data to main table, lucene index files will be generated for
all the
+index_columns(String Columns) given in DMProperties which contains information
about the data
+location of index_columns. These index files will be written inside a folder
named with datamap name
+inside each segment folders.</p>
+<p>A system level configuration carbon.lucene.compression.mode can be added
for best compression of
+lucene index files. The default value is speed, where the index writing speed
will be more. If the
+value is compression, the index file size will be compressed.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Lucene indexes cannot be queried
directly.
+Queries are to be made on main table. when a query with TEXT_MATCH('name:c10')
or
+TEXT_MATCH_WITH_LIMIT('name:n10',10)[the second parameter represents the
number of result to be
+returned, if user does not specify this value, all results will be returned
without any limit] is
+fired, two jobs are fired. The first job writes the temporary files in folder
created at table level
+which contains lucene's seach results and these files will be read in second
job to give faster
+results. These temporary files will be cleared once the query finishes.</p>
+<p>User can verify whether a query can leverage Lucene datamap or not by
executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check
whether TEXT_MATCH()
+filter is applied on query or not.</p>
+<p><strong>Note:</strong></p>
+<ol>
+<li>
+<p>The filter columns in TEXT_MATCH or TEXT_MATCH_WITH_LIMIT must be always in
lower case and
+filter condition like 'AND','OR' must be in upper case.</p>
+<p>Ex:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+</code></pre>
+</li>
+<li>
+<p>Query supports only one TEXT_MATCH udf for filter condition and not
multiple udfs.</p>
+<p>The following query is supported:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*')
+</code></pre>
+<p>The following query is not supported:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:*10) AND
TEXT_MATCH(name:*n*')
+</code></pre>
+</li>
+</ol>
+<p>Below like queries can be converted to text_match queries as following:</p>
+<pre><code>select * from datamap_test where name='n10'
+
+select * from datamap_test where name like 'n1%'
+
+select * from datamap_test where name like '%10'
+
+select * from datamap_test where name like '%n%'
+
+select * from datamap_test where name like '%10' and name not like '%n%'
+</code></pre>
+<p>Lucene TEXT_MATCH Queries:</p>
+<pre><code>select * from datamap_test where TEXT_MATCH('name:n10')
+
+select * from datamap_test where TEXT_MATCH('name:n1*')
+
+select * from datamap_test where TEXT_MATCH('name:*10')
+
+select * from datamap_test where TEXT_MATCH('name:*n*')
+
+select * from datamap_test where TEXT_MATCH('name:*10 -name:*n*')
+</code></pre>
+<p><strong>Note:</strong> For lucene queries and syntax, refer to <a
href="http://www.lucenetutorial.com/lucene-query-syntax.html"; target=_blank
rel="nofollow">lucene-syntax</a></p>
+<h2>
+<a id="data-management-with-lucene-datamap" class="anchor"
href="#data-management-with-lucene-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with
lucene datamap</h2>
+<p>Once there is lucene datamap is created on the main table, following
command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>,
<code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>.</li>
+</ol>
+<p><strong>Note</strong>: Adding a new column is supported, and for dropping
columns and change datatype
+command, CarbonData will check whether it will impact the lucene datamap, if
not, the operation
+is allowed, otherwise operation will be rejected by throwing exception.</p>
+<ol start="3">
+<li>Partition management command: <code>ALTER TABLE ADD/DROP
PARTITION</code>.</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in
current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the lucene datamap by <code>DROP DATAMAP</code> command.</li>
+<li>Carry out the data management operation on main table.</li>
+<li>Create the lucene datamap again by <code>CREATE DATAMAP</code> command.
+Basically, user can manually trigger the operation by re-building the
datamap.</li>
+</ol>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if
(!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/preaggregate-datamap-guide.html
b/content/preaggregate-datamap-guide.html
new file mode 100644
index 0000000..e4f1c91
--- /dev/null
+++ b/content/preaggregate-datamap-guide.html
@@ -0,0 +1,502 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-pre-aggregate-datamap" class="anchor"
href="#carbondata-pre-aggregate-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData
Pre-aggregate DataMap</h1>
+<ul>
+<li><a href="#quick-example">Quick Example</a></li>
+<li><a href="#datamap-management">DataMap Management</a></li>
+<li><a href="#preaggregate-datamap-introduction">Pre-aggregate Table</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#querying-data">Querying Data</a></li>
+<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li>
+<li><a href="#data-management-with-pre-aggregate-tables">Data
Management</a></li>
+</ul>
+<h2>
+<a id="quick-example" class="anchor" href="#quick-example"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Quick example</h2>
+<p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p>
+<p>Package carbon jar, and copy
assembly/target/scala-2.11/carbondata_2.11-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar
to $SPARK_HOME/jars</p>
+<div class="highlight highlight-source-shell"><pre>mvn clean package
-DskipTests -Pspark-2.2</pre></div>
+<p>Start spark-shell in new terminal, type :paste, then copy and run the
following code.</p>
+<div class="highlight highlight-source-scala"><pre> <span
class="pl-k">import</span> <span class="pl-en">java</span>.<span
class="pl-en">io</span>.<span class="pl-en">File</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span
class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span
class="pl-en">sql</span>.{<span class="pl-en">CarbonEnv</span>, <span
class="pl-en">SparkSession</span>}
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span
class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span
class="pl-en">sql</span>.<span class="pl-en">CarbonSession</span>.<span
class="pl-en">_</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span
class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span
class="pl-en">sql</span>.<span class="pl-en">streaming</span>.{<span
class="pl-en">ProcessingTime</span>, <span class="pl-en">StreamingQuery</span>}
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span
class="pl-en">apache</span>.<span class="pl-en">carbondata</span>.<span
class="pl-en">core</span>.<span class="pl-en">util</span>.<span
class="pl-en">path</span>.<span class="pl-en">CarbonStorePath</span>
+
+ <span class="pl-k">val</span> <span class="pl-smi">warehouse</span> <span
class="pl-k">=</span> <span class="pl-k">new</span> <span
class="pl-en">File</span>(<span class="pl-s"><span
class="pl-pds">"</span>./warehouse<span
class="pl-pds">"</span></span>).getCanonicalPath
+ <span class="pl-k">val</span> <span class="pl-smi">metastore</span> <span
class="pl-k">=</span> <span class="pl-k">new</span> <span
class="pl-en">File</span>(<span class="pl-s"><span
class="pl-pds">"</span>./metastore<span
class="pl-pds">"</span></span>).getCanonicalPath
+
+ <span class="pl-k">val</span> <span class="pl-smi">spark</span> <span
class="pl-k">=</span> <span class="pl-en">SparkSession</span>
+ .builder()
+ .master(<span class="pl-s"><span class="pl-pds">"</span>local<span
class="pl-pds">"</span></span>)
+ .appName(<span class="pl-s"><span
class="pl-pds">"</span>preAggregateExample<span class="pl-pds">"</span></span>)
+ .config(<span class="pl-s"><span
class="pl-pds">"</span>spark.sql.warehouse.dir<span
class="pl-pds">"</span></span>, warehouse)
+ .getOrCreateCarbonSession(warehouse, metastore)
+
+ spark.sparkContext.setLogLevel(<span class="pl-s"><span
class="pl-pds">"</span>ERROR<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> drop table if exists
previously</span>
+ spark.sql(s<span class="pl-s"><span class="pl-pds">"</span>DROP TABLE IF
EXISTS sales<span class="pl-pds">"</span></span>)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create main table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE TABLE sales (</span>
+<span class="pl-s"> | user_id string,</span>
+<span class="pl-s"> | country string,</span>
+<span class="pl-s"> | quantity int,</span>
+<span class="pl-s"> | price bigint)</span>
+<span class="pl-s"> | STORED AS carbondata</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-c"><span class="pl-c">//</span> Create pre-aggregate table on
the main table</span>
+ <span class="pl-c"><span class="pl-c">//</span> If main table already have
data, following command </span>
+ <span class="pl-c"><span class="pl-c">//</span> will trigger one immediate
load to the pre-aggregate table</span>
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> | CREATE DATAMAP agg_sales</span>
+<span class="pl-s"> | ON TABLE sales</span>
+<span class="pl-s"> | USING "preaggregate"</span>
+<span class="pl-s"> | AS</span>
+<span class="pl-s"> | SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | FROM sales</span>
+<span class="pl-s"> | GROUP BY country</span>
+<span class="pl-s"> <span class="pl-pds">"""</span></span>.stripMargin)
+
+ <span class="pl-k">import</span> <span class="pl-en">spark</span>.<span
class="pl-en">implicits</span>.<span class="pl-en">_</span>
+ <span class="pl-k">import</span> <span class="pl-en">org</span>.<span
class="pl-en">apache</span>.<span class="pl-en">spark</span>.<span
class="pl-en">sql</span>.<span class="pl-en">SaveMode</span>
+ <span class="pl-k">import</span> <span class="pl-en">scala</span>.<span
class="pl-en">util</span>.<span class="pl-en">Random</span>
+
+ <span class="pl-c"><span class="pl-c">//</span> Load data to the main table,
it will also</span>
+ <span class="pl-c"><span class="pl-c">//</span> trigger immediate load to
pre-aggregate table.</span>
+ <span class="pl-c"><span class="pl-c">//</span> These two loading operation
is carried out in a</span>
+ <span class="pl-c"><span class="pl-c">//</span> transactional manner,
meaning that the whole </span>
+ <span class="pl-c"><span class="pl-c">//</span> operation will fail if one
of the loading fails</span>
+ <span class="pl-k">val</span> <span class="pl-smi">r</span> <span
class="pl-k">=</span> <span class="pl-k">new</span> <span
class="pl-en">Random</span>()
+ spark.sparkContext.parallelize(<span class="pl-c1">1</span> to <span
class="pl-c1">10</span>)
+ .map(x <span class="pl-k">=></span> (<span class="pl-s"><span
class="pl-pds">"</span>ID.<span class="pl-pds">"</span></span> <span
class="pl-k">+</span> r.nextInt(<span class="pl-c1">100000</span>), <span
class="pl-s"><span class="pl-pds">"</span>country<span
class="pl-pds">"</span></span> <span class="pl-k">+</span> x <span
class="pl-k">%</span> <span class="pl-c1">8</span>, x <span
class="pl-k">%</span> <span class="pl-c1">50</span>, x <span
class="pl-k">%</span> <span class="pl- [...]
+ .toDF(<span class="pl-s"><span class="pl-pds">"</span>user_id<span
class="pl-pds">"</span></span>, <span class="pl-s"><span
class="pl-pds">"</span>country<span class="pl-pds">"</span></span>, <span
class="pl-s"><span class="pl-pds">"</span>quantity<span
class="pl-pds">"</span></span>, <span class="pl-s"><span
class="pl-pds">"</span>price<span class="pl-pds">"</span></span>)
+ .write
+ .format(<span class="pl-s"><span class="pl-pds">"</span>carbondata<span
class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>tableName<span
class="pl-pds">"</span></span>, <span class="pl-s"><span
class="pl-pds">"</span>sales<span class="pl-pds">"</span></span>)
+ .option(<span class="pl-s"><span class="pl-pds">"</span>compress<span
class="pl-pds">"</span></span>, <span class="pl-s"><span
class="pl-pds">"</span>true<span class="pl-pds">"</span></span>)
+ .mode(<span class="pl-en">SaveMode</span>.<span class="pl-en">Append</span>)
+ .save()
+
+ spark.sql(
+ s<span class="pl-s"><span class="pl-pds">"""</span></span>
+<span class="pl-s"> |SELECT country, sum(quantity), avg(price)</span>
+<span class="pl-s"> | from sales GROUP BY country</span>
+<span class="pl-s"> <span
class="pl-pds">"""</span></span>.stripMargin).show
+
+ spark.stop</pre></div>
+<h4>
+<a id="datamap-management" class="anchor" href="#datamap-management"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>DataMap Management</h4>
+<p>DataMap can be created using following DDL</p>
+<pre><code>CREATE DATAMAP [IF NOT EXISTS] datamap_name
+ON TABLE main_table
+USING "datamap_provider"
+DMPROPERTIES ('key'='value', ...)
+AS
+ SELECT statement
+</code></pre>
+<p>The string followed by USING is called DataMap Provider, in this version
CarbonData supports two
+kinds of DataMap:</p>
+<ol>
+<li>preaggregate, for pre-aggregate table. Pre-Aggregate table supports two
values for DMPROPERTIES.
+a. 'path' is used to specify the store location of the
datamap.('path'='/location/').
+b. 'partitioning' when set to false enables user to disable partitioning of
the datamap.
+Default value is true for this property.</li>
+<li>timeseries, for timeseries roll-up table. Please refer to <a
href="./timeseries-datamap-guide.html">Timeseries DataMap</a>
+</li>
+</ol>
+<p>DataMap can be dropped using following DDL</p>
+<pre><code>DROP DATAMAP [IF EXISTS] datamap_name
+ON TABLE main_table
+</code></pre>
+<p>To show all DataMaps created, use:</p>
+<pre><code>SHOW DATAMAP
+ON TABLE main_table
+</code></pre>
+<p>It will show all DataMaps created on main table.</p>
+<h2>
+<a id="preaggregate-datamap-introduction" class="anchor"
href="#preaggregate-datamap-introduction" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Preaggregate DataMap
Introduction</h2>
+<p>Pre-aggregate tables are created as DataMaps and managed as tables
internally by CarbonData.
+User can create as many pre-aggregate datamaps required to improve query
performance,
+provided the storage requirements and loading speeds are acceptable.</p>
+<p>Once pre-aggregate datamaps are created, CarbonData's SparkSQL optimizer
extension supports to
+select the most efficient pre-aggregate datamap and rewrite the SQL to query
against the selected
+datamap instead of the main table. Since the data size of pre-aggregate
datamap is smaller,
+user queries are much faster. In our previous experience, we have seen 5X to
100X times faster
+in production SQLs.</p>
+<p>For instance, main table called <strong>sales</strong> which is defined
as</p>
+<pre><code>CREATE TABLE sales (
+ order_time timestamp,
+ user_id string,
+ sex string,
+ country string,
+ quantity int,
+ price bigint)
+STORED AS carbondata
+</code></pre>
+<p>User can create pre-aggregate tables using the Create DataMap DDL</p>
+<pre><code>CREATE DATAMAP agg_sales
+ON TABLE sales
+USING "preaggregate"
+AS
+ SELECT country, sex, sum(quantity), avg(price)
+ FROM sales
+ GROUP BY country, sex
+</code></pre>
+<h4>
+<a id="functions-supported-in-pre-aggregate-table" class="anchor"
href="#functions-supported-in-pre-aggregate-table" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Functions supported
in pre-aggregate table</h4>
+<table>
+<thead>
+<tr>
+<th>Function</th>
+<th align="center">Rollup supported</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>SUM</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>AVG</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MAX</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>MIN</td>
+<td align="center">Yes</td>
+</tr>
+<tr>
+<td>COUNT</td>
+<td align="center">Yes</td>
+</tr>
+</tbody>
+</table>
+<h4>
+<a id="how-pre-aggregate-tables-are-selected" class="anchor"
href="#how-pre-aggregate-tables-are-selected" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>How pre-aggregate
tables are selected</h4>
+<p>When a user query is submitted, during query planning phase, CarbonData
will collect all matched
+pre-aggregate tables as candidates according to Relational Algebra
transformation rules. Then, the
+best pre-aggregate table for this query will be selected among the candidates
based on cost.
+For simplicity, current cost estimation is based on the data size of the
pre-aggregate table. (We
+assume that query will be faster on smaller table)</p>
+<p>For the main table <strong>sales</strong> and pre-aggregate table
<strong>agg_sales</strong> created above, following queries</p>
+<pre><code>SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY
country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT avg(price), country from sales GROUP BY country
+</code></pre>
+<p>will be transformed by CarbonData's query planner to query against
pre-aggregate table
+<strong>agg_sales</strong> instead of the main table <strong>sales</strong></p>
+<p>However, for following queries</p>
+<pre><code>SELECT user_id, country, sex, sum(quantity), avg(price) from sales
GROUP BY user_id, country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT country, max(price) from sales GROUP BY country
+</code></pre>
+<p>will query against main table <strong>sales</strong> only, because it does
not satisfy pre-aggregate table
+selection logic.</p>
+<h2>
+<a id="loading-data" class="anchor" href="#loading-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Loading data</h2>
+<p>For existing table with loaded data, data load to pre-aggregate table will
be triggered by the
+CREATE DATAMAP statement when user creates the pre-aggregate table. For
incremental loads after
+aggregates tables are created, loading data to main table triggers the load to
pre-aggregate tables
+once main table loading is complete.</p>
+<p>These loads are transactional
+meaning that data on main table and pre-aggregate tables are only visible to
the user after all
+tables are loaded successfully, if one of these loads fails, new data are not
visible in all tables
+as if the load operation is not happened.</p>
+<h2>
+<a id="querying-data" class="anchor" href="#querying-data"
aria-hidden="true"><span aria-hidden="true" class="octicon
octicon-link"></span></a>Querying data</h2>
+<p>As a technique for query acceleration, Pre-aggregate tables cannot be
queried directly.
+Queries are to be made on main table. While doing query planning, internally
CarbonData will check
+associated pre-aggregate tables with the main table, and do query plan
transformation accordingly.</p>
+<p>User can verify whether a query can leverage pre-aggregate table or not by
executing <code>EXPLAIN</code>
+command, which will show the transformed logical plan, and thus user can check
whether pre-aggregate
+table is selected.</p>
+<h2>
+<a id="compacting-pre-aggregate-tables" class="anchor"
href="#compacting-pre-aggregate-tables" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Compacting
pre-aggregate tables</h2>
+<p>Running Compaction command (<code>ALTER TABLE COMPACT</code>) on main table
will <strong>not automatically</strong>
+compact the pre-aggregate tables created on the main table. User need to run
Compaction command
+separately on each pre-aggregate table to compact them.</p>
+<p>Compaction is an optional operation for pre-aggregate table. If compaction
is performed on
+main table but not performed on pre-aggregate table, all queries still can
benefit from
+pre-aggregate tables. To further improve the query performance, compaction on
pre-aggregate tables
+can be triggered to merge the segments and files in the pre-aggregate
tables.</p>
+<h2>
+<a id="data-management-with-pre-aggregate-tables" class="anchor"
href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with
pre-aggregate tables</h2>
+<p>In current implementation, data consistency needs to be maintained for both
main table and pre-aggregate
+tables. Once there is pre-aggregate table created on the main table, following
command on the main
+table
+is not supported:</p>
+<ol>
+<li>Data management command: <code>UPDATE/DELETE/DELETE SEGMENT</code>.</li>
+<li>Schema management command: <code>ALTER TABLE DROP COLUMN</code>,
<code>ALTER TABLE CHANGE DATATYPE</code>,
+<code>ALTER TABLE RENAME</code>. Note that adding a new column is supported,
and for dropping columns and
+change datatype command, CarbonData will check whether it will impact the
pre-aggregate table, if
+not, the operation is allowed, otherwise operation will be rejected by
throwing exception.</li>
+<li>Partition management command: <code>ALTER TABLE ADD/DROP PARTITION</code>
+</li>
+<li>Complex Datatypes for preaggregate is not supported.</li>
+</ol>
+<p>However, there is still way to support these operations on main table, in
current CarbonData
+release, user can do as following:</p>
+<ol>
+<li>Remove the pre-aggregate table by <code>DROP DATAMAP</code> command</li>
+<li>Carry out the data management operation on main table</li>
+<li>Create the pre-aggregate table again by <code>CREATE DATAMAP</code> command
+Basically, user can manually trigger the operation by re-building the
datamap.</li>
+</ol>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if
(!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
diff --git a/content/timeseries-datamap-guide.html
b/content/timeseries-datamap-guide.html
new file mode 100644
index 0000000..a9137d0
--- /dev/null
+++ b/content/timeseries-datamap-guide.html
@@ -0,0 +1,369 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'>
+ <!-- The above 3 meta tags *must* come first in the head; any other head
content must come *after* these tags -->
+ <title>CarbonData</title>
+ <style>
+
+ </style>
+ <!-- Bootstrap -->
+
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link href="css/style.css" rel="stylesheet">
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media
queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script
src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script>
+ <script
src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script>
+ <![endif]-->
+ <script src="js/jquery.min.js"></script>
+ <script src="js/bootstrap.min.js"></script>
+ <script defer
src="https://use.fontawesome.com/releases/v5.0.8/js/all.js";></script>
+
+
+</head>
+<body>
+<header>
+ <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper">
+ <div class="container">
+ <div class="navbar-header">
+ <button aria-controls="navbar" aria-expanded="false"
data-target="#navbar" data-toggle="collapse"
+ class="navbar-toggle collapsed" type="button">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="index.html" class="logo">
+ <img src="images/CarbonDataLogo.png" alt="CarbonData logo"
title="CarbocnData logo"/>
+ </a>
+ </div>
+ <div class="navbar-collapse collapse cd_navcontnt" id="navbar">
+ <ul class="nav navbar-nav navbar-right navlist-custom">
+ <li><a href="index.html" class="hidden-xs"><i class="fa
fa-home" aria-hidden="true"></i> </a>
+ </li>
+ <li><a href="index.html" class="hidden-lg hidden-md
hidden-sm">Home</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle "
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false"> Download <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/";
+ target="_blank">Apache CarbonData
1.5.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/";
+ target="_blank">Apache CarbonData
1.4.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/";
+ target="_blank">Apache CarbonData
1.4.0</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/";
+ target="_blank">Apache CarbonData
1.3.1</a></li>
+ <li>
+ <a
href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/";
+ target="_blank">Apache CarbonData
1.3.0</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases";
+ target="_blank">Release Archive</a></li>
+ </ul>
+ </li>
+ <li><a href="documentation.html"
class="active">Documentation</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle"
data-toggle="dropdown" role="button" aria-haspopup="true"
+ aria-expanded="false">Community <span
class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/how-to-contribute-to-apache-carbondata.md";
+ target="_blank">Contributing to
CarbonData</a></li>
+ <li>
+ <a
href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md";
+ target="_blank">Release Guide</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list";
+ target="_blank">Project PMC and
Committers</a></li>
+ <li>
+ <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609";
+ target="_blank">CarbonData Meetups</a></li>
+ <li><a href="security.html">Apache CarbonData
Security</a></li>
+ <li><a
href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache
+ Jira</a></li>
+ <li><a href="videogallery.html">CarbonData Videos
</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="apache_link
hidden-xs dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="http://www.apache.org/"; class="hidden-lg
hidden-md hidden-sm dropdown-toggle"
+ data-toggle="dropdown" role="button"
aria-haspopup="true" aria-expanded="false">Apache</a>
+ <ul class="dropdown-menu">
+ <li><a href="http://www.apache.org/";
target="_blank">Apache Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/";
target="_blank">License</a></li>
+ <li><a
href="http://www.apache.org/foundation/sponsorship.html";
+ target="_blank">Sponsorship</a></li>
+ <li><a
href="http://www.apache.org/foundation/thanks.html";
target="_blank">Thanks</a></li>
+ </ul>
+ </li>
+
+ <li>
+ <a href="#" id="search-icon"><i class="fa fa-search"
aria-hidden="true"></i></a>
+
+ </li>
+
+ </ul>
+ </div><!--/.nav-collapse -->
+ <div id="search-box">
+ <form method="get" action="http://www.google.com/search";
target="_blank">
+ <div class="search-block">
+ <table border="0" cellpadding="0" width="100%">
+ <tr>
+ <td style="width:80%">
+ <input type="text" name="q" size=" 5"
maxlength="255" value=""
+ class="search-input"
placeholder="Search...." required/>
+ </td>
+ <td style="width:20%">
+ <input type="submit" value="Search"/></td>
+ </tr>
+ <tr>
+ <td align="left" style="font-size:75%"
colspan="2">
+ <input type="checkbox" name="sitesearch"
value="carbondata.apache.org" checked/>
+ <span style=" position: relative; top:
-3px;"> Only search for CarbonData</span>
+ </td>
+ </tr>
+ </table>
+ </div>
+ </form>
+ </div>
+ </div>
+ </nav>
+</header> <!-- end Header part -->
+
+<div class="fixed-padding"></div> <!-- top padding with fixde header -->
+
+<section><!-- Dashboard nav -->
+ <div class="container-fluid q">
+ <div class="col-sm-12 col-md-12 maindashboard">
+ <div class="verticalnavbar">
+ <nav class="b-sticky-nav">
+ <div class="nav-scroller">
+ <div class="nav__inner">
+ <a class="b-nav__intro nav__item"
href="./introduction.html">introduction</a>
+ <a class="b-nav__quickstart nav__item"
href="./quick-start-guide.html">quick start</a>
+ <a class="b-nav__uses nav__item"
href="./usecases.html">use cases</a>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__docs nav__item
nav__sub__anchor" href="./language-manual.html">Language Reference</a>
+ <a class="nav__item nav__sub__item"
href="./ddl-of-carbondata.html">DDL</a>
+ <a class="nav__item nav__sub__item"
href="./dml-of-carbondata.html">DML</a>
+ <a class="nav__item nav__sub__item"
href="./streaming-guide.html">Streaming</a>
+ <a class="nav__item nav__sub__item"
href="./configuration-parameters.html">Configuration</a>
+ <a class="nav__item nav__sub__item"
href="./datamap-developer-guide.html">Datamaps</a>
+ <a class="nav__item nav__sub__item"
href="./supported-data-types-in-carbondata.html">Data Types</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__datamap nav__item
nav__sub__anchor" href="./datamap-management.html">DataMaps</a>
+ <a class="nav__item nav__sub__item"
href="./bloomfilter-datamap-guide.html">Bloom Filter</a>
+ <a class="nav__item nav__sub__item"
href="./lucene-datamap-guide.html">Lucene</a>
+ <a class="nav__item nav__sub__item"
href="./preaggregate-datamap-guide.html">Pre-Aggregate</a>
+ <a class="nav__item nav__sub__item"
href="./timeseries-datamap-guide.html">Time Series</a>
+ </div>
+
+ <div class="nav__item nav__item__with__subs">
+ <a class="b-nav__api nav__item
nav__sub__anchor" href="./sdk-guide.html">API</a>
+ <a class="nav__item nav__sub__item"
href="./sdk-guide.html">Java SDK</a>
+ <a class="nav__item nav__sub__item"
href="./CSDK-guide.html">C++ SDK</a>
+ </div>
+
+ <a class="b-nav__perf nav__item"
href="./performance-tuning.html">Performance Tuning</a>
+ <a class="b-nav__s3 nav__item"
href="./s3-guide.html">S3 Storage</a>
+ <a class="b-nav__faq nav__item"
href="./faq.html">FAQ</a>
+ <a class="b-nav__contri nav__item"
href="./how-to-contribute-to-apache-carbondata.html">Contribute</a>
+ <a class="b-nav__security nav__item"
href="./security.html">Security</a>
+ <a class="b-nav__release nav__item"
href="./release-guide.html">Release Guide</a>
+ </div>
+ </div>
+ <div class="navindicator">
+ <div class="b-nav__intro navindicator__item"></div>
+ <div class="b-nav__quickstart
navindicator__item"></div>
+ <div class="b-nav__uses navindicator__item"></div>
+ <div class="b-nav__docs navindicator__item"></div>
+ <div class="b-nav__datamap navindicator__item"></div>
+ <div class="b-nav__api navindicator__item"></div>
+ <div class="b-nav__perf navindicator__item"></div>
+ <div class="b-nav__s3 navindicator__item"></div>
+ <div class="b-nav__faq navindicator__item"></div>
+ <div class="b-nav__contri navindicator__item"></div>
+ <div class="b-nav__security navindicator__item"></div>
+ </div>
+ </nav>
+ </div>
+ <div class="mdcontent">
+ <section>
+ <div style="padding:10px 15px;">
+ <div id="viewpage" name="viewpage">
+ <div class="row">
+ <div class="col-sm-12 col-md-12">
+ <div>
+<h1>
+<a id="carbondata-timeseries-datamap" class="anchor"
href="#carbondata-timeseries-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData
Timeseries DataMap</h1>
+<ul>
+<li><a href="#timeseries-datamap-introduction-alpha-feature">Timeseries
DataMap Introduction</a></li>
+<li><a href="#compacting-timeseries-datamp">Compaction</a></li>
+<li><a href="#data-management-on-timeseries-datamap">Data Management</a></li>
+</ul>
+<h2>
+<a id="timeseries-datamap-introduction-alpha-feature" class="anchor"
href="#timeseries-datamap-introduction-alpha-feature" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap
Introduction (Alpha Feature)</h2>
+<p>Timeseries DataMap is a pre-aggregate table implementation based on
'pre-aggregate' DataMap.
+Difference is that Timeseries DataMap has built-in understanding of time
hierarchy and
+levels: year, month, day, hour, minute, so that it supports automatic roll-up
in time dimension
+for query.</p>
+<p><strong>CAUTION:</strong> Current version of CarbonData does not support
roll-up.It will be implemented in future versions.</p>
+<p>The data loading, querying, compaction command and its behavior is the same
as preaggregate DataMap.
+Please refer to <a href="./preaggregate-datamap-guide.html">Pre-aggregate
DataMap</a>
+for more information.</p>
+<p>To use this datamap, user can create multiple timeseries datamap on the
main table which has
+a <em>event_time</em> column, one datamap for one time granularity.</p>
+<p>For example, below statement effectively create multiple pre-aggregate
tables on main table called
+<strong>timeseries</strong></p>
+<pre><code>CREATE DATAMAP agg_year
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'year_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id),
sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_month
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'month_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id),
sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_day
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id),
sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_sales_hour
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id),
sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+CREATE DATAMAP agg_minute
+ON TABLE sales
+USING "timeseries"
+DMPROPERTIES (
+ 'event_time'='order_time',
+ 'minute_granularity'='1',
+) AS
+SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id),
sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>For querying timeseries data, Carbondata has builtin support for following
time related UDF</p>
+<pre><code>timeseries(timeseries column name, 'aggregation level')
+</code></pre>
+<pre><code>SELECT timeseries(order_time, 'hour'), sum(quantity) FROM sales
GROUP BY timeseries(order_time,
+'hour')
+</code></pre>
+<p>It is <strong>not necessary</strong> to create pre-aggregate tables for
each granularity unless required for
+query.</p>
+<p>For Example: For main table <strong>sales</strong> , if following
timeseries datamaps were created for day
+level and hour level pre-aggregate</p>
+<pre><code> CREATE DATAMAP agg_day
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'day_granularity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity),
count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+
+ CREATE DATAMAP agg_sales_hour
+ ON TABLE sales
+ USING "timeseries"
+ DMPROPERTIES (
+ 'event_time'='order_time',
+ 'hour_granularity'='1',
+ ) AS
+ SELECT order_time, country, sex, sum(quantity), max(quantity),
count(user_id), sum(price),
+ avg(price) FROM sales GROUP BY order_time, country, sex
+</code></pre>
+<p>Queries like below will not be rolled-up and hit the main table</p>
+<pre><code>Select timeseries(order_time, 'month'), sum(quantity) from sales
group by timeseries(order_time,
+ 'month')
+
+Select timeseries(order_time, 'year'), sum(quantity) from sales group by
timeseries(order_time,
+ 'year')
+</code></pre>
+<p>NOTE (<b>RESTRICTION</b>):</p>
+<ul>
+<li>Only value of 1 is supported for hierarchy levels. Other hierarchy levels
will be supported in
+the future CarbonData release.</li>
+<li>timeseries datamap for the desired levels needs to be created one after
the other</li>
+<li>timeseries datamaps created for each level needs to be dropped
separately</li>
+</ul>
+<h2>
+<a id="compacting-timeseries-datamp" class="anchor"
href="#compacting-timeseries-datamp" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Compacting
timeseries datamp</h2>
+<p>Refer to Compaction section in <a
href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<h2>
+<a id="data-management-on-timeseries-datamap" class="anchor"
href="#data-management-on-timeseries-datamap" aria-hidden="true"><span
aria-hidden="true" class="octicon octicon-link"></span></a>Data Management on
timeseries datamap</h2>
+<p>Refer to Data Management section in <a
href="./preaggregate-datamap-guide.html">preaggregation datamap</a>.
+Same applies to timeseries datamap.</p>
+<script>
+$(function() {
+ // Show selected style on nav item
+ $('.b-nav__datamap').addClass('selected');
+
+ if
(!$('.b-nav__datamap').parent().hasClass('nav__item__with__subs--expanded')) {
+ // Display datamap subnav items
+
$('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded');
+ }
+});
+</script></div>
+</div>
+</div>
+</div>
+<div class="doc-footer">
+ <a href="#top" class="scroll-top">Top</a>
+</div>
+</div>
+</section>
+</div>
+</div>
+</div>
+</section><!-- End systemblock part -->
+<script src="js/custom.js"></script>
+</body>
+</html>
\ No newline at end of file
