http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/quick-start-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/quick-start-guide.html b/src/main/webapp/quick-start-guide.html index f3e8a8f..8e599d3 100644 --- a/src/main/webapp/quick-start-guide.html +++ b/src/main/webapp/quick-start-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -212,7 +220,7 @@ <div> <h1> <a id="quick-start" class="anchor" href="#quick-start" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick Start</h1> -<p>This tutorial provides a quick introduction to using CarbonData.To follow along with this guide, first download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/"; target=_blank rel="nofollow">CarbonData website</a>.Alternatively it can be created following <a href="https://github.com/apache/carbondata/tree/master/build"; target=_blank>Building CarbonData</a> steps.</p> +<p>This tutorial provides a quick introduction to using CarbonData. To follow along with this guide, first download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/"; target=_blank rel="nofollow">CarbonData website</a>.Alternatively it can be created following <a href="https://github.com/apache/carbondata/tree/master/build"; target=_blank>Building CarbonData</a> steps.</p> <h2> <a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h2> <ul> @@ -233,7 +241,7 @@ EOF </ul> <h2> <a id="integration" class="anchor" href="#integration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration</h2> -<p>CarbonData can be integrated with Spark and Presto Execution Engines.The below documentation guides on Installing and Configuring with these execution engines.</p> +<p>CarbonData can be integrated with Spark and Presto Execution Engines. The below documentation guides on Installing and Configuring with these execution engines.</p> <h3> <a id="spark" class="anchor" href="#spark" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Spark</h3> <p><a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Installing and Configuring CarbonData to run locally with Spark Shell</a></p> @@ -555,26 +563,26 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store </code></pre> <h2> <a id="installing-and-configuring-carbondata-on-presto" class="anchor" href="#installing-and-configuring-carbondata-on-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Presto</h2> -<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto.User need to create CarbonData Table and load data into it +<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto. User need to create CarbonData Table and load data into it either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a>. Once the table is created,it can be queried from Presto.</strong></p> <h3> <a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3> <ol> <li> -<p>Download the 0.187 version of Presto using: -<code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.187/presto-server-0.187.tar.gz</code></p> +<p>Download the 0.210 version of Presto using: +<code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz</code></p> </li> <li> -<p>Extract Presto tar file: <code>tar zxvf presto-server-0.187.tar.gz</code>.</p> +<p>Extract Presto tar file: <code>tar zxvf presto-server-0.210.tar.gz</code>.</p> </li> <li> <p>Download the Presto CLI for the coordinator and name it presto.</p> </li> </ol> -<pre><code> wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.187/presto-cli-0.187-executable.jar +<pre><code> wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.210/presto-cli-0.210-executable.jar - mv presto-cli-0.187-executable.jar presto + mv presto-cli-0.210-executable.jar presto chmod +x presto </code></pre> @@ -582,7 +590,7 @@ Once the table is created,it can be queried from Presto.</strong></p> <a id="create-configuration-files" class="anchor" href="#create-configuration-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Configuration Files</h3> <ol> <li> -<p>Create <code>etc</code> folder in presto-server-0.187 directory.</p> +<p>Create <code>etc</code> folder in presto-server-0.210 directory.</p> </li> <li> <p>Create <code>config.properties</code>, <code>jvm.config</code>, <code>log.properties</code>, and <code>node.properties</code> files.</p> @@ -624,10 +632,15 @@ node.data-dir=/home/ubuntu/data <pre><code>coordinator=true node-scheduler.include-coordinator=false http-server.http.port=8086 -query.max-memory=50GB -query.max-memory-per-node=2GB +query.max-memory=5GB +query.max-total-memory-per-node=5GB +query.max-memory-per-node=3GB +memory.heap-headroom-per-node=1GB discovery-server.enabled=true -discovery.uri=<coordinator_ip>:8086 +discovery.uri=http://localhost:8086 +task.max-worker-threads=4 +optimizer.dictionary-aggregation=true +optimizer.optimize-hash-generation = false </code></pre> <p>The options <code>node-scheduler.include-coordinator=false</code> and <code>coordinator=true</code> indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers.</p> <p><strong>Note</strong>: It is recommended to set <code>query.max-memory-per-node</code> to half of the JVM config max memory, though the workload is highly concurrent, lower value for <code>query.max-memory-per-node</code> is to be used.</p> @@ -640,7 +653,7 @@ Then, <code>query.max-memory=<30GB * number of nodes></code>.</p> <a id="contents-of-your-configproperties-1" class="anchor" href="#contents-of-your-configproperties-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your config.properties</h5> <pre><code>coordinator=false http-server.http.port=8086 -query.max-memory=50GB +query.max-memory=5GB query.max-memory-per-node=2GB discovery.uri=<coordinator_ip>:8086 </code></pre> @@ -663,10 +676,10 @@ discovery.uri=<coordinator_ip>:8086 </ol> <h3> <a id="start-presto-server-on-all-nodes" class="anchor" href="#start-presto-server-on-all-nodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto Server on all nodes</h3> -<pre><code>./presto-server-0.187/bin/launcher start +<pre><code>./presto-server-0.210/bin/launcher start </code></pre> <p>To run it as a background process.</p> -<pre><code>./presto-server-0.187/bin/launcher run +<pre><code>./presto-server-0.210/bin/launcher run </code></pre> <p>To run it in foreground.</p> <h3>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/release-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/release-guide.html b/src/main/webapp/release-guide.html index cb47540..891d897 100644 --- a/src/main/webapp/release-guide.html +++ b/src/main/webapp/release-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/s3-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/s3-guide.html b/src/main/webapp/s3-guide.html index 57af913..62c51c3 100644 --- a/src/main/webapp/s3-guide.html +++ b/src/main/webapp/s3-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -211,7 +219,7 @@ <div class="col-sm-12 col-md-12"> <div> <h1> -<a id="s3-guide-alpha-feature-141" class="anchor" href="#s3-guide-alpha-feature-141" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Guide (Alpha Feature 1.4.1)</h1> +<a id="s3-guide" class="anchor" href="#s3-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Guide</h1> <p>Object storage is the recommended storage format in cloud as it can support storing large data files. S3 APIs are widely used for accessing object stores. This can be used to store or retrieve data on Amazon cloud, Huawei Cloud(OBS) or on any other object http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/sdk-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/sdk-guide.html b/src/main/webapp/sdk-guide.html index a252965..5ddf9f7 100644 --- a/src/main/webapp/sdk-guide.html +++ b/src/main/webapp/sdk-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -259,9 +267,9 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-smi">CarbonProperties</span><span class="pl-k">.</span>getInstance()<span class="pl-k">.</span>addProperty(<span class="pl-s"><span class="pl-pds">"</span>enable.offheap.sort<span class="pl-pds">"</span></span>, enableOffheap); - <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path); + <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withCsvInput(schema); - <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>buildWriterForCSVInput(schema); + <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>build(); <span class="pl-k">int</span> rows <span class="pl-k">=</span> <span class="pl-c1">5</span>; <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> rows; i<span class="pl-k">++</span>) { @@ -314,7 +322,7 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-k">try</span> { <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder() .outputPath(path) - .buildWriterForAvroInput(<span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema)); + .withAvroInput(<span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema))<span class="pl-k">.</span>build(); <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> <span class="pl-c1">100</span>; i<span class="pl-k">++</span>) { writer<span class="pl-k">.</span>write(record); @@ -352,10 +360,10 @@ These SDK writer output contains just a carbondata and carbonindex files. No met <span class="pl-smi">Schema</span> <span class="pl-smi">CarbonSchema</span> <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Schema</span>(fields); - <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path); + <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>outputPath(path)<span class="pl-k">.</span>withJsonInput(<span class="pl-smi">CarbonSchema</span>); <span class="pl-c"><span class="pl-c">//</span> initialize json writer with carbon schema</span> - <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>buildWriterForJsonInput(<span class="pl-smi">CarbonSchema</span>); + <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>build(); <span class="pl-c"><span class="pl-c">//</span> one row of json Data as String</span> <span class="pl-smi">String</span> <span class="pl-smi">JsonRow</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-cce">\"</span>name<span class="pl-cce">\"</span>:<span class="pl-cce">\"</span>abcd<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>age<span class="pl-cce">\"</span>:10}<span class="pl-pds">"</span></span>; @@ -368,59 +376,127 @@ These SDK writer output contains just a carbondata and carbonindex files. No met } </pre></div> <h2> <a id="datatypes-mapping" class="anchor" href="#datatypes-mapping" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Datatypes Mapping</h2> -<p>Each of SQL data types are mapped into data types of SDK. Following are the mapping:</p> +<p>Each of SQL data types and Avro Data Types are mapped into data types of SDK. Following are the mapping:</p> <table> <thead> <tr> <th>SQL DataTypes</th> +<th>Avro DataTypes</th> <th>Mapped SDK DataTypes</th> </tr> </thead> <tbody> <tr> <td>BOOLEAN</td> +<td>BOOLEAN</td> <td>DataTypes.BOOLEAN</td> </tr> <tr> <td>SMALLINT</td> +<td>-</td> <td>DataTypes.SHORT</td> </tr> <tr> <td>INTEGER</td> +<td>INTEGER</td> <td>DataTypes.INT</td> </tr> <tr> <td>BIGINT</td> +<td>LONG</td> <td>DataTypes.LONG</td> </tr> <tr> <td>DOUBLE</td> +<td>DOUBLE</td> <td>DataTypes.DOUBLE</td> </tr> <tr> <td>VARCHAR</td> +<td>-</td> <td>DataTypes.STRING</td> </tr> <tr> +<td>FLOAT</td> +<td>FLOAT</td> +<td>DataTypes.FLOAT</td> +</tr> +<tr> +<td>BYTE</td> +<td>-</td> +<td>DataTypes.BYTE</td> +</tr> +<tr> +<td>DATE</td> <td>DATE</td> <td>DataTypes.DATE</td> </tr> <tr> <td>TIMESTAMP</td> +<td>-</td> <td>DataTypes.TIMESTAMP</td> </tr> <tr> <td>STRING</td> +<td>STRING</td> <td>DataTypes.STRING</td> </tr> <tr> <td>DECIMAL</td> +<td>DECIMAL</td> <td>DataTypes.createDecimalType(precision, scale)</td> </tr> +<tr> +<td>ARRAY</td> +<td>ARRAY</td> +<td>DataTypes.createArrayType(elementType)</td> +</tr> +<tr> +<td>STRUCT</td> +<td>RECORD</td> +<td>DataTypes.createStructType(fields)</td> +</tr> +<tr> +<td>-</td> +<td>ENUM</td> +<td>DataTypes.STRING</td> +</tr> +<tr> +<td>-</td> +<td>UNION</td> +<td>DataTypes.createStructType(types)</td> +</tr> +<tr> +<td>-</td> +<td>MAP</td> +<td>DataTypes.createMapType(keyType, valueType)</td> +</tr> +<tr> +<td>-</td> +<td>TimeMillis</td> +<td>DataTypes.INT</td> +</tr> +<tr> +<td>-</td> +<td>TimeMicros</td> +<td>DataTypes.LONG</td> +</tr> +<tr> +<td>-</td> +<td>TimestampMillis</td> +<td>DataTypes.TIMESTAMP</td> +</tr> +<tr> +<td>-</td> +<td>TimestampMicros</td> +<td>DataTypes.TIMESTAMP</td> +</tr> </tbody> </table> -<p><strong>NOTE:</strong> -Carbon Supports below logical types of AVRO. +<p><strong>NOTE:</strong></p> +<ol> +<li> +<p>Carbon Supports below logical types of AVRO. a. Date The date logical type represents a date within the calendar, with no reference to a particular time zone or time of day. A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch, 1 January 1970 (ISO calendar). @@ -429,10 +505,27 @@ The timestamp-millis logical type represents an instant on the global timeline, A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000 UTC. c. Timestamp (microsecond precision) The timestamp-micros logical type represents an instant on the global timeline, independent of a particular time zone or calendar, with a precision of one microsecond. -A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds from the unix epoch, 1 January 1970 00:00:00.000000 UTC.</p> -<pre><code>Currently the values of logical types are not validated by carbon. -Expect that avro record passed by the user is already validated by avro record generator tools. -</code></pre> +A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds from the unix epoch, 1 January 1970 00:00:00.000000 UTC. +d. Decimal +The decimal logical type represents an arbitrary-precision signed decimal number of the form unscaled � 10-scale. +A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two's-complement representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified using an attribute. +e. Time (millisecond precision) +The time-millis logical type represents a time of day, with no reference to a particular calendar, time zone or date, with a precision of one millisecond. +A time-millis logical type annotates an Avro int, where the int stores the number of milliseconds after midnight, 00:00:00.000. +f. Time (microsecond precision) +The time-micros logical type represents a time of day, with no reference to a particular calendar, time zone or date, with a precision of one microsecond. +A time-micros logical type annotates an Avro long, where the long stores the number of microseconds after midnight, 00:00:00.000000.</p> +<p>Currently the values of logical types are not validated by carbon. +Expect that avro record passed by the user is already validated by avro record generator tools.</p> +</li> +<li> +<p>If the string data is more than 32K in length, use withTableProperties() with "long_string_columns" property +or directly use DataTypes.VARCHAR if it is carbon schema.</p> +</li> +<li> +<p>Avro Bytes, Fixed and Duration data types are not yet supported.</p> +</li> +</ol> <h2> <a id="run-sql-on-files-directly" class="anchor" href="#run-sql-on-files-directly" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Run SQL on files directly</h2> <p>Instead of creating table and query it, you can also query that file directly with SQL.</p> @@ -454,18 +547,6 @@ Expect that avro record passed by the user is already validated by avro record g public CarbonWriterBuilder outputPath(String path); </code></pre> <pre><code>/** -* If set false, writes the carbondata and carbonindex files in a flat folder structure -* @param isTransactionalTable is a boolelan value -* if set to false, then writes the carbondata and carbonindex files -* in a flat folder structure. -* if set to true, then writes the carbondata and carbonindex files -* in segment folder structure.. -* By default set to false. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); -</code></pre> -<pre><code>/** * to set the timestamp in the carbondata and carbonindex index files * @param UUID is a timestamp to be used in the carbondata and carbonindex index files. * By default set to zero. @@ -511,14 +592,6 @@ public CarbonWriterBuilder localDictionaryThreshold(int localDictionaryThreshold public CarbonWriterBuilder sortBy(String[] sortColumns); </code></pre> <pre><code>/** -* If set, create a schema file in metadata folder. -* @param persist is a boolean value, If set to true, creates a schema file in metadata folder. -* By default set to false. will not create metadata folder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder persistSchemaFile(boolean persist); -</code></pre> -<pre><code>/** * sets the taskNo for the writer. SDKs concurrently running * will set taskNo in order to avoid conflicts in file's name during write. * @param taskNo is the TaskNo user wants to specify. @@ -540,7 +613,7 @@ public CarbonWriterBuilder taskNo(long taskNo); * g. complex_delimiter_level_2 -- value to Split the nested complexTypeData * h. quotechar * i. escapechar -* +* * Default values are as follows. * * a. bad_records_logger_enable -- "false" @@ -558,84 +631,76 @@ public CarbonWriterBuilder taskNo(long taskNo); public CarbonWriterBuilder withLoadOptions(Map<String, String> options); </code></pre> <pre><code>/** - * To support the table properties for sdk writer - * - * @param options key,value pair of create table properties. - * supported keys values are - * a. blocksize -- [1-2048] values in MB. Default value is 1024 - * b. blockletsize -- values in MB. Default value is 64 MB - * c. localDictionaryThreshold -- positive value, default is 10000 - * d. enableLocalDictionary -- true / false. Default is false - * e. sortcolumns -- comma separated column. "c1,c2". Default all dimensions are sorted. - * - * @return updated CarbonWriterBuilder - */ +* To support the table properties for sdk writer +* +* @param options key,value pair of create table properties. +* supported keys values are +* a. table_blocksize -- [1-2048] values in MB. Default value is 1024 +* b. table_blocklet_size -- values in MB. Default value is 64 MB +* c. local_dictionary_threshold -- positive value, default is 10000 +* d. local_dictionary_enable -- true / false. Default is false +* e. sort_columns -- comma separated column. "c1,c2". Default all dimensions are sorted. + If empty string "" is passed. No columns are sorted +* j. sort_scope -- "local_sort", "no_sort", "batch_sort". default value is "local_sort" +* k. long_string_columns -- comma separated string columns which are more than 32k length. +* default value is null. +* +* @return updated CarbonWriterBuilder +*/ public CarbonWriterBuilder withTableProperties(Map<String, String> options); </code></pre> <pre><code>/** -* this writer is not thread safe, use buildThreadSafeWriterForCSVInput in multi thread environment -* Build a {@link CarbonWriter}, which accepts row in CSV format object -* @param schema carbon Schema object {org.apache.carbondata.sdk.file.Schema} -* @return CSVCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException +* To make sdk writer thread safe. +* +* @param numOfThreads should number of threads in which writer is called in multi-thread scenario +* default sdk writer is not thread safe. +* can use one writer instance in one thread only. +* @return updated CarbonWriterBuilder */ -public CarbonWriter buildWriterForCSVInput(org.apache.carbondata.sdk.file.Schema schema) throws IOException, InvalidLoadOptionException; +public CarbonWriterBuilder withThreadSafe(short numOfThreads); </code></pre> <pre><code>/** -* Can use this writer in multi-thread instance. -* Build a {@link CarbonWriter}, which accepts row in CSV format -* @param schema carbon Schema object {org.apache.carbondata.sdk.file.Schema} -* @param numOfThreads number of threads() in which .write will be called. -* @return CSVCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException +* To support hadoop configuration +* +* @param conf hadoop configuration support, can set s3a AK,SK,end point and other conf with this +* @return updated CarbonWriterBuilder */ -public CarbonWriter buildThreadSafeWriterForCSVInput(Schema schema, short numOfThreads) - throws IOException, InvalidLoadOptionException; +public CarbonWriterBuilder withHadoopConf(Configuration conf) </code></pre> <pre><code>/** -* this writer is not thread safe, use buildThreadSafeWriterForAvroInput in multi thread environment -* Build a {@link CarbonWriter}, which accepts Avro format object -* @param avroSchema avro Schema object {org.apache.avro.Schema} -* @return AvroCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException +* to build a {@link CarbonWriter}, which accepts row in CSV format +* +* @param schema carbon Schema object {org.apache.carbondata.sdk.file.Schema} +* @return CarbonWriterBuilder */ -public CarbonWriter buildWriterForAvroInput(org.apache.avro.Schema schema) throws IOException, InvalidLoadOptionException; +public CarbonWriterBuilder withCsvInput(Schema schema); </code></pre> <pre><code>/** -* Can use this writer in multi-thread instance. -* Build a {@link CarbonWriter}, which accepts Avro object +* to build a {@link CarbonWriter}, which accepts Avro object +* * @param avroSchema avro Schema object {org.apache.avro.Schema} -* @param numOfThreads number of threads() in which .write will be called. -* @return AvroCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException +* @return CarbonWriterBuilder */ -public CarbonWriter buildThreadSafeWriterForAvroInput(org.apache.avro.Schema avroSchema, short numOfThreads) - throws IOException, InvalidLoadOptionException +public CarbonWriterBuilder withAvroInput(org.apache.avro.Schema avroSchema); </code></pre> <pre><code>/** -* this writer is not thread safe, use buildThreadSafeWriterForJsonInput in multi thread environment -* Build a {@link CarbonWriter}, which accepts Json object +* to build a {@link CarbonWriter}, which accepts Json object +* * @param carbonSchema carbon Schema object -* @return JsonCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException +* @return CarbonWriterBuilder */ -public JsonCarbonWriter buildWriterForJsonInput(Schema carbonSchema); +public CarbonWriterBuilder withJsonInput(Schema carbonSchema); </code></pre> <pre><code>/** -* Can use this writer in multi-thread instance. -* Build a {@link CarbonWriter}, which accepts Json object -* @param carbonSchema carbon Schema object -* @param numOfThreads number of threads() in which .write will be called. -* @return JsonCarbonWriter +* Build a {@link CarbonWriter} +* This writer is not thread safe, +* use withThreadSafe() configuration in multi thread environment +* +* @return CarbonWriter {AvroCarbonWriter/CSVCarbonWriter/JsonCarbonWriter based on Input Type } * @throws IOException * @throws InvalidLoadOptionException */ -public JsonCarbonWriter buildThreadSafeWriterForJsonInput(Schema carbonSchema, short numOfThreads) +public CarbonWriter build() throws IOException, InvalidLoadOptionException; </code></pre> <h3> <a id="class-orgapachecarbondatasdkfilecarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriter</h3> @@ -645,7 +710,6 @@ public JsonCarbonWriter buildThreadSafeWriterForJsonInput(Schema carbonSchema, s * which is one row of data. * If CSVCarbonWriter, object is of type String[], which is one row of data * If JsonCarbonWriter, object is of type String, which is one row of json -* Note: This API is not thread safe if writer is not built with number of threads argument. * @param object * @throws IOException */ @@ -793,17 +857,6 @@ External client can make use of this reader to read CarbonData files without Car */ public CarbonReaderBuilder projection(String[] projectionColumnNames); </code></pre> -<pre><code> /** - * Configure the transactional status of table - * If set to false, then reads the carbondata and carbonindex files from a flat folder structure. - * If set to true, then reads the carbondata and carbonindex files from segment folder structure. - * Default value is false - * - * @param isTransactionalTable whether is transactional table or not - * @return CarbonReaderBuilder object - */ - public CarbonReaderBuilder isTransactionalTable(boolean isTransactionalTable); -</code></pre> <pre><code> /** * Configure the filter expression for carbon reader * @@ -812,56 +865,13 @@ External client can make use of this reader to read CarbonData files without Car */ public CarbonReaderBuilder filter(Expression filterExpression); </code></pre> -<pre><code> /** - * Set the access key for S3 - * - * @param key the string of access key for different S3 type,like: fs.s3a.access.key - * @param value the value of access key - * @return CarbonWriterBuilder - */ - public CarbonReaderBuilder setAccessKey(String key, String value); -</code></pre> -<pre><code> /** - * Set the access key for S3. - * - * @param value the value of access key - * @return CarbonWriterBuilder object - */ - public CarbonReaderBuilder setAccessKey(String value); -</code></pre> -<pre><code> /** - * Set the secret key for S3 - * - * @param key the string of secret key for different S3 type,like: fs.s3a.secret.key - * @param value the value of secret key - * @return CarbonWriterBuilder object - */ - public CarbonReaderBuilder setSecretKey(String key, String value); -</code></pre> -<pre><code> /** - * Set the secret key for S3 - * - * @param value the value of secret key - * @return CarbonWriterBuilder object - */ - public CarbonReaderBuilder setSecretKey(String value); -</code></pre> -<pre><code> /** - * Set the endpoint for S3 - * - * @param key the string of endpoint for different S3 type,like: fs.s3a.endpoint - * @param value the value of endpoint - * @return CarbonWriterBuilder object - */ - public CarbonReaderBuilder setEndPoint(String key, String value); -</code></pre> -<pre><code> /** - * Set the endpoint for S3 - * - * @param value the value of endpoint - * @return CarbonWriterBuilder object - */ - public CarbonReaderBuilder setEndPoint(String value); +<pre><code>/** + * To support hadoop configuration + * + * @param conf hadoop configuration support, can set s3a AK,SK,end point and other conf with this + * @return updated CarbonReaderBuilder + */ + public CarbonReaderBuilder withHadoopConf(Configuration conf); </code></pre> <pre><code> /** * Build CarbonReader @@ -992,8 +1002,15 @@ public String getProperty(String key, String defaultValue); </code></pre> <p>Reference : <a href="./configuration-parameters.html">list of carbon properties</a></p> <script> -// Show selected style on nav item -$(function() { $('.b-nav__api').addClass('selected'); }); +$(function() { + // Show selected style on nav item + $('.b-nav__api').addClass('selected'); + + if (!$('.b-nav__api').parent().hasClass('nav__item__with__subs--expanded')) { + // Display api subnav items + $('.b-nav__api').parent().toggleClass('nav__item__with__subs--expanded'); + } +}); </script></div> </div> </div> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/segment-management-on-carbondata.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/segment-management-on-carbondata.html b/src/main/webapp/segment-management-on-carbondata.html index 1e6f61d..1d70f24 100644 --- a/src/main/webapp/segment-management-on-carbondata.html +++ b/src/main/webapp/segment-management-on-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/streaming-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/streaming-guide.html b/src/main/webapp/streaming-guide.html index 86f0385..e82b84a 100644 --- a/src/main/webapp/streaming-guide.html +++ b/src/main/webapp/streaming-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -360,7 +368,7 @@ streaming table using following DDL.</p> <p>At the begin of streaming ingestion, the system will try to acquire the table level lock of streaming.lock file. If the system isn't able to acquire the lock of this table, it will throw an InterruptedException.</p> <h2> <a id="create-streaming-segment" class="anchor" href="#create-streaming-segment" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create streaming segment</h2> -<p>The input data of streaming will be ingested into a segment of the CarbonData table, the status of this segment is streaming. CarbonData call it a streaming segment. The "tablestatus" file will record the segment status and data size. The user can use ?SHOW SEGMENTS FOR TABLE tableName? to check segment status.</p> +<p>The streaming data will be ingested into a separate segment of carbondata table, this segment is termed as streaming segment. The status of this segment will be recorded as "streaming" in "tablestatus" file along with its data size. You can use "SHOW SEGMENTS FOR TABLE tableName" to check segment status.</p> <p>After the streaming segment reaches the max size, CarbonData will change the segment status to "streaming finish" from "streaming", and create new "streaming" segment to continue to ingest streaming data.</p> <table> <thead> @@ -533,8 +541,9 @@ streaming table using following DDL.</p> | register TIMESTAMP, | updated TIMESTAMP |) - |STORED BY carbondata + |STORED AS carbondata |TBLPROPERTIES ( + | 'streaming'='source', | 'format'='csv', | 'path'='$csvDataDir' |) @@ -553,7 +562,7 @@ streaming table using following DDL.</p> | register TIMESTAMP, | updated TIMESTAMP |) - |STORED BY carbondata + |STORED AS carbondata |TBLPROPERTIES ( | 'streaming'='true' |) @@ -576,7 +585,7 @@ streaming table using following DDL.</p> sql("SHOW STREAMS [ON TABLE tableName]") </code></pre> <p>In above example, two table is created: source and sink. The <code>source</code> table's format is <code>csv</code> and <code>sink</code> table format is <code>carbon</code>. Then a streaming job is created to stream data from source table to sink table.</p> -<p>These two tables are normal carbon table, they can be queried independently.</p> +<p>These two tables are normal carbon tables, they can be queried independently.</p> <h3> <a id="streaming-job-management" class="anchor" href="#streaming-job-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Streaming Job Management</h3> <p>As above example shown:</p> @@ -601,18 +610,22 @@ streaming table using following DDL.</p> name STRING, age <span class="pl-k">INT</span> ) -STORED BY carbondata +STORED <span class="pl-k">AS</span> carbondata TBLPROPERTIES( - <span class="pl-s"><span class="pl-pds">'</span>format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>socket<span class="pl-pds">'</span></span>, - <span class="pl-s"><span class="pl-pds">'</span>host<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>localhost<span class="pl-pds">'</span></span>, - <span class="pl-s"><span class="pl-pds">'</span>port<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>8888<span class="pl-pds">'</span></span> + <span class="pl-s"><span class="pl-pds">'</span>streaming<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>source<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>socket<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>host<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>localhost<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>port<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>8888<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>record_format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>csv<span class="pl-pds">'</span></span>, <span class="pl-k">//</span> can be csv <span class="pl-k">or</span> json, default is csv + <span class="pl-s"><span class="pl-pds">'</span>delimiter<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>|<span class="pl-pds">'</span></span> )</pre></div> <p>will translate to</p> <div class="highlight highlight-source-scala"><pre>spark.readStream .schema(tableSchema) .format(<span class="pl-s"><span class="pl-pds">"</span>socket<span class="pl-pds">"</span></span>) .option(<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>localhost<span class="pl-pds">"</span></span>) - .option(<span class="pl-s"><span class="pl-pds">"</span>port<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>8888<span class="pl-pds">"</span></span>)</pre></div> + .option(<span class="pl-s"><span class="pl-pds">"</span>port<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>8888<span class="pl-pds">"</span></span>) + .option(<span class="pl-s"><span class="pl-pds">"</span>delimiter<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>|<span class="pl-pds">"</span></span>)</pre></div> </li> <li> <p>The sink table should have a TBLPROPERTY <code>'streaming'</code> equal to <code>true</code>, indicating it is a streaming table.</p> @@ -621,9 +634,23 @@ TBLPROPERTIES( <p>In the given STMPROPERTIES, user must specify <code>'trigger'</code>, its value must be <code>ProcessingTime</code> (In future, other value will be supported). User should also specify interval value for the streaming job.</p> </li> <li> -<p>If the schema specifid in sink table is different from CTAS, the streaming job will fail</p> +<p>If the schema specified in sink table is different from CTAS, the streaming job will fail</p> </li> </ul> +<p>For Kafka data source, create the source table by:</p> +<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">source</span>( + name STRING, + age <span class="pl-k">INT</span> +) +STORED <span class="pl-k">AS</span> carbondata +TBLPROPERTIES( + <span class="pl-s"><span class="pl-pds">'</span>streaming<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>source<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>kafka<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>kafka.bootstrap.servers<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>kafkaserver:9092<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>subscribe<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>test<span class="pl-pds">'</span></span> + <span class="pl-s"><span class="pl-pds">'</span>record_format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>csv<span class="pl-pds">'</span></span>, <span class="pl-k">//</span> can be csv <span class="pl-k">or</span> json, default is csv + <span class="pl-s"><span class="pl-pds">'</span>delimiter<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>|<span class="pl-pds">'</span></span> +)</pre></div> <h5> <a id="stop-stream" class="anchor" href="#stop-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>STOP STREAM</h5> <p>When this is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist.</p> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/supported-data-types-in-carbondata.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/supported-data-types-in-carbondata.html b/src/main/webapp/supported-data-types-in-carbondata.html index f346052..6c692b8 100644 --- a/src/main/webapp/supported-data-types-in-carbondata.html +++ b/src/main/webapp/supported-data-types-in-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -223,7 +231,10 @@ <li>BIGINT</li> <li>DOUBLE</li> <li>DECIMAL</li> +<li>FLOAT</li> +<li>BYTE</li> </ul> +<p><strong>NOTE</strong>: Float and Bytes are only supported for SDK and FileFormat.</p> </li> <li> <p>Date/Time Types</p> @@ -249,6 +260,8 @@ Please refer to TBLProperties in <a href="./ddl-of-carbondata.html#create-table" </li> <li>structs: STRUCT<code><col_name : data_type COMMENT col_comment, ...></code> </li> +<li>maps: MAP<code><primitive_type, data_type></code> +</li> </ul> <p><strong>NOTE</strong>: Only 2 level complex type schema is supported for now.</p> </li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/timeseries-datamap-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/timeseries-datamap-guide.html b/src/main/webapp/timeseries-datamap-guide.html index 73a4580..d34a013 100644 --- a/src/main/webapp/timeseries-datamap-guide.html +++ b/src/main/webapp/timeseries-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/main/webapp/usecases.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/usecases.html b/src/main/webapp/usecases.html index cb309dd..3ff114a 100644 --- a/src/main/webapp/usecases.html +++ b/src/main/webapp/usecases.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -258,7 +266,7 @@ <a id="detailed-queries-in-the-telecom-scenario" class="anchor" href="#detailed-queries-in-the-telecom-scenario" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Detailed Queries in the Telecom scenario</h2> <h3> <a id="scenario" class="anchor" href="#scenario" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Scenario</h3> -<p>User wants to analyse all the CHR(Call History Record) and MR(Measurement Records) of the mobile subscribers in order to identify the service failures within 10 secs.Also user wants to run machine learning models on the data to fairly estimate the reasons and time of probable failures and take action ahead to meet the SLA(Service Level Agreements) of VIP customers.</p> +<p>User wants to analyse all the CHR(Call History Record) and MR(Measurement Records) of the mobile subscribers in order to identify the service failures within 10 secs. Also user wants to run machine learning models on the data to fairly estimate the reasons and time of probable failures and take action ahead to meet the SLA(Service Level Agreements) of VIP customers.</p> <h3> <a id="challenges" class="anchor" href="#challenges" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Challenges</h3> <ul> @@ -271,7 +279,7 @@ <a id="solution" class="anchor" href="#solution" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Solution</h3> <p>Setup a Hadoop + Spark + CarbonData cluster managed by YARN.</p> <p>Proposed the following configurations for CarbonData.(These tunings were proposed before CarbonData introduced SORT_COLUMNS parameter using which the sort order and schema order could be different.)</p> -<p>Add the frequently used columns to the left of the table definition.Add it in the increasing order of cardinality.It was suggested to keep msisdn,imsi columns in the beginning of the schema.With latest CarbonData, SORT_COLUMNS needs to be configured msisdn,imsi in the beginning.</p> +<p>Add the frequently used columns to the left of the table definition. Add it in the increasing order of cardinality. It was suggested to keep msisdn,imsi columns in the beginning of the schema. With latest CarbonData, SORT_COLUMNS needs to be configured msisdn,imsi in the beginning.</p> <p>Add timestamp column to the right of the schema as it is naturally increasing.</p> <p>Create two separate YARN queues for Query and Data Loading.</p> <p>Apart from these, the following CarbonData configuration was suggested to be configured in the cluster.</p> @@ -319,7 +327,7 @@ <td>Data Loading</td> <td>carbon.use.local.dir</td> <td>TRUE</td> -<td>yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications.Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance</td> +<td>yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications. Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance</td> </tr> <tr> <td>Data Loading</td> @@ -381,7 +389,7 @@ <a id="detailed-queries-in-the-smart-city-scenario" class="anchor" href="#detailed-queries-in-the-smart-city-scenario" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Detailed Queries in the Smart City scenario</h2> <h3> <a id="scenario-1" class="anchor" href="#scenario-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Scenario</h3> -<p>User wants to analyse the person/vehicle movement and behavior during a certain time period.This output data needs to be joined with a external table for Human details extraction.The query will be run with different time period as filter to identify potential behavior mismatch.</p> +<p>User wants to analyse the person/vehicle movement and behavior during a certain time period. This output data needs to be joined with a external table for Human details extraction. The query will be run with different time period as filter to identify potential behavior mismatch.</p> <h3> <a id="challenges-1" class="anchor" href="#challenges-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Challenges</h3> <p>Data generated per day is very huge.Data needs to be loaded multiple times per day to accomodate the incoming data size.</p> @@ -414,13 +422,13 @@ <td>Data Loading</td> <td>enable.unsafe.sort</td> <td>TRUE</td> -<td>Temporary data generated during sort is huge which causes GC bottlenecks.Using unsafe reduces the pressure on GC</td> +<td>Temporary data generated during sort is huge which causes GC bottlenecks. Using unsafe reduces the pressure on GC</td> </tr> <tr> <td>Data Loading</td> <td>enable.offheap.sort</td> <td>TRUE</td> -<td>Temporary data generated during sort is huge which causes GC bottlenecks.Using offheap reduces the pressure on GC.offheap can be accessed through java unsafe.hence enable.unsafe.sort needs to be true</td> +<td>Temporary data generated during sort is huge which causes GC bottlenecks. Using offheap reduces the pressure on GC.offheap can be accessed through java unsafe.hence enable.unsafe.sort needs to be true</td> </tr> <tr> <td>Data Loading</td> @@ -444,7 +452,7 @@ <td>Data Loading</td> <td>table_blocksize</td> <td>512</td> -<td>To efficiently schedule multiple tasks during query.This size depends on data scenario.If data is such that the filters would select less number of blocklets to scan, keeping higher number works well.If the number blocklets to scan is more, better to reduce the size as more tasks can be scheduled in parallel.</td> +<td>To efficiently schedule multiple tasks during query. This size depends on data scenario.If data is such that the filters would select less number of blocklets to scan, keeping higher number works well.If the number blocklets to scan is more, better to reduce the size as more tasks can be scheduled in parallel.</td> </tr> <tr> <td>Data Loading</td> @@ -456,7 +464,7 @@ <td>Data Loading</td> <td>carbon.use.local.dir</td> <td>TRUE</td> -<td>yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications.Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance</td> +<td>yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications. Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance</td> </tr> <tr> <td>Data Loading</td> @@ -468,7 +476,7 @@ <td>Data Loading</td> <td>sort.inmemory.size.in.mb</td> <td>92160</td> -<td>Memory allocated to do inmemory sorting.When more memory is available in the node, configuring this will retain more sort blocks in memory so that the merge sort is faster due to no/very less IO</td> +<td>Memory allocated to do inmemory sorting. When more memory is available in the node, configuring this will retain more sort blocks in memory so that the merge sort is faster due to no/very less IO</td> </tr> <tr> <td>Compaction</td> @@ -486,7 +494,7 @@ <td>Compaction</td> <td>carbon.enable.auto.load.merge</td> <td>FALSE</td> -<td>Doing auto minor compaction is costly process as data size is huge.Perform manual compaction when the cluster is less loaded</td> +<td>Doing auto minor compaction is costly process as data size is huge.Perform manual compaction when the cluster is less loaded</td> </tr> <tr> <td>Query</td> @@ -498,13 +506,13 @@ <td>Query</td> <td>enable.unsafe.in.query.procressing</td> <td>true</td> -<td>Data that needs to be scanned in huge which in turn generates more short lived Java objects.This cause pressure of GC.using unsafe and offheap will reduce the GC overhead</td> +<td>Data that needs to be scanned in huge which in turn generates more short lived Java objects. This cause pressure of GC.using unsafe and offheap will reduce the GC overhead</td> </tr> <tr> <td>Query</td> <td>use.offheap.in.query.processing</td> <td>true</td> -<td>Data that needs to be scanned in huge which in turn generates more short lived Java objects.This cause pressure of GC.using unsafe and offheap will reduce the GC overhead.offheap can be accessed through java unsafe.hence enable.unsafe.in.query.procressing needs to be true</td> +<td>Data that needs to be scanned in huge which in turn generates more short lived Java objects. This cause pressure of GC.using unsafe and offheap will reduce the GC overhead.offheap can be accessed through java unsafe.hence enable.unsafe.in.query.procressing needs to be true</td> </tr> <tr> <td>Query</td> @@ -516,7 +524,7 @@ <td>Query</td> <td>carbon.unsafe.working.memory.in.mb</td> <td>10240</td> -<td>Amount of memory to use for offheap operations.Can increase this memory based on the data size</td> +<td>Amount of memory to use for offheap operations, you can increase this memory based on the data size</td> </tr> </tbody> </table> @@ -565,7 +573,7 @@ <li>Create pre-aggregate tables for non timestamp based group by queries</li> <li>For queries containing group by date, create timeseries based Datamap(pre-aggregate) tables so that the data is rolled up during creation and fetch is faster</li> <li>Reduce the Spark shuffle partitions.(In our configuration on 14 node cluster, it was reduced to 35 from default of 200)</li> -<li>Enable global dictionary for columns which have less cardinalities.Aggregation can be done on encoded data, there by improving the performance</li> +<li>Enable global dictionary for columns which have less cardinalities. Aggregation can be done on encoded data, there by improving the performance</li> <li>For columns whose cardinality is high,enable the local dictionary so that store size is less and can take dictionary benefit for scan</li> </ul> <h2> @@ -575,13 +583,13 @@ <p>Need to support storing of continously arriving data and make it available immediately for query.</p> <h3> <a id="challenges-3" class="anchor" href="#challenges-3" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Challenges</h3> -<p>When the data ingestion is near real time and the data needs to be available for query immediately, usual scenario is to do data loading in micro batches.But this causes the problem of generating many small files.This poses two problems:</p> +<p>When the data ingestion is near real time and the data needs to be available for query immediately, usual scenario is to do data loading in micro batches.But this causes the problem of generating many small files. This poses two problems:</p> <ol> <li>Small file handling in HDFS is inefficient</li> <li>CarbonData will suffer in query performance as all the small files will have to be queried when filter is on non time column</li> </ol> <p>CarbonData will suffer in query performance as all the small files will have to be queried when filter is on non time column.</p> -<p>Since data is continouly arriving, allocating resources for compaction might not be feasible.</p> +<p>Since data is continously arriving, allocating resources for compaction might not be feasible.</p> <h3> <a id="goal-1" class="anchor" href="#goal-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Goal</h3> <ol> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/site/markdown/CSDK-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/CSDK-guide.md b/src/site/markdown/CSDK-guide.md new file mode 100644 index 0000000..c4f4a31 --- /dev/null +++ b/src/site/markdown/CSDK-guide.md @@ -0,0 +1,197 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CSDK Guide + +CarbonData CSDK provides C++ interface to write and read carbon file. +CSDK use JNI to invoke java SDK in C++ code. + + +# CSDK Reader +This CSDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files in C++ +code and without CarbonSession. + + +In the carbon jars package, there exist a carbondata-sdk.jar, +including SDK reader for CSDK. +## Quick example +``` +// 1. init JVM +JavaVM *jvm; +JNIEnv *initJVM() { + JNIEnv *env; + JavaVMInitArgs vm_args; + int parNum = 3; + int res; + JavaVMOption options[parNum]; + + options[0].optionString = "-Djava.compiler=NONE"; + options[1].optionString = "-Djava.class.path=../../sdk/target/carbondata-sdk.jar"; + options[2].optionString = "-verbose:jni"; + vm_args.version = JNI_VERSION_1_8; + vm_args.nOptions = parNum; + vm_args.options = options; + vm_args.ignoreUnrecognized = JNI_FALSE; + + res = JNI_CreateJavaVM(&jvm, (void **) &env, &vm_args); + if (res < 0) { + fprintf(stderr, "\nCan't create Java VM\n"); + exit(1); + } + + return env; +} + +// 2. create carbon reader and read data +// 2.1 read data from local disk +/** + * test read data from local disk, without projection + * + * @param env jni env + * @return + */ +bool readFromLocalWithoutProjection(JNIEnv *env) { + + CarbonReader carbonReaderClass; + carbonReaderClass.builder(env, "../resources/carbondata", "test"); + carbonReaderClass.build(); + + while (carbonReaderClass.hasNext()) { + jobjectArray row = carbonReaderClass.readNextRow(); + jsize length = env->GetArrayLength(row); + int j = 0; + for (j = 0; j < length; j++) { + jobject element = env->GetObjectArrayElement(row, j); + char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE); + printf("%s\t", str); + } + printf("\n"); + } + carbonReaderClass.close(); +} + +// 2.2 read data from S3 + +/** + * read data from S3 + * parameter is ak sk endpoint + * + * @param env jni env + * @param argv argument vector + * @return + */ +bool readFromS3(JNIEnv *env, char *argv[]) { + CarbonReader reader; + + char *args[3]; + // "your access key" + args[0] = argv[1]; + // "your secret key" + args[1] = argv[2]; + // "your endPoint" + args[2] = argv[3]; + + reader.builder(env, "s3a://sdk/WriterOutput", "test"); + reader.withHadoopConf(3, args); + reader.build(); + printf("\nRead data from S3:\n"); + while (reader.hasNext()) { + jobjectArray row = reader.readNextRow(); + jsize length = env->GetArrayLength(row); + + int j = 0; + for (j = 0; j < length; j++) { + jobject element = env->GetObjectArrayElement(row, j); + char *str = (char *) env->GetStringUTFChars((jstring) element, JNI_FALSE); + printf("%s\t", str); + } + printf("\n"); + } + + reader.close(); +} + +// 3. destory JVM + (jvm)->DestroyJavaVM(); +``` +Find example code at main.cpp of CSDK module + +## API List +``` + /** + * create a CarbonReaderBuilder object for building carbonReader, + * CarbonReaderBuilder object can configure different parameter + * + * @param env JNIEnv + * @param path data store path + * @param tableName table name + * @return CarbonReaderBuilder object + */ + jobject builder(JNIEnv *env, char *path, char *tableName); + + /** + * Configure the projection column names of carbon reader + * + * @param argc argument counter + * @param argv argument vector + * @return CarbonReaderBuilder object + */ + jobject projection(int argc, char *argv[]); + + /** + * build carbon reader with argument vector + * it support multiple parameter + * like: key=value + * for example: fs.s3a.access.key=XXXX, XXXX is user's access key value + * + * @param argc argument counter + * @param argv argument vector + * @return CarbonReaderBuilder object + **/ + jobject withHadoopConf(int argc, char *argv[]); + + /** + * build carbonReader object for reading data + * it support read data from load disk + * + * @return carbonReader object + */ + jobject build(); + + /** + * Whether it has next row data + * + * @return boolean value, if it has next row, return true. if it hasn't next row, return false. + */ + jboolean hasNext(); + + /** + * read next row from data + * + * @return object array of one row + */ + jobjectArray readNextRow(); + + /** + * close the carbon reader + * + * @return boolean value + */ + jboolean close(); + +``` http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/site/markdown/bloomfilter-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/bloomfilter-datamap-guide.md b/src/site/markdown/bloomfilter-datamap-guide.md index b2e7d60..fb244fe 100644 --- a/src/site/markdown/bloomfilter-datamap-guide.md +++ b/src/site/markdown/bloomfilter-datamap-guide.md @@ -15,7 +15,7 @@ limitations under the License. --> -# CarbonData BloomFilter DataMap (Alpha Feature) +# CarbonData BloomFilter DataMap * [DataMap Management](#datamap-management) * [BloomFilter Datamap Introduction](#bloomfilter-datamap-introduction) @@ -46,7 +46,7 @@ Showing all DataMaps on this table ``` Disable Datamap -> The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This will only take effect current session. +> The datamap by default is enabled. To support tuning on query, we can disable a specific datamap during query to observe whether we can gain performance enhancement from it. This is effective only for current session. ``` // disable the datamap @@ -82,7 +82,7 @@ and we always query on `id` and `name` with precise value. since `id` is in the sort_columns and it is orderd, query on it will be fast because CarbonData can skip all the irrelative blocklets. But queries on `name` may be bad since the blocklet minmax may not help, -because in each blocklet the range of the value of `name` may be the same -- all from A*~z*. +because in each blocklet the range of the value of `name` may be the same -- all from A* to z*. In this case, user can create a BloomFilter datamap on column `name`. Moreover, user can also create a BloomFilter datamap on the sort_columns. This is useful if user has too many segments and the range of the value of sort_columns are almost the same. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/src/site/markdown/carbon-as-spark-datasource-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/carbon-as-spark-datasource-guide.md b/src/site/markdown/carbon-as-spark-datasource-guide.md new file mode 100644 index 0000000..bc56a54 --- /dev/null +++ b/src/site/markdown/carbon-as-spark-datasource-guide.md @@ -0,0 +1,99 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData as Spark's Datasource + +The CarbonData fileformat is now integrated as Spark datasource for read and write operation without using CarbonSession. This is useful for users who wants to use carbondata as spark's data source. + +**Note:** You can only apply the functions/features supported by spark datasource APIs, functionalities supported would be similar to Parquet. The carbon session features are not supported. + +# Create Table with DDL + +Now you can create Carbon table using Spark's datasource DDL syntax. + +``` + CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name + [(col_name1 col_type1 [COMMENT col_comment1], ...)] + USING CARBON + [OPTIONS (key1=val1, key2=val2, ...)] + [PARTITIONED BY (col_name1, col_name2, ...)] + [CLUSTERED BY (col_name3, col_name4, ...) INTO num_buckets BUCKETS] + [LOCATION path] + [COMMENT table_comment] + [TBLPROPERTIES (key1=val1, key2=val2, ...)] + [AS select_statement] +``` + +## Supported OPTIONS + +| Property | Default Value | Description | +|-----------|--------------|------------| +| table_blocksize | 1024 | Size of blocks to write onto hdfs. For more details, see [Table Block Size Configuration](./ddl-of-carbondata.md#table-block-size-configuration). | +| table_blocklet_size | 64 | Size of blocklet to write. | +| local_dictionary_threshold | 10000 | Cardinality upto which the local dictionary can be generated. For more details, see [Local Dictionary Configuration](./ddl-of-carbondata.md#local-dictionary-configuration). | +| local_dictionary_enable | false | Enable local dictionary generation. For more details, see [Local Dictionary Configuration](./ddl-of-carbondata.md#local-dictionary-configuration). | +| sort_columns | all dimensions are sorted | Columns to include in sort and its order of sort. For more details, see [Sort Columns Configuration](./ddl-of-carbondata.md#sort-columns-configuration). | +| sort_scope | local_sort | Sort scope of the load.Options include no sort, local sort, batch sort, and global sort. For more details, see [Sort Scope Configuration](./ddl-of-carbondata.md#sort-scope-configuration). | +| long_string_columns | null | Comma separated string/char/varchar columns which are more than 32k length. For more details, see [String longer than 32000 characters](./ddl-of-carbondata.md#string-longer-than-32000-characters). | + +## Example + +``` + CREATE TABLE CARBON_TABLE (NAME STRING) USING CARBON OPTIONS('table_block_size'='256') +``` + +# Using DataFrame + +Carbon format can be used in dataframe also. Following are the ways to use carbon format in dataframe. + +Write carbon using dataframe +``` +df.write.format("carbon").save(path) +``` + +Read carbon using dataframe +``` +val df = spark.read.format("carbon").load(path) +``` + +## Example + +``` +import org.apache.spark.sql.SparkSession + +val spark = SparkSession + .builder() + .appName("Spark SQL basic example") + .config("spark.some.config.option", "some-value") + .getOrCreate() + +// For implicit conversions like converting RDDs to DataFrames +import spark.implicits._ +val df = spark.sparkContext.parallelize(1 to 10 * 10 * 1000) + .map(x => (r.nextInt(100000), "name" + x % 8, "city" + x % 50, BigDecimal.apply(x % 60))) + .toDF("ID", "name", "city", "age") + +// Write to carbon format +df.write.format("carbon").save("/user/person_table") + +// Read carbon using dataframe +val dfread = spark.read.format("carbon").load("/user/person_table") +dfread.show() +``` + +Reference : [list of carbon properties](./configuration-parameters.md) +
