http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/quick-start-guide.html ---------------------------------------------------------------------- diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html index ea88086..f3e8a8f 100644 --- a/content/quick-start-guide.html +++ b/content/quick-start-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -217,10 +217,10 @@ <a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h2> <ul> <li> -<p>Spark 2.2.1 version is installed and running.CarbonData supports Spark versions upto 2.2.1.Please follow steps described in <a href="https://spark.apache.org/docs/latest"; target=_blank rel="nofollow">Spark docs website</a> for installing and running Spark.</p> +<p>CarbonData supports Spark versions upto 2.2.1.Please download Spark package from <a href="https://spark.apache.org/downloads.html"; target=_blank rel="nofollow">Spark website</a></p> </li> <li> -<p>Create a sample.csv file using the following commands. The CSV file is required for loading data into CarbonData.</p> +<p>Create a sample.csv file using the following commands. The CSV file is required for loading data into CarbonData</p> <pre><code>cd carbondata cat > sample.csv << EOF id,name,city,age @@ -232,21 +232,18 @@ EOF </li> </ul> <h2> -<a id="deployment-modes" class="anchor" href="#deployment-modes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deployment modes</h2> +<a id="integration" class="anchor" href="#integration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration</h2> <p>CarbonData can be integrated with Spark and Presto Execution Engines.The below documentation guides on Installing and Configuring with these execution engines.</p> <h3> <a id="spark" class="anchor" href="#spark" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Spark</h3> <p><a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Installing and Configuring CarbonData to run locally with Spark Shell</a></p> <p><a href="#installing-and-configuring-carbondata-on-standalone-spark-cluster">Installing and Configuring CarbonData on Standalone Spark Cluster</a></p> <p><a href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster">Installing and Configuring CarbonData on Spark on YARN Cluster</a></p> +<p><a href="#query-execution-using-carbondata-thrift-server">Installing and Configuring CarbonData Thrift Server for Query Execution</a></p> <h3> <a id="presto" class="anchor" href="#presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Presto</h3> <p><a href="#installing-and-configuring-carbondata-on-presto">Installing and Configuring CarbonData on Presto</a></p> <h2> -<a id="querying-data" class="anchor" href="#querying-data" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying Data</h2> -<p><a href="#query-execution-using-carbondata-thrift-server">Query Execution using CarbonData Thrift Server</a></p> -<h2></h2> -<h2> <a id="installing-and-configuring-carbondata-to-run-locally-with-spark-shell" class="anchor" href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData to run locally with Spark Shell</h2> <p>Apache Spark Shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. Please visit <a href="http://spark.apache.org/docs/latest/"; target=_blank rel="nofollow">Apache Spark Documentation</a> for more details on Spark shell.</p> <h4> @@ -275,12 +272,12 @@ import org.apache.spark.sql.CarbonSession._ <h6> <a id="creating-a-table" class="anchor" href="#creating-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h6> <pre><code>scala>carbon.sql("CREATE TABLE - IF NOT EXISTS test_table( - id string, - name string, - city string, - age Int) - STORED BY 'carbondata'") + IF NOT EXISTS test_table( + id string, + name string, + city string, + age Int) + STORED AS carbondata") </code></pre> <h6> <a id="loading-data-to-a-table" class="anchor" href="#loading-data-to-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h6> @@ -558,12 +555,11 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store </code></pre> <h2> <a id="installing-and-configuring-carbondata-on-presto" class="anchor" href="#installing-and-configuring-carbondata-on-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Presto</h2> -<ul> -<li> +<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto.User need to create CarbonData Table and load data into it +either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a>. +Once the table is created,it can be queried from Presto.</strong></p> <h3> <a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3> -</li> -</ul> <ol> <li> <p>Download the 0.187 version of Presto using: @@ -684,12 +680,23 @@ discovery.uri=<coordinator_ip>:8086 <pre><code>select * from system.runtime.nodes; </code></pre> <p>Now you can use the Presto CLI on the coordinator to query data sources in the catalog using the Presto workers.</p> +<p>List the schemas(databases) available</p> +<pre><code>show schemas; +</code></pre> +<p>Selected the schema where CarbonData table resides</p> +<pre><code>use carbonschema; +</code></pre> +<p>List the available tables</p> +<pre><code>show tables; +</code></pre> +<p>Query from the available tables</p> +<pre><code>select * from carbon_table; +</code></pre> <p><strong>Note :</strong> Create Tables and data loads should be done before executing queries as we can not create carbon table from this interface.</p> <script> // Show selected style on nav item $(function() { $('.b-nav__quickstart').addClass('selected'); }); -</script> -</div> +</script></div> </div> </div> </div> @@ -704,4 +711,4 @@ $(function() { $('.b-nav__quickstart').addClass('selected'); }); </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/release-guide.html ---------------------------------------------------------------------- diff --git a/content/release-guide.html b/content/release-guide.html index fb51368..cb47540 100644 --- a/content/release-guide.html +++ b/content/release-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -553,8 +553,7 @@ contributions.</p> <script> // Show selected style on nav item $(function() { $('.b-nav__release').addClass('selected'); }); -</script> -</div> +</script></div> </div> </div> </div> @@ -569,4 +568,4 @@ $(function() { $('.b-nav__release').addClass('selected'); }); </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/s3-guide.html ---------------------------------------------------------------------- diff --git a/content/s3-guide.html b/content/s3-guide.html index 9042b19..57af913 100644 --- a/content/s3-guide.html +++ b/content/s3-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -277,8 +277,7 @@ to a HDFS directory.</li> <script> // Show selected style on nav item $(function() { $('.b-nav__s3').addClass('selected'); }); -</script> -</div> +</script></div> </div> </div> </div> @@ -293,4 +292,4 @@ $(function() { $('.b-nav__s3').addClass('selected'); }); </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/sdk-guide.html ---------------------------------------------------------------------- diff --git a/content/sdk-guide.html b/content/sdk-guide.html index 95ae214..a252965 100644 --- a/content/sdk-guide.html +++ b/content/sdk-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -212,9 +212,14 @@ <div> <h1> <a id="sdk-guide" class="anchor" href="#sdk-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Guide</h1> -<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader.</p> +<p>CarbonData provides SDK to facilitate</p> +<ol> +<li><a href="#sdk-writer">Writing carbondata files from other application which does not use Spark</a></li> +<li><a href="#sdk-reader">Reading carbondata files from other application which does not use Spark</a></li> +</ol> <h1> <a id="sdk-writer" class="anchor" href="#sdk-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Writer</h1> +<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader.</p> <p>This SDK writer, writes carbondata file and carbonindex file at a given path. External client can make use of this writer to convert other format data or live data to create carbondata and index files. These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present.</p> @@ -989,8 +994,7 @@ public String getProperty(String key, String defaultValue); <script> // Show selected style on nav item $(function() { $('.b-nav__api').addClass('selected'); }); -</script> -</div> +</script></div> </div> </div> </div> @@ -1005,4 +1009,4 @@ $(function() { $('.b-nav__api').addClass('selected'); }); </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/sdk-writer-guide.html ---------------------------------------------------------------------- diff --git a/content/sdk-writer-guide.html b/content/sdk-writer-guide.html deleted file mode 100644 index 5553194..0000000 --- a/content/sdk-writer-guide.html +++ /dev/null @@ -1,549 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> - <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> - <title>CarbonData</title> - <style> - - </style> - <!-- Bootstrap --> - - <link rel="stylesheet" href="css/bootstrap.min.css"> - <link href="css/style.css" rel="stylesheet"> - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> - <![endif]--> - <script src="js/jquery.min.js"></script> - <script src="js/bootstrap.min.js"></script> - - -</head> -<body> -<header> - <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> - <div class="container"> - <div class="navbar-header"> - <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" - class="navbar-toggle collapsed" type="button"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a href="index.html" class="logo"> - <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> - </a> - </div> - <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> - <ul class="nav navbar-nav navbar-right navlist-custom"> - <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> - </li> - <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false"> Download <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; - target="_blank">Apache CarbonData 1.4.0</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; - target="_blank">Apache CarbonData 1.3.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"; - target="_blank">Apache CarbonData 1.2.0</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"; - target="_blank">Apache CarbonData 1.1.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"; - target="_blank">Apache CarbonData 1.1.0</a></li> - <li> - <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"; - target="_blank">Apache CarbonData 1.0.0</a></li> - <li> - <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"; - target="_blank">Apache CarbonData 0.2.0</a></li> - <li> - <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"; - target="_blank">Apache CarbonData 0.1.1</a></li> - <li> - <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"; - target="_blank">Apache CarbonData 0.1.0</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; - target="_blank">Release Archive</a></li> - </ul> - </li> - <li><a href="mainpage.html" class="active">Documentation</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false">Community <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; - target="_blank">Contributing to CarbonData</a></li> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; - target="_blank">Release Guide</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; - target="_blank">Project PMC and Committers</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; - target="_blank">CarbonData Meetups</a></li> - <li><a href="security.html">Apache CarbonData Security</a></li> - <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache - Jira</a></li> - <li><a href="videogallery.html">CarbonData Videos </a></li> - </ul> - </li> - <li class="dropdown"> - <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li class="dropdown"> - <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li> - <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> - - </li> - - </ul> - </div><!--/.nav-collapse --> - <div id="search-box"> - <form method="get" action="http://www.google.com/search"; target="_blank"> - <div class="search-block"> - <table border="0" cellpadding="0" width="100%"> - <tr> - <td style="width:80%"> - <input type="text" name="q" size=" 5" maxlength="255" value="" - class="search-input" placeholder="Search...." required/> - </td> - <td style="width:20%"> - <input type="submit" value="Search"/></td> - </tr> - <tr> - <td align="left" style="font-size:75%" colspan="2"> - <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> - <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> - </td> - </tr> - </table> - </div> - </form> - </div> - </div> - </nav> -</header> <!-- end Header part --> - -<div class="fixed-padding"></div> <!-- top padding with fixde header --> - -<section><!-- Dashboard nav --> - <div class="container-fluid q"> - <div class="col-sm-12 col-md-12 maindashboard"> - <div class="row"> - <section> - <div style="padding:10px 15px;"> - <div id="viewpage" name="viewpage"> - <div class="row"> - <div class="col-sm-12 col-md-12"> - <div><h1> -<a id="sdk-writer-guide" class="anchor" href="#sdk-writer-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Writer Guide</h1> -<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar. -This SDK writer, writes carbondata file and carbonindex file at a given path. -External client can make use of this writer to convert other format data or live data to create carbondata and index files. -These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present.</p> -<h2> -<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2> -<h3> -<a id="example-with-csv-format" class="anchor" href="#example-with-csv-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example with csv format</h3> -<div class="highlight highlight-source-java"><pre> <span class="pl-k">import</span> <span class="pl-smi">java.io.IOException</span>; - - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException</span>; - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.DataTypes</span>; - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>; - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriterBuilder</span>; - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>; - <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Schema</span>; - - <span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdk</span> { - - <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">main</span>(<span class="pl-k">String</span>[] <span class="pl-v">args</span>) <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { - testSdkWriter(); - } - - <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">testSdkWriter</span>() <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { - <span class="pl-smi">String</span> path <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/home/root1/Documents/ab/temp<span class="pl-pds">"</span></span>; - - <span class="pl-k">Field</span>[] fields <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>[<span class="pl-c1">2</span>]; - fields[<span class="pl-c1">0</span>] <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>(<span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>, <span class="pl-smi">DataTypes</span><span class="pl-c1"><span class="pl-k">.</span>STRING</span>); - fields[<span class="pl-c1">1</span>] <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>(<span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>, <span class="pl-smi">DataTypes</span><span class="pl-c1"><span class="pl-k">.</span>INT</span>); - - <span class="pl-smi">Schema</span> schema <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Schema</span>(fields); - - <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>withSchema(schema)<span class="pl-k">.</span>outputPath(path); - - <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>buildWriterForCSVInput(); - - <span class="pl-k">int</span> rows <span class="pl-k">=</span> <span class="pl-c1">5</span>; - <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> rows; i<span class="pl-k">++</span>) { - writer<span class="pl-k">.</span>write(<span class="pl-k">new</span> <span class="pl-smi">String</span>[] { <span class="pl-s"><span class="pl-pds">"</span>robot<span class="pl-pds">"</span></span> <span class="pl-k">+</span> (i <span class="pl-k">%</span> <span class="pl-c1">10</span>), <span class="pl-smi">String</span><span class="pl-k">.</span>valueOf(i) }); - } - writer<span class="pl-k">.</span>close(); - } - }</pre></div> -<h3> -<a id="example-with-avro-format" class="anchor" href="#example-with-avro-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example with Avro format</h3> -<div class="highlight highlight-source-java"><pre><span class="pl-k">import</span> <span class="pl-smi">java.io.IOException</span>; - -<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException</span>; -<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.DataTypes</span>; -<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.AvroCarbonWriter</span>; -<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>; -<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>; - -<span class="pl-k">import</span> <span class="pl-smi">org.apache.avro.generic.GenericData</span>; -<span class="pl-k">import</span> <span class="pl-smi">org.apache.commons.lang.CharEncoding</span>; - -<span class="pl-k">import</span> <span class="pl-smi">tech.allegro.schema.json2avro.converter.JsonAvroConverter</span>; - -<span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdkAvro</span> { - - <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">main</span>(<span class="pl-k">String</span>[] <span class="pl-v">args</span>) <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { - testSdkWriter(); - } - - - <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">testSdkWriter</span>() <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { - <span class="pl-smi">String</span> path <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>./AvroCarbonWriterSuiteWriteFiles<span class="pl-pds">"</span></span>; - <span class="pl-c"><span class="pl-c">//</span> Avro schema</span> - <span class="pl-smi">String</span> avroSchema <span class="pl-k">=</span> - <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-pds">"</span></span> <span class="pl-k">+</span> - <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>record<span class="pl-cce">\"</span>,<span class="pl-pds">"</span></span> <span class="pl-k">+</span> - <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>Acme<span class="pl-cce">\"</span>,<span class="pl-pds">"</span></span> <span class="pl-k">+</span> - <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>fields<span class="pl-cce">\"</span> : [<span class="pl-pds">"</span></span> - <span class="pl-k">+</span> <span class="pl-s"><span class="pl-pds">"</span>{ <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>fname<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>string<span class="pl-cce">\"</span> },<span class="pl-pds">"</span></span> - <span class="pl-k">+</span> <span class="pl-s"><span class="pl-pds">"</span>{ <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>age<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>int<span class="pl-cce">\"</span> }]<span class="pl-pds">"</span></span> <span class="pl-k">+</span> - <span class="pl-s"><span class="pl-pds">"</span>}<span class="pl-pds">"</span></span>; - - <span class="pl-smi">String</span> json <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-cce">\"</span>fname<span class="pl-cce">\"</span>:<span class="pl-cce">\"</span>bob<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>age<span class="pl-cce">\"</span>:10}<span class="pl-pds">"</span></span>; - - <span class="pl-c"><span class="pl-c">//</span> conversion to GenericData.Record</span> - <span class="pl-smi">JsonAvroConverter</span> converter <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">JsonAvroConverter</span>(); - <span class="pl-smi">GenericData</span><span class="pl-k">.</span><span class="pl-smi">Record</span> record <span class="pl-k">=</span> converter<span class="pl-k">.</span>convertToGenericDataRecord( - json<span class="pl-k">.</span>getBytes(<span class="pl-smi">CharEncoding</span><span class="pl-c1"><span class="pl-k">.</span>UTF_8</span>), <span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema)); - - <span class="pl-c"><span class="pl-c">//</span> prepare carbon schema from avro schema </span> - <span class="pl-smi">org.apache.carbondata.sdk.file<span class="pl-k">.</span>Schema</span> carbonSchema <span class="pl-k">=</span> - <span class="pl-smi">AvroCarbonWriter</span><span class="pl-k">.</span>getCarbonSchemaFromAvroSchema(avroSchema); - - <span class="pl-k">try</span> { - <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder() - .withSchema(carbonSchema) - .outputPath(path) - .buildWriterForAvroInput(); - - <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> <span class="pl-c1">100</span>; i<span class="pl-k">++</span>) { - writer<span class="pl-k">.</span>write(record); - } - writer<span class="pl-k">.</span>close(); - } <span class="pl-k">catch</span> (<span class="pl-smi">Exception</span> e) { - e<span class="pl-k">.</span>printStackTrace(); - } - } -}</pre></div> -<h2> -<a id="datatypes-mapping" class="anchor" href="#datatypes-mapping" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Datatypes Mapping</h2> -<p>Each of SQL data types are mapped into data types of SDK. Following are the mapping:</p> -<table> -<thead> -<tr> -<th>SQL DataTypes</th> -<th>Mapped SDK DataTypes</th> -</tr> -</thead> -<tbody> -<tr> -<td>BOOLEAN</td> -<td>DataTypes.BOOLEAN</td> -</tr> -<tr> -<td>SMALLINT</td> -<td>DataTypes.SHORT</td> -</tr> -<tr> -<td>INTEGER</td> -<td>DataTypes.INT</td> -</tr> -<tr> -<td>BIGINT</td> -<td>DataTypes.LONG</td> -</tr> -<tr> -<td>DOUBLE</td> -<td>DataTypes.DOUBLE</td> -</tr> -<tr> -<td>VARCHAR</td> -<td>DataTypes.STRING</td> -</tr> -<tr> -<td>DATE</td> -<td>DataTypes.DATE</td> -</tr> -<tr> -<td>TIMESTAMP</td> -<td>DataTypes.TIMESTAMP</td> -</tr> -<tr> -<td>STRING</td> -<td>DataTypes.STRING</td> -</tr> -<tr> -<td>DECIMAL</td> -<td>DataTypes.createDecimalType(precision, scale)</td> -</tr> -</tbody> -</table> -<h2> -<a id="api-list" class="anchor" href="#api-list" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2> -<h3> -<a id="class-orgapachecarbondatasdkfilecarbonwriterbuilder" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriterbuilder" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriterBuilder</h3> -<pre><code>/** -* prepares the builder with the schema provided -* @param schema is instance of Schema -* This method must be called when building CarbonWriterBuilder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withSchema(Schema schema); -</code></pre> -<pre><code>/** -* Sets the output path of the writer builder -* @param path is the absolute path where output files are written -* This method must be called when building CarbonWriterBuilder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder outputPath(String path); -</code></pre> -<pre><code>/** -* If set false, writes the carbondata and carbonindex files in a flat folder structure -* @param isTransactionalTable is a boolelan value -* if set to false, then writes the carbondata and carbonindex files -* in a flat folder structure. -* if set to true, then writes the carbondata and carbonindex files -* in segment folder structure.. -* By default set to false. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); -</code></pre> -<pre><code>/** -* to set the timestamp in the carbondata and carbonindex index files -* @param UUID is a timestamp to be used in the carbondata and carbonindex index files. -* By default set to zero. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder uniqueIdentifier(long UUID); -</code></pre> -<pre><code>/** -* To set the carbondata file size in MB between 1MB-2048MB -* @param blockSize is size in MB between 1MB to 2048 MB -* default value is 1024 MB -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withBlockSize(int blockSize); -</code></pre> -<pre><code>/** -* To set the blocklet size of carbondata file -* @param blockletSize is blocklet size in MB -* default value is 64 MB -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withBlockletSize(int blockletSize); -</code></pre> -<pre><code>/** -* sets the list of columns that needs to be in sorted order -* @param sortColumns is a string array of columns that needs to be sorted. -* If it is null or by default all dimensions are selected for sorting -* If it is empty array, no columns are sorted -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder sortBy(String[] sortColumns); -</code></pre> -<pre><code>/** -* If set, create a schema file in metadata folder. -* @param persist is a boolean value, If set to true, creates a schema file in metadata folder. -* By default set to false. will not create metadata folder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder persistSchemaFile(boolean persist); -</code></pre> -<pre><code>/** -* sets the taskNo for the writer. SDKs concurrently running -* will set taskNo in order to avoid conflicts in file's name during write. -* @param taskNo is the TaskNo user wants to specify. -* by default it is system time in nano seconds. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder taskNo(String taskNo); -</code></pre> -<pre><code>/** -* To support the load options for sdk writer -* @param options key,value pair of load options. -* supported keys values are -* a. bad_records_logger_enable -- true (write into separate logs), false -* b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT -* c. bad_record_path -- path -* d. dateformat -- same as JAVA SimpleDateFormat -* e. timestampformat -- same as JAVA SimpleDateFormat -* f. complex_delimiter_level_1 -- value to Split the complexTypeData -* g. complex_delimiter_level_2 -- value to Split the nested complexTypeData -* h. quotechar -* i. escapechar -* -* Default values are as follows. -* -* a. bad_records_logger_enable -- "false" -* b. bad_records_action -- "FAIL" -* c. bad_record_path -- "" -* d. dateformat -- "" , uses from carbon.properties file -* e. timestampformat -- "", uses from carbon.properties file -* f. complex_delimiter_level_1 -- "$" -* g. complex_delimiter_level_2 -- ":" -* h. quotechar -- "\"" -* i. escapechar -- "\\" -* -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withLoadOptions(Map<String, String> options); -</code></pre> -<pre><code>/** -* Build a {@link CarbonWriter}, which accepts row in CSV format object -* @return CSVCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException -*/ -public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOptionException; -</code></pre> -<pre><code>/** -* Build a {@link CarbonWriter}, which accepts Avro format object -* @return AvroCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException -*/ -public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException; -</code></pre> -<h3> -<a id="class-orgapachecarbondatasdkfilecarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriter</h3> -<pre><code>/** -* Write an object to the file, the format of the object depends on the implementation -* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record -* If CSVCarbonWriter, object is of type String[] -* Note: This API is not thread safe -* @param object -* @throws IOException -*/ -public abstract void write(Object object) throws IOException; -</code></pre> -<pre><code>/** -* Flush and close the writer -*/ -public abstract void close() throws IOException; -</code></pre> -<pre><code>/** -* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} -*/ -public static CarbonWriterBuilder builder() { -return new CarbonWriterBuilder(); -} -</code></pre> -<h3> -<a id="class-orgapachecarbondatasdkfilefield" class="anchor" href="#class-orgapachecarbondatasdkfilefield" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Field</h3> -<pre><code>/** -* Field Constructor -* @param name name of the field -* @param type datatype of field, specified in strings. -*/ -public Field(String name, String type); -</code></pre> -<pre><code>/** -* Field constructor -* @param name name of the field -* @param type datatype of the field of class DataType -*/ -public Field(String name, DataType type); -</code></pre> -<h3> -<a id="class-orgapachecarbondatasdkfileschema" class="anchor" href="#class-orgapachecarbondatasdkfileschema" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Schema</h3> -<pre><code>/** -* construct a schema with fields -* @param fields -*/ -public Schema(Field[] fields); -</code></pre> -<pre><code>/** -* Create a Schema using JSON string, for example: -* [ -* {"name":"string"}, -* {"age":"int"} -* ] -* @param json specified as string -* @return Schema -*/ -public static Schema parseJson(String json); -</code></pre> -<h3> -<a id="class-orgapachecarbondatasdkfileavrocarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfileavrocarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.AvroCarbonWriter</h3> -<pre><code>/** -* converts avro schema to carbon schema, required by carbonWriter -* -* @param avroSchemaString json formatted avro schema as string -* @return carbon sdk schema -*/ -public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); -</code></pre> -</div> -</div> -</div> -</div> -<div class="doc-footer"> - <a href="#top" class="scroll-top">Top</a> -</div> -</div> -</section> -</div> -</div> -</div> -</section><!-- End systemblock part --> -<script src="js/custom.js"></script> -</body> -</html> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/security.html ---------------------------------------------------------------------- diff --git a/content/security.html b/content/security.html index 2aa4b8e..9168fa3 100644 --- a/content/security.html +++ b/content/security.html @@ -205,4 +205,4 @@ <script src="js/custom.js"></script> <script src="js/mdNavigation.js" type="text/javascript"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/segment-management-on-carbondata.html ---------------------------------------------------------------------- diff --git a/content/segment-management-on-carbondata.html b/content/segment-management-on-carbondata.html index 528ee9d..1e6f61d 100644 --- a/content/segment-management-on-carbondata.html +++ b/content/segment-management-on-carbondata.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -307,8 +307,7 @@ $(function() { $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); } }); -</script> -</div> +</script></div> </div> </div> </div> @@ -323,4 +322,4 @@ $(function() { </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/streaming-guide.html ---------------------------------------------------------------------- diff --git a/content/streaming-guide.html b/content/streaming-guide.html index c6d8391..86f0385 100644 --- a/content/streaming-guide.html +++ b/content/streaming-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -212,6 +212,37 @@ <div> <h1> <a id="carbondata-streaming-ingestion" class="anchor" href="#carbondata-streaming-ingestion" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Streaming Ingestion</h1> +<ul> +<li> +<a href="#quick-example">Streaming Table Management</a> +<ul> +<li><a href="#create-table-with-streaming-property">Create table with streaming property</a></li> +<li><a href="#alter-streaming-property">Alter streaming property</a></li> +<li><a href="#acquire-streaming-lock">Acquire streaming lock</a></li> +<li><a href="#create-streaming-segment">Create streaming segment</a></li> +<li><a href="#change-segment-status">Change Stream segment status</a></li> +<li><a href="#handoff-streaming-finish-segment-to-columnar-segment">Handoff "streaming finish" segment to columnar segment</a></li> +<li><a href="#auto-handoff-streaming-segment">Auto handoff streaming segment</a></li> +<li><a href="#stream-data-parser">Stream data parser</a></li> +<li><a href="#close-streaming-table">Close streaming table</a></li> +<li><a href="#constraint">Constraints</a></li> +</ul> +</li> +<li> +<a href="#streamsql">StreamSQL</a> +<ul> +<li><a href="#streaming-table">Defining Streaming Table</a></li> +<li> +<a href="#streaming-job-management">Streaming Job Management</a> +<ul> +<li><a href="#start-stream">START STREAM</a></li> +<li><a href="#stop-stream">STOP STREAM</a></li> +<li><a href="#show-streams">SHOW STREAMS</a></li> +</ul> +</li> +</ul> +</li> +</ul> <h2> <a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2> <p>Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME</p> @@ -255,7 +286,7 @@ <span class="pl-s"> | col1 INT,</span> <span class="pl-s"> | col2 STRING</span> <span class="pl-s"> | )</span> -<span class="pl-s"> | STORED BY 'carbondata'</span> +<span class="pl-s"> | STORED AS carbondata</span> <span class="pl-s"> | TBLPROPERTIES('streaming'='true')<span class="pl-pds">"""</span></span>.stripMargin) <span class="pl-k">val</span> <span class="pl-smi">carbonTable</span> <span class="pl-k">=</span> <span class="pl-en">CarbonEnv</span>.getCarbonTable(<span class="pl-en">Some</span>(<span class="pl-s"><span class="pl-pds">"</span>default<span class="pl-pds">"</span></span>), <span class="pl-s"><span class="pl-pds">"</span>carbon_table<span class="pl-pds">"</span></span>)(spark) @@ -300,7 +331,7 @@ streaming table using following DDL.</p> col1 <span class="pl-k">INT</span>, col2 STRING ) - STORED BY <span class="pl-s"><span class="pl-pds">'</span>carbondata<span class="pl-pds">'</span></span> + STORED <span class="pl-k">AS</span> carbondata TBLPROPERTIES(<span class="pl-s"><span class="pl-pds">'</span>streaming<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>true<span class="pl-pds">'</span></span>)</pre></div> <table> <thead> @@ -483,6 +514,145 @@ streaming table using following DDL.</p> <li>block delete "streaming" segment while the streaming ingestion is running.</li> <li>block drop the streaming table while the streaming ingestion is running.</li> </ol> +<h2> +<a id="streamsql" class="anchor" href="#streamsql" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>StreamSQL</h2> +<h3> +<a id="streaming-table" class="anchor" href="#streaming-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Streaming Table</h3> +<p><strong>Example</strong></p> +<p>Following example shows how to start a streaming ingest job</p> +<pre><code> sql( + s""" + |CREATE TABLE source( + | id INT, + | name STRING, + | city STRING, + | salary FLOAT, + | tax DECIMAL(8,2), + | percent double, + | birthday DATE, + | register TIMESTAMP, + | updated TIMESTAMP + |) + |STORED BY carbondata + |TBLPROPERTIES ( + | 'format'='csv', + | 'path'='$csvDataDir' + |) + """.stripMargin) + + sql( + s""" + |CREATE TABLE sink( + | id INT, + | name STRING, + | city STRING, + | salary FLOAT, + | tax DECIMAL(8,2), + | percent double, + | birthday DATE, + | register TIMESTAMP, + | updated TIMESTAMP + |) + |STORED BY carbondata + |TBLPROPERTIES ( + | 'streaming'='true' + |) + """.stripMargin) + + sql( + """ + |START STREAM job123 ON TABLE sink + |STMPROPERTIES( + | 'trigger'='ProcessingTime', + | 'interval'='1 seconds') + |AS + | SELECT * + | FROM source + | WHERE id % 2 = 1 + """.stripMargin) + + sql("STOP STREAM job123") + + sql("SHOW STREAMS [ON TABLE tableName]") +</code></pre> +<p>In above example, two table is created: source and sink. The <code>source</code> table's format is <code>csv</code> and <code>sink</code> table format is <code>carbon</code>. Then a streaming job is created to stream data from source table to sink table.</p> +<p>These two tables are normal carbon table, they can be queried independently.</p> +<h3> +<a id="streaming-job-management" class="anchor" href="#streaming-job-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Streaming Job Management</h3> +<p>As above example shown:</p> +<ul> +<li> +<code>START STREAM jobName ON TABLE tableName</code> is used to start a streaming ingest job.</li> +<li> +<code>STOP STREAM jobName</code> is used to stop a streaming job by its name</li> +<li> +<code>SHOW STREAMS [ON TABLE tableName]</code> is used to print streaming job information</li> +</ul> +<h5> +<a id="start-stream" class="anchor" href="#start-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>START STREAM</h5> +<p>When this is issued, carbon will start a structured streaming job to do the streaming ingestion. Before launching the job, system will validate:</p> +<ul> +<li> +<p>The format of table specified in CTAS FROM clause must be one of: csv, json, text, parquet, kafka, socket. These are formats supported by spark 2.2.0 structured streaming</p> +</li> +<li> +<p>User should pass the options of the streaming source table in its TBLPROPERTIES when creating it. StreamSQL will pass them transparently to spark when creating the streaming job. For example:</p> +<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">source</span>( + name STRING, + age <span class="pl-k">INT</span> +) +STORED BY carbondata +TBLPROPERTIES( + <span class="pl-s"><span class="pl-pds">'</span>format<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>socket<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>host<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>localhost<span class="pl-pds">'</span></span>, + <span class="pl-s"><span class="pl-pds">'</span>port<span class="pl-pds">'</span></span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>8888<span class="pl-pds">'</span></span> +)</pre></div> +<p>will translate to</p> +<div class="highlight highlight-source-scala"><pre>spark.readStream + .schema(tableSchema) + .format(<span class="pl-s"><span class="pl-pds">"</span>socket<span class="pl-pds">"</span></span>) + .option(<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>localhost<span class="pl-pds">"</span></span>) + .option(<span class="pl-s"><span class="pl-pds">"</span>port<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>8888<span class="pl-pds">"</span></span>)</pre></div> +</li> +<li> +<p>The sink table should have a TBLPROPERTY <code>'streaming'</code> equal to <code>true</code>, indicating it is a streaming table.</p> +</li> +<li> +<p>In the given STMPROPERTIES, user must specify <code>'trigger'</code>, its value must be <code>ProcessingTime</code> (In future, other value will be supported). User should also specify interval value for the streaming job.</p> +</li> +<li> +<p>If the schema specifid in sink table is different from CTAS, the streaming job will fail</p> +</li> +</ul> +<h5> +<a id="stop-stream" class="anchor" href="#stop-stream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>STOP STREAM</h5> +<p>When this is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist.</p> +<h5> +<a id="show-streams" class="anchor" href="#show-streams" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SHOW STREAMS</h5> +<p><code>SHOW STREAMS ON TABLE tableName</code> command will print the streaming job information as following</p> +<table> +<thead> +<tr> +<th>Job name</th> +<th>status</th> +<th>Source</th> +<th>Sink</th> +<th>start time</th> +<th>time elapsed</th> +</tr> +</thead> +<tbody> +<tr> +<td>job123</td> +<td>Started</td> +<td>device</td> +<td>fact</td> +<td>2018-02-03 14:32:42</td> +<td>10d2h32m</td> +</tr> +</tbody> +</table> +<p><code>SHOW STREAMS</code> command will show all stream jobs in the system.</p> <script> $(function() { // Show selected style on nav item @@ -493,8 +663,7 @@ $(function() { $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); } }); -</script> -</div> +</script></div> </div> </div> </div> @@ -509,4 +678,4 @@ $(function() { </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/supported-data-types-in-carbondata.html ---------------------------------------------------------------------- diff --git a/content/supported-data-types-in-carbondata.html b/content/supported-data-types-in-carbondata.html index e0ad647..f346052 100644 --- a/content/supported-data-types-in-carbondata.html +++ b/content/supported-data-types-in-carbondata.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -269,8 +269,7 @@ $(function() { $('.b-nav__docs').parent().toggleClass('nav__item__with__subs--expanded'); } }); -</script> -</div> +</script></div> </div> </div> </div> @@ -285,4 +284,4 @@ $(function() { </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/a51dc596/content/timeseries-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/timeseries-datamap-guide.html b/content/timeseries-datamap-guide.html index 497b02f..73a4580 100644 --- a/content/timeseries-datamap-guide.html +++ b/content/timeseries-datamap-guide.html @@ -159,7 +159,7 @@ <div class="nav__inner"> <a class="b-nav__intro nav__item" href="./introduction.html">introduction</a> <a class="b-nav__quickstart nav__item" href="./quick-start-guide.html">quick start</a> - <a class="b-nav__uses nav__item" href="./usescases.html">use cases</a> + <a class="b-nav__uses nav__item" href="./usecases.html">use cases</a> <div class="nav__item nav__item__with__subs"> <a class="b-nav__docs nav__item nav__sub__anchor" href="./language-manual.html">Language Reference</a> @@ -179,9 +179,9 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Support</a> <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> + <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> <a class="b-nav__contri nav__item" href="./how-to-contribute-to-apache-carbondata.html">Contribute</a> <a class="b-nav__security nav__item" href="./security.html">Security</a> @@ -194,9 +194,9 @@ <div class="b-nav__uses navindicator__item"></div> <div class="b-nav__docs navindicator__item"></div> <div class="b-nav__datamap navindicator__item"></div> - <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__api navindicator__item"></div> <div class="b-nav__perf navindicator__item"></div> + <div class="b-nav__s3 navindicator__item"></div> <div class="b-nav__faq navindicator__item"></div> <div class="b-nav__contri navindicator__item"></div> <div class="b-nav__security navindicator__item"></div> @@ -213,9 +213,9 @@ <h1> <a id="carbondata-timeseries-datamap" class="anchor" href="#carbondata-timeseries-datamap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Timeseries DataMap</h1> <ul> -<li><a href="#timeseries-datamap-intoduction">Timeseries DataMap Introduction</a></li> -<li><a href="#compacting-pre-aggregate-tables">Compaction</a></li> -<li><a href="#data-management-with-pre-aggregate-tables">Data Management</a></li> +<li><a href="#timeseries-datamap-introduction-alpha-feature">Timeseries DataMap Introduction</a></li> +<li><a href="#compacting-timeseries-datamp">Compaction</a></li> +<li><a href="#data-management-on-timeseries-datamap">Data Management</a></li> </ul> <h2> <a id="timeseries-datamap-introduction-alpha-feature" class="anchor" href="#timeseries-datamap-introduction-alpha-feature" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timeseries DataMap Introduction (Alpha Feature)</h2> @@ -343,8 +343,7 @@ $(function() { $('.b-nav__datamap').parent().toggleClass('nav__item__with__subs--expanded'); } }); -</script> -</div> +</script></div> </div> </div> </div> @@ -359,4 +358,4 @@ $(function() { </section><!-- End systemblock part --> <script src="js/custom.js"></script> </body> -</html> +</html> \ No newline at end of file
