http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/main/webapp/quick-start-guide.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/quick-start-guide.html b/src/main/webapp/quick-start-guide.html new file mode 100644 index 0000000..377d343 --- /dev/null +++ b/src/main/webapp/quick-start-guide.html @@ -0,0 +1,321 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/1.0.0-incubating"; + target="_blank">Apache CarbonData 1.0.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.2.0-incubating"; + target="_blank">Apache CarbonData 0.2.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.1-incubating"; + target="_blank">Apache CarbonData 0.1.1</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.0-incubating"; + target="_blank">Apache CarbonData 0.1.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PPMC+and+Committers+member+list"; + target="_blank">Project PPMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="../../security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search";> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input"/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + Only search for CarbonData + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> + +<h1> +<a id="quick-start" class="anchor" href="#quick-start" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick Start</h1> + +<p>This tutorial provides a quick introduction to using CarbonData.</p> + +<h2> +<a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h2> + +<ul> +<li> +<a href="https://github.com/apache/incubator-carbondata/blob/master/build";>Installation and building CarbonData</a>.</li> +<li> +<p>Create a sample.csv file using the following commands. The CSV file is required for loading data into CarbonData.</p> + +<pre><code>cd carbondata +cat > sample.csv << EOF +id,name,city,age +1,david,shenzhen,31 +2,eason,shenzhen,27 +3,jarry,wuhan,35 +EOF +</code></pre> +</li> +</ul> + +<h2> +<a id="interactive-analysis-with-spark-shell" class="anchor" href="#interactive-analysis-with-spark-shell" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Interactive Analysis with Spark Shell</h2> + +<h2> +<a id="version-21" class="anchor" href="#version-21" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Version 2.1</h2> + +<p>Apache Spark Shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. Please visit <a href="http://spark.apache.org/docs/latest/";>Apache Spark Documentation</a> for more details on Spark shell.</p> + +<h4> +<a id="basics" class="anchor" href="#basics" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Basics</h4> + +<p>Start Spark shell by running the following command in the Spark directory:</p> + +<pre><code>./bin/spark-shell --jars <carbondata assembly jar path> +</code></pre> + +<p>In this shell, SparkSession is readily available as <code>spark</code> and Spark context is readily available as <code>sc</code>.</p> + +<p>In order to create a CarbonSession we will have to configure it explicitly in the following manner :</p> + +<ul> +<li>Import the following :</li> +</ul> + +<pre><code>import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.CarbonSession._ +</code></pre> + +<ul> +<li>Create a CarbonSession :</li> +</ul> + +<pre><code>val carbon = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<hdfs store path>") +</code></pre> + +<p><strong>NOTE</strong>: By default metastore location is pointed to <code>../carbon.metastore</code>, user can provide own metastore location to CarbonSession like <code>SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession("<hdfs store path>", "<local metastore path>")</code></p> + +<h4> +<a id="executing-queries" class="anchor" href="#executing-queries" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Executing Queries</h4> + +<h5> +<a id="creating-a-table" class="anchor" href="#creating-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h5> + +<pre><code>scala>carbon.sql("CREATE TABLE IF NOT EXISTS test_table(id string, name string, city string, age Int) STORED BY 'carbondata'") +</code></pre> + +<h5> +<a id="loading-data-to-a-table" class="anchor" href="#loading-data-to-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h5> + +<pre><code>scala>carbon.sql("LOAD DATA INPATH 'sample.csv file path' INTO TABLE test_table") +</code></pre> + +<p><strong>NOTE</strong>: Please provide the real file path of <code>sample.csv</code> for the above script.</p> + +<h6> +<a id="query-data-from-a-table" class="anchor" href="#query-data-from-a-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data from a Table</h6> + +<pre><code>scala>carbon.sql("SELECT * FROM test_table").show() + +scala>carbon.sql("SELECT city, avg(age), sum(age) FROM test_table GROUP BY city").show() +</code></pre> + +<h2> +<a id="interactive-analysis-with-spark-shell-1" class="anchor" href="#interactive-analysis-with-spark-shell-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Interactive Analysis with Spark Shell</h2> + +<h2> +<a id="version-16" class="anchor" href="#version-16" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Version 1.6</h2> + +<h4> +<a id="basics-1" class="anchor" href="#basics-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Basics</h4> + +<p>Start Spark shell by running the following command in the Spark directory:</p> + +<pre><code>./bin/spark-shell --jars <carbondata assembly jar path> +</code></pre> + +<p><strong>NOTE</strong>: In this shell, SparkContext is readily available as <code>sc</code>.</p> + +<ul> +<li>In order to execute the Queries we need to import CarbonContext:</li> +</ul> + +<pre><code>import org.apache.spark.sql.CarbonContext +</code></pre> + +<ul> +<li>Create an instance of CarbonContext in the following manner :</li> +</ul> + +<pre><code>val cc = new CarbonContext(sc, "<hdfs store path>") +</code></pre> + +<p><strong>NOTE</strong>: If running on local machine without hdfs, configure the local machine's store path instead of hdfs store path</p> + +<h4> +<a id="executing-queries-1" class="anchor" href="#executing-queries-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Executing Queries</h4> + +<h5> +<a id="creating-a-table-1" class="anchor" href="#creating-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Table</h5> + +<pre><code>scala>cc.sql("CREATE TABLE IF NOT EXISTS test_table (id string, name string, city string, age Int) STORED BY 'carbondata'") +</code></pre> + +<p>To see the table created :</p> + +<pre><code>scala>cc.sql("SHOW TABLES").show() +</code></pre> + +<h5> +<a id="loading-data-to-a-table-1" class="anchor" href="#loading-data-to-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Loading Data to a Table</h5> + +<pre><code>scala>cc.sql("LOAD DATA INPATH 'sample.csv file path' INTO TABLE test_table") +</code></pre> + +<p><strong>NOTE</strong>: Please provide the real file path of <code>sample.csv</code> for the above script.</p> + +<h5> +<a id="query-data-from-a-table-1" class="anchor" href="#query-data-from-a-table-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Data from a Table</h5> + +<pre><code>scala>cc.sql("SELECT * FROM test_table").show() +scala>cc.sql("SELECT city, avg(age), sum(age) FROM test_table GROUP BY city").show() +</code></pre> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/main/webapp/security.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/security.html b/src/main/webapp/security.html index 205eba4..7031278 100644 --- a/src/main/webapp/security.html +++ b/src/main/webapp/security.html @@ -12,7 +12,7 @@ <link rel="stylesheet" href="css/bootstrap.min.css"> <link href="css/style.css" rel="stylesheet"> - <link href="css/print.css" rel="stylesheet" > + <link href="css/print.css" rel="stylesheet"> <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> <!--[if lt IE 9]> @@ -22,17 +22,18 @@ </head> <body> <header> - <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper" > + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> <div class="container"> <div class="navbar-header"> - <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" class="navbar-toggle collapsed" type="button"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a href="index.html" class="logo"> - <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo" /> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> </a> </div> <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> @@ -40,7 +41,8 @@ <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a></li> <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Download <span class="caret"></span></a> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/1.0.0-incubating"; @@ -59,53 +61,71 @@ target="_blank">Release Archive</a></li> </ul> </li> - <li><a href="docs/latest/mainpage.html?page=userguide" class="">Documentation</a></li> + <li><a href="mainpage.html" class="">Documentation</a></li> <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> <ul class="dropdown-menu"> - <li><a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; - target="_blank">Contributing to CarbonData</a></li></li> - <li><a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Committers"; target="_blank">Project Committers</a></li> - <li><a href="meetup.html">CarbonData Meetups </a></li> - <li><a href="security.html">Apache CarbonData Security </a></li> + <li> + <a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PPMC+and+Committers+member+list"; + target="_blank">Project PPMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> </ul> </li> <li class="dropdown"> - <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> </ul> </li> <li class="dropdown"> - <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> </ul> </li> <li> - <a href="#" id="search-icon" ><i class="fa fa-search" aria-hidden="true"></i></a> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> </li> </ul> </div><!--/.nav-collapse --> - <div id="search-box" > + <div id="search-box"> <form method="get" action="http://www.google.com/search";> <div class="search-block"> <table border="0" cellpadding="0" width="100%"> <tr> <td style="width:80%"> - <input type="text" name="q" size=" 5" maxlength="255" value="" class="search-input" /> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input"/> </td> <td style="width:20%"> - <input type="submit" value="Search" /></td></tr> - <tr><td align="left" style="font-size:75%" colspan="2"> - <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked /> Only search for CarbonData </td></tr> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + Only search for CarbonData + </td> + </tr> </table> </div> </form> @@ -119,7 +139,7 @@ <section style="position:relative; overflow:hidden "> <div class="meetup-banber-bg"> - <img src="images/CDbannerSliderbg.jpg" alt="CarbonData banner" /> + <img src="images/CDbannerSliderbg.jpg" alt="CarbonData banner"/> </div> <div class="meetup-banber" style="padding:30px"> @@ -145,14 +165,24 @@ </h1> <p> - The Apache Software Foundation takes a rigorous standpoint in annihilating the security issues in its software projects. Apache CarbonData is highly sensitive and forthcoming to issues pertaining to its features and functionality. + The Apache Software Foundation takes a rigorous standpoint in annihilating the security issues + in its software projects. Apache CarbonData is highly sensitive and forthcoming to issues + pertaining to its features and functionality. </p> - <p>If you have apprehensions regarding CarbonDataâs security or you discover vulnerability or potential threat, donât hesitate to get in touch with the Apache Security Team by dropping a mail at <a href="mailto:secur...@apache.org";> secur...@apache.org</a>. In the mail, specify the project name CarbonData with the description of the issue or potential threat. You are also urged to recommend the way to reproduce and replicate the issue. The security team and the CarbonData community will get back to you after assessing and analysing the findings.</p> + <p>If you have apprehensions regarding CarbonDataâs security or you discover vulnerability or + potential threat, donât hesitate to get in touch with the Apache Security Team by dropping a + mail at <a href="mailto:secur...@apache.org";> secur...@apache.org</a>. In the mail, specify the + project name CarbonData with the description of the issue or potential threat. You are also + urged to recommend the way to reproduce and replicate the issue. The security team and the + CarbonData community will get back to you after assessing and analysing the findings.</p> <p> - PLEASE PAY ATTENTION to report the security issue on the security email before disclosing it on public domain. + PLEASE PAY ATTENTION to report the security issue on the security email before disclosing it on + public domain. </p> <p> - The ASF Security Team maintains a page with the description of how vulnerabilities and potential threats are handled, check their <a href="http://www.apache.org/security/"; target="_blank">Web page</a> for more Details. + The ASF Security Team maintains a page with the description of how vulnerabilities and potential + threats are handled, check their <a href="http://www.apache.org/security/"; target="_blank">Web + page</a> for more Details. </p> </div> http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/main/webapp/supported-data-types-in-carbondata.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/supported-data-types-in-carbondata.html b/src/main/webapp/supported-data-types-in-carbondata.html new file mode 100644 index 0000000..3357820 --- /dev/null +++ b/src/main/webapp/supported-data-types-in-carbondata.html @@ -0,0 +1,219 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/1.0.0-incubating"; + target="_blank">Apache CarbonData 1.0.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.2.0-incubating"; + target="_blank">Apache CarbonData 0.2.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.1-incubating"; + target="_blank">Apache CarbonData 0.1.1</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.0-incubating"; + target="_blank">Apache CarbonData 0.1.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PPMC+and+Committers+member+list"; + target="_blank">Project PPMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="../../security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search";> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input"/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + Only search for CarbonData + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> + +<h1> +<a id="data-types" class="anchor" href="#data-types" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Types</h1> + +<h4> +<a id="carbondata-supports-the-following-data-types" class="anchor" href="#carbondata-supports-the-following-data-types" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData supports the following data types:</h4> + +<ul> +<li> +<p>Numeric Types</p> + +<ul> +<li>SMALLINT</li> +<li>INT/INTEGER</li> +<li>BIGINT</li> +<li>DOUBLE</li> +<li>DECIMAL</li> +</ul> +</li> +<li> +<p>Date/Time Types</p> + +<ul> +<li>TIMESTAMP</li> +<li>DATE</li> +</ul> +</li> +<li> +<p>String Types</p> + +<ul> +<li>STRING</li> +<li>CHAR</li> +</ul> +</li> +<li> +<p>Complex Types</p> + +<ul> +<li>arrays: ARRAY<code><data_type></code> +</li> +<li>structs: STRUCT<code><col_name : data_type COMMENT col_comment, ...></code> +</li> +</ul> +</li> +</ul> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/main/webapp/troubleshooting.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/troubleshooting.html b/src/main/webapp/troubleshooting.html new file mode 100644 index 0000000..7ee56a2 --- /dev/null +++ b/src/main/webapp/troubleshooting.html @@ -0,0 +1,418 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/1.0.0-incubating"; + target="_blank">Apache CarbonData 1.0.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.2.0-incubating"; + target="_blank">Apache CarbonData 0.2.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.1-incubating"; + target="_blank">Apache CarbonData 0.1.1</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.0-incubating"; + target="_blank">Apache CarbonData 0.1.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PPMC+and+Committers+member+list"; + target="_blank">Project PPMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="../../security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search";> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input"/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + Only search for CarbonData + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> + +<h1> +<a id="troubleshooting" class="anchor" href="#troubleshooting" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Troubleshooting</h1> + +<p>This tutorial is designed to provide troubleshooting for end users and developers +who are building, deploying, and using CarbonData.</p> + +<h2> +<a id="failed-to-load-thrift-libraries" class="anchor" href="#failed-to-load-thrift-libraries" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to load thrift libraries</h2> + +<p><strong>Symptom</strong></p> + +<p>Thrift throws following exception :</p> + +<pre><code> thrift: error while loading shared libraries: + libthriftc.so.0: cannot open shared object file: No such file or directory +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The complete path to the directory containing the libraries is not configured correctly.</p> + +<p><strong>Procedure</strong></p> + +<p>Follow the Apache thrift docs at <a href="https://thrift.apache.org/docs/install";>https://thrift.apache.org/docs/install</a> to install thrift correctly.</p> + +<h2> +<a id="failed-to-launch-the-spark-shell" class="anchor" href="#failed-to-launch-the-spark-shell" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to launch the Spark Shell</h2> + +<p><strong>Symptom</strong></p> + +<p>The shell prompts the following error :</p> + +<pre><code> org.apache.spark.sql.CarbonContext$$anon$$apache$spark$sql$catalyst$analysis + $OverrideCatalog$_setter_$org$apache$spark$sql$catalyst$analysis + $OverrideCatalog$$overrides_$e +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The Spark Version and the selected Spark Profile do not match.</p> + +<p><strong>Procedure</strong></p> + +<ol> +<li><p>Ensure your spark version and selected profile for spark are correct.</p></li> +<li> +<p>Use the following command :</p> + +<pre><code> "mvn -Pspark-2.1 -Dspark.version {yourSparkVersion} clean package" +</code></pre> + +<p>Note : Refrain from using "mvn clean package" without specifying the profile.</p> +</li> +</ol> + +<h2> +<a id="failed-to-execute-load-query-on-cluster" class="anchor" href="#failed-to-execute-load-query-on-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute load query on cluster.</h2> + +<p><strong>Symptom</strong></p> + +<p>Load query failed with the following exception:</p> + +<pre><code> Dictionary file is locked for updation. +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The carbon.properties file is not identical in all the nodes of the cluster.</p> + +<p><strong>Procedure</strong></p> + +<p>Follow the steps to ensure the carbon.properties file is consistent across all the nodes:</p> + +<ol> +<li><p>Copy the carbon.properties file from the master node to all the other nodes in the cluster. + For example, you can use ssh to copy this file to all the nodes.</p></li> +<li><p>For the changes to take effect, restart the Spark cluster.</p></li> +</ol> + +<h2> +<a id="failed-to-execute-insert-query-on-cluster" class="anchor" href="#failed-to-execute-insert-query-on-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute insert query on cluster.</h2> + +<p><strong>Symptom</strong></p> + +<p>Load query failed with the following exception:</p> + +<pre><code> Dictionary file is locked for updation. +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The carbon.properties file is not identical in all the nodes of the cluster.</p> + +<p><strong>Procedure</strong></p> + +<p>Follow the steps to ensure the carbon.properties file is consistent across all the nodes:</p> + +<ol> +<li><p>Copy the carbon.properties file from the master node to all the other nodes in the cluster. + For example, you can use scp to copy this file to all the nodes.</p></li> +<li><p>For the changes to take effect, restart the Spark cluster.</p></li> +</ol> + +<h2> +<a id="failed-to-connect-to-hiveuser-with-thrift" class="anchor" href="#failed-to-connect-to-hiveuser-with-thrift" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to connect to hiveuser with thrift</h2> + +<p><strong>Symptom</strong></p> + +<p>We get the following exception :</p> + +<pre><code> Cannot connect to hiveuser. +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The external process does not have permission to access.</p> + +<p><strong>Procedure</strong></p> + +<p>Ensure that the Hiveuser in mysql must allow its access to the external processes.</p> + +<h2> +<a id="failure-to-read-the-metastore-db-during-table-creation" class="anchor" href="#failure-to-read-the-metastore-db-during-table-creation" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failure to read the metastore db during table creation.</h2> + +<p><strong>Symptom</strong></p> + +<p>We get the following exception on trying to connect :</p> + +<pre><code> Cannot read the metastore db +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The metastore db is dysfunctional.</p> + +<p><strong>Procedure</strong></p> + +<p>Remove the metastore db from the carbon.metastore in the Spark Directory.</p> + +<h2> +<a id="failed-to-load-data-on-the-cluster" class="anchor" href="#failed-to-load-data-on-the-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to load data on the cluster</h2> + +<p><strong>Symptom</strong></p> + +<p>Data loading fails with the following exception :</p> + +<pre><code> Data Load failure exeception +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The following issue can cause the failure :</p> + +<ol> +<li><p>The core-site.xml, hive-site.xml, yarn-site and carbon.properties are not consistent across all nodes of the cluster.</p></li> +<li> +<p>Path to hdfs ddl is not configured correctly in the carbon.properties.</p> + +<p><strong>Procedure</strong></p> + +<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p> + +<ol> +<li> +<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster. +For example, you can use scp to copy this file to all the nodes.</p> + +<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p> +</li> +<li><p>For the changes to take effect, restart the Spark cluster.</p></li> +</ol> +</li> +</ol> + +<h2> +<a id="failed-to-insert-data-on-the-cluster" class="anchor" href="#failed-to-insert-data-on-the-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to insert data on the cluster</h2> + +<p><strong>Symptom</strong></p> + +<p>Insertion fails with the following exception :</p> + +<pre><code> Data Load failure exeception +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>The following issue can cause the failure :</p> + +<ol> +<li><p>The core-site.xml, hive-site.xml, yarn-site and carbon.properties are not consistent across all nodes of the cluster.</p></li> +<li> +<p>Path to hdfs ddl is not configured correctly in the carbon.properties.</p> + +<p><strong>Procedure</strong></p> + +<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p> + +<ol> +<li> +<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster. +For example, you can use scp to copy this file to all the nodes.</p> + +<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p> +</li> +<li><p>For the changes to take effect, restart the Spark cluster.</p></li> +</ol> +</li> +</ol> + +<h2> +<a id="failed-to-execute-concurrent-operationsloadinsertupdate-on-table-by-multiple-workers" class="anchor" href="#failed-to-execute-concurrent-operationsloadinsertupdate-on-table-by-multiple-workers" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute Concurrent Operations(Load,Insert,Update) on table by multiple workers.</h2> + +<p><strong>Symptom</strong></p> + +<p>Execution fails with the following exception :</p> + +<pre><code> Table is locked for updation. +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>Concurrency not supported.</p> + +<p><strong>Procedure</strong></p> + +<p>Worker must wait for the query execution to complete and the table to release the lock for another query execution to succeed..</p> + +<h2> +<a id="failed-to-create-a-table-with-a-single-numeric-column" class="anchor" href="#failed-to-create-a-table-with-a-single-numeric-column" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to create a table with a single numeric column.</h2> + +<p><strong>Symptom</strong></p> + +<p>Execution fails with the following exception :</p> + +<pre><code> Table creation fails. +</code></pre> + +<p><strong>Possible Cause</strong></p> + +<p>Behavior not supported.</p> + +<p><strong>Procedure</strong></p> + +<p>A single column that can be considered as dimension is mandatory for table creation.</p> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/main/webapp/useful-tips-on-carbondata.html ---------------------------------------------------------------------- diff --git a/src/main/webapp/useful-tips-on-carbondata.html b/src/main/webapp/useful-tips-on-carbondata.html new file mode 100644 index 0000000..a213951 --- /dev/null +++ b/src/main/webapp/useful-tips-on-carbondata.html @@ -0,0 +1,447 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/1.0.0-incubating"; + target="_blank">Apache CarbonData 1.0.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.2.0-incubating"; + target="_blank">Apache CarbonData 0.2.0</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.1-incubating"; + target="_blank">Apache CarbonData 0.1.1</a></li> + <li> + <a href="https://www.apache.org/dyn/closer.lua/incubator/carbondata/0.1.0-incubating"; + target="_blank">Apache CarbonData 0.1.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/incubator-carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PPMC+and+Committers+member+list"; + target="_blank">Project PPMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="../../security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search";> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input"/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + Only search for CarbonData + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> + +<h1> +<a id="useful-tips" class="anchor" href="#useful-tips" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Useful Tips</h1> + +<p>This tutorial guides you to create CarbonData Tables and optimize performance. +The following sections will elaborate on the above topics :</p> + +<ul> +<li><a href="#suggestions-to-create-carbondata-table">Suggestions to create CarbonData Table</a></li> +<li><a href="#configurations-for-optimizing-carbondata-performance">Configurations For Optimizing CarbonData Performance</a></li> +</ul> + +<h2> +<a id="suggestions-to-create-carbondata-table" class="anchor" href="#suggestions-to-create-carbondata-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Suggestions to Create CarbonData Table</h2> + +<p>Recently CarbonData was used to analyze performance of Telecommunication field. +The results of the analysis for table creation with dimensions ranging from +10 thousand to 10 billion rows and 100 to 300 columns have been summarized below. </p> + +<p>The following table describes some of the columns from the table used.</p> + +<p><strong>Table Column Description</strong></p> + +<table> +<thead> +<tr> +<th>Column Name</th> +<th>Data Type</th> +<th>Cardinality</th> +<th>Attribution</th> +</tr> +</thead> +<tbody> +<tr> +<td>msisdn</td> +<td>String</td> +<td>30 million</td> +<td>Dimension</td> +</tr> +<tr> +<td>BEGIN_TIME</td> +<td>BigInt</td> +<td>10 Thousand</td> +<td>Dimension</td> +</tr> +<tr> +<td>HOST</td> +<td>String</td> +<td>1 million</td> +<td>Dimension</td> +</tr> +<tr> +<td>Dime_1</td> +<td>String</td> +<td>1 Thousand</td> +<td>Dimension</td> +</tr> +<tr> +<td>counter_1</td> +<td>Numeric(20,0)</td> +<td>NA</td> +<td>Measure</td> +</tr> +<tr> +<td>...</td> +<td>...</td> +<td>NA</td> +<td>Measure</td> +</tr> +<tr> +<td>counter_100</td> +<td>Numeric(20,0)</td> +<td>NA</td> +<td>Measure</td> +</tr> +</tbody> +</table> + +<p>CarbonData has more than 50 test cases, on the basis of these we have following suggestions to enhance the query performance :</p> + +<ul> +<li> +<p><strong>Put the frequently-used column filter in the beginning</strong></p> + +<p>For example, MSISDN filter is used in most of the query then we must put the MSISDN in the first column. +The create table command can be modified as suggested below :</p> +</li> +</ul> + +<pre><code> create table carbondata_table( + msisdn String, + ... + )STORED BY 'org.apache.carbondata.format' + TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,..', + 'DICTIONARY_INCLUDE'='...'); +</code></pre> + +<p>Now the query with MSISDN in the filter will be more efficient.</p> + +<ul> +<li> +<p><strong>Put the frequently-used columns in the order of low to high cardinality</strong></p> + +<p>If the table in the specified query has multiple columns which are frequently used to filter the results, it is suggested to put +the columns in the order of cardinality low to high. This ordering of frequently used columns improves the compression ratio and +enhances the performance of queries with filter on these columns.</p> + +<p>For example if MSISDN, HOST and Dime_1 are frequently-used columns, then the column order of table is suggested as +Dime_1>HOST>MSISDN as Dime_1 has the lowest cardinality. +The create table command can be modified as suggested below :</p> +</li> +</ul> + +<pre><code> create table carbondata_table( + Dime_1 String, + HOST String, + MSISDN String, + ... + )STORED BY 'org.apache.carbondata.format' + TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST..', + 'DICTIONARY_INCLUDE'='Dime_1..'); +</code></pre> + +<ul> +<li> +<p><strong>Put the Dimension type columns in order of low to high cardinality</strong></p> + +<p>If the columns used to filter are not frequently used, then it is suggested to order all the columns of dimension type in order of low to high cardinality. +The create table command can be modified as below :</p> +</li> +</ul> + +<pre><code> create table carbondata_table( + Dime_1 String, + BEGIN_TIME bigint + HOST String, + MSISDN String, + ... + )STORED BY 'org.apache.carbondata.format' + TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI..', + 'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME..'); +</code></pre> + +<ul> +<li> +<p><strong>For measure type columns with non high accuracy, replace Numeric(20,0) data type with Double data type</strong></p> + +<p>For columns of measure type, not requiring high accuracy, it is suggested to replace Numeric data type with Double to enhance +query performance. The create table command can be modified as below :</p> +</li> +</ul> + +<pre><code> create table carbondata_table( + Dime_1 String, + BEGIN_TIME bigint + HOST String, + MSISDN String, + counter_1 double, + counter_2 double, + ... + counter_100 double + )STORED BY 'org.apache.carbondata.format' + TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI', + 'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME'); +</code></pre> + +<p>The result of performance analysis of test-case shows reduction in query execution time from 15 to 3 seconds, thereby improving performance by nearly 5 times.</p> + +<ul> +<li> +<p><strong>Columns of incremental character should be re-arranged at the end of dimensions</strong></p> + +<p>Consider the following scenario where data is loaded each day and the start_time is incremental for each load, it is +suggested to put start_time at the end of dimensions. </p> + +<p>Incremental values are efficient in using min/max index. The create table command can be modified as below :</p> +</li> +</ul> + +<pre><code> create table carbondata_table( + Dime_1 String, + HOST String, + MSISDN String, + counter_1 double, + counter_2 double, + BEGIN_TIME bigint, + ... + counter_100 double + )STORED BY 'org.apache.carbondata.format' + TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI', + 'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME'); +</code></pre> + +<ul> +<li> +<p><strong>Avoid adding high cardinality columns to dictionary</strong></p> + +<p>If the system has low memory configuration, then it is suggested to exclude high cardinality columns from the dictionary to +enhance load performance. Creation of dictionary for high cardinality columns at time of load will degrade load performance due to +excessive memory usage. </p> + +<p>By default CarbonData determines the cardinality at the first data load and allows for dictionary creation only if the cardinality is less than +1 million.</p> +</li> +</ul> + +<h2> +<a id="configurations-for-optimizing-carbondata-performance" class="anchor" href="#configurations-for-optimizing-carbondata-performance" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configurations for Optimizing CarbonData Performance</h2> + +<p>Recently we did some performance POC on CarbonData for Finance and telecommunication Field. It involved detailed queries and aggregation +scenarios. After the completion of POC, some of the configurations impacting the performance have been identified and tabulated below :</p> + +<table> +<thead> +<tr> +<th>Parameter</th> +<th>Location</th> +<th>Used For</th> +<th>Description</th> +<th>Tuning</th> +</tr> +</thead> +<tbody> +<tr> +<td>carbon.sort.intermediate.files.limit</td> +<td>spark/carbonlib/carbon.properties</td> +<td>Data loading</td> +<td>During the loading of data, local temp is used to sort the data. This number specifies the minimum number of intermediate files after which the merge sort has to be initiated.</td> +<td>Increasing the parameter to a higher value will improve the load performance. For example, when we increase the value from 20 to 100, it increases the data load performance from 35MB/S to more than 50MB/S. Higher values of this parameter consumes more memory during the load.</td> +</tr> +<tr> +<td>carbon.number.of.cores.while.loading</td> +<td>spark/carbonlib/carbon.properties</td> +<td>Data loading</td> +<td>Specifies the number of cores used for data processing during data loading in CarbonData.</td> +<td>If you have more number of CPUs, then you can increase the number of CPUs, which will increase the performance. For example if we increase the value from 2 to 4 then the CSV reading performance can increase about 1 times</td> +</tr> +<tr> +<td>carbon.compaction.level.threshold</td> +<td>spark/carbonlib/carbon.properties</td> +<td>Data loading and Querying</td> +<td>For minor compaction, specifies the number of segments to be merged in stage 1 and number of compacted segments to be merged in stage 2.</td> +<td>Each CarbonData load will create one segment, if every load is small in size it will generate many small file over a period of time impacting the query performance. Configuring this parameter will merge the small segment to one big segment which will sort the data and improve the performance. For Example in one telecommunication scenario, the performance improves about 2 times after minor compaction.</td> +</tr> +<tr> +<td>spark.sql.shuffle.partitions</td> +<td>spark/con/spark-defaults.conf</td> +<td>Querying</td> +<td>The number of task started when spark shuffle.</td> +<td>The value can be 1 to 2 times as much as the executor cores. In an aggregation scenario, reducing the number from 200 to 32 reduced the query time from 17 to 9 seconds.</td> +</tr> +<tr> +<td>num-executors/executor-cores/executor-memory</td> +<td>spark/con/spark-defaults.conf</td> +<td>Querying</td> +<td>The number of executors, CPU cores, and memory used for CarbonData query.</td> +<td>In the bank scenario, we provide the 4 CPUs cores and 15 GB for each executor which can get good performance. This 2 value does not mean more the better. It needs to be configured properly in case of limited resources. For example, In the bank scenario, it has enough CPU 32 cores each node but less memory 64 GB each node. So we cannot give more CPU but less memory. For example, when 4 cores and 12GB for each executor. It sometimes happens GC during the query which impact the query performance very much from the 3 second to more than 15 seconds. In this scenario need to increase the memory or decrease the CPU cores.</td> +</tr> +<tr> +<td>carbon.detail.batch.size</td> +<td>spark/carbonlib/carbon.properties</td> +<td>Data loading</td> +<td>The buffer size to store records, returned from the block scan.</td> +<td>In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000.</td> +</tr> +<tr> +<td>carbon.use.local.dir</td> +<td>spark/carbonlib/carbon.properties</td> +<td>Data loading</td> +<td>Whether use YARN local directories for multi-table load disk load balance</td> +<td>If this is set it to true CarbonData will use YARN local directories for multi-table load disk load balance, that will improve the data load performance.</td> +</tr> +</tbody> +</table> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-carbondata-site/blob/4f8753c1/src/site/markdown/How-to-contribute-to-Apache-CarbonData.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/How-to-contribute-to-Apache-CarbonData.md b/src/site/markdown/How-to-contribute-to-Apache-CarbonData.md deleted file mode 100644 index 7d0136e..0000000 --- a/src/site/markdown/How-to-contribute-to-Apache-CarbonData.md +++ /dev/null @@ -1,193 +0,0 @@ -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -# How to contribute to Apache CarbonData - -The Apache CarbonData community welcomes all kinds of contributions from anyone with a passion for -faster data format! Apache CarbonData is a new file format for faster interactive query using -advanced columnar storage, index, compression and encoding techniques to improve computing -efficiency,in turn it will help speedup queries an order of magnitude faster over PetaBytes of data. - -We use a review-then-commit workflow in CarbonData for all contributions. - -* Engage -> Design -> Code -> Review -> Commit - -## Engage - -### Mailing list(s) - -We discuss design and implementation issues on d...@carbondata.incubator.apache.org Join by -emailing dev-subscr...@carbondata.incubator.apache.org - -### Apache JIRA - -We use [Apache JIRA](https://issues.apache.org/jira/browse/CARBONDATA) as an issue tracking and -project management tool, as well as a way to communicate among a very diverse and distributed set -of contributors. To be able to gather feedback, avoid frustration, and avoid duplicated efforts all -CarbonData-related work should be tracked there. - -If you do not already have an Apache JIRA account, sign up [here](https://issues.apache.org/jira/). - -If a quick search doesnât turn up an existing JIRA issue for the work you want to contribute, -create it. Please discuss your proposal with a committer or the component lead in JIRA or, -alternatively, on the developer mailing list(d...@carbondata.incubator.apache.org). - -If thereâs an existing JIRA issue for your intended contribution, please comment about your -intended work. Once the work is understood, a committer will assign the issue to you. -(If you donât have a JIRA role yet, youâll be added to the âcontributorâ role.) If an issue is -currently assigned, please check with the current assignee before reassigning. - -For moderate or large contributions, you should not start coding or writing a design doc unless -there is a corresponding JIRA issue assigned to you for that work. Simple changes, -like fixing typos, do not require an associated issue. - -### Design - -To avoid potential frustration during the code review cycle, we encourage you to clearly scope and -design non-trivial contributions with the CarbonData community before you start coding. - -Generally, the JIRA issue is the best place to gather relevant design docs, comments, or references. -Itâs great to explicitly include relevant stakeholders early in the conversation. For designs that -may be generally interesting, we also encourage conversations on the developerâs mailing list. - -### Code - -We use GitHubâs pull request functionality to review proposed code changes. -If you do not already have a personal GitHub account, sign up [here](https://github.com). - -### Git config - -Ensure to finish the below config(user.email, user.name) before starting PR works. -``` -$ git config --global user.email "y...@example.com" -$ git config --global user.name "Your Name" -``` - -#### Fork the repository on GitHub - -Go to the [Apache CarbonData GitHub mirror](https://github.com/apache/incubator-carbondata) and -fork the repository to your own private account. -This will be your private workspace for staging changes. - -#### Clone the repository locally - -You are now ready to create the development environment on your local machine. -Clone CarbonDataâs read-only GitHub mirror. -``` -$ git clone https://github.com/apache/incubator-carbondata.git -$ cd incubator-carbondata -``` -Add your forked repository as an additional Git remote, where youâll push your changes. -``` -$ git remote add <GitHub_user> https://github.com/<GitHub_user>/incubator-carbondata.git -``` -You are now ready to start developing! - -#### Create a branch in your fork - -Youâll work on your contribution in a branch in your own (forked) repository. Create a local branch, -initialized with the state of the branch you expect your changes to be merged into. -Keep in mind that we use several branches, including master, feature-specific, and -release-specific branches. If you are unsure, initialize with the state of the master branch. -``` -$ git fetch --all -$ git checkout -b <my-branch> origin/master -``` -At this point, you can start making and committing changes to this branch in a standard way. - -#### Syncing and pushing your branch - -Periodically while you work, and certainly before submitting a pull request, you should update -your branch with the most recent changes to the target branch. -``` -$ git pull --rebase -``` -Remember to always use --rebase parameter to avoid extraneous merge commits. - -To push your local, committed changes to your (forked) repository on GitHub, run: -``` -$ git push <GitHub_user> <my-branch> -``` -#### Testing - -All code should have appropriate unit testing coverage. New code should have new tests in the -same contribution. Bug fixes should include a regression test to prevent the issue from reoccurring. - -For contributions to the Java code, run unit tests locally via Maven. -``` -$ mvn clean verify -``` - -### Review - -Once the initial code is complete and the tests pass, itâs time to start the code review process. -We review and discuss all code, no matter who authors it. Itâs a great way to build community, -since you can learn from other developers, and they become familiar with your contribution. -It also builds a strong project by encouraging a high quality bar and keeping code consistent -throughout the project. - -#### Create a pull request - -Organize your commits to make your reviewerâs job easier. Use the following command to -re-order, squash, edit, or change description of individual commits. -``` -$ git rebase -i origin/master -``` -Navigate to the CarbonData GitHub mirror to create a pull request. The title of the pull request -should be strictly in the following format: -``` -[CARBONDATA-issue number>] Title of the pull request -``` -Please include a descriptive pull request message to help make the reviewerâs job easier. - -If you know a good committer to review your pull request, please make a comment like the following. -If not, donât worry, a committer will pick it up. -``` -Hi @<committer/reviewer name>, can you please take a look? -``` - -#### Code Review and Revision - -During the code review process, donât rebase your branch or otherwise modify published commits, -since this can remove existing comment history and be confusing to the reviewer, -When you make a revision, always push it in a new commit. - -Our GitHub mirror automatically provides pre-commit testing coverage using Jenkins. -Please make sure those tests pass,the contribution cannot be merged otherwise. - -#### LGTM -Once the reviewer is happy with the change, theyâll respond with an LGTM (âlooks good to me!â). -At this point, the committer will take over, possibly make some additional touch ups, -and merge your changes into the codebase. - -In the case both the author and the reviewer are committers, either can merge the pull request. -Just be sure to communicate clearly whose responsibility it is in this particular case. - -Thank you for your contribution to Apache CarbonData! - -#### Deleting your branch(optional) -Once the pull request is merged into the Apache CarbonData repository, you can safely delete the -branch locally and purge it from your forked repository. - -From another local branch, run: -``` -$ git fetch --all -$ git branch -d <my-branch> -$ git push <GitHub_user> --delete <my-branch> -```
