date:20171027

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread mohammadshahidkhan

Github user mohammadshahidkhan commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
+1 LGTM


---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1344/



---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1343/



---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/708/



---

[GitHub] carbondata issue #1443: [CARBONDATA-1524][CARBONDATA-1525] Added support for...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1443
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1342/



---

[GitHub] carbondata issue #1443: [CARBONDATA-1524][CARBONDATA-1525] Added support for...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1443
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/706/



---

[GitHub] carbondata pull request #1443: [CARBONDATA-1524[CARBONDATA-1525] Added suppo...

2017-10-27 Thread kunal642

GitHub user kunal642 opened a pull request:

https://github.com/apache/carbondata/pull/1443

[CARBONDATA-1524[CARBONDATA-1525] Added support for aggregate table drop

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[CARBONDATA-] Description of pull request`
   
 - [X] Make sure to add PR description including

- the root cause/problem statement
- What is the implemented solution

 - [X] Any interfaces changed?
 Metastore interface is changed to support child table drop

 - [X] Any backward compatibility impacted?
 No
 - [X] Document update required?
No
 - [X] Testing done
 Test cases are added for child and parent table drop scenarios.
 
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
 
---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kunal642/carbondata drop_agg_support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1443


commit 4400f65d3ad588230a84ca67bb96c6675d60ef02
Author: kunal642 
Date:   2017-10-18T14:39:04Z

added support for aggregate table drop




---

[GitHub] carbondata issue #1429: [WIP] Add StructType and ArrayType class

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1429
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1341/



---

[GitHub] carbondata issue #1442: [WIP] Add examples for Carbon usage when integrating...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1442
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1340/



---

[GitHub] carbondata pull request #1428: [WIP] Drop agg support

2017-10-27 Thread kunal642

Github user kunal642 closed the pull request at:

https://github.com/apache/carbondata/pull/1428


---

[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1429
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/705/



---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1339/



---

[GitHub] carbondata issue #1442: [WIP] Add examples for Carbon usage when integrating...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1442
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/704/



---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/703/



---

[GitHub] carbondata pull request #1442: [WIP] Add examples

2017-10-27 Thread jackylk

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/1442

[WIP] Add examples

It is good to have more examples for user reference. 
This PR adds back examples from spark-example module in earlier spark 1 
integration

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata example

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1442






---

[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1418
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1338/



---

[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1418
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/702/



---

[GitHub] carbondata pull request #1435: [CARBONDATA-1626]add data size and index size...

2017-10-27 Thread mohammadshahidkhan

Github user mohammadshahidkhan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1435#discussion_r147471059
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ---
@@ -1376,6 +1376,32 @@
 
   public static final String BITSET_PIPE_LINE_DEFAULT = "true";
 
+  /**
+   * The total size of carbon data
+   */
+  public static final String CARBON_TOTAL_DATA_SIZE = "datasize";
+
+  /**
+   * The total size of carbon index
+   */
+  public static final String CARBON_TOTAL_INDEX_SIZE = "indexsize";
+
+  /**
+   * ENABLE_CALCULATE_DATA_INDEX_SIZE
+   */
+  @CarbonProperty public static final String ENABLE_CALCULATE_SIZE = 
"carbon.enable.calculate.size";
+
+  /**
+   * DEFAULT_ENABLE_CALCULATE_DATA_INDEX_SIZE
+   */
+  @CarbonProperty public static final String DEFAULT_ENABLE_CALCULATE_SIZE 
= "true";
--- End diff --

for constant variable/keys CarbonProperty anotation not required


---

[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...

2017-10-27 Thread mohammadshahidkhan

Github user mohammadshahidkhan commented on the issue:

https://github.com/apache/carbondata/pull/1418
  
retest this please


---

[jira] [Resolved] (CARBONDATA-1517) 1. Support CTAS in carbon and support creating aggregation tables using CTAS.And update aggregation table information to main table schema.

2017-10-27 Thread Ravindra Pesala (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-1517.
-
   Resolution: Fixed
 Assignee: kumar vishal
Fix Version/s: 1.3.0

> 1. Support CTAS in carbon and support creating aggregation tables using 
> CTAS.And update aggregation table information to main table schema.
> ---
>
> Key: CARBONDATA-1517
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1517
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Ravindra Pesala
>Assignee: kumar vishal
> Fix For: 1.3.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> User should add aggregation tables with following syntax
> {code}
> CREATE TABLE agg_sales 
> STORED BY 'carbondata'
> TBLPROPERTIES ('parent_table'='sales') 
> AS SELECT user_id, sum(quantity), avg(price) FROM sales GROUP BY user_id
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
LGTM


---

[GitHub] carbondata pull request #1439: [CARBONDATA-1628] Re-factory LoadTableCommand...

2017-10-27 Thread kumarvishal09

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1439#discussion_r147454328
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
 ---
@@ -831,4 +835,207 @@ object CommonUtil {
 LOGGER.error(s)
 }
   }
+
+  def getFinalOptions(carbonProperty: CarbonProperties,
--- End diff --

Please change the method name


---

[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1437
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1337/



---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1336/



---

[jira] [Updated] (CARBONDATA-1651) Unsupported Spark2 BooleanType

2017-10-27 Thread Roman Timrov (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Timrov updated CARBONDATA-1651:
-
Description: 
Unable to save Dataset if it contains field with BooleanType
class CarbonDataFrameWriter
method convertToCarbonType doesn't support it


> Unsupported Spark2 BooleanType
> --
>
> Key: CARBONDATA-1651
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1651
> Project: CarbonData
>  Issue Type: Bug
>  Components: spark-integration
>Affects Versions: 1.3.0
>Reporter: Roman Timrov
>
> Unable to save Dataset if it contains field with BooleanType
> class CarbonDataFrameWriter
> method convertToCarbonType doesn't support it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1437
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/701/



---

[jira] [Created] (CARBONDATA-1651) Unsupported Spark2 BooleanType

2017-10-27 Thread Roman Timrov (JIRA)

Roman Timrov created CARBONDATA-1651:


 Summary: Unsupported Spark2 BooleanType
 Key: CARBONDATA-1651
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1651
 Project: CarbonData
  Issue Type: Bug
  Components: spark-integration
Affects Versions: 1.3.0
Reporter: Roman Timrov






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread chenerlu

Github user chenerlu commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147437929
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
 ---
@@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends 
AbstractCarbonSparkSQLParser {
   , tableName: String, fields: Seq[Field],
   partitionCols: Seq[PartitionerField],
   tableProperties: mutable.Map[String, String],
-  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): 
TableModel = {
+  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false,
+  comment: Option[String] = None): TableModel = {
--- End diff --

Have renamed.


---

[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1429
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1335/



---

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread chenerlu

Github user chenerlu commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147436112
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableWithTableComment.scala
 ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.createTable
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+/**
+ * test functionality for create table with table comment
+ */
+class TestCreateTableWithTableComment extends QueryTest with 
BeforeAndAfterAll {
+
+  override def beforeAll {
+sql("use default")
+sql("drop table if exists withTableComment")
+sql("drop table if exists withoutTableComment")
+  }
+
+  test("test create table with table comment") {
+sql(
+  s"""
+ | create table withTableComment(
+ | id int,
+ | name string
+ | )
+ | comment "This table has table comment"
+ | STORED BY 'carbondata'
+   """.stripMargin
+)
+
+val result = sql("describe formatted withTableComment")
+
+checkExistence(result, true, "Comment:")
+checkExistence(result, true, "This table has table comment")
+  }
+
+  test("test create table without table comment") {
+sql(
+  s"""
+ | create table withoutTableComment(
+ | id int,
+ | name string
+ | )
+ | STORED BY 'carbondata'
+   """.stripMargin
+)
+
--- End diff --

This PR not contains this functions.


---

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread chenerlu

Github user chenerlu commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147435775
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
 ---
@@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends 
AbstractCarbonSparkSQLParser {
   , tableName: String, fields: Seq[Field],
   partitionCols: Seq[PartitionerField],
   tableProperties: mutable.Map[String, String],
-  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): 
TableModel = {
+  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false,
+  comment: Option[String] = None): TableModel = {
--- End diff --

Carbon already support column comment.


---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/700/



---

[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1429
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1334/



---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1333/



---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread sraghunandan

Github user sraghunandan commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
@mohammadshahidkhan @kumarvishal09 please help to review this PR


---

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1332/



---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1331/



---

[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1429
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/699/



---

[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1417
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/698/



---

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/697/



---

[GitHub] carbondata issue #1441: [CARBONDATA-1489] Added testcases Insert and Delete ...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1441
  
Can one of the admins verify this patch?


---

[GitHub] carbondata issue #1441: [CARBONDATA-1489] Added testcases Insert and Delete ...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1441
  
Can one of the admins verify this patch?


---

[GitHub] carbondata pull request #1441: [CARBONDATA-1489] Added testcases Insert and ...

2017-10-27 Thread ashwini-krishnakumar

GitHub user ashwini-krishnakumar opened a pull request:

https://github.com/apache/carbondata/pull/1441

[CARBONDATA-1489]  Added testcases  Insert and Delete testcases for Struct 
and Array data type

Added testcases  Insert and Delete testcases for Struct and Array data type

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[CARBONDATA-] Description of pull request`
   
 - [ ] Make sure to add PR description including

- the root cause/problem statement
- What is the implemented solution

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
 
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
 
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
 
---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ashwini-krishnakumar/carbondata 
CARBONDATA-1489

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1441






---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1330/



---

[GitHub] carbondata issue #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement St...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1440
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1329/



---

[GitHub] carbondata pull request #1430: [CARBONDATA-1444] Update documentation about ...

2017-10-27 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1430


---

[GitHub] carbondata issue #1430: [CARBONDATA-1444] Update documentation about Boolean...

2017-10-27 Thread chenliang613

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1430
  
LGTM


---

[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores

2017-10-27 Thread Ravindra Pesala (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222176#comment-16222176
 ] 

Ravindra Pesala commented on CARBONDATA-1624:
-

Ok, I got your point. You mean just get the configured spark.executor.cores 
from spark while initializing session and set to carbon properties.
I was thinking little more about it, get the currently available cores from 
spark for every load and block those cores to control the parallel loads.  Any 
way we can do your way first later we can extend to get cores for each load.

> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 
> 'carbon.number.of.cores.while.loading' dynamically as per the available 
> executor cores 
> 
>
> Key: CARBONDATA-1624
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
> Project: CarbonData
>  Issue Type: Improvement
>  Components: data-load, spark-integration
>Affects Versions: 1.3.0
>Reporter: Zhichao  Zhang
>Assignee: Zhichao  Zhang
>Priority: Minor
>
> If we are using carbondata + spark to load data, we can set 
> carbon.number.of.cores.while.loading to the  number of executor cores. 
> For example, when set the number of executor cores to 6, it shows that there 
> are at 
> least 6 cores per node for loading data, so we can set 
> carbon.number.of.cores.while.loading to 6 automatically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1328/



---

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread sounakr

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147354669
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
 ---
@@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends 
AbstractCarbonSparkSQLParser {
   , tableName: String, fields: Seq[Field],
   partitionCols: Seq[PartitionerField],
   tableProperties: mutable.Map[String, String],
-  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): 
TableModel = {
+  bucketFields: Option[BucketFields], isAlterFlow: Boolean = false,
+  comment: Option[String] = None): TableModel = {
--- End diff --

Better to rename it to tableComment in order to avoid confusion with Column 
Comment which will come in future.


---

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread sounakr

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147350873
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableWithTableComment.scala
 ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.createTable
+
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+/**
+ * test functionality for create table with table comment
+ */
+class TestCreateTableWithTableComment extends QueryTest with 
BeforeAndAfterAll {
+
+  override def beforeAll {
+sql("use default")
+sql("drop table if exists withTableComment")
+sql("drop table if exists withoutTableComment")
+  }
+
+  test("test create table with table comment") {
+sql(
+  s"""
+ | create table withTableComment(
+ | id int,
+ | name string
+ | )
+ | comment "This table has table comment"
+ | STORED BY 'carbondata'
+   """.stripMargin
+)
+
+val result = sql("describe formatted withTableComment")
+
+checkExistence(result, true, "Comment:")
+checkExistence(result, true, "This table has table comment")
+  }
+
+  test("test create table without table comment") {
+sql(
+  s"""
+ | create table withoutTableComment(
+ | id int,
+ | name string
+ | )
+ | STORED BY 'carbondata'
+   """.stripMargin
+)
+
--- End diff --

Can existing table will be able to add comment through Alter table command?


---

[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...

2017-10-27 Thread sounakr

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1437#discussion_r147354409
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
 ---
@@ -287,7 +288,8 @@ abstract class CarbonDDLSqlParser extends 
AbstractCarbonSparkSQLParser {
   groupCols,
   Some(colProps),
   bucketFields: Option[BucketFields],
-  partitionInfo)
+  partitionInfo,
+  comment)
--- End diff --

Please check if the all the callers for TableModel case class is covered 
the "comment" parameter or have handled it. For e.g. in def 
createTableInfoFromParam.


---

[GitHub] carbondata pull request #1434: [CARBONDATA-1593]Add partition to table cause...

2017-10-27 Thread sounakr

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1434#discussion_r147375453
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/AlterTableSplitCarbonPartitionCommand.scala
 ---
@@ -109,7 +109,7 @@ case class AlterTableSplitCarbonPartitionCommand(
 CarbonUtil.writeThriftTableToSchemaFile(schemaFilePath, thriftTable)
 // update the schema modified time
 carbonMetaStore.updateAndTouchSchemasUpdatedTime(storePath)
-sparkSession.catalog.refreshTable(tableName)
+sparkSession.catalog.refreshTable(s"$dbName.$tableName")
--- End diff --

Better to call SessionCatalog.refreshTable rather than Catalog refreshTable 
as it will honour the DBName.
sparkSession.sessionState.catalog.refreshTable() and pass TableIdentifier. 


---

[jira] [Closed] (CARBONDATA-1630) load data into hive table fail

2017-10-27 Thread xujie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xujie closed CARBONDATA-1630.
-
Resolution: Fixed

> load data into hive table fail
> --
>
> Key: CARBONDATA-1630
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1630
> Project: CarbonData
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0
> Environment: hive.version:1.1.0-cdh5.10.0
> hadoop:version:2.6.0-cdh5.10.0
>Reporter: xujie
>Priority: Critical
>
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.CarbonSession._
> val rootPath = "hdfs://namenodeb:8020/app/carbondata"
> val storeLocation = s"$rootPath/store"
> val warehouse = s"$rootPath/warehouse"
> val metastoredb = s"$rootPath/metastore_db"
> val carbon = 
> SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", 
> warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION,
>  storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb)
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.Row
> val rdd = sc.textFile("/data/home/hadoop/test.txt");
> val schemaString = "id name city"
> val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, 
> StringType, nullable = true))
> val schema = StructType(fields)
> val rowRDD = rdd.map(_.split(",")).map(attributes => 
> Row(attributes(0),attributes(1),attributes(2)))
> val peopleDF = spark.createDataFrame(rowRDD, schema)
> peopleDF.createOrReplaceTempView("tmp_table")
> spark.sql("insert into target_table SELECT * FROM tmp_table")
> java.lang.RuntimeException: Failed to add entry in table status for 
> default.target_table
>   at scala.sys.package$.error(package.scala:27)
>   at 
> org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533)
>   at 
> org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928)
>   at 
> org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754)
>   at 
> org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651)
>   at 
> org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637)
>   at 
> org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98)
>   at 
> org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
>   at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
>   at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
>   at 
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at 
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
>   at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
>   at 
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
>   at 
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
>   at org.apache.spark.sql.Dataset.(Dataset.scala:185)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
>   at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
>   ... 52 elided



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1327/



---

[jira] [Created] (CARBONDATA-1650) load data into hive table fail

2017-10-27 Thread xujie (JIRA)

xujie created CARBONDATA-1650:
-

 Summary: load data into hive table fail
 Key: CARBONDATA-1650
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1650
 Project: CarbonData
  Issue Type: Bug
  Components: hive-integration
Affects Versions: 1.2.0
 Environment: hive.version:1.1.0-cdh5.10.0
hadoop:version:2.6.0-cdh5.10.0
Reporter: xujie
Priority: Critical


import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val rootPath = "hdfs://namenodeb:8020/app/carbondata"
val storeLocation = s"$rootPath/store"
val warehouse = s"$rootPath/warehouse"
val metastoredb = s"$rootPath/metastore_db"
val carbon = 
SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", 
warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION,
 storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb)
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val rdd = sc.textFile("/data/home/hadoop/test.txt");
val schemaString = "id name city"
val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, 
StringType, nullable = true))
val schema = StructType(fields)
val rowRDD = rdd.map(_.split(",")).map(attributes => 
Row(attributes(0),attributes(1),attributes(2)))
val peopleDF = spark.createDataFrame(rowRDD, schema)
peopleDF.createOrReplaceTempView("tmp_table")
spark.sql("insert into target_table SELECT * FROM tmp_table")

java.lang.RuntimeException: Failed to add entry in table status for 
default.target_table
  at scala.sys.package$.error(package.scala:27)
  at 
org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533)
  at 
org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928)
  at 
org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754)
  at 
org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651)
  at 
org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637)
  at 
org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98)
  at 
org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
  at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
  at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
  at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
  at org.apache.spark.sql.Dataset.(Dataset.scala:185)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
  ... 52 elided



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1326/



---

[GitHub] carbondata issue #1438: [WIP][CARBONDATA-1649]insert overwrite fix during jo...

2017-10-27 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1438
  
@ravipesala @gvramana @QiangCai @manishgupta88 please review this PR


---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/696/



---

[jira] [Created] (CARBONDATA-1648) Change alter table drop partition to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1648:
---

 Summary: Change alter table drop partition to support two level 
partitions
 Key: CARBONDATA-1648
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1648
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1647) Change show partition to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1647:
---

 Summary: Change show partition to support two level partitions
 Key: CARBONDATA-1647
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1647
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1436: [WIP][CARBONDATA-1617] Merging carbonindex files wit...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1436
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1325/



---

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/695/



---

[GitHub] carbondata issue #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement St...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1440
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/694/



---

[jira] [Created] (CARBONDATA-1646) Concurrent performance testing of partition tables

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1646:
---

 Summary: Concurrent performance testing of partition tables
 Key: CARBONDATA-1646
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1646
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1645) Change alter table add/split partition to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1645:
---

 Summary: Change alter table add/split partition to support two 
level partitions
 Key: CARBONDATA-1645
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1645
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1644) Change query process to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1644:
---

 Summary: Change query process to support two level partitions
 Key: CARBONDATA-1644
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1644
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1643) Change load process to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1643:
---

 Summary: Change load process to support two level partitions
 Key: CARBONDATA-1643
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1643
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1642) Implement Hash-Hash partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1642:
---

 Summary: Implement Hash-Hash partitioner
 Key: CARBONDATA-1642
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1642
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1641) Implement Hash-List partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1641:
---

 Summary: Implement Hash-List partitioner
 Key: CARBONDATA-1641
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1641
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1640) Implement Hash-Range partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1640:
---

 Summary: Implement Hash-Range partitioner
 Key: CARBONDATA-1640
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1640
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1639) Implement List-Hash partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1639:
---

 Summary: Implement List-Hash partitioner
 Key: CARBONDATA-1639
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1639
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1638) Implement List-List partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1638:
---

 Summary: Implement List-List partitioner
 Key: CARBONDATA-1638
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1638
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1637) Implement List-Range partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1637:
---

 Summary: Implement List-Range partitioner
 Key: CARBONDATA-1637
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1637
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1636) Implement Range-Hash partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1636:
---

 Summary: Implement Range-Hash partitioner
 Key: CARBONDATA-1636
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1636
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1635) Implement Range-List partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1635:
---

 Summary: Implement Range-List partitioner
 Key: CARBONDATA-1635
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1635
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1634) Implement Range-Range partitioner

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1634:
---

 Summary: Implement Range-Range partitioner
 Key: CARBONDATA-1634
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1634
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1633) Change parser to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1633:
---

 Summary: Change parser to support two level partitions
 Key: CARBONDATA-1633
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1633
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (CARBONDATA-1631) Implement Range Interval Partition

2017-10-27 Thread Cao, Lionel (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cao, Lionel reassigned CARBONDATA-1631:
---

Assignee: (was: Cao, Lionel)

> Implement Range Interval Partition
> --
>
> Key: CARBONDATA-1631
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1631
> Project: CarbonData
>  Issue Type: Sub-task
>  Components: core, spark-integration, sql
>Reporter: Cao, Lionel
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata pull request #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Imple...

2017-10-27 Thread QiangCai

GitHub user QiangCai opened a pull request:

https://github.com/apache/carbondata/pull/1440

[WIP][CARBONDATA-1581][CARBONDATA-1582] Implement StreamSinkProvider and 
stream file writer

1. Change hadoop.version to 2.7.2 as default
Require using truncate operation of the filesystem.

2. CarbonSource extend StreamSinkProvider
Provide stream sink to support streaming ingest

3. Implement CarbonStreamOutputFormat and CarbonStreamRecordWriter
CarbonStreamRecordWriter write input data to CarbonData stream file.

4. Avoid Small file issue
Append new blocklet to old file to avoid small file issue

5. Support fault tolerant
Stream segment has a CarbonIndex file, this index file record the 
information of the CarbonData files.
We can recover data to last successful commit.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/carbondata streaming

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1440


commit 6c94c9311ea1b260e75bf576eec75aea17ce8984
Author: QiangCai 
Date:   2017-10-18T03:13:00Z

support streaming ingest




---

[jira] [Created] (CARBONDATA-1632) Change PartitionInfo and related model to support two level partitions

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1632:
---

 Summary: Change PartitionInfo and related model to support two 
level partitions
 Key: CARBONDATA-1632
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1632
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1631) Implement Range Interval Partition

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1631:
---

 Summary: Implement Range Interval Partition
 Key: CARBONDATA-1631
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1631
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Cao, Lionel
Assignee: Cao, Lionel






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1437
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1324/



---

[jira] [Created] (CARBONDATA-1630) load data into hive table fail

2017-10-27 Thread xujie (JIRA)

xujie created CARBONDATA-1630:
-

 Summary: load data into hive table fail
 Key: CARBONDATA-1630
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1630
 Project: CarbonData
  Issue Type: Bug
  Components: hive-integration
Affects Versions: 1.2.0
 Environment: hive.version:1.1.0-cdh5.10.0
hadoop:version:2.6.0-cdh5.10.0
Reporter: xujie
Priority: Critical


import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val rootPath = "hdfs://namenodeb:8020/app/carbondata"
val storeLocation = s"$rootPath/store"
val warehouse = s"$rootPath/warehouse"
val metastoredb = s"$rootPath/metastore_db"
val carbon = 
SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", 
warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION,
 storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb)
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val rdd = sc.textFile("/data/home/hadoop/test.txt");
val schemaString = "id name city"
val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, 
StringType, nullable = true))
val schema = StructType(fields)
val rowRDD = rdd.map(_.split(",")).map(attributes => 
Row(attributes(0),attributes(1),attributes(2)))
val peopleDF = spark.createDataFrame(rowRDD, schema)
peopleDF.createOrReplaceTempView("tmp_table")
spark.sql("insert into target_table SELECT * FROM tmp_table")

java.lang.RuntimeException: Failed to add entry in table status for 
default.target_table
  at scala.sys.package$.error(package.scala:27)
  at 
org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533)
  at 
org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928)
  at 
org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754)
  at 
org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651)
  at 
org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637)
  at 
org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98)
  at 
org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
  at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
  at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
  at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
  at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
  at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
  at org.apache.spark.sql.Dataset.(Dataset.scala:185)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
  ... 52 elided



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (CARBONDATA-1629) Partition Function Enhancement

2017-10-27 Thread Cao, Lionel (JIRA)

Cao, Lionel created CARBONDATA-1629:
---

 Summary: Partition Function Enhancement
 Key: CARBONDATA-1629
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1629
 Project: CarbonData
  Issue Type: New Feature
  Components: core, spark-integration, sql
Affects Versions: 1.3.0
Reporter: Cao, Lionel
Assignee: Cao, Lionel
 Fix For: 1.3.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1439
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/693/



---

[GitHub] carbondata pull request #1439: [CARBONDATA-1628] Re-factory LoadTableCommand...

2017-10-27 Thread QiangCai

GitHub user QiangCai opened a pull request:

https://github.com/apache/carbondata/pull/1439

[CARBONDATA-1628] Re-factory LoadTableCommand to reuse code for streaming 
ingest in the future

Re-factory LoadTableCommand to reuse code for streaming ingest in the future

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/carbondata refactory_loadtable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1439


commit 53e4ad765cc530446980626a638dd6f90da5a692
Author: QiangCai 
Date:   2017-10-27T08:06:06Z

refactory_loadtable




---

[jira] [Created] (CARBONDATA-1628) Re-factory LoadTableCommand to reuse code for streaming ingest in the future

2017-10-27 Thread QiangCai (JIRA)

QiangCai created CARBONDATA-1628:


 Summary: Re-factory LoadTableCommand to reuse code for streaming 
ingest in the future
 Key: CARBONDATA-1628
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1628
 Project: CarbonData
  Issue Type: Improvement
  Components: spark-integration
Reporter: QiangCai
Priority: Minor


Re-factory LoadTableCommand to reuse code for streaming ingest in the future



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1438: [WIP]insert overwrite fix

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1438
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1323/



---

[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1433
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/692/



---

[GitHub] carbondata issue #1436: [WIP][CARBONDATA-1617] Merging carbonindex files wit...

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1436
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/691/



---

[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores

2017-10-27 Thread Zhichao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221856#comment-16221856
 ] 

Zhichao  Zhang commented on CARBONDATA-1624:


Hi [~ravi.pesala], why does it impact other loads?  I will implement this as 
following steps:
1. Add a parameter called 'carbon.number.of.cores.in.executor' to 
CarbonCommonConstants;
2. In 'CarbonSession.getOrCreateCarbonSession', get the core number of executor 
from SparkConf (using sparkContext.getConf.get("spark.executor.cores")) , and 
then set this number to CarbonProperties, the key is 
'carbon.number.of.cores.in.executor';
3. Change CarbonProperties.getNumberOfCores(), use the value of 
'carbon.number.of.cores.in.executor' as the default value of 
'carbon.number.of.cores.while.loading'; if the value of 
'carbon.number.of.cores.in.executor' is null or empty, use 
'NUM_CORES_DEFAULT_VAL' as the default value.

Are the above steps OK?

Do you mean that it can't change the below code snippet to 
'CarbonProperties.getNumberOfCores()'?


> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 
> 'carbon.number.of.cores.while.loading' dynamically as per the available 
> executor cores 
> 
>
> Key: CARBONDATA-1624
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
> Project: CarbonData
>  Issue Type: Improvement
>  Components: data-load, spark-integration
>Affects Versions: 1.3.0
>Reporter: Zhichao  Zhang
>Assignee: Zhichao  Zhang
>Priority: Minor
>
> If we are using carbondata + spark to load data, we can set 
> carbon.number.of.cores.while.loading to the  number of executor cores. 
> For example, when set the number of executor cores to 6, it shows that there 
> are at 
> least 6 cores per node for loading data, so we can set 
> carbon.number.of.cores.while.loading to 6 automatically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...

2017-10-27 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1418
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/1322/



---

[GitHub] carbondata issue #1438: [WIP]insert overwrite fix

2017-10-27 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1438
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/690/



---

[GitHub] carbondata issue #1432: [WIP][CARBONDATA-1608]Support Column Comment for Cre...

2017-10-27 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1432
  
@ravipesala please review


---

[GitHub] carbondata issue #1435: [WIP][CARBONDATA-1626]add data size and index size i...

2017-10-27 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1435
  
@ravipesala please review


---

[GitHub] carbondata issue #1435: [WIP][CARBONDATA-1626]add data size and index size i...

2017-10-27 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1435
  
@gvramana please review


---

[GitHub] carbondata issue #1432: [WIP][CARBONDATA-1608]Support Column Comment for Cre...

2017-10-27 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1432
  
@gvramana please review


---

[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores

2017-10-27 Thread Ravindra Pesala (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221758#comment-16221758
 ] 

Ravindra Pesala commented on CARBONDATA-1624:
-

Welcome to contribute.
We should not use the carbonproperties anymore for this dynamic cores as it 
impacts other loads. First find the available cores which we can allocate for 
loading per executor before  submitting and pass the same information to carbon 
in RDD compute.

> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 
> 'carbon.number.of.cores.while.loading' dynamically as per the available 
> executor cores 
> 
>
> Key: CARBONDATA-1624
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
> Project: CarbonData
>  Issue Type: Improvement
>  Components: data-load, spark-integration
>Affects Versions: 1.3.0
>Reporter: Zhichao  Zhang
>Assignee: Zhichao  Zhang
>Priority: Minor
>
> If we are using carbondata + spark to load data, we can set 
> carbon.number.of.cores.while.loading to the  number of executor cores. 
> For example, when set the number of executor cores to 6, it shows that there 
> are at 
> least 6 cores per node for loading data, so we can set 
> carbon.number.of.cores.while.loading to 6 automatically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

99 matches

Mail list logo