[jira] [Created] (CARBONDATA-1108) Support delete operation in vector reader of Spark 2.1

[Ravindra Pesala (JIRA)]

Ravindra Pesala created CARBONDATA-1108:
---

 Summary: Support delete operation in vector reader of Spark 2.1
 Key: CARBONDATA-1108
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1108
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Ravindra Pesala


Support delete operation in vector reader of Spark 2.1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata pull request #954: [CARBONDATA-947][CARBONDATA-948] IUD support i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/954


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #954: [CARBONDATA-947][CARBONDATA-948] IUD support in 2.1

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/954
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #975: [Documentation] Single pass condition for high cardin...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/975
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #975: [Documentation] Single pass condition for high cardin...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/975
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #945: [CARBONDATA-1088] Minimize the driver side blo...

[ravipesala]

Github user ravipesala closed the pull request at:

https://github.com/apache/carbondata/pull/945


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (CARBONDATA-1102) Selecting Int type in hive from carbon table is showing class cast exception

[anubhav tarar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029633#comment-16029633
 ] 

anubhav tarar edited comment on CARBONDATA-1102 at 5/30/17 4:21 PM:


[~cenyuhai] i am working on some prs for hive, all of them are failing due to 
this issue specialy the carbondata vs orc performance pr i guess earlier 
short,int were taking as long but now they are taking their respective ranges 
in master branch, so i think its better to remove this issue earlier so i can 
easily continue ,can i create a seperate pr for correcting them?


was (Author: anubhavtarar):
[~cenyuhai] i am working on some prs for hive, all of them are failing due to 
this issue specialy the hive vs orc performance pr i guess earlier short,int 
were taking as long but now they are taking their respective ranges in master 
branch, so i think its better to remove this issue earlier so i can easily 
continue ,can i create a seperate pr for correcting them?

> Selecting Int type in hive from carbon table is showing class cast exception
> 
>
> Key: CARBONDATA-1102
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1102
> Project: CarbonData
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0
> Environment: hive,spark 2.1
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>Priority: Trivial
>
> in carbon
> 0: jdbc:hive2://localhost:1> CREATE TABLE ALLDATATYPETEST(ID INT,NAME 
> STRING,SALARY DECIMAL,MARKS DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) 
> STORED BY 'CARBONDATA' ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (3.702 seconds)
> 0: jdbc:hive2://localhost:1> LOAD DATA INPATH 
> 'hdfs://localhost:54310/alldatatypetest.csv' into table alldatatypetest;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (7.16 seconds)
> 0: jdbc:hive2://localhost:1> SELECT * FROM ALLDATATYPETEST;
> +-++-++--++--+
> | ID  |NAME| SALARY  | MARKS  | JOININGDATE  |  LEAVINGDATE   
> |
> +-++-++--++--+
> | 1   | 'ANUBHAV'  | 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> | 2   | 'LIANG'| 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> +-++-++--++--+
> 2 rows selected (1.978 seconds)
> in hive
> hive> CREATE TABLE ALLDATATYPETEST(ID INT,NAME STRING,SALARY DECIMAL,MARKS 
> DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) ROW FORMAT SERDE 
> 'org.apache.carbondata.hive.CarbonHiveSerDe' STORED AS INPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonInputFormat' OUTPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonOutputFormat' TBLPROPERTIES 
> ('spark.sql.sources.provider'='org.apache.spark.sql.CarbonSource');
> OK
> Time taken: 1.934 seconds
> hive> ALTER TABLE ALLDATATYPETEST SET LOCATION 
> 'hdfs://localhost:54310/opt/carbonStore/default/alldatatypetest';
> OK
> Time taken: 1.192 seconds
> hive> SELECT * FROM ALLDATATYPETEST;
> OK
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> java.lang.Integer cannot be cast to java.lang.Long
> Time taken: 0.174 seconds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CARBONDATA-1102) Selecting Int type in hive from carbon table is showing class cast exception

[anubhav tarar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029633#comment-16029633
 ] 

anubhav tarar commented on CARBONDATA-1102:
---

[~cenyuhai] i am working on some prs for hive, all of them are failing due to 
this issue specialy the hive vs orc performance pr i guess earlier short,int 
were taking as long but now they are taking their respective ranges in master 
branch, so i think its better to remove this issue earlier so i can easily 
continue ,can i create a seperate pr for correcting them?

> Selecting Int type in hive from carbon table is showing class cast exception
> 
>
> Key: CARBONDATA-1102
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1102
> Project: CarbonData
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0
> Environment: hive,spark 2.1
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>Priority: Trivial
>
> in carbon
> 0: jdbc:hive2://localhost:1> CREATE TABLE ALLDATATYPETEST(ID INT,NAME 
> STRING,SALARY DECIMAL,MARKS DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) 
> STORED BY 'CARBONDATA' ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (3.702 seconds)
> 0: jdbc:hive2://localhost:1> LOAD DATA INPATH 
> 'hdfs://localhost:54310/alldatatypetest.csv' into table alldatatypetest;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (7.16 seconds)
> 0: jdbc:hive2://localhost:1> SELECT * FROM ALLDATATYPETEST;
> +-++-++--++--+
> | ID  |NAME| SALARY  | MARKS  | JOININGDATE  |  LEAVINGDATE   
> |
> +-++-++--++--+
> | 1   | 'ANUBHAV'  | 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> | 2   | 'LIANG'| 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> +-++-++--++--+
> 2 rows selected (1.978 seconds)
> in hive
> hive> CREATE TABLE ALLDATATYPETEST(ID INT,NAME STRING,SALARY DECIMAL,MARKS 
> DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) ROW FORMAT SERDE 
> 'org.apache.carbondata.hive.CarbonHiveSerDe' STORED AS INPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonInputFormat' OUTPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonOutputFormat' TBLPROPERTIES 
> ('spark.sql.sources.provider'='org.apache.spark.sql.CarbonSource');
> OK
> Time taken: 1.934 seconds
> hive> ALTER TABLE ALLDATATYPETEST SET LOCATION 
> 'hdfs://localhost:54310/opt/carbonStore/default/alldatatypetest';
> OK
> Time taken: 1.192 seconds
> hive> SELECT * FROM ALLDATATYPETEST;
> OK
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> java.lang.Integer cannot be cast to java.lang.Long
> Time taken: 0.174 seconds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata pull request #976: [WIP] Added set command in carbon to update pr...

[ManoharVanam]

GitHub user ManoharVanam opened a pull request:

https://github.com/apache/carbondata/pull/976

[WIP] Added set command in carbon to update properties dynamically

Added set command in carbon to update properties dynamically

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ManoharVanam/incubator-carbondata 
dynamic_manohar_dummy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #976


commit f60e36679b7ee3e524e1ab7ffbc135b935bb8946
Author: ravipesala 
Date:   2017-05-18T09:34:17Z

1. added log message for addition of property value
2. throwing exception when attempt to add key, value for non dynamic carbon 
parameter.

Refactored the code

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java

Added thresholdLocal session params

validation

# Conflicts:
#   
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

Added ThreadLocalSession params

Dynamic parameter support

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java

Adding session based properties

Added set command in carbon to update properties dynamically




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (CARBONDATA-1102) Selecting Int type in hive from carbon table is showing class cast exception

[cen yuhai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029567#comment-16029567
 ] 

cen yuhai commented on CARBONDATA-1102:
---

I will fix it in CARBON-1008

> Selecting Int type in hive from carbon table is showing class cast exception
> 
>
> Key: CARBONDATA-1102
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1102
> Project: CarbonData
>  Issue Type: Bug
>  Components: hive-integration
>Affects Versions: 1.2.0
> Environment: hive,spark 2.1
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>Priority: Trivial
>
> in carbon
> 0: jdbc:hive2://localhost:1> CREATE TABLE ALLDATATYPETEST(ID INT,NAME 
> STRING,SALARY DECIMAL,MARKS DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) 
> STORED BY 'CARBONDATA' ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (3.702 seconds)
> 0: jdbc:hive2://localhost:1> LOAD DATA INPATH 
> 'hdfs://localhost:54310/alldatatypetest.csv' into table alldatatypetest;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (7.16 seconds)
> 0: jdbc:hive2://localhost:1> SELECT * FROM ALLDATATYPETEST;
> +-++-++--++--+
> | ID  |NAME| SALARY  | MARKS  | JOININGDATE  |  LEAVINGDATE   
> |
> +-++-++--++--+
> | 1   | 'ANUBHAV'  | 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> | 2   | 'LIANG'| 20  | 100.0  | 2016-04-14   | 2016-04-14 15:00:09.0  
> |
> +-++-++--++--+
> 2 rows selected (1.978 seconds)
> in hive
> hive> CREATE TABLE ALLDATATYPETEST(ID INT,NAME STRING,SALARY DECIMAL,MARKS 
> DOUBLE,JOININGDATE DATE,LEAVINGDATE TIMESTAMP) ROW FORMAT SERDE 
> 'org.apache.carbondata.hive.CarbonHiveSerDe' STORED AS INPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonInputFormat' OUTPUTFORMAT 
> 'org.apache.carbondata.hive.MapredCarbonOutputFormat' TBLPROPERTIES 
> ('spark.sql.sources.provider'='org.apache.spark.sql.CarbonSource');
> OK
> Time taken: 1.934 seconds
> hive> ALTER TABLE ALLDATATYPETEST SET LOCATION 
> 'hdfs://localhost:54310/opt/carbonStore/default/alldatatypetest';
> OK
> Time taken: 1.192 seconds
> hive> SELECT * FROM ALLDATATYPETEST;
> OK
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> java.lang.Integer cannot be cast to java.lang.Long
> Time taken: 0.174 seconds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata pull request #945: [CARBONDATA-1088] Minimize the driver side blo...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/945#discussion_r119111383
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datastore/impl/array/BlockArrayIndexBuilder.java
 ---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.datastore.impl.array;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.carbondata.core.datastore.BTreeBuilderInfo;
+import org.apache.carbondata.core.datastore.BtreeBuilder;
+import org.apache.carbondata.core.datastore.DataRefNode;
+import org.apache.carbondata.core.datastore.block.BlockInfo;
+import org.apache.carbondata.core.memory.CarbonUnsafe;
+import org.apache.carbondata.core.memory.MemoryAllocator;
+import org.apache.carbondata.core.memory.MemoryAllocatorFactory;
+import org.apache.carbondata.core.memory.MemoryBlock;
+import org.apache.carbondata.core.metadata.blocklet.DataFileFooter;
+import 
org.apache.carbondata.core.metadata.blocklet.index.BlockletMinMaxIndex;
+
+/**
+ * Builder for storing block index in unsafe array format
+ */
+public class BlockArrayIndexBuilder implements BtreeBuilder {
--- End diff --

We should use interface defined by #958 
Suggest to merge #958 first then rebase on it, ok?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #909: [CARBONDATA-1049] avoid logging raw data into ...

[kumarvishal09]

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/909#discussion_r118451182
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
 ---
@@ -522,4 +522,16 @@ public static String 
checkAndCreateCarbonStoreLocation(String factStoreLocation,
 return aggType;
   }
 
+  /**
+   * the method prepares and return the message mentioning the reason of 
badrecord
+   *
+   * @param columnName
+   * @param dataType
+   * @return
+   */
+  public static String prepareFailureReason(String columnName, DataType 
dataType) {
--- End diff --

Once data loading is done or step which is handling bad record is finished 
map will be discarded so it is ok(Only when bad record is enabled). My 
intention for this comment was to create this message only once, don't create 
this for each bad record. Or you can do lazy creation, when you found first bad 
record then create the message, from next time use the same message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #952: [CARBONDATA-1094] Wrong results returned by the query...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/952
  
restest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #952: [CARBONDATA-1094] Wrong results returned by the query...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/952
  
LGTM
@sounakr Please have a look on range filter changes. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-1107) Multi User load on same table is failing with NullPointerException

[Naresh P R (JIRA)]

Naresh P R created CARBONDATA-1107:
--

 Summary: Multi User load on same table is failing with 
NullPointerException
 Key: CARBONDATA-1107
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1107
 Project: CarbonData
  Issue Type: Bug
 Environment: Spark-2.1
Reporter: Naresh P R
Assignee: Naresh P R
Priority: Minor


User1 in beeline terminal1 performs below operation
Create table t1
load data into t1
User2(Does not have privilege to read t1) in beeline terminal2 performs below 
operation
load data into t1 -- This command will fail for table folder permission and 
carbonTable will be removed from CarbonMetadata
User1 in beeline terminal1 performs below operation
load data into t1 -- This command will throw NullPointerException for 
carbonTable.
from CarbonMetadata.
This is corrected by fetching the CarbonTable from CarbonMetastore of 
corresponding Session.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #973: [WIP] Multi User load on same table is failing with N...

[nareshpr]

Github user nareshpr commented on the issue:

https://github.com/apache/carbondata/pull/973
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (CARBONDATA-1047) Add load options to perform batch sort and add more testcases

[Pallavi Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029349#comment-16029349
 ] 

Pallavi Singh commented on CARBONDATA-1047:
---

There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1086
(https://issues.apache.org/jira/browse/CARBONDATA-1086)

> Add load options to perform batch sort and add more testcases
> -
>
> Key: CARBONDATA-1047
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1047
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
> Fix For: 1.2.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Add load options to perform batch sort and add more testcases.
> Add options like below to the load command for batch sort.
> {code}
> LOAD DATA LOCAL INPATH '$filePath' into table carbon_load1 
> OPTIONS('batch_sort'='true', 'batch_sort_size_inmb'='1')
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CARBONDATA-742) Add batch sort to improve the loading performance

[Pallavi Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029348#comment-16029348
 ] 

Pallavi Singh commented on CARBONDATA-742:
--

There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1086
(https://issues.apache.org/jira/browse/CARBONDATA-1086)

> Add batch sort to improve the loading performance
> -
>
> Key: CARBONDATA-742
> URL: https://issues.apache.org/jira/browse/CARBONDATA-742
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
>Assignee: Ravindra Pesala
> Fix For: 1.1.0
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Current Problem:
> Sort step is major issue as it is blocking step. It needs to receive all data 
> and write down the sort temp files to disk, after that only data writer step 
> can start.
> Solution: 
> Make sort step as non blocking step so it avoids waiting of Data writer step.
> Process the data in sort step in batches with size of in-memory capability of 
> the machine. For suppose if machine can allocate 4 GB to process data 
> in-memory, then Sort step can sorts the data with batch size of 2GB and gives 
> it to the data writer step. By the time data writer step consumes the data, 
> sort step receives and sorts the data. So here all steps are continuously 
> working and absolutely there is no disk IO in sort step.
> So there would not be any waiting of data writer step for sort step, As and 
> when sort step sorts the data in memory data writer can start writing it.
> It can significantly improves the performance.
> Advantages:
> Increases the loading performance as there is no intermediate IO and no 
> blocking of Sort step.
> There is no extra effort for compaction, the current flow can handle it.
> Disadvantages:
> Number of driver side btrees will increase. So the memory might increase but 
> it could be controlled by current LRU cache implementation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CARBONDATA-766) Size based blocklet for V3

[Pallavi Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029346#comment-16029346
 ] 

Pallavi Singh commented on CARBONDATA-766:
--

There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1085
(https://issues.apache.org/jira/browse/CARBONDATA-1085)

> Size based blocklet for V3
> --
>
> Key: CARBONDATA-766
> URL: https://issues.apache.org/jira/browse/CARBONDATA-766
> Project: CarbonData
>  Issue Type: Bug
>Reporter: kumar vishal
>Assignee: kumar vishal
> Fix For: 1.1.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Currently number of pages is based on configured fixed value(number of pages 
> per blocklet) , problem with this approach is in some cases blocklet size 
> will be less and it will cause more number of IO, to avoid this we can have 
> size based blocklet , in this case how many pages it will fit in blocklet 
> will based on configure size, so number of IO will be uniform



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CARBONDATA-726) Update with V3 format for better IO and processing optimization.

[Pallavi Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029343#comment-16029343
 ] 

Pallavi Singh edited comment on CARBONDATA-726 at 5/30/17 12:24 PM:


There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1084
(https://issues.apache.org/jira/browse/CARBONDATA-1084)


was (Author: pallavisingh_09):
There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1084

> Update with V3 format for better IO and processing optimization.
> 
>
> Key: CARBONDATA-726
> URL: https://issues.apache.org/jira/browse/CARBONDATA-726
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
> Fix For: 1.1.0
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> Problems in current format.
> 1. IO read is slower since it needs to go for multiple seeks on the file to 
> read column blocklets. Current size of blocklet is 12, so it needs to 
> read multiple times from file to scan the data on that column. Alternatively 
> we can increase the blocklet size but it suffers for filter queries as it 
> gets big blocklet to filter.
> 2. Decompression is slower in current format, we are using inverted index for 
> faster filter queries and using NumberCompressor to compress the inverted 
> index in bit wise packing. It becomes slower so we should avoid number 
> compressor. One alternative is to keep blocklet size with in 32000 so that 
> inverted index can be written with short, but IO read suffers a lot.
> To overcome from above 2 issues we are introducing new format V3.
> Here each blocklet has multiple pages with size 32000, number of pages in 
> blocklet is configurable. Since we keep the page with in short limit so no 
> need compress the inverted index here.
> And maintain the max/min for each page to further prune the filter queries.
> Read the blocklet with pages at once and keep in offheap memory.
> During filter first check the max/min range and if it is valid then go for 
> decompressing the page to filter further.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CARBONDATA-726) Update with V3 format for better IO and processing optimization.

[Pallavi Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029343#comment-16029343
 ] 

Pallavi Singh commented on CARBONDATA-726:
--

There was Documentation Impact for the issue. Raised and fixed that in JIRA 
ISSUE: 1084

> Update with V3 format for better IO and processing optimization.
> 
>
> Key: CARBONDATA-726
> URL: https://issues.apache.org/jira/browse/CARBONDATA-726
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
> Fix For: 1.1.0
>
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> Problems in current format.
> 1. IO read is slower since it needs to go for multiple seeks on the file to 
> read column blocklets. Current size of blocklet is 12, so it needs to 
> read multiple times from file to scan the data on that column. Alternatively 
> we can increase the blocklet size but it suffers for filter queries as it 
> gets big blocklet to filter.
> 2. Decompression is slower in current format, we are using inverted index for 
> faster filter queries and using NumberCompressor to compress the inverted 
> index in bit wise packing. It becomes slower so we should avoid number 
> compressor. One alternative is to keep blocklet size with in 32000 so that 
> inverted index can be written with short, but IO read suffers a lot.
> To overcome from above 2 issues we are introducing new format V3.
> Here each blocklet has multiple pages with size 32000, number of pages in 
> blocklet is configurable. Since we keep the page with in short limit so no 
> need compress the inverted index here.
> And maintain the max/min for each page to further prune the filter queries.
> Read the blocklet with pages at once and keep in offheap memory.
> During filter first check the max/min range and if it is valid then go for 
> decompressing the page to filter further.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #791: [CARBONDATA-920] Updated useful-tips-on-carbondata.md

[vandana7]

Github user vandana7 commented on the issue:

https://github.com/apache/carbondata/pull/791
  
@chenliang613 please review, resolved merge conflicts


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #972: [WIP] Added set command in carbon to update pr...

[ManoharVanam]

GitHub user ManoharVanam reopened a pull request:

https://github.com/apache/carbondata/pull/972

[WIP] Added set command in carbon to update properties dynamically

Currently there is no way to set the carbon properties dynamically, always 
need to update the carbon.properties file and restart the driver.
So it will be easy for users to give option to add/update the properties 
dynamically through set command.
The below sql command updates the carbon properties in driver and as well 
as in executor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ManoharVanam/incubator-carbondata dynamic_new

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/972.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #972


commit 48761df4fcf91a8203453b272266f54c2c18
Author: ravipesala 
Date:   2017-05-18T09:34:17Z

Adding session based properties

Added set command in carbon to update properties dynamically

commit cbe6c2f61774cccd1222516774489b7841b2cc0a
Author: Manohar 
Date:   2017-05-25T13:40:21Z

Added ThreadLocalSession params

Dynamic parameter support

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java

commit 762f5564221964941b970e4c216ed019fca286c4
Author: mohammadshahidkhan 
Date:   2017-05-26T17:08:59Z

validation

# Conflicts:
#   
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

commit 74727b5de5d774a9cfc19072d8e1e2653953ef36
Author: Manohar 
Date:   2017-05-29T12:28:01Z

Added thresholdLocal session params

commit da67426ae80e6ac42fdfd1b2df5a2d45c231b548
Author: Manohar 
Date:   2017-05-29T15:46:26Z

Refactored the code

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #973: [WIP] Multi User load on same table is failing with N...

[nareshpr]

Github user nareshpr commented on the issue:

https://github.com/apache/carbondata/pull/973
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #975: [Documentation] Single pass condition for high...

[sgururajshetty]

GitHub user sgururajshetty opened a pull request:

https://github.com/apache/carbondata/pull/975

[Documentation] Single pass condition for high cardinality

Added the information that when single pass is used, then high cardinality 
enable property will be disabled during the data load.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sgururajshetty/carbondata master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/975.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #975


commit bc17d92bb1ea99dcb677abaf95cce411813d9a6f
Author: sgururajshetty 
Date:   2017-05-30T11:12:32Z

condition for single pass




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #972: [WIP] Added set command in carbon to update propertie...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/972
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #970: [CARBONDATA-1104] Query failure while using un...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/970


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #974: [CARBONDATA-1106] Added support for complex da...

[geetikagupta16]

GitHub user geetikagupta16 opened a pull request:

https://github.com/apache/carbondata/pull/974

[CARBONDATA-1106] Added support for complex datatype Array in presto

Added functionality for Complex datatype array

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/geetikagupta16/incubator-carbondata 
feature/PrestoComplexArray

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/974.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #974


commit 722792ac9aa765f18c1d5870aa0115a12cd25b01
Author: chenliang613 
Date:   2017-04-02T02:27:12Z

fix typo issues of integration/ presto

commit 18edcc1591ae1dc8b45dce2b62bd82142fc5730e
Author: ffpeng90 
Date:   2017-04-05T15:00:50Z

[CARBONDATA-872] Fix comment issues of integration/presto for easier reading

fix comments

Add comments for some classes and functions

update annotation format

delete checkstyle directory

commit 18cb4c0fa72c5d7a6993f32883cef738030ee10c
Author: Geetika gupta 
Date:   2017-04-19T10:05:45Z

Added functionality for Complex datatype array

Refactored code in record cursor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-1106) Add support for Complex Datatype Array in presto

[Geetika Gupta (JIRA)]

Geetika Gupta created CARBONDATA-1106:
-

 Summary: Add support for Complex Datatype Array in presto
 Key: CARBONDATA-1106
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1106
 Project: CarbonData
  Issue Type: Sub-task
  Components: presto-integration
Affects Versions: 1.2.0
Reporter: Geetika Gupta






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #966: [CARBONDATA-1101] Avoid widening between wrapper clas...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/966
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #927: [CARBONDATA-1066] ArrayIndexOutBoundException ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/927


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #973: [WIP] Multi User load on same table is failing...

[nareshpr]

GitHub user nareshpr opened a pull request:

https://github.com/apache/carbondata/pull/973

[WIP] Multi User load on same table is failing with NullPointerException.

User1 in beeline terminal1 performs below operation
Create table t1
load data into t1
User2(Does not have privilege to read t1) in beeline terminal2 performs 
below operation
load data into t1 -- This command will fail for table folder permission and 
carbonTable will be removed from CarbonMetadata
User1 in beeline terminal1 performs below operation
load data into t1 -- This command will throw NullPointerException for 
carbonTable.
from CarbonMetadata.
This is corrected by fetching the CarbonTable from CarbonMetastore of 
corresponding Session.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nareshpr/incubator-carbondata 
ConcurrentOperationOnSameTable

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/973.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #973


commit 7c3020d093ea8e79a55e110b4b2b89f6e5f5fc3a
Author: nareshpr 
Date:   2017-05-30T09:18:10Z

MultiClient Load is failing




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #909: [CARBONDATA-1049] avoid logging raw data into ...

[mohammadshahidkhan]

GitHub user mohammadshahidkhan reopened a pull request:

https://github.com/apache/carbondata/pull/909

[CARBONDATA-1049] avoid logging raw data into driver and executor log



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata 
logger_optimaztion

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #909


commit 733c9abaee56a381056f9e284346582f236d45ca
Author: mohammadshahidkhan 
Date:   2017-05-12T07:06:42Z

[CARBONDATA-1049] Avoid logging raw data into driver and executor log.

commit b93f7a287b3da9327bfd6a1d84249b8274cbb0d7
Author: mohammadshahidkhan 
Date:   2017-05-24T06:53:22Z

[CARBONDATA-1049] Avoid logging raw data into driver and executor log.// 
added isDebuggEnabled check




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #909: [CARBONDATA-1049] avoid logging raw data into ...

[mohammadshahidkhan]

Github user mohammadshahidkhan closed the pull request at:

https://github.com/apache/carbondata/pull/909


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #909: [CARBONDATA-1049] avoid logging raw data into ...

[mohammadshahidkhan]

GitHub user mohammadshahidkhan reopened a pull request:

https://github.com/apache/carbondata/pull/909

[CARBONDATA-1049] avoid logging raw data into driver and executor log



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata 
logger_optimaztion

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/909.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #909


commit 733c9abaee56a381056f9e284346582f236d45ca
Author: mohammadshahidkhan 
Date:   2017-05-12T07:06:42Z

[CARBONDATA-1049] Avoid logging raw data into driver and executor log.

commit fea634f57bd8cdaf398c6614f5d61be08461a07b
Author: mohammadshahidkhan 
Date:   2017-05-24T06:53:22Z

[CARBONDATA-1049] Avoid logging raw data into driver and executor log.// 
added isDebuggEnabled check




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #909: [CARBONDATA-1049] avoid logging raw data into ...

[mohammadshahidkhan]

Github user mohammadshahidkhan closed the pull request at:

https://github.com/apache/carbondata/pull/909


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (CARBONDATA-1105) ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf

[Liang Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028811#comment-16028811
 ] 

Liang Chen commented on CARBONDATA-1105:


[~cenyuhai]  Currently, CarbonData doesn't support Spark2.1.1

> ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
> --
>
> Key: CARBONDATA-1105
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1105
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.2.0
> Environment: spark 2.1.1
>Reporter: cen yuhai
>
> I think it is related to SPARK-19944
> https://github.com/apache/spark/pull/17301
> {code}
> scala> carbon.sql("create table temp.test_carbon(id int, name string, scale 
> decimal, country string, salary double) STORED BY 'carbondata'")
> java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
>   at 
> org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
>   at 
> org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
>   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
>   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
>   at 
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>   at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:593)
>   ... 52 elided
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.sql.catalyst.CatalystConf
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   ... 59 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CARBONDATA-1105) ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf

[cen yuhai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028810#comment-16028810
 ] 

cen yuhai commented on CARBONDATA-1105:
---

we should rebuild carbon with spark2.1.1. mvn clean package 
-Dspark.version=2.1.1 -Pspark-2.1 -DskipTests

> ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf
> --
>
> Key: CARBONDATA-1105
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1105
> Project: CarbonData
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.2.0
> Environment: spark 2.1.1
>Reporter: cen yuhai
>
> I think it is related to SPARK-19944
> https://github.com/apache/spark/pull/17301
> {code}
> scala> carbon.sql("create table temp.test_carbon(id int, name string, scale 
> decimal, country string, salary double) STORED BY 'carbondata'")
> java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
>   at 
> org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
>   at 
> org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
>   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
>   at 
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
>   at 
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
>   at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:593)
>   ... 52 elided
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.sql.catalyst.CatalystConf
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   ... 59 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata pull request #972: [WIP] Added set command in carbon to update pr...

[ManoharVanam]

GitHub user ManoharVanam opened a pull request:

https://github.com/apache/carbondata/pull/972

[WIP] Added set command in carbon to update properties dynamically

Currently there is no way to set the carbon properties dynamically, always 
need to update the carbon.properties file and restart the driver.
So it will be easy for users to give option to add/update the properties 
dynamically through set command.
The below sql command updates the carbon properties in driver and as well 
as in executor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ManoharVanam/incubator-carbondata dynamic_new

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/972.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #972


commit 018e9b43130b92d293c2090146c4c84597be4965
Author: ravipesala 
Date:   2017-05-18T09:34:17Z

Adding session based properties

Added set command in carbon to update properties dynamically

commit 6daf608b1ae701ce6a4c1d98236a24c3be98bbf1
Author: Manohar 
Date:   2017-05-25T13:40:21Z

Added ThreadLocalSession params

Dynamic parameter support

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java

commit 13b52243502a0211de66271253753e864512ba25
Author: mohammadshahidkhan 
Date:   2017-05-26T17:08:59Z

validation

# Conflicts:
#   
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

commit a48ea746fd595df0ea005879318a0be1df10b0d5
Author: Manohar 
Date:   2017-05-29T12:28:01Z

Added thresholdLocal session params

commit 63b68c0835c3a063cbf8ab499f499a43dc2754e7
Author: Manohar 
Date:   2017-05-29T15:46:26Z

Refactored the code

# Conflicts:
#   
processing/src/main/java/org/apache/carbondata/processing/model/CarbonLoadModel.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (CARBONDATA-1065) Implement set command in carbon to update carbon properties dynamically

[Manohar Vanam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manohar Vanam reassigned CARBONDATA-1065:
-

Assignee: Manohar Vanam  (was: Ravindra Pesala)

> Implement set command in carbon to update carbon properties dynamically
> ---
>
> Key: CARBONDATA-1065
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1065
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Ravindra Pesala
>Assignee: Manohar Vanam
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently there is no way to set the carbon properties dynamically, always 
> need to update the carbon.properties file and restart the driver.
> So it will be easy for users to give option to add/update the properties 
> dynamically through set command.
> The below sql command updates the carbon properties in driver and as well as 
> in executor
> {code}
> set key1=value1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #970: [CARBONDATA-1104] Query failure while using unsafe fo...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/970
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #927: [CARBONDATA-1066] ArrayIndexOutBoundException for Dic...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/927
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #927: [CARBONDATA-1066] ArrayIndexOutBoundException for Dic...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/927
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-1105) ClassNotFoundException: org.apache.spark.sql.catalyst.CatalystConf

[cen yuhai (JIRA)]

cen yuhai created CARBONDATA-1105:
-

 Summary: ClassNotFoundException: 
org.apache.spark.sql.catalyst.CatalystConf
 Key: CARBONDATA-1105
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1105
 Project: CarbonData
  Issue Type: Bug
  Components: core
Affects Versions: 1.2.0
 Environment: spark 2.1.1
Reporter: cen yuhai


I think it is related to SPARK-19944
https://github.com/apache/spark/pull/17301
{code}
scala> carbon.sql("create table temp.test_carbon(id int, name string, scale 
decimal, country string, salary double) STORED BY 'carbondata'")
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
  at 
org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
  at 
org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
  at 
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
  at 
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
  at 
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:593)
  ... 52 elided
Caused by: java.lang.ClassNotFoundException: 
org.apache.spark.sql.catalyst.CatalystConf
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  ... 59 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #950: [CARBONDATA-1092] alter table should support no_inver...

[rahulforallp]

Github user rahulforallp commented on the issue:

https://github.com/apache/carbondata/pull/950
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---