[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2411
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5534/



---


[GitHub] carbondata issue #2411: [CARBONDATA-2654][Datamap] Optimize output for expla...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2411
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5504/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6677/



---


[GitHub] carbondata issue #2410: [CARBONDATA-2650][Datamap] Fix bugs in negative numb...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2410
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5503/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5533/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5532/



---


[GitHub] carbondata issue #2410: [CARBONDATA-2650][Datamap] Fix bugs in negative numb...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2410
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6676/



---


[GitHub] carbondata issue #2410: [CARBONDATA-2650][Datamap] Fix bugs in negative numb...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2410
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5531/



---


[GitHub] carbondata issue #2408: [CARBONDATA-2653][BloomDataMap] Fix bugs in incorrec...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2408
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5502/



---


[GitHub] carbondata issue #2408: [CARBONDATA-2653][BloomDataMap] Fix bugs in incorrec...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2408
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6675/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5501/



---


[GitHub] carbondata issue #2408: [CARBONDATA-2653][BloomDataMap] Fix bugs in incorrec...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2408
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5530/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6674/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5500/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
@KanakaKumar CI pass, please check again.


---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5529/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6672/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5499/



---


[GitHub] carbondata pull request #2410: [CARBONDATA-2650][Datamap] Fix bugs in negati...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2410#discussion_r199310663
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -433,46 +434,57 @@ protected Expression 
getFilterPredicates(Configuration configuration) {
 // First prune using default datamap on driver side.
 DataMapExprWrapper dataMapExprWrapper = DataMapChooser
 .getDefaultDataMap(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
-List prunedBlocklets =
+List finalPrunedBlocklets =
 dataMapExprWrapper.prune(segmentIds, partitionsToPrune);
-
 ExplainCollector.recordDefaultDataMapPruning(
-dataMapExprWrapper.getDataMapSchema(), prunedBlocklets.size());
+dataMapExprWrapper.getDataMapSchema(), 
finalPrunedBlocklets.size());
+if (finalPrunedBlocklets.size() == 0) {
+  return finalPrunedBlocklets;
+}
 
 DataMapChooser chooser = new 
DataMapChooser(getOrCreateCarbonTable(job.getConfiguration()));
 
 // Get the available CG datamaps and prune further.
 DataMapExprWrapper cgDataMapExprWrapper = 
chooser.chooseCGDataMap(resolver);
 if (cgDataMapExprWrapper != null) {
   // Prune segments from already pruned blocklets
-  pruneSegments(segmentIds, prunedBlocklets);
+  pruneSegments(segmentIds, finalPrunedBlocklets);
+  List cgPrunedBlocklets = new ArrayList<>();
   // Again prune with CG datamap.
   if (distributedCG && dataMapJob != null) {
-prunedBlocklets = DataMapUtil
+cgPrunedBlocklets = DataMapUtil
 .executeDataMapJob(carbonTable, resolver, segmentIds, 
cgDataMapExprWrapper, dataMapJob,
--- End diff --

fixed


---


[GitHub] carbondata pull request #2410: [CARBONDATA-2650][Datamap] Fix bugs in negati...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2410#discussion_r199310661
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -433,46 +434,57 @@ protected Expression 
getFilterPredicates(Configuration configuration) {
 // First prune using default datamap on driver side.
 DataMapExprWrapper dataMapExprWrapper = DataMapChooser
 .getDefaultDataMap(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
-List prunedBlocklets =
+List finalPrunedBlocklets =
--- End diff --

fixed. Use the origin variable name 'prunedBlocklets' to represent the 
final output


---


[GitHub] carbondata pull request #2410: [CARBONDATA-2650][Datamap] Fix bugs in negati...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2410#discussion_r199310665
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -433,46 +434,57 @@ protected Expression 
getFilterPredicates(Configuration configuration) {
 // First prune using default datamap on driver side.
 DataMapExprWrapper dataMapExprWrapper = DataMapChooser
 .getDefaultDataMap(getOrCreateCarbonTable(job.getConfiguration()), 
resolver);
-List prunedBlocklets =
+List finalPrunedBlocklets =
 dataMapExprWrapper.prune(segmentIds, partitionsToPrune);
-
 ExplainCollector.recordDefaultDataMapPruning(
-dataMapExprWrapper.getDataMapSchema(), prunedBlocklets.size());
+dataMapExprWrapper.getDataMapSchema(), 
finalPrunedBlocklets.size());
+if (finalPrunedBlocklets.size() == 0) {
+  return finalPrunedBlocklets;
+}
 
 DataMapChooser chooser = new 
DataMapChooser(getOrCreateCarbonTable(job.getConfiguration()));
 
 // Get the available CG datamaps and prune further.
 DataMapExprWrapper cgDataMapExprWrapper = 
chooser.chooseCGDataMap(resolver);
 if (cgDataMapExprWrapper != null) {
   // Prune segments from already pruned blocklets
-  pruneSegments(segmentIds, prunedBlocklets);
+  pruneSegments(segmentIds, finalPrunedBlocklets);
+  List cgPrunedBlocklets = new ArrayList<>();
   // Again prune with CG datamap.
   if (distributedCG && dataMapJob != null) {
-prunedBlocklets = DataMapUtil
+cgPrunedBlocklets = DataMapUtil
 .executeDataMapJob(carbonTable, resolver, segmentIds, 
cgDataMapExprWrapper, dataMapJob,
 partitionsToPrune);
   } else {
-prunedBlocklets = cgDataMapExprWrapper.prune(segmentIds, 
partitionsToPrune);
+cgPrunedBlocklets = cgDataMapExprWrapper.prune(segmentIds, 
partitionsToPrune);
   }
-
+  // since index datamap prune in segment scope,
+  // the result need to intersect with previous pruned result
+  finalPrunedBlocklets = (List) CollectionUtils.intersection(
+  finalPrunedBlocklets, cgPrunedBlocklets);
   ExplainCollector.recordCGDataMapPruning(
-  cgDataMapExprWrapper.getDataMapSchema(), prunedBlocklets.size());
+  cgDataMapExprWrapper.getDataMapSchema(), 
finalPrunedBlocklets.size());
+}
+
+if (finalPrunedBlocklets.size() == 0) {
+  return finalPrunedBlocklets;
 }
 // Now try to prune with FG DataMap.
 if (isFgDataMapPruningEnable(job.getConfiguration()) && dataMapJob != 
null) {
   DataMapExprWrapper fgDataMapExprWrapper = 
chooser.chooseFGDataMap(resolver);
   if (fgDataMapExprWrapper != null) {
 // Prune segments from already pruned blocklets
-pruneSegments(segmentIds, prunedBlocklets);
-prunedBlocklets = DataMapUtil
+pruneSegments(segmentIds, finalPrunedBlocklets);
+List fgPrunedBlocklets = DataMapUtil
 .executeDataMapJob(carbonTable, resolver, segmentIds, 
fgDataMapExprWrapper, dataMapJob,
 partitionsToPrune);
-
+finalPrunedBlocklets = (List) CollectionUtils.intersection(
+finalPrunedBlocklets, fgPrunedBlocklets);
 ExplainCollector.recordFGDataMapPruning(
-fgDataMapExprWrapper.getDataMapSchema(), 
prunedBlocklets.size());
+fgDataMapExprWrapper.getDataMapSchema(), 
finalPrunedBlocklets.size());
   }
 } // TODO: add a else branch to push FGDataMap pruning to reader side
--- End diff --

fixed


---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6673/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5498/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
retest this please


---


[GitHub] carbondata issue #2405: [CARBONDATA-2635][BloomDataMap] Support different in...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2405
  
@jackylk review comments are resolved


---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6671/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
retest this please


---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5496/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5497/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
retest this please


---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6670/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6669/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
retest this please


---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
retest this please


---


[GitHub] carbondata issue #2430: [WIP]Local dictionary fallback issue

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2430
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5495/



---


[GitHub] carbondata issue #2430: [WIP]Local dictionary fallback issue

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2430
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6668/



---


[GitHub] carbondata issue #2430: [WIP]Local dictionary fallback issue

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2430
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5528/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5494/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6667/



---


[GitHub] carbondata pull request #2430: [WIP]Local dictionary fallback issue

2018-06-29 Thread kumarvishal09
GitHub user kumarvishal09 opened a pull request:

https://github.com/apache/carbondata/pull/2430

[WIP]Local dictionary fallback issue

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kumarvishal09/incubator-carbondata 
localdictperformance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2430


commit f4e3bd8615a28d0bda998a03dd24c4e0b3c0ebe8
Author: kumarvishal09 
Date:   2018-06-29T20:15:54Z

Local dictionary fallback issue




---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5527/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread kunal642
Github user kunal642 commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
retest this please


---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5526/



---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5493/



---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1//



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5525/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5492/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5489/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6665/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5524/



---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5523/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6662/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6664/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5491/



---


[GitHub] carbondata issue #2420: [CARBONDATA-2666] updated rename command so that tab...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2420
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5522/



---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5490/



---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6663/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5521/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6661/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5520/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5488/



---


[GitHub] carbondata issue #2403: [CARBONDATA-2633][BloomDataMap] Fix bugs in bloomfil...

2018-06-29 Thread xuchuanyin
Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2403
  
@jackylk I refactored the commit based on our discussion, please check.

Some tests are added to clarify the scenarios


---


[GitHub] carbondata pull request #2428: [HOTFIX] Fixed compilation issues and bloom c...

2018-06-29 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2428


---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread kumarvishal09
Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
LGTM


---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5487/



---


[GitHub] carbondata issue #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2422
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5519/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6660/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5518/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5517/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5483/



---


[GitHub] carbondata issue #2429: [CARBONDATA-2674][Streaming]Streaming with merge ind...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2429
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5485/



---


[GitHub] carbondata issue #2429: [CARBONDATA-2674][Streaming]Streaming with merge ind...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2429
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6658/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6656/



---


[GitHub] carbondata issue #2429: [CARBONDATA-2674][Streaming]Streaming with merge ind...

2018-06-29 Thread mohammadshahidkhan
Github user mohammadshahidkhan commented on the issue:

https://github.com/apache/carbondata/pull/2429
  
retest this please


---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread kumarvishal09
Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
retest this please 


---


[GitHub] carbondata issue #2406: [CARBONDATA-2642] Added configurable Lock path prope...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2406
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5516/



---


[GitHub] carbondata issue #2417: [WIP][Complex Column Enhancements]Primitive DataType...

2018-06-29 Thread ravipesala
Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2417
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/5515/



---


[GitHub] carbondata issue #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2422
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6657/



---


[GitHub] carbondata issue #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2422
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5484/



---


[GitHub] carbondata issue #2414: [CARBONDATA-2658][DataLoad]No difference in memory s...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2414
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5482/



---


[GitHub] carbondata issue #2429: [CARBONDATA-2674][Streaming]Streaming with merge ind...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2429
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5481/



---


[GitHub] carbondata issue #2414: [CARBONDATA-2658][DataLoad]No difference in memory s...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2414
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6655/



---


[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6647/



---


[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...

2018-06-29 Thread kunal642
Github user kunal642 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2422#discussion_r199126782
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala
 ---
@@ -2585,67 +2585,445 @@ class LocalDictionarySupportCreateTableTest 
extends QueryTest with BeforeAndAfte
 }
   }
 
-  test("test preaggregate table local dictionary enabled table")
+  test("test local dictionary foer varchar datatype columns") {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_include'='city',
+| 'LONG_STRING_COLUMNS'='city')
+  """.stripMargin)
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("city"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+  }
+
+  test("test local dictionary describe formatted only with default 
configs")
   {
 sql("drop table if exists local1")
-sql("CREATE TABLE local1 (id Int, date date, country string, phonetype 
string, " +
-"serialname String,salary int ) STORED BY 
'org.apache.carbondata.format' " +
-
"tblproperties('dictionary_include'='country','local_dictionary_enable'='true','local_dictionary_include'
 = 'phonetype','local_dictionary_exclude' ='serialname')")
-sql("create datamap PreAggCount on table local1 using 'preaggregate' 
as " +
-"select country,count(salary) as count from local1 group by 
country")
-val descLoc = sql("describe formatted local1_PreAggCount").collect
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata'
+  """.stripMargin)
+
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("true"))
+}
 descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
   case Some(row) => assert(row.get(1).toString.contains("1"))
 }
 descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
-  case Some(row) => assert(row.get(1).toString.contains("phonetype"))
+  case Some(row) => assert(row.get(1).toString.contains("name,city"))
 }
-descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude")) 
match {
-  case Some(row) => assert(row.get(1).toString.contains("serialname"))
+  }
+
+  test("test local dictionary for invalid threshold")
+  {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata' 
tblproperties('local_dictionary_threshold'='30')
+  """.stripMargin)
+
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
 }
-descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  }
+
+  test("test alter set for local dictionary enable to disable")
+  {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata' 
tblproperties('local_dictionary_threshold'='30')
+  """.stripMargin)
+
+val descLoc1 = sql("describe formatted local1").collect
+descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
   case Some(row) => assert(row.get(1).toString.contains("true"))
 }
+descLoc1.find(_.get(0).toString.contains("Local Dictionary 
Threshold")) match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+sql("alter table local1 set 
tblproperties('local_dictionary_enable'='false')")
+val descLoc2 = sql("describe formatted local1").collect
+descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("false"))
+}
+checkExistence(sql("DESC FORMATTED local1"), false,
+  "Local Dictionary Threshold")
   

[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...

2018-06-29 Thread kunal642
Github user kunal642 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2422#discussion_r199126397
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/localdictionary/LocalDictionarySupportCreateTableTest.scala
 ---
@@ -2585,67 +2585,445 @@ class LocalDictionarySupportCreateTableTest 
extends QueryTest with BeforeAndAfte
 }
   }
 
-  test("test preaggregate table local dictionary enabled table")
+  test("test local dictionary foer varchar datatype columns") {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_include'='city',
+| 'LONG_STRING_COLUMNS'='city')
+  """.stripMargin)
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("city"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+  }
+
+  test("test local dictionary describe formatted only with default 
configs")
   {
 sql("drop table if exists local1")
-sql("CREATE TABLE local1 (id Int, date date, country string, phonetype 
string, " +
-"serialname String,salary int ) STORED BY 
'org.apache.carbondata.format' " +
-
"tblproperties('dictionary_include'='country','local_dictionary_enable'='true','local_dictionary_include'
 = 'phonetype','local_dictionary_exclude' ='serialname')")
-sql("create datamap PreAggCount on table local1 using 'preaggregate' 
as " +
-"select country,count(salary) as count from local1 group by 
country")
-val descLoc = sql("describe formatted local1_PreAggCount").collect
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata'
+  """.stripMargin)
+
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("true"))
+}
 descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
   case Some(row) => assert(row.get(1).toString.contains("1"))
 }
 descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
-  case Some(row) => assert(row.get(1).toString.contains("phonetype"))
+  case Some(row) => assert(row.get(1).toString.contains("name,city"))
 }
-descLoc.find(_.get(0).toString.contains("Local Dictionary Exclude")) 
match {
-  case Some(row) => assert(row.get(1).toString.contains("serialname"))
+  }
+
+  test("test local dictionary for invalid threshold")
+  {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata' 
tblproperties('local_dictionary_threshold'='30')
+  """.stripMargin)
+
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
 }
-descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  }
+
+  test("test alter set for local dictionary enable to disable")
+  {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'carbondata' 
tblproperties('local_dictionary_threshold'='30')
+  """.stripMargin)
+
+val descLoc1 = sql("describe formatted local1").collect
+descLoc1.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
   case Some(row) => assert(row.get(1).toString.contains("true"))
 }
+descLoc1.find(_.get(0).toString.contains("Local Dictionary 
Threshold")) match {
+  case Some(row) => assert(row.get(1).toString.contains("1"))
+}
+sql("alter table local1 set 
tblproperties('local_dictionary_enable'='false')")
+val descLoc2 = sql("describe formatted local1").collect
+descLoc2.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("false"))
+}
+checkExistence(sql("DESC FORMATTED local1"), false,
+  "Local Dictionary Threshold")
   

[GitHub] carbondata issue #2428: [HOTFIX] Fixed compilation issues and bloom clear is...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2428
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5471/



---


[GitHub] carbondata pull request #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dict...

2018-06-29 Thread kunal642
Github user kunal642 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2422#discussion_r199126014
  
--- Diff: 
integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/CreateTableWithLocalDictionaryTestCase.scala
 ---
@@ -2096,6 +2096,264 @@ class CreateTableWithLocalDictionaryTestCase 
extends QueryTest with BeforeAndAft
 }
   }
 
+  test("test alter table add column") {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_enable'='true',
+| 
'local_dictionary_threshold'='2','local_dictionary_include'='city','no_inverted_index'='name')
+  """.stripMargin)
+sql("alter table local1 add columns (alt string) 
tblproperties('local_dictionary_include'='alt')")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("2"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("city,alt"))
+}
+  }
+
+  test("test alter table add column default configs for local dictionary") 
{
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_enable'='true',
+| 'local_dictionary_threshold'='2','no_inverted_index'='name')
+  """.stripMargin)
+sql("alter table local1 add columns (alt string)")
+val descLoc = sql("describe formatted local1").collect
+descLoc.find(_.get(0).toString.contains("Local Dictionary Threshold")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("2"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Enabled")) 
match {
+  case Some(row) => assert(row.get(1).toString.contains("true"))
+}
+descLoc.find(_.get(0).toString.contains("Local Dictionary Include")) 
match {
+  case Some(row) => 
assert(row.get(1).toString.contains("name,city,alt"))
+}
+  }
+
+  test("test alter table add column where same column is in dictionary 
include and local dictionary include") {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_enable'='true',
+| 
'local_dictionary_threshold'='2','local_dictionary_include'='city','no_inverted_index'='name')
+  """.stripMargin)
+val exception = intercept[MalformedCarbonCommandException] {
+  sql(
+"alter table local1 add columns (alt string) 
tblproperties('local_dictionary_include'='alt','dictionary_include'='alt')")
+}
+assert(exception.getMessage
+  .contains(
+"LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE column: alt 
specified in Dictionary " +
+"include. Local Dictionary will not be generated for Dictionary 
include columns. Please " +
+"check create table statement."))
+  }
+
+  test("test alter table add column where duplicate columns present in 
local dictionary include") {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, age int)
+| STORED BY 'org.apache.carbondata.format' 
tblproperties('local_dictionary_enable'='true',
+| 
'local_dictionary_threshold'='2','local_dictionary_include'='city','no_inverted_index'='name')
+  """.stripMargin)
+val exception = intercept[MalformedCarbonCommandException] {
+  sql(
+"alter table local1 add columns (alt string) 
tblproperties('local_dictionary_include'='alt,alt')")
+}
+assert(exception.getMessage
+  .contains(
+"LOCAL_DICTIONARY_INCLUDE/LOCAL_DICTIONARY_EXCLUDE contains 
Duplicate Columns: alt. " +
+"Please check create table statement."))
+  }
+
+  test("test alter table add column where duplicate columns present in 
local dictionary include/exclude")
+  {
+sql("drop table if exists local1")
+sql(
+  """
+| CREATE TABLE local1(id int, name string, city string, 

[GitHub] carbondata issue #2429: [CARBONDATA-2674][Streaming]Streaming with merge ind...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2429
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6654/



---


[GitHub] carbondata pull request #2429: [CARBONDATA-2674][Streaming]Streaming with me...

2018-06-29 Thread mohammadshahidkhan
GitHub user mohammadshahidkhan opened a pull request:

https://github.com/apache/carbondata/pull/2429

[CARBONDATA-2674][Streaming]Streaming with merge index enabled does not 
consider the merge index file while pruning.

…

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed?
 None
 - [X] Any backward compatibility impacted?
 None
 - [X] Document update required?
None
 - [X] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   Added test to varify the merge index scenarios.
 - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
  NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata 
streaming_with_mi

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2429.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2429


commit 7fb282da1c51fca55a4c8c93e06c93b04a3a29e2
Author: mohammadshahidkhan 
Date:   2018-06-29T10:44:08Z

[CARBONDATA-2674][Streaming]Streaming with merge index enabled does not 
consider the merge index file while pruning.




---


[jira] [Updated] (CARBONDATA-2674) Streaming with merge index enabled does not consider the merge index file while pruning.

2018-06-29 Thread Mohammad Shahid Khan (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohammad Shahid Khan updated CARBONDATA-2674:
-
Summary: Streaming with merge index enabled does not consider the merge 
index file while pruning.   (was: Streaming with merge index enabled does not 
consider the index file while pruning.)

> Streaming with merge index enabled does not consider the merge index file 
> while pruning. 
> -
>
> Key: CARBONDATA-2674
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2674
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Mohammad Shahid Khan
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (CARBONDATA-2674) Streaming with merge index enabled does not consider the index file while pruning.

2018-06-29 Thread Mohammad Shahid Khan (JIRA)
Mohammad Shahid Khan created CARBONDATA-2674:


 Summary: Streaming with merge index enabled does not consider the 
index file while pruning.
 Key: CARBONDATA-2674
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2674
 Project: CarbonData
  Issue Type: Bug
Reporter: Mohammad Shahid Khan






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5480/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6653/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5479/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5478/



---


[GitHub] carbondata issue #2421: [CARBONDATA-2606] [Complex DataType Enhancements]Fix...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2421
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5477/



---


[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6652/



---


[GitHub] carbondata issue #2399: [CARBONDATA-2629] Support SDK carbon reader read dat...

2018-06-29 Thread xubo245
Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/2399
  
retest this please


---


[GitHub] carbondata issue #2422: [CARBONDATA-2585][CARBONDATA-2586][Local Dictionary]...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2422
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5476/



---


[GitHub] carbondata issue #2397: [CARBONDATA-2644][DataLoad]ADD carbon.load.sortMemor...

2018-06-29 Thread CarbonDataQA
Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2397
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5475/



---


  1   2   >