[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2606#discussion_r208460810
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
---
@@ -169,38 +171,59 @@ private void doExecute(Iterator
iterator, int iteratorIndex) thr
if (rowsNotExist) {
rowsNotExist = false;
dataHandler =
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
dataHandler.initialise();
}
processBatch(iterator.next(), dataHandler, iteratorIndex);
}
-if (!rowsNotExist) {
- finish(dataHandler, iteratorIndex);
+try {
+ if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+ }
+} catch (CarbonDataWriterException e) {
--- End diff --
remove this catch block
---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6555/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7830/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2606 retest this please ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6554/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6553/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6551/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7829/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2615 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6552/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6550/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2615 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7828/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2616 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6203/ ---
[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2616 [HOTFIX]Fixed int overflow and comparison gone wrong during blocklet min/max Problem: During calculating min/max for blocklet, it needs to calculate from all the pages. During that comparison, it is typecasting to int and overflows, so there is a chance the negative becomes positive and positive become negative. That's why min max of long comes wrong for bigger values. Solution: Don't typecast directly, instead check first the negative or positive and then return. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata blocklet-min-max-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2616.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2616 commit a99d0516ee06a285f8be3880d7005bf161f1c940 Author: ravipesala Date: 2018-08-07T15:49:36Z Fixed int overflow and comparasion gone wrong during blocklet min/max ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7827/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7826/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2612 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6547/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6548/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2615 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6202/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2612 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7824/ ---
[GitHub] carbondata pull request #2615: [HOTFIX] [presto] presto code cleanup
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2615 [HOTFIX] [presto] presto code cleanup Changes in the PR: a. Removed spark jars from release jars dependency b. enable check style for presto module c. fixed all the checkstyle for presto module Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2615.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2615 commit 399ca66addcd6767bc43ac7fa15514eb705546d4 Author: ajantha-bhat Date: 2018-08-06T12:21:03Z [HOTFIX] presto code cleanup a. Removed spark jars from release jars dependency b. enable check style for presto module c. fixed all the checkstyle for presto module ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7825/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7822/ ---
[jira] [Resolved] (CARBONDATA-2807) Fixed data load performance issue with more number of records
[ https://issues.apache.org/jira/browse/CARBONDATA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2807. - Resolution: Fixed Fix Version/s: 1.4.1 > Fixed data load performance issue with more number of records > - > > Key: CARBONDATA-2807 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2807 > Project: CarbonData > Issue Type: Improvement >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.4.1 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > **Problem:**Data Loading is taking more time when number of records are high. > **Root cause:** As number of records are high intermediate merger is taking > more time. > **Solution:** Checking the number of files present in file list is done is > synchronized block because of this > each intermediate request is taking sometime and when number of records are > high it impacting overall data loading performance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2588 ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2588 LGTM ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2602 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6544/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6546/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6545/ ---
[jira] [Resolved] (CARBONDATA-2831) Support Merge index files read from non transactional table.
[ https://issues.apache.org/jira/browse/CARBONDATA-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2831. - Resolution: Fixed Fix Version/s: 1.4.1 > Support Merge index files read from non transactional table. > > > Key: CARBONDATA-2831 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2831 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.1 > > Time Spent: 3h > Remaining Estimate: 0h > > problem : Currently SDK read/ nontransactional table read from external > table gives null output when carbonMergeindex file is present instead of > carobnindex files. > cause : In LatestFileReadCommitted, while taking snapshot, merge index files > were not considered. > solution: consider the merge index files while taking snapshot > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2610: [CARBONDATA-2831] Added Support Merge index f...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2610 ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6200/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7823/ ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2602 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7820/ ---
[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2537 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6199/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7819/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2564 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6198/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6543/ ---
[GitHub] carbondata pull request #2537: [CARBONDATA-2768][CarbonStore] Fix error in t...
Github user xuchuanyin closed the pull request at: https://github.com/apache/carbondata/pull/2537 ---
[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2415#discussion_r208224446
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
---
@@ -92,11 +89,38 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val
dataFrame: DataFrame) {
).filter(_._2.isDefined)
.map(property => s"'${property._1}' =
'${property._2.get}'").mkString(",")
+val partition: Seq[String] = if (options.partitionColumns.isDefined) {
+ if (options.partitionColumns.get.toSet.size !=
options.partitionColumns.get.length) {
+throw new MalformedCarbonCommandException(s"repeated partition
column")
+ }
+ options.partitionColumns.get.map { column =>
+val field = schema.fields.find(_.name.equalsIgnoreCase(column))
+if (field.isEmpty) {
+ throw new MalformedCarbonCommandException(s"invalid partition
column: $column")
+}
+s"$column ${field.get.dataType.typeName}"
+ }
+} else {
+ Seq()
+}
+
+val schemaWithoutPartition = if (options.partitionColumns.isDefined) {
+ val fields = schema.filterNot(field =>
options.partitionColumns.get.contains(field.name))
--- End diff --
fixed
---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2614 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6542/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2610 LGTM ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6197/ ---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2614 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7818/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6541/ ---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/2589 ---
[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2391 @xubo245 Please close it as it is handled in https://github.com/apache/carbondata/pull/2441 ---
[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2537 LGTM merged into external-format ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 retest this please ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6196/ ---
[GitHub] carbondata issue #2423: [CARBONDATA-2530][MV] Fix wrong data displayed when ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2423 @xubo245 Please close it, it is handled in https://github.com/apache/carbondata/pull/2474 ---
[GitHub] carbondata pull request #2602: [CARBONDATA-2539]Fix mv classcast exception i...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2602 ---
[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2605 ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2602 LGTM ---
[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2612#discussion_r208211297
--- Diff:
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
---
@@ -779,6 +779,32 @@ public UpdateVO getInvalidTimestampRange(String
segmentId) {
}
return range;
}
+
+ /**
+ * Returns the invalid timestamp range of a segment.
+ * @return
+ */
+ public List getInvalidTimestampRange() {
+List ranges = new ArrayList();
+for (LoadMetadataDetails segment : segmentDetails) {
+ if (!(segment.getSegmentStatus().equals(SegmentStatus.SUCCESS) ||
+
segment.getSegmentStatus().equals(SegmentStatus.LOAD_PARTIAL_SUCCESS))) {
--- End diff --
Please refer SegmentStatusManager for checking the invalid segments
---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7817/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 @kunal642 Please fix checkstyle ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/2605 2.1 build has random failure, which is not related to this change ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2605 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7816/ ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2605 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6540/ ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2605 LGTM ---
[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2614#discussion_r208206503
--- Diff: examples/spark2/pom.xml ---
@@ -49,6 +49,11 @@
carbondata-store-sdk
${project.version}
+
+ org.apache.carbondata
+ carbondata-mv-core
--- End diff --
This is adding always or in case of `mv` profile is selected?
---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2602 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6195/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2588 retest this please ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6539/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7815/ ---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2614 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6194/ ---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2588#discussion_r208199760
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
}
public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
- if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
--- End diff --
ok
---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2588#discussion_r208199601
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
}
public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
- if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+ synchronized (lockObject) {
fileList = procFiles.toArray(new File[procFiles.size()]);
this.procFiles = new ArrayList();
-if (LOGGER.isDebugEnabled()) {
- LOGGER
- .debug("Submitting request for intermediate merging no of
files: " + fileList.length);
-}
}
-}
-if (null != fileList) {
+ if (LOGGER.isDebugEnabled()) {
+LOGGER.debug("Sumitting request for intermediate merging no of
files: " + fileList.length);
+ }
startIntermediateMerging(fileList);
--- End diff --
ok
---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2588#discussion_r208198390
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
}
public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
- if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >=
parameters.getNumberOfIntermediateFileToBeMerged()) {
--- End diff --
@mohammadshahidkhan Yes you are right but in this case UnsafeSortDataRow
processing will be slower as it will read/ sort and write so chances of above
condition is negligible, because of this double check is not added here
---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2606 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6193/ ---
[jira] [Resolved] (CARBONDATA-2836) Fixed data loading performance issue
[ https://issues.apache.org/jira/browse/CARBONDATA-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2836. - Resolution: Fixed Fix Version/s: 1.4.1 > Fixed data loading performance issue > > > Key: CARBONDATA-2836 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2836 > Project: CarbonData > Issue Type: Improvement >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.4.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Problem: Data Loading is taking more time when number of records are high(3.5 > billion) records > Root Cause: In case of Final merge sort temp row conversion is done in main > thread because of this final step processing became slower. > Solution: Mode conversion logic to pre-fetch thread for parallel processing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2611: [CARBONDATA-2836]Fixed data loading performan...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2611 ---
[GitHub] carbondata issue #2611: [CARBONDATA-2836]Fixed data loading performance issu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2611 LGTM ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2610 retest this please ---
[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2614 [CARBONDATA-2837] Added MVExample in example module Added MVExample in example module Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata mvexample Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2614.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2614 commit f50e72201dd1d0da720bcfaf68c5e4df14a51624 Author: ravipesala Date: 2018-08-07T11:07:17Z Added MVExample ---
[jira] [Created] (CARBONDATA-2837) Add MV Example in examples module
Ravindra Pesala created CARBONDATA-2837: --- Summary: Add MV Example in examples module Key: CARBONDATA-2837 URL: https://issues.apache.org/jira/browse/CARBONDATA-2837 Project: CarbonData Issue Type: Improvement Reporter: Ravindra Pesala -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2605 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6192/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6191/ ---
[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2605#discussion_r208186221
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
---
@@ -641,6 +641,78 @@ object CarbonScalaUtil {
}
}
+ /**
+ * This method validates all the child columns of complex column
recursively to check whether
+ * any of the child column is of string dataType or not
+ *
+ * @param field
+ */
+ def validateChildColumnsRecursively(field: Field): Boolean = {
+if (field.children.isDefined && null != field.children.get) {
+ field.children.get.exists { childColumn =>
+if (childColumn.children.isDefined && null !=
childColumn.children.get) {
+ validateChildColumnsRecursively(childColumn)
+} else {
+ childColumn.dataType.get.equalsIgnoreCase("string")
+}
+ }
+} else {
+ false
+}
+ }
+
+ /**
+ * This method validates the local dictionary configured columns
+ *
+ * @param fields
--- End diff --
please add parameter description for all newly added parameter
---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6536/ ---
[jira] [Closed] (CARBONDATA-2822) Carbon Configuration - "carbon.invisible.segments.preserve.count" configuration property is not working as expected.
[ https://issues.apache.org/jira/browse/CARBONDATA-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran closed CARBONDATA-2822. - > Carbon Configuration - "carbon.invisible.segments.preserve.count" > configuration property is not working as expected. > - > > Key: CARBONDATA-2822 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2822 > Project: CarbonData > Issue Type: Bug > Components: core, file-format > Environment: 3 Node ANT cluster. >Reporter: Prasanna Ravichandran >Priority: Minor > Attachments: configuration.png > > > For the *carbon.invisible.segments.preserve.count* configuration, it is not > working as expected. > +*Steps to reproduce:*+ > 1) Setting up "*carbon.invisible.segments.preserve.count=20"* in > carbon.properties and restarting the thrift server. > > 2) After performing Loading 40 times and Compaction 4 times. > 3) Perform clean files, so that the tablestatus.history file would be > generated with invisible segments details. > So Total 44 segments would be created including visible and invisible > segments.(40 load segment (like segment ID from 0,1,2...39) + 4 compacted new > segment(like 0.1,20.1,22.1,0.2)) > In that, *41 segments information are present in the "tablestatus.history" > file(*which holds invisible(marked for delete and compacted) segments > details) and 3 segments information are present in the "tablestatus" > file(which holds visible segments(0 .2 -final compacted segment) along with > (1^st^ segment - 0th segment) and (last segment-39th segment)). *But > invisible segment preserve count is configured to 20, which is not followed > for the tablestatus.history file.* > +*Expected result:*+ > tablestatus.history file should preserve only the latest 20 segments, as per > the configuration. > +*Actual result:*+ > tablestatus.history file is having 41 invisible segments details.(which is > above the configured value: 20) > > This is tested with ANT cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7812/ ---
[jira] [Resolved] (CARBONDATA-2822) Carbon Configuration - "carbon.invisible.segments.preserve.count" configuration property is not working as expected.
[ https://issues.apache.org/jira/browse/CARBONDATA-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna Ravichandran resolved CARBONDATA-2822. --- Resolution: Invalid Working fine. > Carbon Configuration - "carbon.invisible.segments.preserve.count" > configuration property is not working as expected. > - > > Key: CARBONDATA-2822 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2822 > Project: CarbonData > Issue Type: Bug > Components: core, file-format > Environment: 3 Node ANT cluster. >Reporter: Prasanna Ravichandran >Priority: Minor > Attachments: configuration.png > > > For the *carbon.invisible.segments.preserve.count* configuration, it is not > working as expected. > +*Steps to reproduce:*+ > 1) Setting up "*carbon.invisible.segments.preserve.count=20"* in > carbon.properties and restarting the thrift server. > > 2) After performing Loading 40 times and Compaction 4 times. > 3) Perform clean files, so that the tablestatus.history file would be > generated with invisible segments details. > So Total 44 segments would be created including visible and invisible > segments.(40 load segment (like segment ID from 0,1,2...39) + 4 compacted new > segment(like 0.1,20.1,22.1,0.2)) > In that, *41 segments information are present in the "tablestatus.history" > file(*which holds invisible(marked for delete and compacted) segments > details) and 3 segments information are present in the "tablestatus" > file(which holds visible segments(0 .2 -final compacted segment) along with > (1^st^ segment - 0th segment) and (last segment-39th segment)). *But > invisible segment preserve count is configured to 20, which is not followed > for the tablestatus.history file.* > +*Expected result:*+ > tablestatus.history file should preserve only the latest 20 segments, as per > the configuration. > +*Actual result:*+ > tablestatus.history file is having 41 invisible segments details.(which is > above the configured value: 20) > > This is tested with ANT cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CARBONDATA-2822) Carbon Configuration - "carbon.invisible.segments.preserve.count" configuration property is not working as expected.
[ https://issues.apache.org/jira/browse/CARBONDATA-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571437#comment-16571437 ] Prasanna Ravichandran commented on CARBONDATA-2822: --- The property *"carbon.invisible.segments.preserve.count"* is actually for TableStatusFile only. When we set this property, if the number of invisible segments in tablestatus file exceeds that configured *carbon.invisible.segments.preserve.count value,* then it is moving all the invisible segments to the tablestatus.history file. It is working fine as expected. > Carbon Configuration - "carbon.invisible.segments.preserve.count" > configuration property is not working as expected. > - > > Key: CARBONDATA-2822 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2822 > Project: CarbonData > Issue Type: Bug > Components: core, file-format > Environment: 3 Node ANT cluster. >Reporter: Prasanna Ravichandran >Priority: Minor > Attachments: configuration.png > > > For the *carbon.invisible.segments.preserve.count* configuration, it is not > working as expected. > +*Steps to reproduce:*+ > 1) Setting up "*carbon.invisible.segments.preserve.count=20"* in > carbon.properties and restarting the thrift server. > > 2) After performing Loading 40 times and Compaction 4 times. > 3) Perform clean files, so that the tablestatus.history file would be > generated with invisible segments details. > So Total 44 segments would be created including visible and invisible > segments.(40 load segment (like segment ID from 0,1,2...39) + 4 compacted new > segment(like 0.1,20.1,22.1,0.2)) > In that, *41 segments information are present in the "tablestatus.history" > file(*which holds invisible(marked for delete and compacted) segments > details) and 3 segments information are present in the "tablestatus" > file(which holds visible segments(0 .2 -final compacted segment) along with > (1^st^ segment - 0th segment) and (last segment-39th segment)). *But > invisible segment preserve count is configured to 20, which is not followed > for the tablestatus.history file.* > +*Expected result:*+ > tablestatus.history file should preserve only the latest 20 segments, as per > the configuration. > +*Actual result:*+ > tablestatus.history file is having 41 invisible segments details.(which is > above the configured value: 20) > > This is tested with ANT cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2613: [HOTFIX] Modified code to fix the degrade in ...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2613#discussion_r208182587
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/result/iterator/RawResultIterator.java
---
@@ -53,153 +39,124 @@
*/
private CarbonIterator detailRawQueryResultIterator;
- private boolean prefetchEnabled;
- private List currentBuffer;
- private List backupBuffer;
- private int currentIdxInBuffer;
- private ExecutorService executorService;
- private Future fetchFuture;
- private Object[] currentRawRow = null;
- private boolean isBackupFilled = false;
+ /**
+ * Counter to maintain the row counter.
+ */
+ private int counter = 0;
+
+ private Object[] currentConveretedRawRow = null;
+
+ /**
+ * LOGGER
+ */
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(RawResultIterator.class.getName());
+
+ /**
+ * batch of the result.
+ */
+ private RowBatch batch;
public RawResultIterator(CarbonIterator
detailRawQueryResultIterator,
- SegmentProperties sourceSegProperties, SegmentProperties
destinationSegProperties,
- boolean isStreamingHandOff) {
+ SegmentProperties sourceSegProperties, SegmentProperties
destinationSegProperties) {
this.detailRawQueryResultIterator = detailRawQueryResultIterator;
this.sourceSegProperties = sourceSegProperties;
this.destinationSegProperties = destinationSegProperties;
-this.executorService = Executors.newFixedThreadPool(1);
-
-if (!isStreamingHandOff) {
- init();
-}
}
- private void init() {
-this.prefetchEnabled = CarbonProperties.getInstance().getProperty(
-CarbonCommonConstants.CARBON_COMPACTION_PREFETCH_ENABLE,
-
CarbonCommonConstants.CARBON_COMPACTION_PREFETCH_ENABLE_DEFAULT).equalsIgnoreCase("true");
-try {
- new RowsFetcher(false).call();
- if (prefetchEnabled) {
-this.fetchFuture = executorService.submit(new RowsFetcher(true));
- }
-} catch (Exception e) {
- LOGGER.error(e, "Error occurs while fetching records");
- throw new RuntimeException(e);
-}
- }
+ @Override public boolean hasNext() {
- /**
- * fetch rows
- */
- private final class RowsFetcher implements Callable {
-private boolean isBackupFilling;
-
-private RowsFetcher(boolean isBackupFilling) {
- this.isBackupFilling = isBackupFilling;
-}
-
-@Override
-public Void call() throws Exception {
- if (isBackupFilling) {
-backupBuffer = fetchRows();
-isBackupFilled = true;
+if (null == batch || checkIfBatchIsProcessedCompletely(batch)) {
+ if (detailRawQueryResultIterator.hasNext()) {
+batch = null;
+batch = detailRawQueryResultIterator.next();
+counter = 0; // batch changed so reset the counter.
} else {
-currentBuffer = fetchRows();
+return false;
}
- return null;
}
- }
- private List fetchRows() {
-if (detailRawQueryResultIterator.hasNext()) {
- return detailRawQueryResultIterator.next().getRows();
+if (!checkIfBatchIsProcessedCompletely(batch)) {
+ return true;
} else {
- return new ArrayList<>();
+ return false;
}
}
- private void fillDataFromPrefetch() {
-try {
- if (currentIdxInBuffer >= currentBuffer.size() && 0 !=
currentIdxInBuffer) {
-if (prefetchEnabled) {
- if (!isBackupFilled) {
-fetchFuture.get();
- }
- // copy backup buffer to current buffer and fill backup buffer
asyn
- currentIdxInBuffer = 0;
- currentBuffer = backupBuffer;
- isBackupFilled = false;
- fetchFuture = executorService.submit(new RowsFetcher(true));
-} else {
- currentIdxInBuffer = 0;
- new RowsFetcher(false).call();
+ @Override public Object[] next() {
--- End diff --
ok
---
[GitHub] carbondata issue #2611: [CARBONDATA-2836]Fixed data loading performance issu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2611 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6533/ ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2605 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6190/ ---
[GitHub] carbondata pull request #2613: [HOTFIX] Modified code to fix the degrade in ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2613#discussion_r208178398
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/result/iterator/RawResultIterator.java
---
@@ -53,153 +39,124 @@
*/
private CarbonIterator detailRawQueryResultIterator;
- private boolean prefetchEnabled;
- private List currentBuffer;
- private List backupBuffer;
- private int currentIdxInBuffer;
- private ExecutorService executorService;
- private Future fetchFuture;
- private Object[] currentRawRow = null;
- private boolean isBackupFilled = false;
+ /**
+ * Counter to maintain the row counter.
+ */
+ private int counter = 0;
+
+ private Object[] currentConveretedRawRow = null;
+
+ /**
+ * LOGGER
+ */
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(RawResultIterator.class.getName());
+
+ /**
+ * batch of the result.
+ */
+ private RowBatch batch;
public RawResultIterator(CarbonIterator
detailRawQueryResultIterator,
- SegmentProperties sourceSegProperties, SegmentProperties
destinationSegProperties,
- boolean isStreamingHandOff) {
+ SegmentProperties sourceSegProperties, SegmentProperties
destinationSegProperties) {
this.detailRawQueryResultIterator = detailRawQueryResultIterator;
this.sourceSegProperties = sourceSegProperties;
this.destinationSegProperties = destinationSegProperties;
-this.executorService = Executors.newFixedThreadPool(1);
-
-if (!isStreamingHandOff) {
- init();
-}
}
- private void init() {
-this.prefetchEnabled = CarbonProperties.getInstance().getProperty(
-CarbonCommonConstants.CARBON_COMPACTION_PREFETCH_ENABLE,
-
CarbonCommonConstants.CARBON_COMPACTION_PREFETCH_ENABLE_DEFAULT).equalsIgnoreCase("true");
-try {
- new RowsFetcher(false).call();
- if (prefetchEnabled) {
-this.fetchFuture = executorService.submit(new RowsFetcher(true));
- }
-} catch (Exception e) {
- LOGGER.error(e, "Error occurs while fetching records");
- throw new RuntimeException(e);
-}
- }
+ @Override public boolean hasNext() {
- /**
- * fetch rows
- */
- private final class RowsFetcher implements Callable {
-private boolean isBackupFilling;
-
-private RowsFetcher(boolean isBackupFilling) {
- this.isBackupFilling = isBackupFilling;
-}
-
-@Override
-public Void call() throws Exception {
- if (isBackupFilling) {
-backupBuffer = fetchRows();
-isBackupFilled = true;
+if (null == batch || checkIfBatchIsProcessedCompletely(batch)) {
+ if (detailRawQueryResultIterator.hasNext()) {
+batch = null;
+batch = detailRawQueryResultIterator.next();
+counter = 0; // batch changed so reset the counter.
} else {
-currentBuffer = fetchRows();
+return false;
}
- return null;
}
- }
- private List fetchRows() {
-if (detailRawQueryResultIterator.hasNext()) {
- return detailRawQueryResultIterator.next().getRows();
+if (!checkIfBatchIsProcessedCompletely(batch)) {
+ return true;
} else {
- return new ArrayList<>();
+ return false;
}
}
- private void fillDataFromPrefetch() {
-try {
- if (currentIdxInBuffer >= currentBuffer.size() && 0 !=
currentIdxInBuffer) {
-if (prefetchEnabled) {
- if (!isBackupFilled) {
-fetchFuture.get();
- }
- // copy backup buffer to current buffer and fill backup buffer
asyn
- currentIdxInBuffer = 0;
- currentBuffer = backupBuffer;
- isBackupFilled = false;
- fetchFuture = executorService.submit(new RowsFetcher(true));
-} else {
- currentIdxInBuffer = 0;
- new RowsFetcher(false).call();
+ @Override public Object[] next() {
--- End diff --
Move @Override to previous line
---
[jira] [Resolved] (CARBONDATA-2823) Alter table set local dictionary include after bloom creation fails throwing incorrect error
[
https://issues.apache.org/jira/browse/CARBONDATA-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacky Li resolved CARBONDATA-2823.
--
Resolution: Fixed
Fix Version/s: 1.4.1
1.5.0
> Alter table set local dictionary include after bloom creation fails throwing
> incorrect error
>
>
> Key: CARBONDATA-2823
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2823
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.4.1
> Environment: Spark 2.1
>Reporter: Chetan Bhat
>Assignee: xuchuanyin
>Priority: Minor
> Fix For: 1.5.0, 1.4.1
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Steps :
> # create table
> # create bloom/lucene datamap
> # load data
> # alter table set tblProperties
> 0: jdbc:hive2://10.18.98.101:22550/default> CREATE TABLE uniqdata_load
> (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ
> timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1
> decimal(30,10), DECIMAL_COLUMN2 decimal(36,36),Double_COLUMN1 double,
> Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY
> 'org.apache.carbondata.format';
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (1.43 seconds)
> 0: jdbc:hive2://10.18.98.101:22550/default> CREATE DATAMAP
> dm_uniqdata1_tmstmp6 ON TABLE uniqdata_load USING 'bloomfilter' DMPROPERTIES
> ('INDEX_COLUMNS' = 'DOJ', 'BLOOM_SIZE'='64', 'BLOOM_FPP'='0.1');
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (0.828 seconds)
> 0: jdbc:hive2://10.18.98.101:22550/default> LOAD DATA INPATH
> 'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata_load
> OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (4.903 seconds)
> 0: jdbc:hive2://10.18.98.101:22550/default> alter table uniqdata_load set
> tblproperties('local_dictionary_include'='CUST_NAME');
> Error:
> org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException:
> streaming is not supported for index datamap (state=,code=0)
>
> Issue : Alter table set local dictionary include fails with incorrect error.
> 0: jdbc:hive2://10.18.98.101:22550/default> alter table uniqdata_load set
> tblproperties('local_dictionary_include'='CUST_NAME');
> *Error:
> org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException:
> streaming is not supported for index datamap (state=,code=0)*
>
> Expected : Operation should be success. If the operation is unsupported it
> should throw correct error message.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2609: [CARBONDATA-2823] Support streaming property ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2609 ---
[GitHub] carbondata issue #2609: [CARBONDATA-2823] Support streaming property with da...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2609 LGTM ---
[GitHub] carbondata pull request #2609: [CARBONDATA-2823] Support streaming property ...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2609#discussion_r208175586
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala
---
@@ -257,20 +256,13 @@ class DDLStrategy(sparkSession: SparkSession) extends
SparkStrategy {
if CarbonEnv.getInstance(sparkSession).carbonMetastore
.tableExists(tableName)(sparkSession) => {
-// TODO remove this limiation after streaming table support
'preaggregate' DataMap
-// if the table has 'preaggregate' DataMap, it doesn't support
streaming now
val carbonTable =
CarbonEnv.getInstance(sparkSession).carbonMetastore
.lookupRelation(tableName)(sparkSession).asInstanceOf[CarbonRelation].carbonTable
if (carbonTable != null &&
!carbonTable.getTableInfo.isTransactionalTable) {
throw new MalformedCarbonCommandException(
"Unsupported operation on non transactional table")
}
-if (carbonTable != null && !carbonTable.canAllow(carbonTable,
TableOperation.STREAMING)) {
--- End diff --
ok
---
[GitHub] carbondata pull request #2594: [CARBONDATA-2809][DataMap] Block rebuilding f...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2594 ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6189/ ---
[GitHub] carbondata issue #2611: [CARBONDATA-2836]Fixed data loading performance issu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2611 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7809/ ---
[GitHub] carbondata issue #2594: [CARBONDATA-2809][DataMap] Block rebuilding for bloo...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2594 LGTM ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6188/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6187/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2612 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7811/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2610 retest this please ---
