[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6558/ ---
[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2619 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7833/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2616 LGTM ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user praveenmeenakshi56 commented on the issue: https://github.com/apache/carbondata/pull/2617 retest SDV please ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6205/ ---
[GitHub] carbondata pull request #2619: [CARBONDATA-2819] Fixed cannot drop preagg da...
GitHub user Sssan520 opened a pull request: https://github.com/apache/carbondata/pull/2619 [CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tab⦠1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} = {true or false}",if datamap provider is 'preagg',block this configuration. 2.When create preagg datamap, now also create its datamap schema file in system folder. 3.Use command "show datamap on table {tableName}" will get its all datamaps from datamap schema files Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed?No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata droppreagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2619.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2619 commit 7884ba86f13b89fb23895400bbac12eaa4e62546 Author: Sssan520 Date: 2018-08-08T06:27:41Z [CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has other index datamaps ---
[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...
GitHub user lianganping opened a pull request: https://github.com/apache/carbondata/pull/2618 [CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tab⦠1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} = {true or false}",if datamap provider is 'preagg',block this configuration. 2.When create preagg datamap, now also create its datamap schema file in system folder. 3.Use command "show datamap on table {tableName}" will get its all datamaps from datamap schema files Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done NA Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sssan520/carbondata droppreagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2618.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2618 commit 7884ba86f13b89fb23895400bbac12eaa4e62546 Author: Sssan520 Date: 2018-08-08T06:27:41Z [CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has other index datamaps ---
[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...
Github user lianganping closed the pull request at: https://github.com/apache/carbondata/pull/2618 ---
[GitHub] carbondata issue #2618: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2618 Can one of the admins verify this patch? ---
[jira] [Resolved] (CARBONDATA-2659) Support partitioned carbon table by DataFrame.write
[ https://issues.apache.org/jira/browse/CARBONDATA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2659. - Resolution: Fixed Fix Version/s: (was: 1.5.0) > Support partitioned carbon table by DataFrame.write > --- > > Key: CARBONDATA-2659 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2659 > Project: CarbonData > Issue Type: Improvement >Reporter: Jacky Li >Priority: Major > Fix For: 1.4.1 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently only partition table is only supported by SQL, it should be > supported by Spark DataFrame API also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2415 ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 LGTM ---
[GitHub] carbondata pull request #2613: [HOTFIX] Modified code to fix the degrade in ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2613 ---
[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2617 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6204/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 LGTM ---
[jira] [Resolved] (CARBONDATA-2834) Refactor code to remove nested for loop to extract invalidTimestampRange.
[ https://issues.apache.org/jira/browse/CARBONDATA-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2834. - Resolution: Fixed Fix Version/s: 1.4.1 > Refactor code to remove nested for loop to extract invalidTimestampRange. > - > > Key: CARBONDATA-2834 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2834 > Project: CarbonData > Issue Type: Bug >Reporter: Kunal Kapoor >Assignee: Kunal Kapoor >Priority: Major > Fix For: 1.4.1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Reactor getInvalidTimestampRange method in SegmentUpdateStatusManager because > it has an unnecessary nested loop to get timestamp from invalid segments. > This will cause query performance degradation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2612 ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 LGTM ---
[GitHub] carbondata pull request #2617: [CARBONDATA-2838] Added SDV test cases for Lo...
GitHub user praveenmeenakshi56 opened a pull request: https://github.com/apache/carbondata/pull/2617 [CARBONDATA-2838] Added SDV test cases for Local Dictionary Support Added SDV test cases for Local Dictionary Support - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. Test cases are added - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_sdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2617.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2617 commit d05ba074b239e3d52672e1638eece39367fddd40 Author: praveenmeenakshi56 Date: 2018-08-08T06:15:49Z Added SDV test cases for Local Dictionary Support ---
[jira] [Created] (CARBONDATA-2838) Add SDV test cases for Local Dictionary Support
Praveen M P created CARBONDATA-2838: --- Summary: Add SDV test cases for Local Dictionary Support Key: CARBONDATA-2838 URL: https://issues.apache.org/jira/browse/CARBONDATA-2838 Project: CarbonData Issue Type: Sub-task Reporter: Praveen M P Assignee: Praveen M P -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2606#discussion_r208460810 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java --- @@ -169,38 +171,59 @@ private void doExecute(Iterator iterator, int iteratorIndex) thr if (rowsNotExist) { rowsNotExist = false; dataHandler = CarbonFactHandlerFactory.createCarbonFactHandler(model); +this.carbonFactHandlers.add(dataHandler); dataHandler.initialise(); } processBatch(iterator.next(), dataHandler, iteratorIndex); } -if (!rowsNotExist) { - finish(dataHandler, iteratorIndex); +try { + if (!rowsNotExist) { +finish(dataHandler, iteratorIndex); + } +} catch (CarbonDataWriterException e) { --- End diff -- remove this catch block ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6555/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7830/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2606 retest this please ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6554/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6553/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6551/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2616 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7829/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2615 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6552/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6550/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2615 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7828/ ---
[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2616 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6203/ ---
[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2616 [HOTFIX]Fixed int overflow and comparison gone wrong during blocklet min/max Problem: During calculating min/max for blocklet, it needs to calculate from all the pages. During that comparison, it is typecasting to int and overflows, so there is a chance the negative becomes positive and positive become negative. That's why min max of long comes wrong for bigger values. Solution: Don't typecast directly, instead check first the negative or positive and then return. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata blocklet-min-max-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2616.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2616 commit a99d0516ee06a285f8be3880d7005bf161f1c940 Author: ravipesala Date: 2018-08-07T15:49:36Z Fixed int overflow and comparasion gone wrong during blocklet min/max ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7827/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7826/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2612 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6547/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6548/ ---
[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2615 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6202/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2612 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7824/ ---
[GitHub] carbondata pull request #2615: [HOTFIX] [presto] presto code cleanup
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2615 [HOTFIX] [presto] presto code cleanup Changes in the PR: a. Removed spark jars from release jars dependency b. enable check style for presto module c. fixed all the checkstyle for presto module Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NA - [ ] Any backward compatibility impacted? NA - [ ] Document update required? NA - [ ] Testing done. NA - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2615.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2615 commit 399ca66addcd6767bc43ac7fa15514eb705546d4 Author: ajantha-bhat Date: 2018-08-06T12:21:03Z [HOTFIX] presto code cleanup a. Removed spark jars from release jars dependency b. enable check style for presto module c. fixed all the checkstyle for presto module ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2564 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7825/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7822/ ---
[jira] [Resolved] (CARBONDATA-2807) Fixed data load performance issue with more number of records
[ https://issues.apache.org/jira/browse/CARBONDATA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2807. - Resolution: Fixed Fix Version/s: 1.4.1 > Fixed data load performance issue with more number of records > - > > Key: CARBONDATA-2807 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2807 > Project: CarbonData > Issue Type: Improvement >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.4.1 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > **Problem:**Data Loading is taking more time when number of records are high. > **Root cause:** As number of records are high intermediate merger is taking > more time. > **Solution:** Checking the number of files present in file list is done is > synchronized block because of this > each intermediate request is taking sometime and when number of records are > high it impacting overall data loading performance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2588 ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2588 LGTM ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2602 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6544/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6546/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6545/ ---
[jira] [Resolved] (CARBONDATA-2831) Support Merge index files read from non transactional table.
[ https://issues.apache.org/jira/browse/CARBONDATA-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2831. - Resolution: Fixed Fix Version/s: 1.4.1 > Support Merge index files read from non transactional table. > > > Key: CARBONDATA-2831 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2831 > Project: CarbonData > Issue Type: Bug >Reporter: Ajantha Bhat >Assignee: Ajantha Bhat >Priority: Major > Fix For: 1.4.1 > > Time Spent: 3h > Remaining Estimate: 0h > > problem : Currently SDK read/ nontransactional table read from external > table gives null output when carbonMergeindex file is present instead of > carobnindex files. > cause : In LatestFileReadCommitted, while taking snapshot, merge index files > were not considered. > solution: consider the merge index files while taking snapshot > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2610: [CARBONDATA-2831] Added Support Merge index f...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2610 ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6200/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2588 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7823/ ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2602 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7820/ ---
[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2537 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6199/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7819/ ---
[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2564 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6198/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6543/ ---
[GitHub] carbondata pull request #2537: [CARBONDATA-2768][CarbonStore] Fix error in t...
Github user xuchuanyin closed the pull request at: https://github.com/apache/carbondata/pull/2537 ---
[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2415#discussion_r208224446 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala --- @@ -92,11 +89,38 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) { ).filter(_._2.isDefined) .map(property => s"'${property._1}' = '${property._2.get}'").mkString(",") +val partition: Seq[String] = if (options.partitionColumns.isDefined) { + if (options.partitionColumns.get.toSet.size != options.partitionColumns.get.length) { +throw new MalformedCarbonCommandException(s"repeated partition column") + } + options.partitionColumns.get.map { column => +val field = schema.fields.find(_.name.equalsIgnoreCase(column)) +if (field.isEmpty) { + throw new MalformedCarbonCommandException(s"invalid partition column: $column") +} +s"$column ${field.get.dataType.typeName}" + } +} else { + Seq() +} + +val schemaWithoutPartition = if (options.partitionColumns.isDefined) { + val fields = schema.filterNot(field => options.partitionColumns.get.contains(field.name)) --- End diff -- fixed ---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2614 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6542/ ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2610 LGTM ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6197/ ---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2614 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7818/ ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6541/ ---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Github user jackylk closed the pull request at: https://github.com/apache/carbondata/pull/2589 ---
[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2391 @xubo245 Please close it as it is handled in https://github.com/apache/carbondata/pull/2441 ---
[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2537 LGTM merged into external-format ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 retest this please ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6196/ ---
[GitHub] carbondata issue #2423: [CARBONDATA-2530][MV] Fix wrong data displayed when ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2423 @xubo245 Please close it, it is handled in https://github.com/apache/carbondata/pull/2474 ---
[GitHub] carbondata pull request #2602: [CARBONDATA-2539]Fix mv classcast exception i...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2602 ---
[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2605 ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2602 LGTM ---
[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2612#discussion_r208211297 --- Diff: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java --- @@ -779,6 +779,32 @@ public UpdateVO getInvalidTimestampRange(String segmentId) { } return range; } + + /** + * Returns the invalid timestamp range of a segment. + * @return + */ + public List getInvalidTimestampRange() { +List ranges = new ArrayList(); +for (LoadMetadataDetails segment : segmentDetails) { + if (!(segment.getSegmentStatus().equals(SegmentStatus.SUCCESS) || + segment.getSegmentStatus().equals(SegmentStatus.LOAD_PARTIAL_SUCCESS))) { --- End diff -- Please refer SegmentStatusManager for checking the invalid segments ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2606 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7817/ ---
[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2612 @kunal642 Please fix checkstyle ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/2605 2.1 build has random failure, which is not related to this change ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2605 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7816/ ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2605 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6540/ ---
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2605 LGTM ---
[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2614#discussion_r208206503 --- Diff: examples/spark2/pom.xml --- @@ -49,6 +49,11 @@ carbondata-store-sdk ${project.version} + + org.apache.carbondata + carbondata-mv-core --- End diff -- This is adding always or in case of `mv` profile is selected? ---
[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2602 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6195/ ---
[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2588 retest this please ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6539/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2613 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7815/ ---
[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2614 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6194/ ---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user mohammadshahidkhan commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2588#discussion_r208199760 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java --- @@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) { } public void startFileMergingIfPossible() { -File[] fileList = null; -synchronized (lockObject) { - if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { +File[] fileList; +if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { --- End diff -- ok ---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user mohammadshahidkhan commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2588#discussion_r208199601 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java --- @@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) { } public void startFileMergingIfPossible() { -File[] fileList = null; -synchronized (lockObject) { - if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { +File[] fileList; +if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { + synchronized (lockObject) { fileList = procFiles.toArray(new File[procFiles.size()]); this.procFiles = new ArrayList(); -if (LOGGER.isDebugEnabled()) { - LOGGER - .debug("Submitting request for intermediate merging no of files: " + fileList.length); -} } -} -if (null != fileList) { + if (LOGGER.isDebugEnabled()) { +LOGGER.debug("Sumitting request for intermediate merging no of files: " + fileList.length); + } startIntermediateMerging(fileList); --- End diff -- ok ---
[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2588#discussion_r208198390 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java --- @@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) { } public void startFileMergingIfPossible() { -File[] fileList = null; -synchronized (lockObject) { - if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { +File[] fileList; +if (procFiles.size() >= parameters.getNumberOfIntermediateFileToBeMerged()) { --- End diff -- @mohammadshahidkhan Yes you are right but in this case UnsafeSortDataRow processing will be slower as it will read/ sort and write so chances of above condition is negligible, because of this double check is not added here ---
[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2606 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6193/ ---
[jira] [Resolved] (CARBONDATA-2836) Fixed data loading performance issue
[ https://issues.apache.org/jira/browse/CARBONDATA-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2836. - Resolution: Fixed Fix Version/s: 1.4.1 > Fixed data loading performance issue > > > Key: CARBONDATA-2836 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2836 > Project: CarbonData > Issue Type: Improvement >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.4.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Problem: Data Loading is taking more time when number of records are high(3.5 > billion) records > Root Cause: In case of Final merge sort temp row conversion is done in main > thread because of this final step processing became slower. > Solution: Mode conversion logic to pre-fetch thread for parallel processing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2611: [CARBONDATA-2836]Fixed data loading performan...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2611 ---
[GitHub] carbondata issue #2611: [CARBONDATA-2836]Fixed data loading performance issu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2611 LGTM ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2610 retest this please ---
[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2614 [CARBONDATA-2837] Added MVExample in example module Added MVExample in example module Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata mvexample Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2614.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2614 commit f50e72201dd1d0da720bcfaf68c5e4df14a51624 Author: ravipesala Date: 2018-08-07T11:07:17Z Added MVExample ---
[jira] [Created] (CARBONDATA-2837) Add MV Example in examples module
Ravindra Pesala created CARBONDATA-2837: --- Summary: Add MV Example in examples module Key: CARBONDATA-2837 URL: https://issues.apache.org/jira/browse/CARBONDATA-2837 Project: CarbonData Issue Type: Improvement Reporter: Ravindra Pesala -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2605 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6192/ ---
[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2613 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6191/ ---
[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2605#discussion_r208186221 --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala --- @@ -641,6 +641,78 @@ object CarbonScalaUtil { } } + /** + * This method validates all the child columns of complex column recursively to check whether + * any of the child column is of string dataType or not + * + * @param field + */ + def validateChildColumnsRecursively(field: Field): Boolean = { +if (field.children.isDefined && null != field.children.get) { + field.children.get.exists { childColumn => +if (childColumn.children.isDefined && null != childColumn.children.get) { + validateChildColumnsRecursively(childColumn) +} else { + childColumn.dataType.get.equalsIgnoreCase("string") +} + } +} else { + false +} + } + + /** + * This method validates the local dictionary configured columns + * + * @param fields --- End diff -- please add parameter description for all newly added parameter ---
[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2610 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6536/ ---