[GitHub] asfgit closed pull request #3065: [HOTFIX] Optimize presto-guide
asfgit closed pull request #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sraghunandan commented on issue #3065: [HOTFIX] Optimize presto-guide
sraghunandan commented on issue #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065#issuecomment-455756135 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sraghunandan commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration
sraghunandan commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration URL: https://github.com/apache/carbondata/pull/3054#issuecomment-455756049 LGTM. @xuchuanyin your opinion please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide
CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065#issuecomment-455748669 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2578/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide
CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065#issuecomment-455748656 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10613/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chenliang613 commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration
chenliang613 commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration URL: https://github.com/apache/carbondata/pull/3054#issuecomment-455746877 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide
CarbonDataQA commented on issue #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065#issuecomment-455745671 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2354/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chenliang613 commented on a change in pull request #3065: [HOTFIX] Optimize presto-guide
chenliang613 commented on a change in pull request #3065: [HOTFIX] Optimize presto-guide URL: https://github.com/apache/carbondata/pull/3065#discussion_r249232561 ## File path: docs/presto-guide.md ## @@ -220,7 +220,8 @@ Now you can use the Presto CLI on the coordinator to query data sources in the c Secondly: Create a folder named 'carbondata' under $PRESTO_HOME$/plugin and copy all jars from carbondata/integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT to $PRESTO_HOME$/plugin/carbondata - + **NOTE:** Not copy one assemble jar, need to copy many jars from integration/presto/target/carbondata-presto-x.x.x-SNAPSHOT Review comment: > Why is this NOTE needed? I think line#221 has already made it clear. Many community guys made mistakes, this is why add one more note. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3253) Remove test case of bloom datamap using search mode
[ https://issues.apache.org/jira/browse/CARBONDATA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-3253: --- Fix Version/s: 1.5.2 > Remove test case of bloom datamap using search mode > --- > > Key: CARBONDATA-3253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3253 > Project: CarbonData > Issue Type: Improvement >Reporter: jiangmanhua >Assignee: jiangmanhua >Priority: Minor > Fix For: 1.5.2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-3253) Remove test case of bloom datamap using search mode
[ https://issues.apache.org/jira/browse/CARBONDATA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen reassigned CARBONDATA-3253: -- Assignee: jiangmanhua Priority: Minor (was: Major) > Remove test case of bloom datamap using search mode > --- > > Key: CARBONDATA-3253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3253 > Project: CarbonData > Issue Type: Improvement >Reporter: jiangmanhua >Assignee: jiangmanhua >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-3253) Remove test case of bloom datamap using search mode
[ https://issues.apache.org/jira/browse/CARBONDATA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-3253. Resolution: Fixed > Remove test case of bloom datamap using search mode > --- > > Key: CARBONDATA-3253 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3253 > Project: CarbonData > Issue Type: Improvement >Reporter: jiangmanhua >Assignee: jiangmanhua >Priority: Minor > Fix For: 1.5.2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3077: [CARBONDATA-3253] Remove test case of bloom datamap using search mode
asfgit closed pull request #3077: [CARBONDATA-3253] Remove test case of bloom datamap using search mode URL: https://github.com/apache/carbondata/pull/3077 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xuchuanyin commented on issue #3046: [CARBONDATA-3231] Fix OOM exception when dictionary map size is too huge in case of varchar columns
xuchuanyin commented on issue #3046: [CARBONDATA-3231] Fix OOM exception when dictionary map size is too huge in case of varchar columns URL: https://github.com/apache/carbondata/pull/3046#issuecomment-455742622 Fine. Please check the code again. I am just curious about the implementation not the details of this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] chenliang613 commented on issue #3077: [CARBONDATA-3253] Remove test case of bloom datamap using search mode
chenliang613 commented on issue #3077: [CARBONDATA-3253] Remove test case of bloom datamap using search mode URL: https://github.com/apache/carbondata/pull/3077#issuecomment-455741790 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression
CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression URL: https://github.com/apache/carbondata/pull/3080#issuecomment-455741671 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10612/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression
CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression URL: https://github.com/apache/carbondata/pull/3080#issuecomment-455741657 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2577/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression
CarbonDataQA commented on issue #3080: [CARBONDATA-3256] Fix that MV datamap doesn't affect using avg expression and count expression URL: https://github.com/apache/carbondata/pull/3080#issuecomment-455736630 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2353/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (CARBONDATA-3246) SDK reader fails if vectorReader is false for concurrent read scenario and batch size is zero.
[ https://issues.apache.org/jira/browse/CARBONDATA-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-3246. -- Resolution: Fixed Fix Version/s: 1.5.2 > SDK reader fails if vectorReader is false for concurrent read scenario and > batch size is zero. > -- > > Key: CARBONDATA-3246 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3246 > Project: CarbonData > Issue Type: Bug >Reporter: Shardul Singh >Assignee: Shardul Singh >Priority: Minor > Fix For: 1.5.2 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > SDK reader fails if vectorReader is false for concurrent read scenario and > batch size is zero. > If the batch size is zero or less , we should throw a proper exception > stating that batch size cannot be less than zero. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False.
asfgit closed pull request #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False. URL: https://github.com/apache/carbondata/pull/3070 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455633340 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10611/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455630797 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2576/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 edited a comment on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
xubo245 edited a comment on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455592722 @ravipesala For the S3, even though the CI and UT pass for each PR, but sometimes the code failed when I run with S3, I have meet many times. we should add base test code for S3, HDFS, Alluxio UT if the technology can support. I only addd the web application in test module, it won't affect other module. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
CarbonDataQA commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455612868 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2352/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kumarvishal09 commented on issue #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False.
kumarvishal09 commented on issue #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False. URL: https://github.com/apache/carbondata/pull/3070#issuecomment-455600169 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 edited a comment on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
xubo245 edited a comment on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455594345 @ravipesala For the alluxio and hive, they can work in carbonData before, but Recent months they can't work in carbondata project because someone change the code but CI/UT/SDV didn't find the failure. It's better to add base test case in CI for them if it's possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
xubo245 commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455594345 @ravipesala For the alluxio and hive, they can work in carbonData before, but Recent months they can't work in carbondata project because someone change the code but CI/UT/SDV didn't find the failure. We should add base test case in CI for them. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
xubo245 commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455592722 @ravipesala I tried to discuss with Alluxio PMC member， they said alluxio can‘t support alluxio mini cluster without webapp inother project (out of alluxio project). Maybe they can support in the future. For the S3, even though the CI and UT pass for each PR, but sometimes the code failed when I run with S3, I have meet many times. we should add base test code for S3, HDFS, Alluxio UT if the technology can support. I only addd the web application in test module, it won't affect other module. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ravipesala commented on issue #3063: [CARBONDATA-3242] Move Range_Column into the table level properties
ravipesala commented on issue #3063: [CARBONDATA-3242] Move Range_Column into the table level properties URL: https://github.com/apache/carbondata/pull/3063#issuecomment-455579142 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (CARBONDATA-3238) Throw StackOverflowError exception using MV datamap
[
https://issues.apache.org/jira/browse/CARBONDATA-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Pesala resolved CARBONDATA-3238.
-
Resolution: Fixed
Fix Version/s: 1.5.2
> Throw StackOverflowError exception using MV datamap
> ---
>
> Key: CARBONDATA-3238
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3238
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.5.1
>Reporter: Chenjian Qiu
>Priority: Blocker
> Fix For: 1.5.2
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Exception:
> java.lang.StackOverflowError
> at
> org.apache.spark.sql.catalyst.expressions.AttributeMap$$anonfun$get$1.apply(AttributeMap.scala:34)
> at
> org.apache.spark.sql.catalyst.expressions.AttributeMap$$anonfun$get$1.apply(AttributeMap.scala:34)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.spark.sql.catalyst.expressions.AttributeMap.get(AttributeMap.scala:34)
> at
> org.apache.spark.sql.catalyst.expressions.AttributeMap.contains(AttributeMap.scala:36)
> TestCase:
> sql("drop datamap if exists all_table_mv")
> sql("drop table if exists all_table")
> sql("create table all_table(x1 bigint,x2 bigint,x3 string,x4 bigint,x5
> bigint,x6 int,x7 string,x8 int, x9 int,x10 bigint," +
> "x11 bigint, x12 bigint,x13 bigint,x14 bigint,x15 bigint,x16 bigint,x17
> bigint,x18 bigint,x19 bigint) stored by 'carbondata'")
> sql("insert into all_table select
> 1,1,null,1,1,1,null,1,1,1,1,1,1,1,1,1,1,1,1")
> sql("create datamap all_table_mv on table all_table using 'mv' " +
> "as select sum(x12) as y1, sum(x13) as y2, sum(x14) as y3,sum(x15) as
> y4,X8,x9,x2 from all_table group by X8,x9,x2")
> sql("rebuild datamap all_table_mv")
> sql("explain select sum(x12) as y1, sum(x13) as y2, sum(x14) as
> y3,sum(x15) as y4,X8,x9,x2 from all_table group by X8,x9,x2")
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #3068: [HOTFIX] Fixed NPE during query with Local Dictionary
asfgit closed pull request #3068: [HOTFIX] Fixed NPE during query with Local Dictionary URL: https://github.com/apache/carbondata/pull/3068 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ravipesala commented on issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionary
ravipesala commented on issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionary URL: https://github.com/apache/carbondata/pull/3068#issuecomment-455573538 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap
asfgit closed pull request #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap URL: https://github.com/apache/carbondata/pull/3058 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ravipesala commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap
ravipesala commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap URL: https://github.com/apache/carbondata/pull/3058#issuecomment-455570917 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455544620 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2575/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455544592 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10610/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455529327 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2351/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455516337 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2574/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455512505 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10609/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455492939 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2350/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257]
Fix for NO_SORT load and describe formatted being in NO_SORT flow even with
Sort Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248983821
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
##
@@ -54,8 +54,12 @@ private[sql] case class CarbonDescribeFormattedCommand(
val carbonTable = relation.carbonTable
val tblProps =
carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+// If Sort Columns are given and Sort Scope is not given then pass
LOCAL_SORT
+// as the sort scope, else pass NO_SORT
val sortScope = if (carbonTable.getNumberOfSortColumns == 0) {
"NO_SORT"
+} else if (tblProps.get("sort_scope") != "NO_SORT") {
+ "LOCAL_SORT"
Review comment:
@qiuchenjian When there is no sort scope specified by the user and we have
sort columns present then it will take LOCAL_SORT.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455483700 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2349/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix
for NO_SORT load and describe formatted being in NO_SORT flow even with Sort
Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248974683
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
##
@@ -54,8 +54,12 @@ private[sql] case class CarbonDescribeFormattedCommand(
val carbonTable = relation.carbonTable
val tblProps =
carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+// If Sort Columns are given and Sort Scope is not given then pass
LOCAL_SORT
+// as the sort scope, else pass NO_SORT
val sortScope = if (carbonTable.getNumberOfSortColumns == 0) {
"NO_SORT"
+} else if (tblProps.get("sort_scope") != "NO_SORT") {
+ "LOCAL_SORT"
Review comment:
@manishnalla1994 why return local_sort when sort_scope isn't no_sort ?
what about global_sort and batch_sort ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257]
Fix for NO_SORT load and describe formatted being in NO_SORT flow even with
Sort Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248973171
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
##
@@ -54,8 +54,12 @@ private[sql] case class CarbonDescribeFormattedCommand(
val carbonTable = relation.carbonTable
val tblProps =
carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+// If Sort Columns are given and Sort Scope is not given then pass
LOCAL_SORT
+// as the sort scope, else pass NO_SORT
val sortScope = if (carbonTable.getNumberOfSortColumns == 0) {
"NO_SORT"
+} else if (tblProps.get("sort_scope") != "NO_SORT") {
+ "LOCAL_SORT"
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455479177 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2348/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #2047: [CARBONDATA-2240] Refactored TestPreaggregateExpressions to remove duplicate test case to improve CI Time
CarbonDataQA commented on issue #2047: [CARBONDATA-2240] Refactored TestPreaggregateExpressions to remove duplicate test case to improve CI Time URL: https://github.com/apache/carbondata/pull/2047#issuecomment-455478699 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2570/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix
for NO_SORT load and describe formatted being in NO_SORT flow even with Sort
Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248970244
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDescribeFormattedCommand.scala
##
@@ -54,8 +54,12 @@ private[sql] case class CarbonDescribeFormattedCommand(
val carbonTable = relation.carbonTable
val tblProps =
carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+// If Sort Columns are given and Sort Scope is not given then pass
LOCAL_SORT
+// as the sort scope, else pass NO_SORT
val sortScope = if (carbonTable.getNumberOfSortColumns == 0) {
"NO_SORT"
+} else if (tblProps.get("sort_scope") != "NO_SORT") {
+ "LOCAL_SORT"
Review comment:
please use "! equalsIgnoreCase" instead of "!="
can you describe this branch clearly ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
manishnalla1994 commented on a change in pull request #3083: [CARBONDATA-3257]
Fix for NO_SORT load and describe formatted being in NO_SORT flow even with
Sort Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248969359
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##
@@ -209,16 +209,24 @@ case class CarbonLoadDataCommand(
* 4. Session property CARBON_OPTIONS_SORT_SCOPE
* 5. Default Sort Scope LOAD_SORT_SCOPE
*/
-optionsFinal.put("sort_scope",
- options.getOrElse("sort_scope",
-carbonProperty.getProperty(
- CarbonLoadOptionConstants.CARBON_TABLE_LOAD_SORT_SCOPE +
table.getDatabaseName + "." +
- table.getTableName,
- tableProperties.asScala.getOrElse("sort_scope",
-
carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_OPTIONS_SORT_SCOPE,
- carbonProperty.getProperty(CarbonCommonConstants.LOAD_SORT_SCOPE,
-CarbonCommonConstants.LOAD_SORT_SCOPE_DEFAULT))
-
+if (tableProperties.get("sort_columns").size > 0 ||
Review comment:
Changed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
qiuchenjian commented on a change in pull request #3083: [CARBONDATA-3257] Fix
for NO_SORT load and describe formatted being in NO_SORT flow even with Sort
Columns given
URL: https://github.com/apache/carbondata/pull/3083#discussion_r248968200
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
##
@@ -209,16 +209,24 @@ case class CarbonLoadDataCommand(
* 4. Session property CARBON_OPTIONS_SORT_SCOPE
* 5. Default Sort Scope LOAD_SORT_SCOPE
*/
-optionsFinal.put("sort_scope",
- options.getOrElse("sort_scope",
-carbonProperty.getProperty(
- CarbonLoadOptionConstants.CARBON_TABLE_LOAD_SORT_SCOPE +
table.getDatabaseName + "." +
- table.getTableName,
- tableProperties.asScala.getOrElse("sort_scope",
-
carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_OPTIONS_SORT_SCOPE,
- carbonProperty.getProperty(CarbonCommonConstants.LOAD_SORT_SCOPE,
-CarbonCommonConstants.LOAD_SORT_SCOPE_DEFAULT))
-
+if (tableProperties.get("sort_columns").size > 0 ||
Review comment:
```suggestion
if (tableProperties.get("sort_columns").size > 0 &&
```
why "||"
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
CarbonDataQA commented on issue #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083#issuecomment-455474819 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2347/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] manishnalla1994 opened a new pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given
manishnalla1994 opened a new pull request #3083: [CARBONDATA-3257] Fix for NO_SORT load and describe formatted being in NO_SORT flow even with Sort Columns given URL: https://github.com/apache/carbondata/pull/3083 Problem: Data Load is in No sort flow when version is upgraded even if sort columns are given. Also describe formatted displays wrong sort scope after refresh. Solution: Added a condition to check for the presence of Sort Columns. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command to list the cache sizes for all tables
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command to list the cache sizes for all tables
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248964249
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/lru/CarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.lru
+
+import scala.collection.mutable
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
+import org.apache.spark.sql.execution.command.DataCommand
+import org.apache.spark.sql.types.{LongType, StringType}
+
+import org.apache.carbondata.core.cache.CacheProvider
+import org.apache.carbondata.core.cache.dictionary.AbstractColumnDictionaryInfo
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.indexstore.BlockletDataMapIndexWrapper
+
+/**
+ * SHOW LRU
+ */
+case class CarbonDataShowLRUCommand(tableIdentifier: Option[TableIdentifier])
+ extends DataCommand {
+
+ override def output: Seq[Attribute] = {
+Seq(AttributeReference("database", StringType, nullable = false)(),
+ AttributeReference("table", StringType, nullable = false)(),
+ AttributeReference("index size", LongType, nullable = false)(),
+ AttributeReference("dictionary size", LongType, nullable = false)())
+ }
+
+ override protected def opName: String = "SHOW LRU"
+
+ def showLRUOfAllTables(sparkSession: SparkSession): Seq[Row] = {
+val currentDatabase = sparkSession.sessionState.catalog.getCurrentDatabase
+val lruCache = CacheProvider.getInstance().getCarbonLRUCache()
+if (lruCache == null) {
+ Seq(Row("ALL", "ALL", 0L, 0L),
+Row(currentDatabase, "ALL", 0L, 0L))
+} else {
+ val tableIdents =
sparkSession.sessionState.catalog.listTables(currentDatabase).toArray
+ val dbLocation = CarbonEnv.getDatabaseLocation(currentDatabase,
sparkSession)
+ val tempLocation = dbLocation.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePaths = tableIdents.map { tableIdent =>
+(tempLocation + CarbonCommonConstants.FILE_SEPARATOR +
+ tableIdent.table + CarbonCommonConstants.FILE_SEPARATOR,
+ tableIdent.database.get + "." + tableIdent.table)
+ }
+
+ val dictIds = tableIdents.flatMap { tableIdent =>
+CarbonEnv
+ .getCarbonTable(tableIdent)(sparkSession)
+ .getAllDimensions
+ .asScala
+ .filter(_.isGlobalDictionaryEncoding)
+ .toArray
+ .map(dim => (dim.getColumnId, tableIdent.database.get + "." +
tableIdent.table))
+ }
+
+ // all databases
+ var (allIndexSize, allDictSize) = (0L, 0L)
+ // current database
+ var (dbIndexSize, dbDictSize) = (0L, 0L)
+ val tableMapIndexSize = mutable.HashMap[String, Long]()
+ val tableMapDictSize = mutable.HashMap[String, Long]()
+ val cacheIterator = lruCache.getLruCacheMap.entrySet().iterator()
+ while (cacheIterator.hasNext) {
+val entry = cacheIterator.next()
+val cache = entry.getValue
+if (cache.isInstanceOf[BlockletDataMapIndexWrapper]) {
+ allIndexSize = allIndexSize + cache.getMemorySize
+ val indexPath = entry.getKey.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePath = tablePaths.find(path =>
indexPath.startsWith(path._1))
+ if (tablePath.isDefined) {
+dbIndexSize = dbIndexSize + cache.getMemorySize
+val memorySize = tableMapIndexSize.get(tablePath.get._2)
+if (memorySize.isEmpty) {
+ tableMapIndexSize.put(tablePath.get._2, cache.getMemorySize)
+} else {
+ tableMapIndexSize.put(tablePath.get._2, memorySize.get +
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command to list the cache sizes for all tables
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command to list the cache sizes for all tables
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248964012
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/lru/CarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.lru
+
+import scala.collection.mutable
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
+import org.apache.spark.sql.execution.command.DataCommand
+import org.apache.spark.sql.types.{LongType, StringType}
+
+import org.apache.carbondata.core.cache.CacheProvider
+import org.apache.carbondata.core.cache.dictionary.AbstractColumnDictionaryInfo
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.indexstore.BlockletDataMapIndexWrapper
+
+/**
+ * SHOW LRU
+ */
+case class CarbonDataShowLRUCommand(tableIdentifier: Option[TableIdentifier])
+ extends DataCommand {
+
+ override def output: Seq[Attribute] = {
+Seq(AttributeReference("database", StringType, nullable = false)(),
+ AttributeReference("table", StringType, nullable = false)(),
+ AttributeReference("index size", LongType, nullable = false)(),
+ AttributeReference("dictionary size", LongType, nullable = false)())
+ }
+
+ override protected def opName: String = "SHOW LRU"
+
+ def showLRUOfAllTables(sparkSession: SparkSession): Seq[Row] = {
+val currentDatabase = sparkSession.sessionState.catalog.getCurrentDatabase
+val lruCache = CacheProvider.getInstance().getCarbonLRUCache()
+if (lruCache == null) {
+ Seq(Row("ALL", "ALL", 0L, 0L),
+Row(currentDatabase, "ALL", 0L, 0L))
+} else {
+ val tableIdents =
sparkSession.sessionState.catalog.listTables(currentDatabase).toArray
+ val dbLocation = CarbonEnv.getDatabaseLocation(currentDatabase,
sparkSession)
+ val tempLocation = dbLocation.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePaths = tableIdents.map { tableIdent =>
+(tempLocation + CarbonCommonConstants.FILE_SEPARATOR +
+ tableIdent.table + CarbonCommonConstants.FILE_SEPARATOR,
+ tableIdent.database.get + "." + tableIdent.table)
+ }
+
+ val dictIds = tableIdents.flatMap { tableIdent =>
+CarbonEnv
+ .getCarbonTable(tableIdent)(sparkSession)
+ .getAllDimensions
+ .asScala
+ .filter(_.isGlobalDictionaryEncoding)
+ .toArray
+ .map(dim => (dim.getColumnId, tableIdent.database.get + "." +
tableIdent.table))
+ }
+
+ // all databases
+ var (allIndexSize, allDictSize) = (0L, 0L)
+ // current database
+ var (dbIndexSize, dbDictSize) = (0L, 0L)
+ val tableMapIndexSize = mutable.HashMap[String, Long]()
+ val tableMapDictSize = mutable.HashMap[String, Long]()
+ val cacheIterator = lruCache.getLruCacheMap.entrySet().iterator()
+ while (cacheIterator.hasNext) {
+val entry = cacheIterator.next()
+val cache = entry.getValue
+if (cache.isInstanceOf[BlockletDataMapIndexWrapper]) {
+ allIndexSize = allIndexSize + cache.getMemorySize
+ val indexPath = entry.getKey.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePath = tablePaths.find(path =>
indexPath.startsWith(path._1))
+ if (tablePath.isDefined) {
+dbIndexSize = dbIndexSize + cache.getMemorySize
+val memorySize = tableMapIndexSize.get(tablePath.get._2)
+if (memorySize.isEmpty) {
+ tableMapIndexSize.put(tablePath.get._2, cache.getMemorySize)
+} else {
+ tableMapIndexSize.put(tablePath.get._2, memorySize.get +
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command to list the cache sizes for all tables
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command to list the cache sizes for all tables
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248964012
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/lru/CarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.lru
+
+import scala.collection.mutable
+import scala.collection.JavaConverters._
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
+import org.apache.spark.sql.execution.command.DataCommand
+import org.apache.spark.sql.types.{LongType, StringType}
+
+import org.apache.carbondata.core.cache.CacheProvider
+import org.apache.carbondata.core.cache.dictionary.AbstractColumnDictionaryInfo
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.indexstore.BlockletDataMapIndexWrapper
+
+/**
+ * SHOW LRU
+ */
+case class CarbonDataShowLRUCommand(tableIdentifier: Option[TableIdentifier])
+ extends DataCommand {
+
+ override def output: Seq[Attribute] = {
+Seq(AttributeReference("database", StringType, nullable = false)(),
+ AttributeReference("table", StringType, nullable = false)(),
+ AttributeReference("index size", LongType, nullable = false)(),
+ AttributeReference("dictionary size", LongType, nullable = false)())
+ }
+
+ override protected def opName: String = "SHOW LRU"
+
+ def showLRUOfAllTables(sparkSession: SparkSession): Seq[Row] = {
+val currentDatabase = sparkSession.sessionState.catalog.getCurrentDatabase
+val lruCache = CacheProvider.getInstance().getCarbonLRUCache()
+if (lruCache == null) {
+ Seq(Row("ALL", "ALL", 0L, 0L),
+Row(currentDatabase, "ALL", 0L, 0L))
+} else {
+ val tableIdents =
sparkSession.sessionState.catalog.listTables(currentDatabase).toArray
+ val dbLocation = CarbonEnv.getDatabaseLocation(currentDatabase,
sparkSession)
+ val tempLocation = dbLocation.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePaths = tableIdents.map { tableIdent =>
+(tempLocation + CarbonCommonConstants.FILE_SEPARATOR +
+ tableIdent.table + CarbonCommonConstants.FILE_SEPARATOR,
+ tableIdent.database.get + "." + tableIdent.table)
+ }
+
+ val dictIds = tableIdents.flatMap { tableIdent =>
+CarbonEnv
+ .getCarbonTable(tableIdent)(sparkSession)
+ .getAllDimensions
+ .asScala
+ .filter(_.isGlobalDictionaryEncoding)
+ .toArray
+ .map(dim => (dim.getColumnId, tableIdent.database.get + "." +
tableIdent.table))
+ }
+
+ // all databases
+ var (allIndexSize, allDictSize) = (0L, 0L)
+ // current database
+ var (dbIndexSize, dbDictSize) = (0L, 0L)
+ val tableMapIndexSize = mutable.HashMap[String, Long]()
+ val tableMapDictSize = mutable.HashMap[String, Long]()
+ val cacheIterator = lruCache.getLruCacheMap.entrySet().iterator()
+ while (cacheIterator.hasNext) {
+val entry = cacheIterator.next()
+val cache = entry.getValue
+if (cache.isInstanceOf[BlockletDataMapIndexWrapper]) {
+ allIndexSize = allIndexSize + cache.getMemorySize
+ val indexPath = entry.getKey.replace(
+CarbonCommonConstants.WINDOWS_FILE_SEPARATOR,
CarbonCommonConstants.FILE_SEPARATOR)
+ val tablePath = tablePaths.find(path =>
indexPath.startsWith(path._1))
+ if (tablePath.isDefined) {
+dbIndexSize = dbIndexSize + cache.getMemorySize
+val memorySize = tableMapIndexSize.get(tablePath.get._2)
+if (memorySize.isEmpty) {
+ tableMapIndexSize.put(tablePath.get._2, cache.getMemorySize)
+} else {
+ tableMapIndexSize.put(tablePath.get._2, memorySize.get +
[GitHub] ajantha-bhat commented on issue #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False.
ajantha-bhat commented on issue #3070: [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False. URL: https://github.com/apache/carbondata/pull/3070#issuecomment-455470370 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kumarvishal09 commented on a change in pull request #3068: [HOTFIX] Fixed NPE during query with Local Dictionary
kumarvishal09 commented on a change in pull request #3068: [HOTFIX] Fixed NPE
during query with Local Dictionary
URL: https://github.com/apache/carbondata/pull/3068#discussion_r248963600
##
File path:
integration/spark-datasource/src/main/spark2.1andspark2.2/org/apache/spark/sql/CarbonVectorProxy.java
##
@@ -311,6 +311,7 @@ public void setDictionary(CarbonDictionary dictionary) {
vector.setDictionary(dictionaryWrapper);
this.dictionary = dictionaryWrapper;
} else {
+this.dictionary = null;
Review comment:
In 2.3 proper interface are exposed from spark and same has been
implemented, so its not required
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (CARBONDATA-3257) Data Load is in No sort flowwhen version is upgraded even if sort columns are given. Also describe formatted displays wrong sort scope after refresh.
MANISH NALLA created CARBONDATA-3257: Summary: Data Load is in No sort flowwhen version is upgraded even if sort columns are given. Also describe formatted displays wrong sort scope after refresh. Key: CARBONDATA-3257 URL: https://issues.apache.org/jira/browse/CARBONDATA-3257 Project: CarbonData Issue Type: Bug Reporter: MANISH NALLA Assignee: MANISH NALLA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command to list the cache sizes for all tables
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command to list the cache sizes for all tables
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248961960
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sql.commands
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCarbonDataShowLRUCommand extends QueryTest with BeforeAndAfterAll {
+ override protected def beforeAll(): Unit = {
+// use new database
+sql("drop database if exists lru_db cascade").collect()
+sql("drop database if exists lru_empty_db cascade").collect()
+sql("create database lru_db").collect()
+sql("create database lru_empty_db").collect()
+dropTable
+sql("use lru_db").collect()
+sql(
+ """
+| CREATE TABLE lru_db.lru_1
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('DICTIONARY_INCLUDE'='deptname')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_1 ")
+
+sql(
+ """
+| CREATE TABLE lru_2
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE
lru_db.lru_2 ")
+sql("insert into table lru_2 select * from lru_1").collect()
+
+sql(
+ """
+| CREATE TABLE lru_3
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_3 ")
+
+// use default database
+sql("use default").collect()
+sql(
+ """
+| CREATE TABLE lru_4
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql("insert into table lru_4 select * from lru_db.lru_2").collect()
+
+// standard partition table
+sql(
+ """
+| CREATE TABLE lru_5
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptname String, projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| PARTITIONED BY (deptno int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(
+ "insert into table lru_5 select
empno,empname,designation,doj,workgroupcategory," +
+
"workgroupcategoryname,deptname,projectcode,projectjoindate,projectenddate,attendance,"
+
+ "utilization,salary,deptno from lru_4").collect()
+
+// count star to cache index
+sql("select max(deptname) from lru_db.lru_1").collect()
+sql("select count(*) from lru_db.lru_2").collect()
+sql("select count(*) from
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command to list the cache sizes for all tables
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command to list the cache sizes for all tables
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248961960
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sql.commands
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCarbonDataShowLRUCommand extends QueryTest with BeforeAndAfterAll {
+ override protected def beforeAll(): Unit = {
+// use new database
+sql("drop database if exists lru_db cascade").collect()
+sql("drop database if exists lru_empty_db cascade").collect()
+sql("create database lru_db").collect()
+sql("create database lru_empty_db").collect()
+dropTable
+sql("use lru_db").collect()
+sql(
+ """
+| CREATE TABLE lru_db.lru_1
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('DICTIONARY_INCLUDE'='deptname')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_1 ")
+
+sql(
+ """
+| CREATE TABLE lru_2
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE
lru_db.lru_2 ")
+sql("insert into table lru_2 select * from lru_1").collect()
+
+sql(
+ """
+| CREATE TABLE lru_3
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_3 ")
+
+// use default database
+sql("use default").collect()
+sql(
+ """
+| CREATE TABLE lru_4
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql("insert into table lru_4 select * from lru_db.lru_2").collect()
+
+// standard partition table
+sql(
+ """
+| CREATE TABLE lru_5
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptname String, projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| PARTITIONED BY (deptno int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(
+ "insert into table lru_5 select
empno,empname,designation,doj,workgroupcategory," +
+
"workgroupcategoryname,deptname,projectcode,projectjoindate,projectenddate,attendance,"
+
+ "utilization,salary,deptno from lru_4").collect()
+
+// count star to cache index
+sql("select max(deptname) from lru_db.lru_1").collect()
+sql("select count(*) from lru_db.lru_2").collect()
+sql("select count(*) from
[GitHub] ravipesala commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT
ravipesala commented on issue #3069: [CARBONDATA-3232] Add test framework and test case for alluxio UT URL: https://github.com/apache/carbondata/pull/3069#issuecomment-455465986 @xubo245 It is not recommended to add web applications in carbon. Better find some other way. Because we don't have UT for S3 and HDFS also, all these are we are handling through sdv. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap
qiuchenjian commented on a change in pull request #3058: [CARBONDATA-3238]
Solve StackOverflowError using MV datamap
URL: https://github.com/apache/carbondata/pull/3058#discussion_r248959581
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/rewrite/DefaultMatchMaker.scala
##
@@ -677,8 +677,18 @@ object SelectSelectGroupbyChildDelta extends
DefaultMatchPattern with PredicateH
val aliasMap_exp = AttributeMap(
gb_2c.outputList.collect {
case a: Alias => (a.toAttribute, a) })
+
+ // avoid to transform a expression more than twrice
+ // accept for select and having
+ val transformedExpFlags = scala.collection.mutable.Map[ExprId,
Int]()
+ aliasMap_exp.keySet.map(alias =>
+transformedExpFlags += (alias.exprId -> 0))
val sel_3q_exp = sel_3q.transformExpressions({
-case attr: Attribute if aliasMap_exp.contains(attr) =>
aliasMap_exp(attr)
+case attr: Attribute if aliasMap_exp.contains(attr) &&
+ transformedExpFlags(attr.exprId) < 2 => {
+ transformedExpFlags(attr.exprId) += 1
+ aliasMap_exp(attr)
+}
Review comment:
@ravipesala done! thanks for your help
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] CarbonDataQA commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap
CarbonDataQA commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap URL: https://github.com/apache/carbondata/pull/3058#issuecomment-455465837 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2569/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru command
qiuchenjian commented on a change in pull request #3078: [WIP] Support show lru
command
URL: https://github.com/apache/carbondata/pull/3078#discussion_r248958992
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/sql/commands/TestCarbonDataShowLRUCommand.scala
##
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sql.commands
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCarbonDataShowLRUCommand extends QueryTest with BeforeAndAfterAll {
+ override protected def beforeAll(): Unit = {
+// use new database
+sql("drop database if exists lru_db cascade").collect()
+sql("drop database if exists lru_empty_db cascade").collect()
+sql("create database lru_db").collect()
+sql("create database lru_empty_db").collect()
+dropTable
+sql("use lru_db").collect()
+sql(
+ """
+| CREATE TABLE lru_db.lru_1
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+| TBLPROPERTIES('DICTIONARY_INCLUDE'='deptname')
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_1 ")
+
+sql(
+ """
+| CREATE TABLE lru_2
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE
lru_db.lru_2 ")
+sql("insert into table lru_2 select * from lru_1").collect()
+
+sql(
+ """
+| CREATE TABLE lru_3
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE lru_3 ")
+
+// use default database
+sql("use default").collect()
+sql(
+ """
+| CREATE TABLE lru_4
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptno int, deptname String,
projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql("insert into table lru_4 select * from lru_db.lru_2").collect()
+
+// standard partition table
+sql(
+ """
+| CREATE TABLE lru_5
+| (empno int, empname String, designation String, doj Timestamp,
workgroupcategory int,
+| workgroupcategoryname String, deptname String, projectcode int,
+| projectjoindate Timestamp, projectenddate Timestamp,attendance
int,utilization int,
+| salary int)
+| PARTITIONED BY (deptno int)
+| STORED BY 'org.apache.carbondata.format'
+ """.stripMargin)
+sql(
+ "insert into table lru_5 select
empno,empname,designation,doj,workgroupcategory," +
+
"workgroupcategoryname,deptname,projectcode,projectjoindate,projectenddate,attendance,"
+
+ "utilization,salary,deptno from lru_4").collect()
+
+// count star to cache index
+sql("select max(deptname) from lru_db.lru_1").collect()
+sql("select count(*) from lru_db.lru_2").collect()
+sql("select count(*) from lru_4").collect()
+sql("select count(*)
[GitHub] gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071#issuecomment-455462605 Previous versions of this problem have been fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang closed pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang closed pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap
CarbonDataQA commented on issue #3058: [CARBONDATA-3238] Solve StackOverflowError using MV datamap URL: https://github.com/apache/carbondata/pull/3058#issuecomment-455462257 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10605/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang opened a new pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang opened a new pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071 remove the version setting:4.5.2 because Linux compiler cannot pass, Error message org.apache.carbondata -hive:jar:1.5.2 - the SNAPSHOT:Failure to find org).apache httpcomponents: httpcore:jar:4.5.2 in http://maven.aliyun.com/nexus/content/groups/public/ Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071#issuecomment-455461872 Previous versions of this problem have been fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071#issuecomment-455461861 Previous versions of this problem have been fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang commented on issue #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071#issuecomment-455461839 Previous versions of this problem have been fixed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] gouyangyang closed pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8
gouyangyang closed pull request #3071: [HOTFIX] Fix the compile error when profile is hadoop-2.8 URL: https://github.com/apache/carbondata/pull/3071 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3075: [CARBONDATA-3251] Fix spark-2.1 UT errors
asfgit closed pull request #3075: [CARBONDATA-3251] Fix spark-2.1 UT errors URL: https://github.com/apache/carbondata/pull/3075 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 edited a comment on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration
xubo245 edited a comment on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration URL: https://github.com/apache/carbondata/pull/3054#issuecomment-455460680 @chenliang613 @jacky @ravipesala @QiangCai @sraghunandan @xuchuanyin Please review it again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration
xubo245 commented on issue #3054: [CARBONDATA-3232] Add example and doc for alluxio integration URL: https://github.com/apache/carbondata/pull/3054#issuecomment-455460680 @chenliang613 @jack @ravipesala @qiang Please review it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
