[GitHub] qiuchenjian opened a new pull request #3120: [CARBONDATA-3286] MV datamap doesn't take effect when query SQL has oalesce with all projections and no filter condition
qiuchenjian opened a new pull request #3120: [CARBONDATA-3286] MV datamap doesn't take effect when query SQL has oalesce with all projections and no filter condition URL: https://github.com/apache/carbondata/pull/3120 [Problem] MV datamap doesn't take effect when query SQL has oalesce with all projections and no filter condition test case: `create table test_main(id int,name string,height int,weight int) using carbondata` `create datamap test_main_mv using 'mv' as select sum(id) as sum_id, sum(height) as sum_height,name as myname from test_main group by name` `select coalesce(sum(id),12) as sumid, sum(height) as sum_height from test_main group by name` [Solution] Fix this bug to support the scene. `[== Physical Plan == *HashAggregate(keys=[myname#269], functions=[sum(sum_id#267L), sum(sum_height#268L)]) +- Exchange hashpartitioning(myname#269, 200) +- *HashAggregate(keys=[myname#269], functions=[partial_sum(sum_id#267L), partial_sum(sum_height#268L)]) +- *FileScan carbondata default.test_main_mv_table[sum_id#267L,sum_height#268L,myname#269]]` Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (CARBONDATA-3286) MV datamap doesn't take effect when query SQL has coalesce with all projections and no filter condition
Chenjian Qiu created CARBONDATA-3286: Summary: MV datamap doesn't take effect when query SQL has coalesce with all projections and no filter condition Key: CARBONDATA-3286 URL: https://issues.apache.org/jira/browse/CARBONDATA-3286 Project: CarbonData Issue Type: Bug Components: core Affects Versions: 1.5.1 Reporter: Chenjian Qiu MV datamap doesn't take effect when query SQL has coalesce with all projections and no filter condition test case : create table test_main(id int,name string,height int,weight int) using carbondata create datamap test_main_mv using 'mv' as select sum(id) as sum_id, sum(height) as sum_height,name as myname from test_main group by name select coalesce(sum(id),12) as sumid, sum(height) as sum_height from test_main group by name -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] xubo245 commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3
xubo245 commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3 URL: https://github.com/apache/carbondata/pull/3107#issuecomment-459941113 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 closed pull request #3079: [WIP][CARBONDATA-3255] Support binary data type
xubo245 closed pull request #3079: [WIP][CARBONDATA-3255] Support binary data type URL: https://github.com/apache/carbondata/pull/3079 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 closed pull request #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3
xubo245 closed pull request #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3 URL: https://github.com/apache/carbondata/pull/3107 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3271) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3271: Summary: WIP (was: CarboData provide python SDK) > WIP > --- > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: Sub-task >Affects Versions: 1.5.1 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3283) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-3283. --- Resolution: Incomplete > WIP > --- > > Key: CARBONDATA-3283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3283 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > > WIP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3254) [WIP]
[ https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-3254. --- Resolution: Incomplete > [WIP] > -- > > Key: CARBONDATA-3254 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3254 > Project: CarbonData > Issue Type: New Feature >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3255) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-3255. --- Resolution: Incomplete > WIP > --- > > Key: CARBONDATA-3255 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3255 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3283) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3283: Summary: WIP (was: Support write data with different data type) > WIP > --- > > Key: CARBONDATA-3283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3283 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > > CarbonData support AI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3283) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3283: Description: WIP (was: CarbonData support AI ) > WIP > --- > > Key: CARBONDATA-3283 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3283 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > > WIP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3255) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3255: Description: (was: Support binary data type) Summary: WIP (was: Support binary data type) > WIP > --- > > Key: CARBONDATA-3255 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3255 > Project: CarbonData > Issue Type: Sub-task >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3254) [WIP]
[ https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3254: Summary: [WIP] (was: [WIP] CarbonData supports deep learning framework to write and read image/voice data) > [WIP] > -- > > Key: CARBONDATA-3254 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3254 > Project: CarbonData > Issue Type: New Feature >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-3271) WIP
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-3271. --- Resolution: Incomplete > WIP > --- > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: Sub-task >Affects Versions: 1.5.1 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3254) [WIP] CarbonData supports deep learning framework to write and read image/voice data
[ https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3254: Description: (was: CarbonData supports deep learning framework to write and read image/voice data * Supports write and read image in CarbonData * Provide Carbon python SDK for read, which can be used for deep learning framework tensorflow/MXNet or others * Support write data by different data type * Support read data by file or file lists) > [WIP] CarbonData supports deep learning framework to write and read > image/voice data > > > Key: CARBONDATA-3254 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3254 > Project: CarbonData > Issue Type: New Feature >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3271) CarboData provide python SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3271: Description: (was: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # https://issues.apache.org/jira/browse/SPARK-3789 ) > CarboData provide python SDK > > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: Sub-task >Affects Versions: 1.5.1 >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3
CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3 URL: https://github.com/apache/carbondata/pull/3107#issuecomment-459936581 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10801/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3
CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3 URL: https://github.com/apache/carbondata/pull/3107#issuecomment-459936322 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2771/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3
CarbonDataQA commented on issue #3107: [WIP][Test][CARBONDATA-3271] Provide python SDK for reading CarbonData from S3 URL: https://github.com/apache/carbondata/pull/3107#issuecomment-459933772 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2542/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap
CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap URL: https://github.com/apache/carbondata/pull/3108#issuecomment-459933610 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2770/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian edited a comment on issue #3084: [CARBONDATA-3258] Add more test case for mv datamap
qiuchenjian edited a comment on issue #3084: [CARBONDATA-3258] Add more test case for mv datamap URL: https://github.com/apache/carbondata/pull/3084#issuecomment-459932384 @QiangCai @ravipesala @jackylk @xuchuanyin @xubo245 @chenliang613 @kumarvishal09 please help me to merge this PR, so that i can submit other code using this PR's class, thank you very much This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on issue #3084: [CARBONDATA-3258] Add more test case for mv datamap
qiuchenjian commented on issue #3084: [CARBONDATA-3258] Add more test case for mv datamap URL: https://github.com/apache/carbondata/pull/3084#issuecomment-459932384 @QiangCai @ravipesala @jackylk @xuchuanyin @xubo245 @chenliang613 @kumarvishal09 please help me to merge this PR, so that i can submit other code using this PR's class This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap
CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap URL: https://github.com/apache/carbondata/pull/3108#issuecomment-459931966 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10800/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap
CarbonDataQA commented on issue #3108: [CARBONDATA-3277] Support to show subquery for datamap URL: https://github.com/apache/carbondata/pull/3108#issuecomment-459928322 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2541/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kevinjmh commented on a change in pull request #3108: [CARBONDATA-3277] Support to show subquery for datamap
kevinjmh commented on a change in pull request #3108: [CARBONDATA-3277] Support
to show subquery for datamap
URL: https://github.com/apache/carbondata/pull/3108#discussion_r253249667
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala
##
@@ -43,7 +44,8 @@ case class CarbonDataMapShowCommand(tableIdentifier:
Option[TableIdentifier])
Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
AttributeReference("ClassName", StringType, nullable = false)(),
AttributeReference("Associated Table", StringType, nullable = false)(),
- AttributeReference("DataMap Properties", StringType, nullable = false)())
+ AttributeReference("DataMap Properties", StringType, nullable = false)(),
+ AttributeReference("Subquery", StringType, nullable = false)())
Review comment:
Good suggestion. I'll change that
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] xuchuanyin commented on issue #3117: [DOC] Update the doc of "Show DataMap"
xuchuanyin commented on issue #3117: [DOC] Update the doc of "Show DataMap" URL: https://github.com/apache/carbondata/pull/3117#issuecomment-459925861 @qiuchenjian take take care of PR #3108 while implementing this one This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xuchuanyin commented on a change in pull request #3108: [CARBONDATA-3277] Support to show subquery for datamap
xuchuanyin commented on a change in pull request #3108: [CARBONDATA-3277]
Support to show subquery for datamap
URL: https://github.com/apache/carbondata/pull/3108#discussion_r253248782
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala
##
@@ -43,7 +44,8 @@ case class CarbonDataMapShowCommand(tableIdentifier:
Option[TableIdentifier])
Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
AttributeReference("ClassName", StringType, nullable = false)(),
AttributeReference("Associated Table", StringType, nullable = false)(),
- AttributeReference("DataMap Properties", StringType, nullable = false)())
+ AttributeReference("DataMap Properties", StringType, nullable = false)(),
+ AttributeReference("Subquery", StringType, nullable = false)())
Review comment:
How about setting this field nullable so that in case of other type of
datamap we can just return null instead of empty.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] xuchuanyin commented on a change in pull request #3113: [CARBONDATA-3284] [CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix
xuchuanyin commented on a change in pull request #3113: [CARBONDATA-3284]
[CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix
URL: https://github.com/apache/carbondata/pull/3113#discussion_r253247971
##
File path: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
##
@@ -3338,4 +3338,20 @@ public static boolean
hasEncoding(List en
public static String generateUUID() {
return UUID.randomUUID().toString();
}
+
+ /**
+ * Below method will be used to get the datamap schema name from datamap
table name
+ * it will split name based on _ and get the last name
+ * This is only for pre aggregate and timeseries tables
+ *
+ * @param tableName
Review comment:
please make sure that the comments all are useful and remove the useless ones
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] xuchuanyin commented on issue #3116: [CARBONDATA-3229]Validate the true/false for all boolean parameters
xuchuanyin commented on issue #3116: [CARBONDATA-3229]Validate the true/false for all boolean parameters URL: https://github.com/apache/carbondata/pull/3116#issuecomment-459923278 @SteNicholas I don't think this modification is needed to group all irrelevant parameters together just because they are all boolean. As long as they are not fatal in case of illegal value, we can always fallback using the default value. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3113: [CARBONDATA-3284] [CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix
qiuchenjian commented on a change in pull request #3113: [CARBONDATA-3284]
[CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix
URL: https://github.com/apache/carbondata/pull/3113#discussion_r253243631
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
##
@@ -77,7 +77,37 @@ case class CarbonDropTableCommand(
}
val relationIdentifiers =
carbonTable.getTableInfo.getParentRelationIdentifiers
if (relationIdentifiers != null && !relationIdentifiers.isEmpty) {
-if (!dropChildTable) {
+var ignoreParentTableCheck = false
+if (carbonTable.getTableInfo.getParentRelationIdentifiers.size() == 1)
{
+ // below handling in case when pre aggregation creation failed in
below scenario
+ // while creating a pre aggregate data map it created pre aggregate
table and registered
+ // in hive, but failed to register in main table because of some
exception.
+ // in this case if it will not allow user to drop datamap and data
map table
+ // for this if user run drop table command for pre aggregate it
should allow user to drop
+ // the same
+ val parentDbName =
carbonTable.getTableInfo.getParentRelationIdentifiers.get(0)
Review comment:
better to add some logs
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3113: [CARBONDATA-3284] [CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix
qiuchenjian commented on a change in pull request #3113: [CARBONDATA-3284] [CARBONDATA-3285] Workaround for Create-PreAgg Datamap Fail & Sort-Columns Fix URL: https://github.com/apache/carbondata/pull/3113#discussion_r253243580 ## File path: docs/faq.md ## @@ -43,6 +43,7 @@ - [Failed to insert data on the cluster](#failed-to-insert-data-on-the-cluster) - [Failed to execute Concurrent Operations(Load,Insert,Update) on table by multiple workers](#failed-to-execute-concurrent-operations-on-table-by-multiple-workers) - [Failed to create a table with a single numeric column](#failed-to-create-a-table-with-a-single-numeric-column) +- [Failed to create datamap and drop-datamap is also not working](#failed-to-create-datamap-and-drop-datamap-is-also-not-working) Review comment: ```suggestion - [Failed to create datamap and drop datamap is also not working](#failed-to-create-datamap-and-drop-datamap-is-also-not-working) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281] limit the LRU cache size
qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281]
limit the LRU cache size
URL: https://github.com/apache/carbondata/pull/3118#discussion_r253034132
##
File path:
core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##
@@ -305,4 +312,15 @@ public void clear() {
lruCacheMap.clear();
}
}
+
+ /**
+ * Check if LRU cache setting is bigger than max memory of jvm.
+ * if LRU cache is bigger than max memory of jvm when query for a big
segments table,may cause JDBC server crash.
+ * @return true LRU cache is bigger than max memory of jvm, false otherwise
+ */
+ private boolean isBeyondMaxMemory(){
+long mSize = Runtime.getRuntime().maxMemory();
+long lruSize = lruCacheMemorySize * 1024 * 1024;
Review comment:
better to use BYTE_CONVERSION_CONSTANT instead of 1024 * 1024
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281] limit the LRU cache size
qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281]
limit the LRU cache size
URL: https://github.com/apache/carbondata/pull/3118#discussion_r253034452
##
File path:
core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##
@@ -305,4 +312,15 @@ public void clear() {
lruCacheMap.clear();
}
}
+
+ /**
+ * Check if LRU cache setting is bigger than max memory of jvm.
+ * if LRU cache is bigger than max memory of jvm when query for a big
segments table,may cause JDBC server crash.
+ * @return true LRU cache is bigger than max memory of jvm, false otherwise
+ */
+ private boolean isBeyondMaxMemory(){
+long mSize = Runtime.getRuntime().maxMemory();
+long lruSize = lruCacheMemorySize * 1024 * 1024;
Review comment:
better to use BYTE_CONVERSION_CONSTANT instead of 1024 * 1024
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281] limit the LRU cache size
qiuchenjian commented on a change in pull request #3118: [CARBONDATA-3281]
limit the LRU cache size
URL: https://github.com/apache/carbondata/pull/3118#discussion_r253034132
##
File path:
core/src/main/java/org/apache/carbondata/core/cache/CarbonLRUCache.java
##
@@ -305,4 +312,15 @@ public void clear() {
lruCacheMap.clear();
}
}
+
+ /**
+ * Check if LRU cache setting is bigger than max memory of jvm.
+ * if LRU cache is bigger than max memory of jvm when query for a big
segments table,may cause JDBC server crash.
+ * @return true LRU cache is bigger than max memory of jvm, false otherwise
+ */
+ private boolean isBeyondMaxMemory(){
+long mSize = Runtime.getRuntime().maxMemory();
+long lruSize = lruCacheMemorySize * 1024 * 1024;
Review comment:
better to use BYTE_CONVERSION_CONSTANT instead of 1024 * 1024
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
