[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708914537 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4452/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
ajantha-bhat commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-708913452 @marchpure and @QiangCai : The solution was to use segmentid_UUID right ? I didn't see UUID in the PR changes. please explain the solution or update the PR description This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat removed a comment on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat removed a comment on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912408 retest this plese This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912408 retest this plese This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912511 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
Karan980 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708897384 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708880660 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2697/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708879401 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4451/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Labels: easyfix features newbie (was: newbie patch)
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Nguyen Dinh Huynh
>Priority: Major
> Labels: easyfix, features, newbie
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3. Is this a bug ?
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
> My code is:
> {code:java}
> val map = Map(
> col("_external_op") -> col("A._external_op"),
> col("_external_ts_sec") -> col("A._external_ts_sec"),
> col("_external_row") -> col("A._external_row"),
> col("_external_pos") -> col("A._external_pos"),
> col("id") -> col("A.id"),
> col("order") -> col("A.order"),
> col("shop_code") -> col("A.shop_code"),
> col("customer_tel") -> col("A.customer_tel"),
> col("channel") -> col("A.channel"),
> col("batch_session_id") -> col("A.batch_session_id"),
> col("deleted_at") -> col("A.deleted_at"),
> col("created") -> col("A.created"))
> .asInstanceOf[Map[Any, Any]]
> val testDf =
> spark.sqlContext.read.format("carbondata")
> .option("tableName", "package_drafts")
> .option("schemaName", "db")
> .option("dbName", "db")
> .option("databaseName", "d")b
> .load()
> .as("B")
> testDf.printSchema()
> testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id")))
> .whenMatched(col("A._external_op") === "u")
> .updateExpr(map)
> .whenMatched(col("A._external_op") === "c")
> .insertExpr(map)
> .whenMatched(col("A._external_op") === "d")
> .delete()
> .execute()
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Flags: Important
Labels: newbie patch (was: )
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Nguyen Dinh Huynh
>Priority: Major
> Labels: newbie, patch
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3. Is this a bug ?
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
> My code is:
> {code:java}
> val map = Map(
> col("_external_op") -> col("A._external_op"),
> col("_external_ts_sec") -> col("A._external_ts_sec"),
> col("_external_row") -> col("A._external_row"),
> col("_external_pos") -> col("A._external_pos"),
> col("id") -> col("A.id"),
> col("order") -> col("A.order"),
> col("shop_code") -> col("A.shop_code"),
> col("customer_tel") -> col("A.customer_tel"),
> col("channel") -> col("A.channel"),
> col("batch_session_id") -> col("A.batch_session_id"),
> col("deleted_at") -> col("A.deleted_at"),
> col("created") -> col("A.created"))
> .asInstanceOf[Map[Any, Any]]
> val testDf =
> spark.sqlContext.read.format("carbondata")
> .option("tableName", "package_drafts")
> .option("schemaName", "db")
> .option("dbName", "db")
> .option("databaseName", "d")b
> .load()
> .as("B")
> testDf.printSchema()
> testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id")))
> .whenMatched(col("A._external_op") === "u")
> .updateExpr(map)
> .whenMatched(col("A._external_op") === "c")
> .insertExpr(map)
> .whenMatched(col("A._external_op") === "d")
> .delete()
> .execute()
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Affects Version/s: 2.0.1
2.0.0
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Nguyen Dinh Huynh
>Priority: Major
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3. Is this a bug ?
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
> My code is:
> {code:java}
> val map = Map(
> col("_external_op") -> col("A._external_op"),
> col("_external_ts_sec") -> col("A._external_ts_sec"),
> col("_external_row") -> col("A._external_row"),
> col("_external_pos") -> col("A._external_pos"),
> col("id") -> col("A.id"),
> col("order") -> col("A.order"),
> col("shop_code") -> col("A.shop_code"),
> col("customer_tel") -> col("A.customer_tel"),
> col("channel") -> col("A.channel"),
> col("batch_session_id") -> col("A.batch_session_id"),
> col("deleted_at") -> col("A.deleted_at"),
> col("created") -> col("A.created"))
> .asInstanceOf[Map[Any, Any]]
> val testDf =
> spark.sqlContext.read.format("carbondata")
> .option("tableName", "package_drafts")
> .option("schemaName", "db")
> .option("dbName", "db")
> .option("databaseName", "d")b
> .load()
> .as("B")
> testDf.printSchema()
> testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id")))
> .whenMatched(col("A._external_op") === "u")
> .updateExpr(map)
> .whenMatched(col("A._external_op") === "c")
> .insertExpr(map)
> .whenMatched(col("A._external_op") === "d")
> .delete()
> .execute()
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [carbondata] Kejian-Li commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Kejian-Li commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708816282 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708720271 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2696/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708718712 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4450/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708693056 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4445/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708690166 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2692/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #3695: [WIP] partition optimization
maheshrajus commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708685238 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708680642 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4449/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708676144 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2693/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708675919 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4446/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708627987 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2691/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
Karan980 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708624107 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
Karan980 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708623745 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708610112 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2690/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708605209 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4442/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708605441 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3// This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708603224 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4443/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708601810 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2689/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708534684 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r504829616
##
File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonIUD.java
##
@@ -173,10 +184,16 @@ public void update(String path, Expression
filterExpression,
RecordWriter deleteDeltaWriter =
CarbonTableOutputFormat.getDeleteDeltaRecordWriter(path);
ObjectArrayWritable writable = new ObjectArrayWritable();
-
+long day = 24L * 3600 * 1000;
while (reader.hasNext()) {
Object[] row = (Object[]) reader.readNextRow();
writable.set(Arrays.copyOfRange(row, row.length - 1, row.length));
+ for (Integer dateIndex : dateIndexes) {
+row[dateIndex] = new Date((day * ((int) row[dateIndex])));
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r504829543
##
File path:
sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonIUDTest.java
##
@@ -72,6 +72,43 @@ public void testDelete() throws Exception {
FileUtils.deleteDirectory(new File(path));
}
+ @Test
+ public void testUpdateOnDateType() throws Exception {
Review comment:
Testcase was added to check conversion of dateType columns to integer
while reading. As date column is present in schema. so no need to perform
update on date column.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r504827820
##
File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonIUD.java
##
@@ -155,7 +158,15 @@ public void update(String path, Expression
filterExpression,
Schema schema =
CarbonSchemaReader.readSchema(indexFiles.get(0)).asOriginOrder();
Field[] fields = schema.getFields();
String[] projectionColumns = new String[fields.length + 1];
+List dateIndexes = new ArrayList<>();
+List timeStampIndexes = new ArrayList<>();
for (int i = 0; i < fields.length; i++) {
+ if (fields[i].getDataType() == DataTypes.DATE) {
+dateIndexes.add(i);
+ }
+ if (fields[i].getDataType() == DataTypes.TIMESTAMP) {
Review comment:
Code removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r504827655
##
File path:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonFileInputFormat.java
##
@@ -279,7 +279,8 @@ public void
setAllColumnProjectionIfNotConfigured(JobContext job, CarbonTable ca
private String[] getDeleteDeltaFiles(String segmentFilePath, List
allDeleteDeltaFiles) {
List deleteDeltaFiles = new ArrayList<>();
String segmentFileName = null;
-String[] pathElements =
segmentFilePath.split(Pattern.quote(File.separator));
+segmentFilePath = segmentFilePath.replace("\\", "/");
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708481924 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2687/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708473034 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4440/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708473030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708457681 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4438/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708448621 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4437/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [WIP]Fix special char issue in create table
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-708442276 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2682/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708436627 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4441/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708434559 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2688/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708434556 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2684/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708420587 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2685/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort
CarbonDataQA1 commented on pull request #3979: URL: https://github.com/apache/carbondata/pull/3979#issuecomment-708407338 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2683/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort
CarbonDataQA1 commented on pull request #3979: URL: https://github.com/apache/carbondata/pull/3979#issuecomment-708398610 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4436/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [WIP]Fix special char issue in create table
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-708395905 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4435/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
CarbonDataQA1 commented on pull request #3984: URL: https://github.com/apache/carbondata/pull/3984#issuecomment-708395934 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2681/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
QiangCai commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708382061 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
marchpure commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708375624 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
marchpure commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708375305 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
CarbonDataQA1 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708366544 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4432/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
CarbonDataQA1 commented on pull request #3984: URL: https://github.com/apache/carbondata/pull/3984#issuecomment-708363702 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4434/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
CarbonDataQA1 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708359537 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2680/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708349031 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2679/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708345630 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2676/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708345192 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4433/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-708341574 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2675/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708340588 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4431/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708337733 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-3880) How to start JDBC service in distributed index
[ https://issues.apache.org/jira/browse/CARBONDATA-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213825#comment-17213825 ] Nihal kumar ojha commented on CARBONDATA-3880: -- Hi, please follow the below steps to configure the distributed index server with JDBC. 1. Add these properties in spark-defaults.conf spark.yarn.keytab= spark.carbon.indexserver.keytab= spark.carbon.indexserver.principal=spark2x/hadoop.hadoop@hadoop.com spark.yarn.principal=spark2x/hadoop.hadoop@hadoop.com 2. Add following configuration in carbon.properties(Ensure the carbon.properties is configured in spark-defaults.conf in driver extra java option) carbon.enable.index.server=true carbon.indexserver.enable.prepriming=true carbon.indexserver.HA.enabled=true carbon.max.executor.lru.cache.size=-1 carbon.disable.index.server.fallback=false carbon.indexserver.zookeeper.dir=/indexserver2x carbon.index.server.port= Then run below spark-submit command at $spark_home location bin/spark-submit --num-executors 2 --master yarn --class org.apache.carbondata.indexserver.indexserver then start spark JDBCserver as usual. Queries should reflect in yarn UI, in the index server and spark JDBC application. > How to start JDBC service in distributed index > --- > > Key: CARBONDATA-3880 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3880 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 2.0.0 >Reporter: li >Priority: Major > Fix For: 2.1.0 > > > How to start JDBC service in distributed index -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708322826 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2678/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708314304 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4430/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-4025) storage space for MV is double to that of a table on which MV has been created.
[ https://issues.apache.org/jira/browse/CARBONDATA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213792#comment-17213792 ] suyash yadav commented on CARBONDATA-4025: -- Hi any update on this? > storage space for MV is double to that of a table on which MV has been > created. > --- > > Key: CARBONDATA-4025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 2.0.1 > Environment: Apcahe carbondata 2.0.1 > Apache spark 2.4.5 > Hadoop 2.7.2 >Reporter: suyash yadav >Priority: Major > > We are doing a POC based on carbondata but we have observed that when we > create n MV on a table with timeseries function of same granualarity the MV > takes double the space of the table. > > In my scenario, My table has 1.3 million records and MV also has same number > of records but the size of the table is 3.6 MB but the size of the MV is > around 6.5 MB. > This is really important for us as critical business decision are getting > affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-708300620 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4429/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-3892) An exception occurred when modifying the table name using SparkSession
[ https://issues.apache.org/jira/browse/CARBONDATA-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213785#comment-17213785 ] Nihal kumar ojha commented on CARBONDATA-3892: -- Hi, I was trying to reproduce this issue but not getting reproduced. I am using the query "ALTER TABLE oldTable RENAME to newTable". please correct me if I am wrong. Or if there is some other configuration then please add here. > An exception occurred when modifying the table name using SparkSession > -- > > Key: CARBONDATA-3892 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3892 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 2.0.0 >Reporter: li >Priority: Blocker > > Exception in thread "main" java.lang.LinkageError: ClassCastException: > attempting to > castjar:file:/usr/hdp/2.6.5.0-292/spark2/carbonlib/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/usr/hdp/2.6.5.0-292/spark2/carbonlib/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar!/javax/ws/rs/ext/RuntimeDelegate.class -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] akashrn5 opened a new pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
akashrn5 opened a new pull request #3984: URL: https://github.com/apache/carbondata/pull/3984 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 opened a new pull request #3983: [WIP]Fix special char issue in create table
akashrn5 opened a new pull request #3983: URL: https://github.com/apache/carbondata/pull/3983 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708287330 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4428/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708285816 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2674/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708286183 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4427/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Pickupolddriver commented on a change in pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
Pickupolddriver commented on a change in pull request #3935:
URL: https://github.com/apache/carbondata/pull/3935#discussion_r504537575
##
File path:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##
@@ -267,9 +266,8 @@ object CarbonDataRDDFactory {
throw new Exception("Exception in compaction " +
exception.getMessage)
}
} finally {
- executor.shutdownNow()
try {
-compactor.deletePartialLoadsInCompaction()
Review comment:
@ajantha-bhat after the merge of #3978 , cleanStaleDeltaFiles will only
be called when there are exceptions during the update process. And it will only
delete the delta and index file created by the exception update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat edited a comment on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708279514 @akashrn5 , @QiangCai : PR is ready (11 random failures are fixed). There are some more random failures are there in CI (flink related). someone can analyze and handle in other PRs I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708279514 @akashrn5 , @QiangCai : PR is ready. There are some more random failures are there in CI. someone can analyze and handle in other PRs I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708275465 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708271365 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2673/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
VenuReddy2103 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708269487 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967:
URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java
##
@@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[]
value) {
}
}
+ @Override
+ public void putAllByteArray(byte[] data, int offset, int length) {
+int[] lengths = getLengths();
+int[] offsets = getOffsets();
+for (int i = 0; i < lengths.length; i++) {
+ if (offsets[i] != 0) {
Review comment:
offset values are 0 by default and put was called, due to which after
select- empty rows were showed. Also select count(*) showed wrong results. Like
you said offset can never be 0, although length may be 0. If offset is 0 then
put should not be called. Hence this check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967:
URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java
##
@@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[]
value) {
}
}
+ @Override
+ public void putAllByteArray(byte[] data, int offset, int length) {
+int[] lengths = getLengths();
+int[] offsets = getOffsets();
+for (int i = 0; i < lengths.length; i++) {
+ if (offsets[i] != 0) {
Review comment:
offset values are 0 by default, due to which after select- empty rows
were showed. Also select count(*) showed wrong results. Like you said offset
can never be 0, although length may be 0. If offset is 0 then put should not be
called. Hence this check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967:
URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java
##
@@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[]
value) {
}
}
+ @Override
+ public void putAllByteArray(byte[] data, int offset, int length) {
+int[] lengths = getLengths();
+int[] offsets = getOffsets();
+for (int i = 0; i < lengths.length; i++) {
+ if (offsets[i] != 0) {
Review comment:
offset values were 0 by default, due to which after select- empty rows
were showed. Also select count(*) showed wrong results. Like you said offset
can never be 0. If it is 0 then put should not be called. Hence this check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [WIP][HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708254236 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4425/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967:
URL: https://github.com/apache/carbondata/pull/3967#discussion_r504504630
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithDeleteDelta.java
##
@@ -242,4 +242,8 @@ public void putArray(int rowId, int offset, int length) {
columnVector.putArray(counter++, offset, length);
}
}
+
+ public CarbonColumnVector getColumnVector() {
Review comment:
Right, I have made the changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967:
URL: https://github.com/apache/carbondata/pull/3967#discussion_r504504712
##
File path:
integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java
##
@@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[]
value) {
}
}
+ @Override
+ public void putAllByteArray(byte[] data, int offset, int length) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708247374 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4424/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708246843 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2670/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Description:
I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
with spark 2.4.3. Is this a bug ?
{code:java}
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91
- Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store
location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
{code}
My code is:
{code:java}
val map = Map(
col("_external_op") -> col("A._external_op"),
col("_external_ts_sec") -> col("A._external_ts_sec"),
col("_external_row") -> col("A._external_row"),
col("_external_pos") -> col("A._external_pos"),
col("id") -> col("A.id"),
col("order") -> col("A.order"),
col("shop_code") -> col("A.shop_code"),
col("customer_tel") -> col("A.customer_tel"),
col("channel") -> col("A.channel"),
col("batch_session_id") -> col("A.batch_session_id"),
col("deleted_at") -> col("A.deleted_at"),
col("created") -> col("A.created"))
.asInstanceOf[Map[Any, Any]]
val testDf =
spark.sqlContext.read.format("carbondata")
.option("tableName", "package_drafts")
.option("schemaName", "db")
.option("dbName", "db")
.option("databaseName", "d")b
.load()
.as("B")
testDf.printSchema()
testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id")))
.whenMatched(col("A._external_op") === "u")
.updateExpr(map)
.whenMatched(col("A._external_op") === "c")
.insertExpr(map)
.whenMatched(col("A._external_op") === "d")
.delete()
.execute()
{code}
was:
I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
with spark 2.4.3. Is this a bug ?
{code:java}
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91
- Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store
location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
{code}
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Reporter: Nguyen Dinh Huynh
>Priority: Major
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3. Is this a bug ?
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
> My code is:
> {code:java}
> val map = Map(
> col("_external_op") -> col("A._external_op"),
> col("_external_ts_sec") -> col("A._external_ts_sec"),
> col("_external_row") -> col("A._external_row"),
> col("_external_pos") -> col("A._external_pos"),
> col("id") -> col("A.id"),
> col("order") -> col("A.order"),
> col("shop_code") -> col("A.shop_code"),
> col("customer_tel") -> col("A.customer_tel"),
> col("channel") -> col("A.channel"),
> col("batch_session_id") -> col("A.batch_session_id"),
> col("deleted_at") -> col("A.deleted_at"),
> col("created") -> col("A.created"))
> .asInstanceOf[Map[Any, Any]]
> val testDf =
> spark.sqlContext.read.format("carbondata")
> .option("tableName", "package_drafts")
> .option("schemaName", "db")
> .option("dbName", "db")
> .option("databaseName", "d")b
> .load()
> .as("B")
> testDf.printSchema()
> testDf.merge(package_draft_view,
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Description:
I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
with spark 2.4.3. Is this a bug ?
{code:java}
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91
- Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store
location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
{code}
was:
I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
with spark 2.4.3
{code:java}
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91
- Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store
location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
{code}
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Reporter: Nguyen Dinh Huynh
>Priority: Major
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3. Is this a bug ?
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[
https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nguyen Dinh Huynh updated CARBONDATA-4033:
--
Description:
I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
with spark 2.4.3
{code:java}
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91
- Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store
location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
{code}
was:
2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to
exception java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed
as it was not found on disk or in memory
2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID
1)
java.lang.RuntimeException: Store location not set for the key
__temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> Error when using merge API with hive table
> --
>
> Key: CARBONDATA-4033
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4033
> Project: CarbonData
> Issue Type: Bug
>Reporter: Nguyen Dinh Huynh
>Priority: Major
>
> I always get this error when trying to upsert hive table. I'm using CDH 6.3.1
> with spark 2.4.3
>
> {code:java}
> 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due
> to exception java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0.
> 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be
> removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR
> Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1)
> java.lang.RuntimeException: Store location not set for the key
> __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-4033) Error when using merge API with hive table
Nguyen Dinh Huynh created CARBONDATA-4033: - Summary: Error when using merge API with hive table Key: CARBONDATA-4033 URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 Project: CarbonData Issue Type: Bug Reporter: Nguyen Dinh Huynh 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467243
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/emptyrow/TestSkipEmptyLines.scala
##
@@ -33,50 +33,65 @@ class TestSkipEmptyLines extends QueryTest with
BeforeAndAfterAll {
test("test load options with true") {
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable " +
-s"OPTIONS('skip_empty_line'='true')")
-checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
Seq(Row("a",25),Row("b",22),Row("c",23)))
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='true')")
+checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
+ Seq(Row("a", 25), Row("b", 22), Row("c", 23)))
}
test("test load options with false") {
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable " +
-s"OPTIONS('skip_empty_line'='false')")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable
OPTIONS('skip_empty_line'='false')")
checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
-
Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null)))
+ Seq(Row("a", 25),
+Row("b", 22),
+Row("c", 23),
+Row(null, null),
+Row(null, null),
+Row(null, null)))
}
test("test carbonproperties with true") {
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE,
"true")
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable")
checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
- Seq(Row("a",25),Row("b",22),Row("c",23)))
+ Seq(Row("a", 25), Row("b", 22), Row("c", 23)))
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE,
CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE_DEFAULT)
}
test("test carbonproperties with false") {
-
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE,
"false")
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE, "false")
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable")
checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
-
Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null)))
+ Seq(Row("a", 25),
Review comment:
changed for all places
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/aggquery/AllDataTypesTestCaseAggregate.scala
##
@@ -46,21 +47,23 @@ class AllDataTypesTestCaseAggregate extends QueryTest with
BeforeAndAfterAll {
"Timestamp, workgroupcategory int, workgroupcategoryname String, deptno
int, deptname " +
"String, projectcode int, projectjoindate Timestamp, projectenddate
Timestamp,attendance " +
"int,utilization int,salary int)row format delimited fields terminated
by ','")
-sql(
- s"LOAD DATA LOCAL INPATH '$resourcesPath/datawithoutheader.csv' INTO
TABLE alldatatypesAGG_hive")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/datawithoutheader.csv' " +
+"INTO TABLE alldatatypesAGG_hive")
}
test(
"select empno,empname,utilization,count(salary),sum(empno) from
alldatatypestableAGG where " +
"empname in ('arvind','ayushi') group by empno,empname,utilization")
{
checkAnswer(
- sql(
-"select empno,empname,utilization,count(salary),sum(empno) from
alldatatypestableAGG where" +
-" empname in ('arvind','ayushi') group by empno,empname,utilization"),
- sql(
-"select empno,empname,utilization,count(salary),sum(empno) from
alldatatypesAGG_hive where" +
-" empname in ('arvind','ayushi') group by
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467565
##
File path:
integration/spark/src/test/scala/org/apache/spark/carbondata/TestStreamingTableQueryFilter.scala
##
@@ -96,25 +98,97 @@ class TestStreamingTableQueryFilter extends QueryTest with
BeforeAndAfterAll {
// filter
checkAnswer(
sql("select * from streaming_table_filter.stream_filter where id = 1"),
- Seq(Row(1, "name_1", "city_1", 1.0, BigDecimal.valueOf(0.01), 80.01,
Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"),
Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue),
Row(wrap(Array("school_1", "school_11")), 1
+ Seq(Row(1,
Review comment:
done, changed for all places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504466643
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexFunctionSuite.scala
##
@@ -803,27 +840,38 @@ class BloomCoarseGrainIndexFunctionSuite extends
QueryTest with BeforeAndAfterA
sql(s"INSERT INTO $bloomSampleTable SELECT 'c1v2', 2, 'c3v2'")
// two segments both has index files
-val carbonTable = CarbonEnv.getCarbonTable(Option("default"),
bloomSampleTable)(SparkTestQueryExecutor.spark)
+val carbonTable = CarbonEnv.getCarbonTable(Option("default"),
bloomSampleTable)(
+ SparkTestQueryExecutor.spark)
import scala.collection.JavaConverters._
(0 to 1).foreach { segId =>
- val indexPath =
CarbonTablePath.getIndexesStorePath(carbonTable.getTablePath, segId.toString,
indexName)
- assert(FileUtils.listFiles(FileUtils.getFile(indexPath),
Array("bloomindexmerge"), true).asScala.nonEmpty)
+ val indexPath =
CarbonTablePath.getIndexesStorePath(carbonTable.getTablePath,
Review comment:
changed for all places
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexSuite.scala
##
@@ -82,16 +82,24 @@ class BloomCoarseGrainIndexSuite extends QueryTest with
BeforeAndAfterAll with B
sql(s"select * from $bloomSampleTable where id = 1 and city='city_1'",
indexName, shouldHit),
sql(s"select * from $normalTable where id = 1 and city='city_1'"))
checkAnswer(
- sql(s"select * from $bloomSampleTable where id = 999 and
city='city_999'", indexName, shouldHit),
+ sql(s"select * from $bloomSampleTable where id = 999 and
city='city_999'",
+indexName,
+shouldHit),
Review comment:
changed for all places
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/index/lucene/LuceneFineGrainIndexSuite.scala
##
@@ -186,12 +191,21 @@ class LuceneFineGrainIndexSuite extends QueryTest with
BeforeAndAfterAll {
sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE index_test1
OPTIONS('header'='false')")
-checkAnswer(sql("SELECT * FROM index_test1 WHERE TEXT_MATCH('name:n10')"),
sql(s"select * from index_test1 where name='n10'"))
-
-var carbonTable = CarbonEnv.getCarbonTable(Some("lucene"),
"index_test1")(sqlContext.sparkSession)
-var indexes =
carbonTable.getIndexMetadata.getIndexesMap.get(IndexType.LUCENE.getIndexProviderName)
- .asScala.filter(p =>
p._2.get(CarbonCommonConstants.INDEX_STATUS).equalsIgnoreCase(IndexStatus.ENABLED.name()))
-assert(indexes.exists(p => p._1.equals("dm12") &&
p._2.get(CarbonCommonConstants.INDEX_STATUS) == IndexStatus.ENABLED.name()))
+checkAnswer(sql("SELECT * FROM index_test1 WHERE TEXT_MATCH('name:n10')"),
+ sql(s"select * from index_test1 where name='n10'"))
+
+var carbonTable = CarbonEnv.getCarbonTable(Some("lucene"),
"index_test1")(sqlContext
+ .sparkSession)
+val indexes = carbonTable.getIndexMetadata
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/index/lucene/LuceneFineGrainIndexSuite.scala
##
@@ -863,8 +891,13 @@ class LuceneFineGrainIndexSuite extends QueryTest with
BeforeAndAfterAll {
| ON index_test_table (name)
| AS 'bloomfilter'
""".stripMargin)
-sql("show indexes on table index_test_table").show(false)
-checkExistence(sql("show indexes on table index_test_table"), true, "dm",
"dm1", "lucene", "bloomfilter")
+sql("show indexes on table index_test_table").collect()
+checkExistence(sql("show indexes on table index_test_table"),
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/aggquery/IntegerDataTypeTestCase.scala
##
@@ -143,9 +147,52 @@ class IntegerDataTypeTestCase extends QueryTest with
BeforeAndAfterAll {
val ff = BigInt(2147484000L)
checkAnswer(
- sql("select
begin_time,begin_time1,begin_time2,begin_time3,begin_time4,begin_time5,begin_time6,begin_time7,begin_time8,begin_time9,begin_time10,begin_time11,begin_time12,begin_time13,begin_time14,begin_time15,begin_time16,begin_time17,begin_time18,begin_time19,begin_time20
from all_encoding_table"),
-
Seq(Row(1497376581,1,8388600,125,1497376581,8386600,1,100,125,1497376581,1497423738,2139095000,1497376581,1497423738,32000,123.4,11.1,3200.1,214744460.2,1497376581,1497376581),
-
Row(1497408581,32000,45000,25,1,55000,32000,75,35,1497423838,1497423838,ff,1497423838,1497423838,31900,838860.7,12.3,127.1,214748360.2,1497408581,1497408581))
+ sql("select
begin_time,begin_time1,begin_time2,begin_time3,begin_time4,begin_time5," +
Review comment:
done
This is an automated
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467105
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataWithCompression.scala
##
@@ -528,8 +571,9 @@ class TestLoadDataWithCompression extends QueryTest with
BeforeAndAfterEach with
var exception = intercept[RuntimeException] {
loadData()
}
+// scalastyle:off lineLength
assertResult("For not carbondata native supported compressor, the result
of method getName() should be the full class name. Expected
'org.apache.carbondata.core.datastore.compression.ZstdCompressor', found
'zstd'")(exception.getMessage)
-
+// scalastyle:on lineLength
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/emptyrow/TestSkipEmptyLines.scala
##
@@ -33,50 +33,65 @@ class TestSkipEmptyLines extends QueryTest with
BeforeAndAfterAll {
test("test load options with true") {
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable " +
-s"OPTIONS('skip_empty_line'='true')")
-checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
Seq(Row("a",25),Row("b",22),Row("c",23)))
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='true')")
+checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
+ Seq(Row("a", 25), Row("b", 22), Row("c", 23)))
}
test("test load options with false") {
sql("drop table if exists skipEmptyRowCarbonTable")
sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS
carbondata")
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table
skipEmptyRowCarbonTable " +
-s"OPTIONS('skip_empty_line'='false')")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " +
+"into table skipEmptyRowCarbonTable
OPTIONS('skip_empty_line'='false')")
checkAnswer(sql("select * from skipEmptyRowCarbonTable"),
-
Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null)))
+ Seq(Row("a", 25),
Review comment:
changed for all places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504466455
##
File path:
integration/spark/src/test/java/org/apache/carbondata/sdk/util/BinaryUtil.java
##
@@ -17,13 +17,16 @@
package org.apache.carbondata.sdk.util;
+import java.io.BufferedInputStream;
Review comment:
done
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexFunctionSuite.scala
##
@@ -77,7 +76,11 @@ class BloomCoarseGrainIndexFunctionSuite extends QueryTest
with BeforeAndAfterA
| properties('BLOOM_SIZE'='64')
""".stripMargin)
-IndexStatusUtil.checkIndexStatus(bloomSampleTable, indexName,
IndexStatus.ENABLED.name(), sqlContext.sparkSession, IndexType.BLOOMFILTER)
+IndexStatusUtil.checkIndexStatus(bloomSampleTable,
Review comment:
changed for all places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464628
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala
##
@@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with
BeforeAndAfterAll {
"projectjoindate Timestamp, projectenddate Timestamp, attendance int,
" +
"utilization int,salary int) STORED AS CARBONDATA")
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " +
-"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')")
+"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " +
+"'BAD_RECORDS_ACTION'='FORCE')")
val withoutIndex =
- sql("select empno from carbontable where empname = 'ayushi' or empname =
'krithin' or empname = 'madhan'")
+ sql("select empno from carbontable " +
Review comment:
changed for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala
##
@@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with
BeforeAndAfterAll {
"projectjoindate Timestamp, projectenddate Timestamp, attendance int,
" +
"utilization int,salary int) STORED AS CARBONDATA")
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " +
-"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')")
+"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " +
+"'BAD_RECORDS_ACTION'='FORCE')")
val withoutIndex =
- sql("select empno from carbontable where empname = 'ayushi' or empname =
'krithin' or empname = 'madhan'")
+ sql("select empno from carbontable " +
+ "where empname = 'ayushi' or empname = 'krithin' or empname =
'madhan'")
.collect().toSeq
-sql("create index empnameindex on table carbontable
(workgroupcategoryname,empname) AS 'carbondata'")
+sql(
+ "create index empnameindex on table carbontable (" +
+ "workgroupcategoryname,empname) AS 'carbondata'")
-checkAnswer(sql("select empno from carbontable where empname = 'ayushi' or
empname = 'krithin' or empname = 'madhan'"),
+checkAnswer(sql(
Review comment:
changed for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestIndexModelWithUnsafeColumnPage.scala
##
@@ -35,8 +35,8 @@ class TestIndexModelWithUnsafeColumnPage extends QueryTest
with BeforeAndAfterAl
}
test("Test secondry index data count") {
-checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable")
-,Seq(Row(1)))
+checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable"),
Review comment:
changed for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala
##
@@ -163,8 +163,11 @@ class TestNIQueryWithIndex extends QueryTest with
BeforeAndAfterAll{
// Query has EqualTo - So SI = Yes
assert(checkSIColumnsSize(ch24, 1)) // EqualTo
-}finally{
- sql(s"set
carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}")
+} finally {
+ sql(s"set carbon.si.lookup.partialstring=${
+CarbonCommonConstants
Review comment:
changed for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala
##
@@ -188,14 +191,17 @@ class TestNIQueryWithIndex extends QueryTest with
BeforeAndAfterAll{
val ch15 = sql("select count(*) from seccust where
c_phone='25-989-741-2988' and c_mktsegment like 'BUI%LDING'")
// equals on c_phone of I1, I2 & (length & startsWith & endswith) on
c_mktsegment of I2 so SI - Yes
- assert(checkSIColumnsSize(ch15, 3)) //size = EqualTo on c_phone, length,
StartsWith
+ assert(checkSIColumnsSize(ch15, 3)) // size = EqualTo on c_phone,
length, StartsWith
val ch16 = sql("select * from seccust where c_phone='25-989-741-2988'")
// Query has EqualTo so SI - Yes
assert(checkSIColumnsSize(ch16, 1)) // size = EqualTo
-} finally{
- sql(s"set
carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}")
+} finally {
+ sql(s"set carbon.si.lookup.partialstring=${
+CarbonCommonConstants
+ .ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT
Review comment:
changed for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithSecondryIndex.scala
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504465234
##
File path:
integration/flink/src/test/scala/org/apache/carbon/flink/TestCarbonPartitionWriter.scala
##
@@ -58,7 +57,11 @@ class TestCarbonPartitionWriter extends QueryTest with
BeforeAndAfterAll{
val dataCount = 1000
val source = getTestSource(dataCount)
- executeStreamingEnvironment(tablePath, writerProperties,
carbonProperties, environment, source)
+ executeStreamingEnvironment(tablePath,
Review comment:
change for all places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464757
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestIndexModelWithUnsafeColumnPage.scala
##
@@ -35,8 +35,8 @@ class TestIndexModelWithUnsafeColumnPage extends QueryTest
with BeforeAndAfterAl
}
test("Test secondry index data count") {
-checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable")
-,Seq(Row(1)))
+checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable"),
Review comment:
change for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala
##
@@ -163,8 +163,11 @@ class TestNIQueryWithIndex extends QueryTest with
BeforeAndAfterAll{
// Query has EqualTo - So SI = Yes
assert(checkSIColumnsSize(ch24, 1)) // EqualTo
-}finally{
- sql(s"set
carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}")
+} finally {
+ sql(s"set carbon.si.lookup.partialstring=${
+CarbonCommonConstants
Review comment:
change for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala
##
@@ -188,14 +191,17 @@ class TestNIQueryWithIndex extends QueryTest with
BeforeAndAfterAll{
val ch15 = sql("select count(*) from seccust where
c_phone='25-989-741-2988' and c_mktsegment like 'BUI%LDING'")
// equals on c_phone of I1, I2 & (length & startsWith & endswith) on
c_mktsegment of I2 so SI - Yes
- assert(checkSIColumnsSize(ch15, 3)) //size = EqualTo on c_phone, length,
StartsWith
+ assert(checkSIColumnsSize(ch15, 3)) // size = EqualTo on c_phone,
length, StartsWith
val ch16 = sql("select * from seccust where c_phone='25-989-741-2988'")
// Query has EqualTo so SI - Yes
assert(checkSIColumnsSize(ch16, 1)) // size = EqualTo
-} finally{
- sql(s"set
carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}")
+} finally {
+ sql(s"set carbon.si.lookup.partialstring=${
+CarbonCommonConstants
+ .ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT
Review comment:
change for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithSecondryIndex.scala
##
@@ -128,25 +129,32 @@ class TestSIWithSecondryIndex extends QueryTest with
BeforeAndAfterAll {
test("test create secondary index global sort on partition table") {
sql("drop table if exists partition_carbon_table")
-sql("create table partition_carbon_table (name string, id string, country
string) PARTITIONED BY(dateofjoin " +
+sql("create table partition_carbon_table (" +
+"name string, id string, country string) PARTITIONED BY(dateofjoin " +
"string) stored as carbondata")
// create SI before the inserting the data
-sql("create index partition_carbon_table_index on table
partition_carbon_table(id, country) as 'carbondata' properties" +
+sql("create index partition_carbon_table_index on table
partition_carbon_table(" +
+"id, country) as 'carbondata' properties" +
"('sort_scope'='global_sort', 'Global_sort_partitions'='3')")
sql("insert into partition_carbon_table select 'xx', '2', 'china', '2020'
" +
"union all select 'xx', '1', 'india', '2021'")
checkAnswerWithoutSort(sql("select id, country from
partition_carbon_table_index"),
Seq(Row("1", "india"), Row("2", "china")))
// check for valid sort_scope
-checkExistence(sql("describe formatted partition_carbon_table_index"),
true, "Sort Scope global_sort")
+checkExistence(sql("describe formatted partition_carbon_table_index"),
+ true,
+ "Sort Scope global_sort")
Review comment:
done
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithSecondryIndex.scala
##
@@ -296,14 +308,18 @@ class TestSIWithSecondryIndex extends QueryTest with
BeforeAndAfterAll {
sql(s"""ALTER TABLE default.index1 SET
|SERDEPROPERTIES ('isSITableEnabled' = 'false')""".stripMargin)
-sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " +
-"TABLE uniqdata OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')")
+sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO TABLE uniqdata
" +
+
"OPTIONS('DELIMITER'=',','BAD_RECORDS_LOGGER_ENABLE'='FALSE','BAD_RECORDS_ACTION'='FORCE')")
val count1 = sql("select * from uniqdata where workgroupcategoryname =
'developer'").count()
-val df1 = sql("select * from uniqdata where
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464628
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala
##
@@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with
BeforeAndAfterAll {
"projectjoindate Timestamp, projectenddate Timestamp, attendance int,
" +
"utilization int,salary int) STORED AS CARBONDATA")
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " +
-"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')")
+"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " +
+"'BAD_RECORDS_ACTION'='FORCE')")
val withoutIndex =
- sql("select empno from carbontable where empname = 'ayushi' or empname =
'krithin' or empname = 'madhan'")
+ sql("select empno from carbontable " +
Review comment:
change for all places
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala
##
@@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with
BeforeAndAfterAll {
"projectjoindate Timestamp, projectenddate Timestamp, attendance int,
" +
"utilization int,salary int) STORED AS CARBONDATA")
sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " +
-"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')")
+"TABLE carbontable OPTIONS('DELIMITER'=',',
'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " +
+"'BAD_RECORDS_ACTION'='FORCE')")
val withoutIndex =
- sql("select empno from carbontable where empname = 'ayushi' or empname =
'krithin' or empname = 'madhan'")
+ sql("select empno from carbontable " +
+ "where empname = 'ayushi' or empname = 'krithin' or empname =
'madhan'")
.collect().toSeq
-sql("create index empnameindex on table carbontable
(workgroupcategoryname,empname) AS 'carbondata'")
+sql(
+ "create index empnameindex on table carbontable (" +
+ "workgroupcategoryname,empname) AS 'carbondata'")
-checkAnswer(sql("select empno from carbontable where empname = 'ayushi' or
empname = 'krithin' or empname = 'madhan'"),
+checkAnswer(sql(
Review comment:
change for all places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464365 ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestBroadCastSIFilterPushJoinWithUDF.scala ## @@ -14,20 +14,21 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + Review comment: reverted This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950:
URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464144
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/mergedata/CarbonDataFileMergeTestCaseOnSI.scala
##
@@ -14,20 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
Review comment:
reverted
##
File path:
index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestAlterTableColumnRenameWithIndex.scala
##
@@ -73,9 +73,12 @@ class TestAlterTableColumnRenameWithIndex extends QueryTest
with BeforeAndAfterA
sql("alter table si_rename change c test string")
sql("alter table si_rename change d testSI string")
sql("show indexes on si_rename").collect
-val query2 = sql("select test,testsi from si_rename where testsi = 'pqr'
or test = 'def'").count()
+val query2 = sql("select test,testsi from si_rename where testsi = 'pqr'
or test = 'def'")
+ .count()
assert(query1 == query2)
-val df = sql("select test,testsi from si_rename where testsi = 'pqr' or
test = 'def'").queryExecution.sparkPlan
+val df = sql("select test,testsi from si_rename where testsi = 'pqr' or
test = 'def'")
+ .queryExecution
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-708216427 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4426/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
