[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708943444 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2699/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
akashrn5 commented on a change in pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#discussion_r505226878 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala ## @@ -149,14 +149,10 @@ private[sql] case class CarbonProjectForDeleteCommand( case e: HorizontalCompactionException => LOGGER.error("Delete operation passed. Exception in Horizontal Compaction." + " Please check logs. " + e.getMessage) -CarbonUpdateUtil.cleanStaleDeltaFiles(carbonTable, e.compactionTimeStamp.toString) Seq(Row(0L)) case e: Exception => LOGGER.error("Exception in Delete data operation " + e.getMessage, e) -// ** start clean up. Review comment: i dont think we can remove this directly, it might create problem, as mentioned in comments ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala ## @@ -374,8 +374,6 @@ object DeleteExecution { blockMappingVO.getSegmentNumberOfBlockMapping) } } else { - // In case of failure , clean all related delete delta files - CarbonUpdateUtil.cleanStaleDeltaFiles(carbonTable, timestamp) Review comment: same as above This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708937365 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4453/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3934: [WIP] Support Global Unique Id for SegmentNo
vikramahuja1001 commented on a change in pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#discussion_r505187328 ## File path: core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ## @@ -297,7 +297,7 @@ private static String getCarbonIndexFileName(String taskNo, int bucketNumber, */ public static String getSegmentPath(String tablePath, String segmentId) { return getPartitionDir(tablePath) + CarbonCommonConstants.FILE_SEPARATOR -+ SEGMENT_PREFIX + segmentId; ++ SEGMENT_PREFIX + segmentId + segmentId; Review comment: why is segmentID used twice instead of a UUID? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708922314 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2698/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3934: [WIP] Support Global Unique Id for SegmentNo
vikramahuja1001 commented on a change in pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#discussion_r505186238 ## File path: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ## @@ -2758,12 +2758,20 @@ public static String getBlockId(AbsoluteTableIdentifier identifier, String fileP String segmentId, boolean isTransactionalTable, boolean isStandardTable, boolean isPartitionTable) { String blockId; -String blockName = filePath.substring(filePath.lastIndexOf("/") + 1); +String blockName; +if (filePath.lastIndexOf("/") != -1) { + blockName = filePath.substring(filePath.lastIndexOf("/") + 1); +} else { + blockName = filePath.substring(filePath.lastIndexOf(File.separator) + 1); + filePath = filePath.replace(File.separator, CarbonCommonConstants.FILE_SEPARATOR); +} + String tablePath = identifier.getTablePath(); if (filePath.startsWith(tablePath)) { if (!isTransactionalTable || isStandardTable) { -blockId = "Part0" + CarbonCommonConstants.FILE_SEPARATOR + "Segment_" + segmentId +blockId = "Part0" + CarbonCommonConstants.FILE_SEPARATOR + "Segment_" ++ segmentId + segmentId Review comment: why is segmentID used twice instead of a UUID? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708914537 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4452/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3934: [WIP] Support Global Unique Id for SegmentNo
ajantha-bhat commented on pull request #3934: URL: https://github.com/apache/carbondata/pull/3934#issuecomment-708913452 @marchpure and @QiangCai : The solution was to use segmentid_UUID right ? I didn't see UUID in the PR changes. please explain the solution or update the PR description This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912408 retest this plese This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912511 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat removed a comment on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat removed a comment on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708912408 retest this plese This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
Karan980 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708897384 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708880660 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2697/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708879401 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4451/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Labels: easyfix features newbie (was: newbie patch) > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Affects Versions: 2.0.0, 2.0.1 >Reporter: Nguyen Dinh Huynh >Priority: Major > Labels: easyfix, features, newbie > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3. Is this a bug ? > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > My code is: > {code:java} > val map = Map( > col("_external_op") -> col("A._external_op"), > col("_external_ts_sec") -> col("A._external_ts_sec"), > col("_external_row") -> col("A._external_row"), > col("_external_pos") -> col("A._external_pos"), > col("id") -> col("A.id"), > col("order") -> col("A.order"), > col("shop_code") -> col("A.shop_code"), > col("customer_tel") -> col("A.customer_tel"), > col("channel") -> col("A.channel"), > col("batch_session_id") -> col("A.batch_session_id"), > col("deleted_at") -> col("A.deleted_at"), > col("created") -> col("A.created")) > .asInstanceOf[Map[Any, Any]] > val testDf = > spark.sqlContext.read.format("carbondata") > .option("tableName", "package_drafts") > .option("schemaName", "db") > .option("dbName", "db") > .option("databaseName", "d")b > .load() > .as("B") > testDf.printSchema() > testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id"))) > .whenMatched(col("A._external_op") === "u") > .updateExpr(map) > .whenMatched(col("A._external_op") === "c") > .insertExpr(map) > .whenMatched(col("A._external_op") === "d") > .delete() > .execute() > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Flags: Important Labels: newbie patch (was: ) > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Affects Versions: 2.0.0, 2.0.1 >Reporter: Nguyen Dinh Huynh >Priority: Major > Labels: newbie, patch > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3. Is this a bug ? > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > My code is: > {code:java} > val map = Map( > col("_external_op") -> col("A._external_op"), > col("_external_ts_sec") -> col("A._external_ts_sec"), > col("_external_row") -> col("A._external_row"), > col("_external_pos") -> col("A._external_pos"), > col("id") -> col("A.id"), > col("order") -> col("A.order"), > col("shop_code") -> col("A.shop_code"), > col("customer_tel") -> col("A.customer_tel"), > col("channel") -> col("A.channel"), > col("batch_session_id") -> col("A.batch_session_id"), > col("deleted_at") -> col("A.deleted_at"), > col("created") -> col("A.created")) > .asInstanceOf[Map[Any, Any]] > val testDf = > spark.sqlContext.read.format("carbondata") > .option("tableName", "package_drafts") > .option("schemaName", "db") > .option("dbName", "db") > .option("databaseName", "d")b > .load() > .as("B") > testDf.printSchema() > testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id"))) > .whenMatched(col("A._external_op") === "u") > .updateExpr(map) > .whenMatched(col("A._external_op") === "c") > .insertExpr(map) > .whenMatched(col("A._external_op") === "d") > .delete() > .execute() > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Affects Version/s: 2.0.1 2.0.0 > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Affects Versions: 2.0.0, 2.0.1 >Reporter: Nguyen Dinh Huynh >Priority: Major > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3. Is this a bug ? > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > My code is: > {code:java} > val map = Map( > col("_external_op") -> col("A._external_op"), > col("_external_ts_sec") -> col("A._external_ts_sec"), > col("_external_row") -> col("A._external_row"), > col("_external_pos") -> col("A._external_pos"), > col("id") -> col("A.id"), > col("order") -> col("A.order"), > col("shop_code") -> col("A.shop_code"), > col("customer_tel") -> col("A.customer_tel"), > col("channel") -> col("A.channel"), > col("batch_session_id") -> col("A.batch_session_id"), > col("deleted_at") -> col("A.deleted_at"), > col("created") -> col("A.created")) > .asInstanceOf[Map[Any, Any]] > val testDf = > spark.sqlContext.read.format("carbondata") > .option("tableName", "package_drafts") > .option("schemaName", "db") > .option("dbName", "db") > .option("databaseName", "d")b > .load() > .as("B") > testDf.printSchema() > testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id"))) > .whenMatched(col("A._external_op") === "u") > .updateExpr(map) > .whenMatched(col("A._external_op") === "c") > .insertExpr(map) > .whenMatched(col("A._external_op") === "d") > .delete() > .execute() > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] Kejian-Li commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Kejian-Li commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708816282 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708720271 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2696/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708718712 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4450/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708693056 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4445/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708690166 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2692/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] maheshrajus commented on pull request #3695: [WIP] partition optimization
maheshrajus commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708685238 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708680642 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4449/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708676144 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2693/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708675919 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4446/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708627987 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2691/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
Karan980 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708624107 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
Karan980 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708623745 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708610112 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2690/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708605209 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4442/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708605441 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3// This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3974: [Carbondata-3999] Fix permission issue of indexServerTmp directory
CarbonDataQA1 commented on pull request #3974: URL: https://github.com/apache/carbondata/pull/3974#issuecomment-708603224 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4443/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708601810 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2689/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708534684 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#discussion_r504829616 ## File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonIUD.java ## @@ -173,10 +184,16 @@ public void update(String path, Expression filterExpression, RecordWriter deleteDeltaWriter = CarbonTableOutputFormat.getDeleteDeltaRecordWriter(path); ObjectArrayWritable writable = new ObjectArrayWritable(); - +long day = 24L * 3600 * 1000; while (reader.hasNext()) { Object[] row = (Object[]) reader.readNextRow(); writable.set(Arrays.copyOfRange(row, row.length - 1, row.length)); + for (Integer dateIndex : dateIndexes) { +row[dateIndex] = new Date((day * ((int) row[dateIndex]))); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#discussion_r504829543 ## File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonIUDTest.java ## @@ -72,6 +72,43 @@ public void testDelete() throws Exception { FileUtils.deleteDirectory(new File(path)); } + @Test + public void testUpdateOnDateType() throws Exception { Review comment: Testcase was added to check conversion of dateType columns to integer while reading. As date column is present in schema. so no need to perform update on date column. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#discussion_r504827820 ## File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonIUD.java ## @@ -155,7 +158,15 @@ public void update(String path, Expression filterExpression, Schema schema = CarbonSchemaReader.readSchema(indexFiles.get(0)).asOriginOrder(); Field[] fields = schema.getFields(); String[] projectionColumns = new String[fields.length + 1]; +List dateIndexes = new ArrayList<>(); +List timeStampIndexes = new ArrayList<>(); for (int i = 0; i < fields.length; i++) { + if (fields[i].getDataType() == DataTypes.DATE) { +dateIndexes.add(i); + } + if (fields[i].getDataType() == DataTypes.TIMESTAMP) { Review comment: Code removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on a change in pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
Karan980 commented on a change in pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#discussion_r504827655 ## File path: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonFileInputFormat.java ## @@ -279,7 +279,8 @@ public void setAllColumnProjectionIfNotConfigured(JobContext job, CarbonTable ca private String[] getDeleteDeltaFiles(String segmentFilePath, List allDeleteDeltaFiles) { List deleteDeltaFiles = new ArrayList<>(); String segmentFileName = null; -String[] pathElements = segmentFilePath.split(Pattern.quote(File.separator)); +segmentFilePath = segmentFilePath.replace("\\", "/"); Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708481924 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2687/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708473034 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4440/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708473030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708457681 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4438/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708448621 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4437/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [WIP]Fix special char issue in create table
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-708442276 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2682/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708436627 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4441/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD
CarbonDataQA1 commented on pull request #3970: URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708434559 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2688/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708434556 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2684/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708420587 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2685/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort
CarbonDataQA1 commented on pull request #3979: URL: https://github.com/apache/carbondata/pull/3979#issuecomment-708407338 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2683/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3979: [Carbondata-3954] Fix insertion from ORC table into carbon table when sort scope is global sort
CarbonDataQA1 commented on pull request #3979: URL: https://github.com/apache/carbondata/pull/3979#issuecomment-708398610 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4436/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3983: [WIP]Fix special char issue in create table
CarbonDataQA1 commented on pull request #3983: URL: https://github.com/apache/carbondata/pull/3983#issuecomment-708395905 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4435/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
CarbonDataQA1 commented on pull request #3984: URL: https://github.com/apache/carbondata/pull/3984#issuecomment-708395934 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2681/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
QiangCai commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708382061 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
marchpure commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708375624 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
marchpure commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708375305 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
CarbonDataQA1 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708366544 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4432/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
CarbonDataQA1 commented on pull request #3984: URL: https://github.com/apache/carbondata/pull/3984#issuecomment-708363702 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4434/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
CarbonDataQA1 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708359537 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2680/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708349031 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2679/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708345630 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2676/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708345192 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4433/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-708341574 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2675/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
CarbonDataQA1 commented on pull request #3981: URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708340588 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4431/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708337733 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-3880) How to start JDBC service in distributed index
[ https://issues.apache.org/jira/browse/CARBONDATA-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213825#comment-17213825 ] Nihal kumar ojha commented on CARBONDATA-3880: -- Hi, please follow the below steps to configure the distributed index server with JDBC. 1. Add these properties in spark-defaults.conf spark.yarn.keytab= spark.carbon.indexserver.keytab= spark.carbon.indexserver.principal=spark2x/hadoop.hadoop@hadoop.com spark.yarn.principal=spark2x/hadoop.hadoop@hadoop.com 2. Add following configuration in carbon.properties(Ensure the carbon.properties is configured in spark-defaults.conf in driver extra java option) carbon.enable.index.server=true carbon.indexserver.enable.prepriming=true carbon.indexserver.HA.enabled=true carbon.max.executor.lru.cache.size=-1 carbon.disable.index.server.fallback=false carbon.indexserver.zookeeper.dir=/indexserver2x carbon.index.server.port= Then run below spark-submit command at $spark_home location bin/spark-submit --num-executors 2 --master yarn --class org.apache.carbondata.indexserver.indexserver then start spark JDBCserver as usual. Queries should reflect in yarn UI, in the index server and spark JDBC application. > How to start JDBC service in distributed index > --- > > Key: CARBONDATA-3880 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3880 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 2.0.0 >Reporter: li >Priority: Major > Fix For: 2.1.0 > > > How to start JDBC service in distributed index -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
CarbonDataQA1 commented on pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#issuecomment-708322826 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2678/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3977: [CARBONDATA-4027] Fix the wrong modifiedtime of loading files in inse…
CarbonDataQA1 commented on pull request #3977: URL: https://github.com/apache/carbondata/pull/3977#issuecomment-708314304 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4430/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-4025) storage space for MV is double to that of a table on which MV has been created.
[ https://issues.apache.org/jira/browse/CARBONDATA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213792#comment-17213792 ] suyash yadav commented on CARBONDATA-4025: -- Hi any update on this? > storage space for MV is double to that of a table on which MV has been > created. > --- > > Key: CARBONDATA-4025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4025 > Project: CarbonData > Issue Type: Improvement > Components: core >Affects Versions: 2.0.1 > Environment: Apcahe carbondata 2.0.1 > Apache spark 2.4.5 > Hadoop 2.7.2 >Reporter: suyash yadav >Priority: Major > > We are doing a POC based on carbondata but we have observed that when we > create n MV on a table with timeseries function of same granualarity the MV > takes double the space of the table. > > In my scenario, My table has 1.3 million records and MV also has same number > of records but the size of the table is 3.6 MB but the size of the MV is > around 6.5 MB. > This is really important for us as critical business decision are getting > affected due to this behaviour. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
CarbonDataQA1 commented on pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#issuecomment-708300620 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4429/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (CARBONDATA-3892) An exception occurred when modifying the table name using SparkSession
[ https://issues.apache.org/jira/browse/CARBONDATA-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213785#comment-17213785 ] Nihal kumar ojha commented on CARBONDATA-3892: -- Hi, I was trying to reproduce this issue but not getting reproduced. I am using the query "ALTER TABLE oldTable RENAME to newTable". please correct me if I am wrong. Or if there is some other configuration then please add here. > An exception occurred when modifying the table name using SparkSession > -- > > Key: CARBONDATA-3892 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3892 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 2.0.0 >Reporter: li >Priority: Blocker > > Exception in thread "main" java.lang.LinkageError: ClassCastException: > attempting to > castjar:file:/usr/hdp/2.6.5.0-292/spark2/carbonlib/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/usr/hdp/2.6.5.0-292/spark2/carbonlib/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar!/javax/ws/rs/ext/RuntimeDelegate.class -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] akashrn5 opened a new pull request #3984: [WIP]Fix MV query issue with aggregation on decimal column
akashrn5 opened a new pull request #3984: URL: https://github.com/apache/carbondata/pull/3984 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akashrn5 opened a new pull request #3983: [WIP]Fix special char issue in create table
akashrn5 opened a new pull request #3983: URL: https://github.com/apache/carbondata/pull/3983 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708287330 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4428/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
CarbonDataQA1 commented on pull request #3982: URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708285816 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2674/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708286183 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4427/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Pickupolddriver commented on a change in pull request #3935: [CARBONDATA-3993] Remove auto data deletion in IUD processs
Pickupolddriver commented on a change in pull request #3935: URL: https://github.com/apache/carbondata/pull/3935#discussion_r504537575 ## File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala ## @@ -267,9 +266,8 @@ object CarbonDataRDDFactory { throw new Exception("Exception in compaction " + exception.getMessage) } } finally { - executor.shutdownNow() try { -compactor.deletePartialLoadsInCompaction() Review comment: @ajantha-bhat after the merge of #3978 , cleanStaleDeltaFiles will only be called when there are exceptions during the update process. And it will only delete the delta and index file created by the exception update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat edited a comment on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708279514 @akashrn5 , @QiangCai : PR is ready (11 random failures are fixed). There are some more random failures are there in CI (flink related). someone can analyze and handle in other PRs I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
ajantha-bhat commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708279514 @akashrn5 , @QiangCai : PR is ready. There are some more random failures are there in CI. someone can analyze and handle in other PRs I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708275465 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
CarbonDataQA1 commented on pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#issuecomment-708271365 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2673/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on pull request #3972: [WIP]Launch same number of task as select query for insert into select and ctas cases when target table is of no_sort
VenuReddy2103 commented on pull request #3972: URL: https://github.com/apache/carbondata/pull/3972#issuecomment-708269487 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java ## @@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[] value) { } } + @Override + public void putAllByteArray(byte[] data, int offset, int length) { +int[] lengths = getLengths(); +int[] offsets = getOffsets(); +for (int i = 0; i < lengths.length; i++) { + if (offsets[i] != 0) { Review comment: offset values are 0 by default and put was called, due to which after select- empty rows were showed. Also select count(*) showed wrong results. Like you said offset can never be 0, although length may be 0. If offset is 0 then put should not be called. Hence this check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java ## @@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[] value) { } } + @Override + public void putAllByteArray(byte[] data, int offset, int length) { +int[] lengths = getLengths(); +int[] offsets = getOffsets(); +for (int i = 0; i < lengths.length; i++) { + if (offsets[i] != 0) { Review comment: offset values are 0 by default, due to which after select- empty rows were showed. Also select count(*) showed wrong results. Like you said offset can never be 0, although length may be 0. If offset is 0 then put should not be called. Hence this check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#discussion_r504507076 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java ## @@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[] value) { } } + @Override + public void putAllByteArray(byte[] data, int offset, int length) { +int[] lengths = getLengths(); +int[] offsets = getOffsets(); +for (int i = 0; i < lengths.length; i++) { + if (offsets[i] != 0) { Review comment: offset values were 0 by default, due to which after select- empty rows were showed. Also select count(*) showed wrong results. Like you said offset can never be 0. If it is 0 then put should not be called. Hence this check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3948: [WIP][HOTFIX] Fix random 11 testcase failure in CI
CarbonDataQA1 commented on pull request #3948: URL: https://github.com/apache/carbondata/pull/3948#issuecomment-708254236 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4425/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#discussion_r504504630 ## File path: core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithDeleteDelta.java ## @@ -242,4 +242,8 @@ public void putArray(int rowId, int offset, int length) { columnVector.putArray(counter++, offset, length); } } + + public CarbonColumnVector getColumnVector() { Review comment: Right, I have made the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3967: [CARBONDATA-4004] [CARBONDATA-4012] Issue with select after update command
akkio-97 commented on a change in pull request #3967: URL: https://github.com/apache/carbondata/pull/3967#discussion_r504504712 ## File path: integration/presto/src/main/prestodb/org/apache/carbondata/presto/readers/SliceStreamReader.java ## @@ -124,6 +124,17 @@ public void putByteArray(int rowId, int count, byte[] value) { } } + @Override + public void putAllByteArray(byte[] data, int offset, int length) { Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708247374 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4424/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3695: [WIP] partition optimization
CarbonDataQA1 commented on pull request #3695: URL: https://github.com/apache/carbondata/pull/3695#issuecomment-708246843 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2670/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Description: I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 with spark 2.4.3. Is this a bug ? {code:java} 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 {code} My code is: {code:java} val map = Map( col("_external_op") -> col("A._external_op"), col("_external_ts_sec") -> col("A._external_ts_sec"), col("_external_row") -> col("A._external_row"), col("_external_pos") -> col("A._external_pos"), col("id") -> col("A.id"), col("order") -> col("A.order"), col("shop_code") -> col("A.shop_code"), col("customer_tel") -> col("A.customer_tel"), col("channel") -> col("A.channel"), col("batch_session_id") -> col("A.batch_session_id"), col("deleted_at") -> col("A.deleted_at"), col("created") -> col("A.created")) .asInstanceOf[Map[Any, Any]] val testDf = spark.sqlContext.read.format("carbondata") .option("tableName", "package_drafts") .option("schemaName", "db") .option("dbName", "db") .option("databaseName", "d")b .load() .as("B") testDf.printSchema() testDf.merge(package_draft_view, col("A.id").equalTo(col("B.id"))) .whenMatched(col("A._external_op") === "u") .updateExpr(map) .whenMatched(col("A._external_op") === "c") .insertExpr(map) .whenMatched(col("A._external_op") === "d") .delete() .execute() {code} was: I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 with spark 2.4.3. Is this a bug ? {code:java} 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 {code} > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Reporter: Nguyen Dinh Huynh >Priority: Major > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3. Is this a bug ? > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > My code is: > {code:java} > val map = Map( > col("_external_op") -> col("A._external_op"), > col("_external_ts_sec") -> col("A._external_ts_sec"), > col("_external_row") -> col("A._external_row"), > col("_external_pos") -> col("A._external_pos"), > col("id") -> col("A.id"), > col("order") -> col("A.order"), > col("shop_code") -> col("A.shop_code"), > col("customer_tel") -> col("A.customer_tel"), > col("channel") -> col("A.channel"), > col("batch_session_id") -> col("A.batch_session_id"), > col("deleted_at") -> col("A.deleted_at"), > col("created") -> col("A.created")) > .asInstanceOf[Map[Any, Any]] > val testDf = > spark.sqlContext.read.format("carbondata") > .option("tableName", "package_drafts") > .option("schemaName", "db") > .option("dbName", "db") > .option("databaseName", "d")b > .load() > .as("B") > testDf.printSchema() > testDf.merge(package_draft_view, c
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Description: I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 with spark 2.4.3. Is this a bug ? {code:java} 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 {code} was: I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 with spark 2.4.3 {code:java} 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 {code} > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Reporter: Nguyen Dinh Huynh >Priority: Major > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3. Is this a bug ? > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-4033) Error when using merge API with hive table
[ https://issues.apache.org/jira/browse/CARBONDATA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nguyen Dinh Huynh updated CARBONDATA-4033: -- Description: I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 with spark 2.4.3 {code:java} 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 {code} was: 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > Error when using merge API with hive table > -- > > Key: CARBONDATA-4033 > URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 > Project: CarbonData > Issue Type: Bug >Reporter: Nguyen Dinh Huynh >Priority: Major > > I always get this error when trying to upsert hive table. I'm using CDH 6.3.1 > with spark 2.4.3 > > {code:java} > 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due > to exception java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. > 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be > removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR > Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) > java.lang.RuntimeException: Store location not set for the key > __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-4033) Error when using merge API with hive table
Nguyen Dinh Huynh created CARBONDATA-4033: - Summary: Error when using merge API with hive table Key: CARBONDATA-4033 URL: https://issues.apache.org/jira/browse/CARBONDATA-4033 Project: CarbonData Issue Type: Bug Reporter: Nguyen Dinh Huynh 2020-10-14 14:59:25 WARN BlockManager:66 - Putting block rdd_21_1 failed due to exception java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0. 2020-10-14 14:59:25 WARN BlockManager:66 - Block rdd_21_1 could not be removed as it was not found on disk or in memory 2020-10-14 14:59:25 ERROR Executor:91 - Exception in task 1.0 in stage 0.0 (TID 1) java.lang.RuntimeException: Store location not set for the key __temptable-7bdfc88b-e5b7-46d5-8492-dfbb98b9a1b0_1602662359786_null_389ec940-ed27-41d1-9038-72ed1cd162e90x0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467243 ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/emptyrow/TestSkipEmptyLines.scala ## @@ -33,50 +33,65 @@ class TestSkipEmptyLines extends QueryTest with BeforeAndAfterAll { test("test load options with true") { sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable " + -s"OPTIONS('skip_empty_line'='true')") -checkAnswer(sql("select * from skipEmptyRowCarbonTable"), Seq(Row("a",25),Row("b",22),Row("c",23))) +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='true')") +checkAnswer(sql("select * from skipEmptyRowCarbonTable"), + Seq(Row("a", 25), Row("b", 22), Row("c", 23))) } test("test load options with false") { sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable " + -s"OPTIONS('skip_empty_line'='false')") +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='false')") checkAnswer(sql("select * from skipEmptyRowCarbonTable"), - Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null))) + Seq(Row("a", 25), +Row("b", 22), +Row("c", 23), +Row(null, null), +Row(null, null), +Row(null, null))) } test("test carbonproperties with true") { CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE, "true") sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable") +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable") checkAnswer(sql("select * from skipEmptyRowCarbonTable"), - Seq(Row("a",25),Row("b",22),Row("c",23))) + Seq(Row("a", 25), Row("b", 22), Row("c", 23))) CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE, CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE_DEFAULT) } test("test carbonproperties with false") { - CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE, "false") +CarbonProperties.getInstance() + .addProperty(CarbonCommonConstants.CARBON_SKIP_EMPTY_LINE, "false") sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable") +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable") checkAnswer(sql("select * from skipEmptyRowCarbonTable"), - Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null))) + Seq(Row("a", 25), Review comment: changed for all places ## File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/aggquery/AllDataTypesTestCaseAggregate.scala ## @@ -46,21 +47,23 @@ class AllDataTypesTestCaseAggregate extends QueryTest with BeforeAndAfterAll { "Timestamp, workgroupcategory int, workgroupcategoryname String, deptno int, deptname " + "String, projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance " + "int,utilization int,salary int)row format delimited fields terminated by ','") -sql( - s"LOAD DATA LOCAL INPATH '$resourcesPath/datawithoutheader.csv' INTO TABLE alldatatypesAGG_hive") +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/datawithoutheader.csv' " + +"INTO TABLE alldatatypesAGG_hive") } test( "select empno,empname,utilization,count(salary),sum(empno) from alldatatypestableAGG where " + "empname in ('arvind','ayushi') group by empno,empname,utilization") { checkAnswer( - sql( -"select empno,empname,utilization,count(salary),sum(empno) from alldatatypestableAGG where" + -" empname in ('arvind','ayushi') group by empno,empname,utilization"), - sql( -"select empno,empname,utilization,count(salary),sum(empno) from alldatatypesAGG_hive where" + -" empname in ('arvind','ayushi') group by empno,empname,utili
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467565 ## File path: integration/spark/src/test/scala/org/apache/spark/carbondata/TestStreamingTableQueryFilter.scala ## @@ -96,25 +98,97 @@ class TestStreamingTableQueryFilter extends QueryTest with BeforeAndAfterAll { // filter checkAnswer( sql("select * from streaming_table_filter.stream_filter where id = 1"), - Seq(Row(1, "name_1", "city_1", 1.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue), Row(wrap(Array("school_1", "school_11")), 1 + Seq(Row(1, Review comment: done, changed for all places This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504466643 ## File path: integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexFunctionSuite.scala ## @@ -803,27 +840,38 @@ class BloomCoarseGrainIndexFunctionSuite extends QueryTest with BeforeAndAfterA sql(s"INSERT INTO $bloomSampleTable SELECT 'c1v2', 2, 'c3v2'") // two segments both has index files -val carbonTable = CarbonEnv.getCarbonTable(Option("default"), bloomSampleTable)(SparkTestQueryExecutor.spark) +val carbonTable = CarbonEnv.getCarbonTable(Option("default"), bloomSampleTable)( + SparkTestQueryExecutor.spark) import scala.collection.JavaConverters._ (0 to 1).foreach { segId => - val indexPath = CarbonTablePath.getIndexesStorePath(carbonTable.getTablePath, segId.toString, indexName) - assert(FileUtils.listFiles(FileUtils.getFile(indexPath), Array("bloomindexmerge"), true).asScala.nonEmpty) + val indexPath = CarbonTablePath.getIndexesStorePath(carbonTable.getTablePath, Review comment: changed for all places ## File path: integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexSuite.scala ## @@ -82,16 +82,24 @@ class BloomCoarseGrainIndexSuite extends QueryTest with BeforeAndAfterAll with B sql(s"select * from $bloomSampleTable where id = 1 and city='city_1'", indexName, shouldHit), sql(s"select * from $normalTable where id = 1 and city='city_1'")) checkAnswer( - sql(s"select * from $bloomSampleTable where id = 999 and city='city_999'", indexName, shouldHit), + sql(s"select * from $bloomSampleTable where id = 999 and city='city_999'", +indexName, +shouldHit), Review comment: changed for all places ## File path: integration/spark/src/test/scala/org/apache/carbondata/index/lucene/LuceneFineGrainIndexSuite.scala ## @@ -186,12 +191,21 @@ class LuceneFineGrainIndexSuite extends QueryTest with BeforeAndAfterAll { sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE index_test1 OPTIONS('header'='false')") -checkAnswer(sql("SELECT * FROM index_test1 WHERE TEXT_MATCH('name:n10')"), sql(s"select * from index_test1 where name='n10'")) - -var carbonTable = CarbonEnv.getCarbonTable(Some("lucene"), "index_test1")(sqlContext.sparkSession) -var indexes = carbonTable.getIndexMetadata.getIndexesMap.get(IndexType.LUCENE.getIndexProviderName) - .asScala.filter(p => p._2.get(CarbonCommonConstants.INDEX_STATUS).equalsIgnoreCase(IndexStatus.ENABLED.name())) -assert(indexes.exists(p => p._1.equals("dm12") && p._2.get(CarbonCommonConstants.INDEX_STATUS) == IndexStatus.ENABLED.name())) +checkAnswer(sql("SELECT * FROM index_test1 WHERE TEXT_MATCH('name:n10')"), + sql(s"select * from index_test1 where name='n10'")) + +var carbonTable = CarbonEnv.getCarbonTable(Some("lucene"), "index_test1")(sqlContext + .sparkSession) +val indexes = carbonTable.getIndexMetadata Review comment: done ## File path: integration/spark/src/test/scala/org/apache/carbondata/index/lucene/LuceneFineGrainIndexSuite.scala ## @@ -863,8 +891,13 @@ class LuceneFineGrainIndexSuite extends QueryTest with BeforeAndAfterAll { | ON index_test_table (name) | AS 'bloomfilter' """.stripMargin) -sql("show indexes on table index_test_table").show(false) -checkExistence(sql("show indexes on table index_test_table"), true, "dm", "dm1", "lucene", "bloomfilter") +sql("show indexes on table index_test_table").collect() +checkExistence(sql("show indexes on table index_test_table"), Review comment: done ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/aggquery/IntegerDataTypeTestCase.scala ## @@ -143,9 +147,52 @@ class IntegerDataTypeTestCase extends QueryTest with BeforeAndAfterAll { val ff = BigInt(2147484000L) checkAnswer( - sql("select begin_time,begin_time1,begin_time2,begin_time3,begin_time4,begin_time5,begin_time6,begin_time7,begin_time8,begin_time9,begin_time10,begin_time11,begin_time12,begin_time13,begin_time14,begin_time15,begin_time16,begin_time17,begin_time18,begin_time19,begin_time20 from all_encoding_table"), - Seq(Row(1497376581,1,8388600,125,1497376581,8386600,1,100,125,1497376581,1497423738,2139095000,1497376581,1497423738,32000,123.4,11.1,3200.1,214744460.2,1497376581,1497376581), - Row(1497408581,32000,45000,25,1,55000,32000,75,35,1497423838,1497423838,ff,1497423838,1497423838,31900,838860.7,12.3,127.1,214748360.2,1497408581,1497408581)) + sql("select begin_time,begin_time1,begin_time2,begin_time3,begin_time4,begin_time5," + Review comment: done This is an automated messa
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504467105 ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataWithCompression.scala ## @@ -528,8 +571,9 @@ class TestLoadDataWithCompression extends QueryTest with BeforeAndAfterEach with var exception = intercept[RuntimeException] { loadData() } +// scalastyle:off lineLength assertResult("For not carbondata native supported compressor, the result of method getName() should be the full class name. Expected 'org.apache.carbondata.core.datastore.compression.ZstdCompressor', found 'zstd'")(exception.getMessage) - +// scalastyle:on lineLength Review comment: done ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/emptyrow/TestSkipEmptyLines.scala ## @@ -33,50 +33,65 @@ class TestSkipEmptyLines extends QueryTest with BeforeAndAfterAll { test("test load options with true") { sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable " + -s"OPTIONS('skip_empty_line'='true')") -checkAnswer(sql("select * from skipEmptyRowCarbonTable"), Seq(Row("a",25),Row("b",22),Row("c",23))) +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='true')") +checkAnswer(sql("select * from skipEmptyRowCarbonTable"), + Seq(Row("a", 25), Row("b", 22), Row("c", 23))) } test("test load options with false") { sql("drop table if exists skipEmptyRowCarbonTable") sql("CREATE TABLE skipEmptyRowCarbonTable (name string, age int) STORED AS carbondata") -sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' into table skipEmptyRowCarbonTable " + -s"OPTIONS('skip_empty_line'='false')") +sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/emptylines.csv' " + +"into table skipEmptyRowCarbonTable OPTIONS('skip_empty_line'='false')") checkAnswer(sql("select * from skipEmptyRowCarbonTable"), - Seq(Row("a",25),Row("b",22),Row("c",23),Row(null,null),Row(null,null),Row(null,null))) + Seq(Row("a", 25), Review comment: changed for all places This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504466455 ## File path: integration/spark/src/test/java/org/apache/carbondata/sdk/util/BinaryUtil.java ## @@ -17,13 +17,16 @@ package org.apache.carbondata.sdk.util; +import java.io.BufferedInputStream; Review comment: done ## File path: integration/spark/src/test/scala/org/apache/carbondata/index/bloom/BloomCoarseGrainIndexFunctionSuite.scala ## @@ -77,7 +76,11 @@ class BloomCoarseGrainIndexFunctionSuite extends QueryTest with BeforeAndAfterA | properties('BLOOM_SIZE'='64') """.stripMargin) -IndexStatusUtil.checkIndexStatus(bloomSampleTable, indexName, IndexStatus.ENABLED.name(), sqlContext.sparkSession, IndexType.BLOOMFILTER) +IndexStatusUtil.checkIndexStatus(bloomSampleTable, Review comment: changed for all places This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3950: [CARBONDATA-3889] Enable scalastyle check for all scala test code
QiangCai commented on a change in pull request #3950: URL: https://github.com/apache/carbondata/pull/3950#discussion_r504464628 ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala ## @@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with BeforeAndAfterAll { "projectjoindate Timestamp, projectenddate Timestamp, attendance int, " + "utilization int,salary int) STORED AS CARBONDATA") sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " + -"TABLE carbontable OPTIONS('DELIMITER'=',', 'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')") +"TABLE carbontable OPTIONS('DELIMITER'=',', 'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " + +"'BAD_RECORDS_ACTION'='FORCE')") val withoutIndex = - sql("select empno from carbontable where empname = 'ayushi' or empname = 'krithin' or empname = 'madhan'") + sql("select empno from carbontable " + Review comment: changed for all places ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestCreateIndexTable.scala ## @@ -262,40 +262,46 @@ class TestCreateIndexTable extends QueryTest with BeforeAndAfterAll { "projectjoindate Timestamp, projectenddate Timestamp, attendance int, " + "utilization int,salary int) STORED AS CARBONDATA") sql(s"LOAD DATA LOCAL INPATH '$resourcesPath/data.csv' INTO " + -"TABLE carbontable OPTIONS('DELIMITER'=',', 'BAD_RECORDS_LOGGER_ENABLE'='FALSE', 'BAD_RECORDS_ACTION'='FORCE')") +"TABLE carbontable OPTIONS('DELIMITER'=',', 'BAD_RECORDS_LOGGER_ENABLE'='FALSE', " + +"'BAD_RECORDS_ACTION'='FORCE')") val withoutIndex = - sql("select empno from carbontable where empname = 'ayushi' or empname = 'krithin' or empname = 'madhan'") + sql("select empno from carbontable " + + "where empname = 'ayushi' or empname = 'krithin' or empname = 'madhan'") .collect().toSeq -sql("create index empnameindex on table carbontable (workgroupcategoryname,empname) AS 'carbondata'") +sql( + "create index empnameindex on table carbontable (" + + "workgroupcategoryname,empname) AS 'carbondata'") -checkAnswer(sql("select empno from carbontable where empname = 'ayushi' or empname = 'krithin' or empname = 'madhan'"), +checkAnswer(sql( Review comment: changed for all places ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestIndexModelWithUnsafeColumnPage.scala ## @@ -35,8 +35,8 @@ class TestIndexModelWithUnsafeColumnPage extends QueryTest with BeforeAndAfterAl } test("Test secondry index data count") { -checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable") -,Seq(Row(1))) +checkAnswer(sql("select count(*) from testSecondryIndex_IndexTable"), Review comment: changed for all places ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala ## @@ -163,8 +163,11 @@ class TestNIQueryWithIndex extends QueryTest with BeforeAndAfterAll{ // Query has EqualTo - So SI = Yes assert(checkSIColumnsSize(ch24, 1)) // EqualTo -}finally{ - sql(s"set carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}") +} finally { + sql(s"set carbon.si.lookup.partialstring=${ +CarbonCommonConstants Review comment: changed for all places ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestNIQueryWithIndex.scala ## @@ -188,14 +191,17 @@ class TestNIQueryWithIndex extends QueryTest with BeforeAndAfterAll{ val ch15 = sql("select count(*) from seccust where c_phone='25-989-741-2988' and c_mktsegment like 'BUI%LDING'") // equals on c_phone of I1, I2 & (length & startsWith & endswith) on c_mktsegment of I2 so SI - Yes - assert(checkSIColumnsSize(ch15, 3)) //size = EqualTo on c_phone, length, StartsWith + assert(checkSIColumnsSize(ch15, 3)) // size = EqualTo on c_phone, length, StartsWith val ch16 = sql("select * from seccust where c_phone='25-989-741-2988'") // Query has EqualTo so SI - Yes assert(checkSIColumnsSize(ch16, 1)) // size = EqualTo -} finally{ - sql(s"set carbon.si.lookup.partialstring=${CarbonCommonConstants.ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT}") +} finally { + sql(s"set carbon.si.lookup.partialstring=${ +CarbonCommonConstants + .ENABLE_SI_LOOKUP_PARTIALSTRING_DEFAULT Review comment: changed for all places ## File path: index/secondary-index/src/test/scala/org/apache/carbondata/spark/testsuite/secondaryindex/TestSIWithSecondryIndex.scala