[GitHub] [carbondata] akashrn5 commented on a change in pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception
akashrn5 commented on a change in pull request #3808:
URL: https://github.com/apache/carbondata/pull/3808#discussion_r448761040
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##
@@ -121,10 +124,17 @@ object Compactor {
segmentIdToLoadStartTimeMapping(validSegments.head),
SegmentStatus.SUCCESS,
carbonLoadModelForMergeDataFiles.getFactTimeStamp,
rebuiltSegments.toList.asJava)
-
+siCompactionIndexList ::= indexCarbonTable
} catch {
case ex: Exception =>
LOGGER.error(s"Compaction failed for SI table
${secondaryIndex.indexName}", ex)
+ siCompactionIndexList.foreach { indexCarbonTable =>
+sparkSession.sql(
+ s"""ALTER TABLE ${carbonLoadModel.getDatabaseName}.${
Review comment:
please correct the format for sparksession.sql(""), please check the
other files where we do alter set for SI, can follow the same
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AbstractDFSCarbonFile.java
##
@@ -455,7 +455,7 @@ public boolean createNewLockFile() throws IOException {
try {
listStatus = fileSystem.listStatus(path);
} catch (IOException e) {
- LOGGER.error("Exception occured: " + e.getMessage(), e);
Review comment:
revert this change if not rerquired
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##
@@ -121,10 +124,17 @@ object Compactor {
segmentIdToLoadStartTimeMapping(validSegments.head),
SegmentStatus.SUCCESS,
carbonLoadModelForMergeDataFiles.getFactTimeStamp,
rebuiltSegments.toList.asJava)
-
+siCompactionIndexList ::= indexCarbonTable
} catch {
case ex: Exception =>
LOGGER.error(s"Compaction failed for SI table
${secondaryIndex.indexName}", ex)
+ siCompactionIndexList.foreach { indexCarbonTable =>
Review comment:
please add a comment here, what you are doing in which scenario
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…
CarbonDataQA1 commented on pull request #3817: URL: https://github.com/apache/carbondata/pull/3817#issuecomment-652542851 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3289/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…
CarbonDataQA1 commented on pull request #3817: URL: https://github.com/apache/carbondata/pull/3817#issuecomment-652542001 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1552/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
CarbonDataQA1 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652497582 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3288/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
akkio-97 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652417278 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652402955 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3286/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652402581 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1549/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3773: [CARBONDATA-3830]Presto complex columns read support
akkio-97 commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r448303815
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedVectorResultCollector.java
##
@@ -98,6 +98,14 @@ void prepareDimensionAndMeasureColumnVectors() {
columnVectorInfo.dimension = queryDimensions[i];
columnVectorInfo.ordinal =
queryDimensions[i].getDimension().getOrdinal();
allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
+ } else if (queryDimensions[i].getDimension().isComplex()) {
+ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
+complexList.add(columnVectorInfo);
+columnVectorInfo.dimension = queryDimensions[i];
+columnVectorInfo.ordinal =
queryDimensions[i].getDimension().getOrdinal();
+columnVectorInfo.genericQueryType =
+
executionInfo.getComlexDimensionInfoMap().get(columnVectorInfo.ordinal);
+allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
Review comment:
Why should that block be removed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
CarbonDataQA1 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652377422 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1550/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
CarbonDataQA1 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652375948 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3287/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] akkio-97 commented on a change in pull request #3773: [CARBONDATA-3830]Presto complex columns read support
akkio-97 commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r448303815
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedVectorResultCollector.java
##
@@ -98,6 +98,14 @@ void prepareDimensionAndMeasureColumnVectors() {
columnVectorInfo.dimension = queryDimensions[i];
columnVectorInfo.ordinal =
queryDimensions[i].getDimension().getOrdinal();
allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
+ } else if (queryDimensions[i].getDimension().isComplex()) {
+ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
+complexList.add(columnVectorInfo);
+columnVectorInfo.dimension = queryDimensions[i];
+columnVectorInfo.ordinal =
queryDimensions[i].getDimension().getOrdinal();
+columnVectorInfo.genericQueryType =
+
executionInfo.getComlexDimensionInfoMap().get(columnVectorInfo.ordinal);
+allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
CarbonDataQA1 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652346497 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1548/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support
CarbonDataQA1 commented on pull request #3773: URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652346179 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3285/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3811: [CARBONDATA-3874] segment mismatch between maintable and SI table when load with concurrency
Indhumathi27 commented on a change in pull request #3811:
URL: https://github.com/apache/carbondata/pull/3811#discussion_r448252248
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/SILoadEventListenerForFailedSegments.scala
##
@@ -78,20 +78,29 @@ class SILoadEventListenerForFailedSegments extends
OperationEventListener with L
.getTableMetadata(TableIdentifier(indexTableName,
Some(carbonLoadModel.getDatabaseName))).storage.properties
.getOrElse("isSITableEnabled", "true").toBoolean
+val indexTable = metaStore
+ .lookupRelation(Some(carbonLoadModel.getDatabaseName),
indexTableName)(
+sparkSession)
+ .asInstanceOf[CarbonRelation]
+ .carbonTable
-if (!isLoadSIForFailedSegments) {
+val metaPathMainTbl: String = carbonTable.getMetadataPath
+val listOfLoadFolderDetailsArrayMainTbl:
Array[LoadMetadataDetails] =
Review comment:
Better, can move getting load meta details of main table and SI to a
method in a scala class like CarbonIndexUtil and reuse
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3811: [CARBONDATA-3874] segment mismatch between maintable and SI table when load with concurrency
Indhumathi27 commented on a change in pull request #3811:
URL: https://github.com/apache/carbondata/pull/3811#discussion_r448251601
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##
@@ -390,8 +391,15 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
}
val indexTablePath = CarbonTablePath
.getMetadataPath(tableInfo.getOrCreateAbsoluteTableIdentifier.getTablePath)
+ val metaPathMainTbl: String = carbonTable.getMetadataPath
Review comment:
If metaPathMainTbl and metaPathSI variable used in one place only, can
replace with
`SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)`
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/SILoadEventListenerForFailedSegments.scala
##
@@ -78,20 +78,29 @@ class SILoadEventListenerForFailedSegments extends
OperationEventListener with L
.getTableMetadata(TableIdentifier(indexTableName,
Some(carbonLoadModel.getDatabaseName))).storage.properties
.getOrElse("isSITableEnabled", "true").toBoolean
+val indexTable = metaStore
+ .lookupRelation(Some(carbonLoadModel.getDatabaseName),
indexTableName)(
+sparkSession)
+ .asInstanceOf[CarbonRelation]
+ .carbonTable
-if (!isLoadSIForFailedSegments) {
+val metaPathMainTbl: String = carbonTable.getMetadataPath
+val listOfLoadFolderDetailsArrayMainTbl:
Array[LoadMetadataDetails] =
Review comment:
Handle Same as above comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…
Indhumathi27 commented on a change in pull request #3817:
URL: https://github.com/apache/carbondata/pull/3817#discussion_r448249998
##
File path:
integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
##
@@ -766,13 +766,13 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
throw new MalformedCarbonCommandException("Invalid table properties")
}
if (options.isBucketingEnabled) {
- if (options.bucketNumber.toString.contains("-") ||
- options.bucketNumber.toString.contains("+") || options.bucketNumber
== 0) {
+ if (options.bucketNumber == None ||
options.bucketNumber.get.toString.contains("-") ||
+ options.bucketNumber.get.toString.contains("+") ||
options.bucketNumber.get == 0) {
Review comment:
Please add testcases for these scenarios
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-65237 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3284/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652297566 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1547/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception
CarbonDataQA1 commented on pull request #3808: URL: https://github.com/apache/carbondata/pull/3808#issuecomment-652279732 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1546/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception
CarbonDataQA1 commented on pull request #3808: URL: https://github.com/apache/carbondata/pull/3808#issuecomment-652278446 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3283/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3818: [Carbondata-3883] Added filtering for the deleted rows for local dictionary fields
CarbonDataQA1 commented on pull request #3818: URL: https://github.com/apache/carbondata/pull/3818#issuecomment-652269811 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1544/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
nihal0107 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652267181 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 removed a comment on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
nihal0107 removed a comment on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652265976 retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3818: [Carbondata-3883] Added filtering for the deleted rows for local dictionary fields
CarbonDataQA1 commented on pull request #3818: URL: https://github.com/apache/carbondata/pull/3818#issuecomment-652266651 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3281/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
nihal0107 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652265976 retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652234633 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3282/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files
CarbonDataQA1 commented on pull request #3819: URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652232880 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1545/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
