[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3811: [CARBONDATA-3874] segment mismatch between maintable and SI table when load with concurrency

2020-07-01 Thread GitBox


Indhumathi27 commented on a change in pull request #3811:
URL: https://github.com/apache/carbondata/pull/3811#discussion_r448251601



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##
@@ -390,8 +391,15 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
   }
   val indexTablePath = CarbonTablePath
 
.getMetadataPath(tableInfo.getOrCreateAbsoluteTableIdentifier.getTablePath)
+  val metaPathMainTbl: String = carbonTable.getMetadataPath

Review comment:
   If metaPathMainTbl and  metaPathSI variable used in one place only, can 
replace with  
`SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)`

##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/SILoadEventListenerForFailedSegments.scala
##
@@ -78,20 +78,29 @@ class SILoadEventListenerForFailedSegments extends 
OperationEventListener with L
   .getTableMetadata(TableIdentifier(indexTableName,
 Some(carbonLoadModel.getDatabaseName))).storage.properties
   .getOrElse("isSITableEnabled", "true").toBoolean
+val indexTable = metaStore
+  .lookupRelation(Some(carbonLoadModel.getDatabaseName), 
indexTableName)(
+sparkSession)
+  .asInstanceOf[CarbonRelation]
+  .carbonTable
 
-if (!isLoadSIForFailedSegments) {
+val metaPathMainTbl: String = carbonTable.getMetadataPath
+val listOfLoadFolderDetailsArrayMainTbl: 
Array[LoadMetadataDetails] =

Review comment:
   Handle Same as above comment





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 commented on a change in pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


akkio-97 commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r448303815



##
File path: 
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedVectorResultCollector.java
##
@@ -98,6 +98,14 @@ void prepareDimensionAndMeasureColumnVectors() {
 columnVectorInfo.dimension = queryDimensions[i];
 columnVectorInfo.ordinal = 
queryDimensions[i].getDimension().getOrdinal();
 allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
+  } else if (queryDimensions[i].getDimension().isComplex()) {
+ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
+complexList.add(columnVectorInfo);
+columnVectorInfo.dimension = queryDimensions[i];
+columnVectorInfo.ordinal = 
queryDimensions[i].getDimension().getOrdinal();
+columnVectorInfo.genericQueryType =
+
executionInfo.getComlexDimensionInfoMap().get(columnVectorInfo.ordinal);
+allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;

Review comment:
   done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 commented on a change in pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


akkio-97 commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r448303815



##
File path: 
core/src/main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedVectorResultCollector.java
##
@@ -98,6 +98,14 @@ void prepareDimensionAndMeasureColumnVectors() {
 columnVectorInfo.dimension = queryDimensions[i];
 columnVectorInfo.ordinal = 
queryDimensions[i].getDimension().getOrdinal();
 allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;
+  } else if (queryDimensions[i].getDimension().isComplex()) {
+ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
+complexList.add(columnVectorInfo);
+columnVectorInfo.dimension = queryDimensions[i];
+columnVectorInfo.ordinal = 
queryDimensions[i].getDimension().getOrdinal();
+columnVectorInfo.genericQueryType =
+
executionInfo.getComlexDimensionInfoMap().get(columnVectorInfo.ordinal);
+allColumnInfo[queryDimensions[i].getOrdinal()] = columnVectorInfo;

Review comment:
   Why should that block be removed? 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652346497


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1548/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…

2020-07-01 Thread GitBox


Indhumathi27 commented on a change in pull request #3817:
URL: https://github.com/apache/carbondata/pull/3817#discussion_r448249998



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
##
@@ -766,13 +766,13 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
   throw new MalformedCarbonCommandException("Invalid table properties")
 }
 if (options.isBucketingEnabled) {
-  if (options.bucketNumber.toString.contains("-") ||
-  options.bucketNumber.toString.contains("+") ||  options.bucketNumber 
== 0) {
+  if (options.bucketNumber == None || 
options.bucketNumber.get.toString.contains("-") ||
+  options.bucketNumber.get.toString.contains("+") ||  
options.bucketNumber.get == 0) {

Review comment:
   Please add testcases for these scenarios





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652375948


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3287/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3811: [CARBONDATA-3874] segment mismatch between maintable and SI table when load with concurrency

2020-07-01 Thread GitBox


Indhumathi27 commented on a change in pull request #3811:
URL: https://github.com/apache/carbondata/pull/3811#discussion_r448252248



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/SILoadEventListenerForFailedSegments.scala
##
@@ -78,20 +78,29 @@ class SILoadEventListenerForFailedSegments extends 
OperationEventListener with L
   .getTableMetadata(TableIdentifier(indexTableName,
 Some(carbonLoadModel.getDatabaseName))).storage.properties
   .getOrElse("isSITableEnabled", "true").toBoolean
+val indexTable = metaStore
+  .lookupRelation(Some(carbonLoadModel.getDatabaseName), 
indexTableName)(
+sparkSession)
+  .asInstanceOf[CarbonRelation]
+  .carbonTable
 
-if (!isLoadSIForFailedSegments) {
+val metaPathMainTbl: String = carbonTable.getMetadataPath
+val listOfLoadFolderDetailsArrayMainTbl: 
Array[LoadMetadataDetails] =

Review comment:
   Better, can move getting load meta details of main table and SI to a 
method in a scala class like CarbonIndexUtil and reuse





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652377422


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1550/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652402581


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1549/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652402955


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3286/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652346179


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3285/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652497582


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3288/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akkio-97 commented on pull request #3773: [CARBONDATA-3830]Presto complex columns read support

2020-07-01 Thread GitBox


akkio-97 commented on pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#issuecomment-652417278


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652234633


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3282/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652232880


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1545/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] akashrn5 commented on a change in pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception

2020-07-01 Thread GitBox


akashrn5 commented on a change in pull request #3808:
URL: https://github.com/apache/carbondata/pull/3808#discussion_r448761040



##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##
@@ -121,10 +124,17 @@ object Compactor {
   segmentIdToLoadStartTimeMapping(validSegments.head),
   SegmentStatus.SUCCESS,
   carbonLoadModelForMergeDataFiles.getFactTimeStamp, 
rebuiltSegments.toList.asJava)
-
+siCompactionIndexList ::= indexCarbonTable
   } catch {
 case ex: Exception =>
   LOGGER.error(s"Compaction failed for SI table 
${secondaryIndex.indexName}", ex)
+  siCompactionIndexList.foreach { indexCarbonTable =>
+sparkSession.sql(
+  s"""ALTER TABLE ${carbonLoadModel.getDatabaseName}.${

Review comment:
   please correct the format for sparksession.sql(""), please check the 
other files where we do alter set for SI, can follow the same

##
File path: 
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AbstractDFSCarbonFile.java
##
@@ -455,7 +455,7 @@ public boolean createNewLockFile() throws IOException {
 try {
   listStatus = fileSystem.listStatus(path);
 } catch (IOException e) {
-  LOGGER.error("Exception occured: " + e.getMessage(), e);

Review comment:
   revert this change if not rerquired

##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##
@@ -121,10 +124,17 @@ object Compactor {
   segmentIdToLoadStartTimeMapping(validSegments.head),
   SegmentStatus.SUCCESS,
   carbonLoadModelForMergeDataFiles.getFactTimeStamp, 
rebuiltSegments.toList.asJava)
-
+siCompactionIndexList ::= indexCarbonTable
   } catch {
 case ex: Exception =>
   LOGGER.error(s"Compaction failed for SI table 
${secondaryIndex.indexName}", ex)
+  siCompactionIndexList.foreach { indexCarbonTable =>

Review comment:
   please add a comment here, what you are doing in which scenario





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3817:
URL: https://github.com/apache/carbondata/pull/3817#issuecomment-652542001


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1552/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3817: [CARBONDATA-3845] Bucket table creation fails with exception for empt…

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3817:
URL: https://github.com/apache/carbondata/pull/3817#issuecomment-652542851


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3289/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


nihal0107 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652265976


   retest



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3808:
URL: https://github.com/apache/carbondata/pull/3808#issuecomment-652278446


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3283/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3808: [CARBONDATA-3873] Secondary index compaction with maintable clean files causing exception

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3808:
URL: https://github.com/apache/carbondata/pull/3808#issuecomment-652279732


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1546/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 removed a comment on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


nihal0107 removed a comment on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652265976


   retest



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3818: [Carbondata-3883] Added filtering for the deleted rows for local dictionary fields

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3818:
URL: https://github.com/apache/carbondata/pull/3818#issuecomment-652266651


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3281/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] nihal0107 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


nihal0107 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652267181


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3818: [Carbondata-3883] Added filtering for the deleted rows for local dictionary fields

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3818:
URL: https://github.com/apache/carbondata/pull/3818#issuecomment-652269811


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1544/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-652297566


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1547/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3819: [CARBONDATA-3855]support carbon SDK to load data from different files

2020-07-01 Thread GitBox


CarbonDataQA1 commented on pull request #3819:
URL: https://github.com/apache/carbondata/pull/3819#issuecomment-65237


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3284/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org