[GitHub] [carbondata] CarbonDataQA2 commented on pull request #3988: [CARBONDATA-4037] Improve the table status and segment file writing

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-788659085


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5105/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-788656644


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5103/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788605734


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3339/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788600828


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5098/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-788600301


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3341/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-788599749


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3343/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4078: [CARBONDATA-4075] Using withEvents instead of fireEvent

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4078:
URL: https://github.com/apache/carbondata/pull/4078#issuecomment-788596393


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5101/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4096: [CARBONDATA-4133] Concurrent Insert Overwrite with static partition on Index server fails

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4096:
URL: https://github.com/apache/carbondata/pull/4096#issuecomment-788594583


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5099/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-788593496


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5100/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #3988: [CARBONDATA-4037] Improve the table status and segment file writing

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-788593208


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3345/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4045: ci_test

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4045:
URL: https://github.com/apache/carbondata/pull/4045#issuecomment-788592826


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5104/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4078: [CARBONDATA-4075] Using withEvents instead of fireEvent

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4078:
URL: https://github.com/apache/carbondata/pull/4078#issuecomment-788592710


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3342/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4098: [WIP] UT with index server

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4098:
URL: https://github.com/apache/carbondata/pull/4098#issuecomment-788592696


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5097/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4098: [WIP] UT with index server

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4098:
URL: https://github.com/apache/carbondata/pull/4098#issuecomment-788592160


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3338/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4096: [CARBONDATA-4133] Concurrent Insert Overwrite with static partition on Index server fails

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4096:
URL: https://github.com/apache/carbondata/pull/4096#issuecomment-788591088


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3340/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4045: ci_test

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4045:
URL: https://github.com/apache/carbondata/pull/4045#issuecomment-788556433


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3344/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-788052788


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3762/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-788049572


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5527/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4098: [WIP] UT with index server

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4098:
URL: https://github.com/apache/carbondata/pull/4098#issuecomment-787995219


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5526/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4098: [WIP] UT with index server

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4098:
URL: https://github.com/apache/carbondata/pull/4098#issuecomment-787993168


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3761/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] vikramahuja1001 removed a comment on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


vikramahuja1001 removed a comment on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-787906686


   @akashrn5 @VenuReddy2103 @ajantha-bhat , can this PR be merged?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #3988: [CARBONDATA-4037] Improve the table status and segment file writing

2021-03-01 Thread GitBox


ShreelekhyaG commented on a change in pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#discussion_r584683576



##
File path: 
core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java
##
@@ -440,6 +393,73 @@ public boolean accept(CarbonFile file) {
 return null;
   }
 
+  /**
+   * Get old and invalid files which have already been merged to a mergeindex 
file.In segment folder
+   * we may have both .index files and .mergeindex files, as we are not 
deleting index files
+   * immediately for old tables, this method reads mergeindex file and adds 
mapped index files to a
+   * list and returns.If more than one mergeindex file is present, considers 
the latest one as valid
+   * Ex: We have 3 files in segment. Segment0/ 1.index , 1.mergeindex file, 
1.carbondata.
+   * 1.index is merged to 1.mergeindex. Here it returns merged index file - 
1.index.
+   */
+  public static Set getInvalidAndMergedIndexFiles(List 
indexFiles)
+  throws IOException {
+SegmentIndexFileStore indexFileStore = new SegmentIndexFileStore();
+Set mergedAndInvalidIndexFiles = new HashSet<>();
+long lastModifiedTime = 0L;
+String validIndexFile = null;
+List mergeIndexFileNames = new ArrayList<>();
+for (String indexFile : indexFiles) {
+  if (indexFile.endsWith(CarbonTablePath.MERGE_INDEX_FILE_EXT)) {
+indexFileStore.readMergeFile(indexFile);
+Map> carbonMergeFileToIndexFilesMap =

Review comment:
   Done

##
File path: 
core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##
@@ -290,6 +290,7 @@ public static boolean 
updateTableMetadataStatus(Set updatedSegmentsList
 
 LoadMetadataDetails[] listOfLoadFolderDetailsArray =
 SegmentStatusManager.readLoadMetadata(metaDataFilepath);
+boolean isUpdateRequired = false;

Review comment:
   Done

##
File path: 
integration/spark/src/main/scala/org/apache/spark/rdd/CarbonMergeFilesRDD.scala
##
@@ -185,13 +193,23 @@ object CarbonMergeFilesRDD {
 val readPath: String = 
CarbonTablePath.getSegmentFilesLocation(tablePath) +
CarbonCommonConstants.FILE_SEPARATOR + 
segmentId + "_" +
segmentFileNameToSegmentIdMap.get(segmentId) + 
".tmp"
+val uuid = String.valueOf(System.currentTimeMillis)
+val newSegmentFileName = 
SegmentFileStore.genSegmentFileName(segmentId, uuid)
 // Merge all partition files into a single file.
-val segmentFileName: String = SegmentFileStore
-  .genSegmentFileName(segmentId, 
segmentFileNameToSegmentIdMap.get(segmentId))
-SegmentFileStore
+val segmentFile = SegmentFileStore
   .mergeSegmentFiles(readPath,
-segmentFileName,
+newSegmentFileName,
 CarbonTablePath.getSegmentFilesLocation(tablePath))
+if (segmentFile != null) {
+  val sfs = new SegmentFileStore(tablePath, newSegmentFileName +
+CarbonTablePath.SEGMENT_EXT)
+  val status = SegmentFileStore.updateTableStatusFile(carbonTable, 
segmentId,

Review comment:
   this flow is called when mergeIndex on old tables. Update happens only 
once.

##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/events/MergeIndexEventListener.scala
##
@@ -140,19 +141,47 @@ class MergeIndexEventListener extends 
OperationEventListener with Logging {
   .get
   .filterNot(streamingSegment.contains(_))
   }
+validSegments.foreach { segment =>
+  if (segmentsToMerge.contains(segment.getSegmentNo)) {
+val segmentFile = segment.getSegmentFileName
+val sfs = new SegmentFileStore(carbonMainTable.getTablePath, 
segmentFile)

Review comment:
   Done.

##
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/events/AlterTableMergeIndexSIEventListener.scala
##
@@ -85,7 +87,21 @@ class AlterTableMergeIndexSIEventListener
   .asScala
 val validSegmentIds: mutable.Buffer[String] = 
mutable.Buffer[String]()
 validSegments.foreach { segment =>
-  validSegmentIds += segment.getSegmentNo
+  val segmentFile = segment.getSegmentFileName
+  val sfs = new 
SegmentFileStore(indexCarbonTable.getTablePath, segmentFile)

Review comment:
   Done.

##
File path: 
core/src/main/java/org/apache/carbondata/core/writer/CarbonIndexFileMergeWriter.java
##
@@ -129,8 +127,13 @@ private String mergeCarbonIndexFilesOfSegment(String 
segmentId,
 String partitionTempPath = "";
 for (String partition : partitionInfo) {
   if (partitionPath.equalsIgnoreCase(partition)) {
-partitionTempPath = partition + "/" + tempFolderPath;
-break;
+if (tempFolderPath != 

[GitHub] [carbondata] vikramahuja1001 commented on pull request #4072: [CARBONDATA-4110] Support clean files dry run operation and show statistics after clean files operation

2021-03-01 Thread GitBox


vikramahuja1001 commented on pull request #4072:
URL: https://github.com/apache/carbondata/pull/4072#issuecomment-787906686


   @akashrn5 @VenuReddy2103 @ajantha-bhat , can this PR be merged?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ShreelekhyaG opened a new pull request #4098: [WIP] UT with index server

2021-03-01 Thread GitBox


ShreelekhyaG opened a new pull request #4098:
URL: https://github.com/apache/carbondata/pull/4098


### Why is this PR needed?
   enable to run UT with index server.

### What changes were proposed in this PR?
   usage of flag `useIndexServer`. excluded some of the test cases to not run 
with index server.
   
### Does this PR introduce any user interface change?
- No
   
### Is any new testcase added?
- No
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787768063


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3760/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4086: [CARBONDATA-4115] Successful load and insert will return segment ID

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #4086:
URL: https://github.com/apache/carbondata/pull/4086#issuecomment-787765521


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5525/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA2 commented on pull request #3988: [CARBONDATA-4037] Improve the table status and segment file writing

2021-03-01 Thread GitBox


CarbonDataQA2 commented on pull request #3988:
URL: https://github.com/apache/carbondata/pull/3988#issuecomment-787744640


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5524/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org