ajantha-bhat commented on a change in pull request #3677: [wip]Fix segment cache issue with parallel spark applications on same store URL: https://github.com/apache/carbondata/pull/3677#discussion_r398315705
########## File path: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ########## @@ -369,6 +379,33 @@ private void modifyColumnSchemaForSortColumn(ColumnSchema columnSchema, boolean return tableBlockIndexUniqueIdentifiers; } + /** + * This case is added for a case where, there are two applications running, and in one application + * operations happened like SI rebuild, update or delete case, then the cache should be updated as + * well. The cache updation happens for same application, but other application may fail to query + * or may give wrong result. Since we overwrite the segment file in these scenarios, check the + * timestamp, and if modified, clear from the cache. + */ + private void clearSegmentMapIfSegmentUpdated(String latestSegmentFilePath, Segment segment) { + SegmentBlockIndexInfo segmentBlockIndexInfo = segmentMap.get(segment.getSegmentNo()); Review comment: As discussed, we can use SegmentRefreshInfo interface in this scenario (no need to list index file, can get info from segment file) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services