ajantha-bhat commented on a change in pull request #3686:
[CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple
application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405437861
##########
File path:
core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -727,32 +730,54 @@ public TableSegmentRefresher
getTableSegmentRefresher(CarbonTable table) {
private Map<String, Boolean> manualSegmentRefresh = new HashMap<>();
TableSegmentRefresher(CarbonTable table) {
- SegmentUpdateStatusManager statusManager = new
SegmentUpdateStatusManager(table);
- SegmentUpdateDetails[] updateStatusDetails =
statusManager.getUpdateStatusDetails();
- for (SegmentUpdateDetails updateDetails : updateStatusDetails) {
- UpdateVO updateVO =
statusManager.getInvalidTimestampRange(updateDetails.getSegmentName());
+ SegmentStatusManager segmentStatusManager =
+ new SegmentStatusManager(table.getAbsoluteTableIdentifier());
+ List<Segment> validSegments;
+ try {
+ validSegments =
segmentStatusManager.getValidAndInvalidSegments().getValidSegments();
+ } catch (IOException e) {
+ LOGGER.error("Error while getting the valid segments.", e);
+ throw new RuntimeException(e);
+ }
+ for (Segment segment : validSegments) {
+ UpdateVO updateVO =
+
SegmentUpdateStatusManager.getInvalidTimestampRange(segment.getLoadMetadataDetails());
SegmentRefreshInfo segmentRefreshInfo;
- if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null) {
- segmentRefreshInfo = new
SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0);
+ if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null
+ || segment.getSegmentFileName() != null) {
+ long segmentFileTimeStamp = FileFactory.getCarbonFile(CarbonTablePath
+ .getSegmentFilePath(table.getTablePath(),
segment.getSegmentFileName()))
+ .getLastModifiedTime();
+ segmentRefreshInfo =
+ new SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0,
segmentFileTimeStamp);
} else {
- segmentRefreshInfo = new SegmentRefreshInfo(0L, 0);
+ segmentRefreshInfo = new SegmentRefreshInfo(0L, 0, 0L);
Review comment:
So, old store (without having segment file). This cache clean up fail ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
