akashrn5 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor
segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405444105
##########
File path:
core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -727,32 +730,54 @@ public TableSegmentRefresher
getTableSegmentRefresher(CarbonTable table) {
private Map<String, Boolean> manualSegmentRefresh = new HashMap<>();
TableSegmentRefresher(CarbonTable table) {
- SegmentUpdateStatusManager statusManager = new
SegmentUpdateStatusManager(table);
- SegmentUpdateDetails[] updateStatusDetails =
statusManager.getUpdateStatusDetails();
- for (SegmentUpdateDetails updateDetails : updateStatusDetails) {
- UpdateVO updateVO =
statusManager.getInvalidTimestampRange(updateDetails.getSegmentName());
+ SegmentStatusManager segmentStatusManager =
+ new SegmentStatusManager(table.getAbsoluteTableIdentifier());
+ List<Segment> validSegments;
+ try {
+ validSegments =
segmentStatusManager.getValidAndInvalidSegments().getValidSegments();
+ } catch (IOException e) {
+ LOGGER.error("Error while getting the valid segments.", e);
+ throw new RuntimeException(e);
+ }
+ for (Segment segment : validSegments) {
+ UpdateVO updateVO =
+
SegmentUpdateStatusManager.getInvalidTimestampRange(segment.getLoadMetadataDetails());
SegmentRefreshInfo segmentRefreshInfo;
- if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null) {
- segmentRefreshInfo = new
SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0);
+ if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null
+ || segment.getSegmentFileName() != null) {
+ long segmentFileTimeStamp = FileFactory.getCarbonFile(CarbonTablePath
+ .getSegmentFilePath(table.getTablePath(),
segment.getSegmentFileName()))
+ .getLastModifiedTime();
+ segmentRefreshInfo =
+ new SegmentRefreshInfo(updateVO.getLatestUpdateTimestamp(), 0,
segmentFileTimeStamp);
} else {
- segmentRefreshInfo = new SegmentRefreshInfo(0L, 0);
+ segmentRefreshInfo = new SegmentRefreshInfo(0L, 0, 0L);
Review comment:
no it wont fail, basically this flow wonttake care of that, because for old
store we operate based on directly index files, and find them in lrucache
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
