ajantha-bhat commented on a change in pull request #3907:
URL: https://github.com/apache/carbondata/pull/3907#discussion_r481755396
##########
File path:
core/src/main/java/org/apache/carbondata/core/index/IndexStoreManager.java
##########
@@ -515,15 +516,16 @@ public TableSegmentRefresher
getTableSegmentRefresher(CarbonTable table) {
UpdateVO updateVO =
SegmentUpdateStatusManager.getInvalidTimestampRange(segment.getLoadMetadataDetails());
SegmentRefreshInfo segmentRefreshInfo;
- if ((updateVO != null && updateVO.getLatestUpdateTimestamp() != null)
- || segment.getSegmentFileName() != null) {
- long segmentFileTimeStamp;
+ if (updateVO.getLatestUpdateTimestamp() != null ||
segment.getSegmentFileName() != null) {
+ long segmentFileTimeStamp = 0L;
if (null != segment.getLoadMetadataDetails()) {
segmentFileTimeStamp =
segment.getLoadMetadataDetails().getLastModifiedTime();
} else {
- segmentFileTimeStamp = FileFactory.getCarbonFile(CarbonTablePath
- .getSegmentFilePath(table.getTablePath(),
segment.getSegmentFileName()))
- .getLastModifiedTime();
+ CarbonFile segmentFile = FileFactory.getCarbonFile(CarbonTablePath
+ .getSegmentFilePath(table.getTablePath(),
segment.getSegmentFileName()));
+ if (segmentFile.exists()) {
Review comment:
In some customer scenarios, we have observed huge segment count (more
than 100K), so here file exist check for all the valid segment not even pruned
segment. so I feel this will slow down the query.
so, I suggest to analyze and make flow to enter `null !=
segment.getLoadMetadataDetails()` block.
I agree that segment refactor might solve this. It might take few months to
finish and we cannot slow down queries till then also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
