QiangCai commented on a change in pull request #3917:
URL: https://github.com/apache/carbondata/pull/3917#discussion_r513883725
##########
File path:
core/src/main/java/org/apache/carbondata/core/util/DeleteLoadFolders.java
##########
@@ -192,11 +208,17 @@ private static boolean
checkIfLoadCanBeDeleted(LoadMetadataDetails oneLoad,
}
private static boolean checkIfLoadCanBeDeletedPhysically(LoadMetadataDetails
oneLoad,
- boolean isForceDelete) {
+ boolean isForceDelete, AbsoluteTableIdentifier absoluteTableIdentifier) {
// Check if the segment is added externally and path is set then do not
delete it
if ((SegmentStatus.MARKED_FOR_DELETE == oneLoad.getSegmentStatus()
- || SegmentStatus.COMPACTED == oneLoad.getSegmentStatus()) &&
(oneLoad.getPath() == null
+ || SegmentStatus.COMPACTED == oneLoad.getSegmentStatus() ||
SegmentStatus
+ .INSERT_IN_PROGRESS == oneLoad.getSegmentStatus()) &&
(oneLoad.getPath() == null
Review comment:
about clean files, because cleaning different load status have different
risk level,
so in my opinion, different load status should have different expiration
time for keeping it at the original place.
1. MARKED_FOR_DELETE and COMPACTED => MaxQueryTimeout is enough
2. INSERT_IN_PROGRESS and INSERT_OVERWRITE_IN_PROGRESS => more than data
processing time(maybe 3 days)
if loading or compaction takes a long time which is longer than
LockingTimeout and MaxQueryTimeout, clean in_progress will have a big risk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
