Hello March, I agree to take a hotfix for data deletion in loading and compaction flow, +1.
Deleting the INSERT_IN_PROGERSS and INSERT_OVERWRITE_IN_PROGRESS is a dangerous activity, so these two kinds of segments should not be automatically deleted. As for MARKED_FOR_DELETE and COMPACTED status segments, these are stale segments, but we can keep them in the file system until the user/admin calls clean file action manually. Since the deletion requires the precision of the table status. So my opinion is to remove all the automatic clean steps in loading/compaction flow first to protect the data from being deleted accidentally. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
