[
https://issues.apache.org/jira/browse/CARBONDATA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayu Shen updated CARBONDATA-4347:
-----------------------------------
Description:
We printed a stack when delete multiple segments, got
!https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2022/6/6/s00494122/40313f6afbd24bdcbb2239bdaf054272/image.png!
When delete multi segments，{_}deletePhysicalPartition{_} will be called each
segment and
_deleteEmptyPartitionFolders_ will be called with each carbonindex file. But
location.getParent (is the partition directory) may be the same between two
segments. So there is repetitive action.
{code:java}
for (Map.Entry<String, List<String>> entry : locationMap.entrySet()) {
if (partitionSpecs != null) {
Path location = new Path(entry.getKey());
boolean exists = pathExistsInPartitionSpec(partitionSpecs, location);
if (!exists) {
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString()));
for (String carbonDataFile : entry.getValue()) {
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(carbonDataFile));
}
}
CarbonFile path =
FileFactory.getCarbonFile(location.getParent().toString());
deleteEmptyPartitionFolders(path);
}
} {code}
Try to collect all partition directories which are related to deleted segments,
delete empty directory after all the segments deleted.
was:
We printed a stack when delete multiple segments, got
!https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2022/6/6/s00494122/40313f6afbd24bdcbb2239bdaf054272/image.png!
When delete multi segments，{_}deletePhysicalPartition{_} will be called each
segment and
_deleteEmptyPartitionFolders_ will be called with each carbonindex file. But
location.getParent (is the partition directory) may be the same between two
segments. So there is repetitive action.
{code:java}
for (Map.Entry<String, List<String>> entry : locationMap.entrySet()) {
if (partitionSpecs != null) {
Path location = new Path(entry.getKey());
boolean exists = pathExistsInPartitionSpec(partitionSpecs, location);
if (!exists) {
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString()));
for (String carbonDataFile : entry.getValue()) {
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(carbonDataFile));
}
}
CarbonFile path =
FileFactory.getCarbonFile(location.getParent().toString());
deleteEmptyPartitionFolders(path);
}
} {code}
Try to collect all partition directories which is relative to deleted segments,
delete them after segments deleted.
> Improve performance when delete empty partition directory
> ---------------------------------------------------------
>
> Key: CARBONDATA-4347
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4347
> Project: CarbonData
> Issue Type: Improvement
> Reporter: Jiayu Shen
> Priority: Major
> Fix For: 2.3.1
>
>
> We printed a stack when delete multiple segments, got
> !https://clouddevops.huawei.com/vision-file-storage/api/file/download/upload-v2/2022/6/6/s00494122/40313f6afbd24bdcbb2239bdaf054272/image.png!
>
> When delete multi segments，{_}deletePhysicalPartition{_} will be called each
> segment and
> _deleteEmptyPartitionFolders_ will be called with each carbonindex file. But
> location.getParent (is the partition directory) may be the same between two
> segments. So there is repetitive action.
>
> {code:java}
> for (Map.Entry<String, List<String>> entry : locationMap.entrySet()) {
> if (partitionSpecs != null) {
> Path location = new Path(entry.getKey());
> boolean exists = pathExistsInPartitionSpec(partitionSpecs, location);
> if (!exists) {
>
> FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString()));
> for (String carbonDataFile : entry.getValue()) {
>
> FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(carbonDataFile));
> }
> }
> CarbonFile path =
> FileFactory.getCarbonFile(location.getParent().toString());
> deleteEmptyPartitionFolders(path);
> }
> } {code}
> Try to collect all partition directories which are related to deleted
> segments, delete empty directory after all the segments deleted.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
