[
https://issues.apache.org/jira/browse/HUDI-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793494#comment-17793494
]
sivabalan narayanan commented on HUDI-7051:
-------------------------------------------
hey [~vmaster] :
sorry I am bit confused.
as per master, filterPartitionPaths in DayBasedCompactionStrategy is as below
{code:java}
@Override
public List<String> filterPartitionPaths(HoodieWriteConfig writeConfig,
List<String> allPartitionPaths) {
return allPartitionPaths.stream().sorted(comparator)
.collect(Collectors.toList()).subList(0,
Math.min(allPartitionPaths.size(),
writeConfig.getTargetPartitionsPerDayBasedCompaction()));
} {code}
Only in
BoundedPartitionAwareCompactionStrategy.filterPartitionPaths I see the replace
operations.
But can you help me understand whats the issue in there. I understand
"dllr_date=2023/10/10" may not be an actual partition present physcially, but
thats interim state used for comparison and later we switch it back.
in other words.
if original partition is hypehnated.
dllr_date=2023-10-10 -> gets converted to "dllr_date=2023/10/10", and then
comparisons are performed to sort them and then converted back to
dllr_date=2023-10-10. So, not sure where is the bug here. can you throw some
light please
> Incorrect replace operation in compaction strategy filter
> ---------------------------------------------------------
>
> Key: HUDI-7051
> URL: https://issues.apache.org/jira/browse/HUDI-7051
> Project: Apache Hudi
> Issue Type: Bug
> Components: compaction
> Reporter: vmaster.cc
> Priority: Major
> Attachments: image-2023-11-08-16-01-46-166.png,
> image-2023-11-08-16-02-39-291.png
>
>
> There are some incorrect replace operation to sort all partition paths.
> {code:java}
> return allPartitionPaths.stream().map(partition -> partition.replace("/",
> "-"))
> .sorted(Comparator.reverseOrder()).map(partitionPath ->
> partitionPath.replace("-", "/")) {code}
> the hive partition before replace is dllr_date=2023-10-10, then after will
> convert to dllr_date=2023/10/10, this is an incorrect partition.
> # org.apache.hudi.table.action.compact.strategy.DayBasedCompactionStrategy
> #
> org.apache.hudi.table.action.compact.strategy.BoundedPartitionAwareCompactionStrategy
> #
> org.apache.hudi.table.action.compact.strategy.UnBoundedPartitionAwareCompactionStrategy
> !image-2023-11-08-16-02-39-291.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
