voonhous opened a new issue, #19881:
URL: https://github.com/apache/hudi/issues/19881
### Bug Description
**What happened:**
`CompactionAdminClient.unscheduleCompactionFileId` removes more operations
from the compaction plan than the one file group it was asked to unschedule.
The filter at master 6ac27904, `CompactionAdminClient.java:149-150`, is
```java
plan.getOperations().stream().filter(op ->
(!op.getFileId().equals(fgId.getFileId())) &&
(!op.getPartitionPath().equals(fgId.getPartitionPath())))
```
which keeps only the operations that differ in both file id and partition
path. Every other pending operation in the same partition is dropped from the
rewritten plan, and its file groups are silently never compacted by that plan.
The intended condition is `!(sameFileId && samePartition)`.
Reachable from the `compaction unscheduleFileId` command in hudi-cli
(through `SparkMain.doCompactUnscheduleFile`) and from any direct caller of
`CompactionAdminClient`.
**What you expected:**
Only the operation for the given file group is removed; the other operations
in the same partition stay in the plan.
**Steps to reproduce:**
1. MOR table with a pending compaction plan holding two operations in the
same partition (file groups A and B).
2. `unscheduleCompactionFileId(fgId(A), skipValidation=true, dryRun=false)`.
3. Reload the plan: B is gone as well.
Found while writing
`TestCompactionCommand.testSparkMainCompactUnscheduleFile` in #19877; that test
only asserts the target file group is gone so it passes with and without the
bug. A regression test belongs in `TestCompactionAdminClient` next to the fix.
Related, same class: `unscheduleCompactionPlan` ignores its `skipValidation`
and `parallelism` arguments and `repairCompaction` ignores `dryRun` and
discards its validation result, so `compaction repair` cannot repair anything.
Worth deciding whether those arguments should be removed or the log-file rename
logic restored.
### Environment
**Hudi version:** master (6ac27904), also present in 1.0.x and 0.15.x
**Query engine:** hudi-cli / any CompactionAdminClient caller
**Relevant configs:** none
### Logs and Stack Trace
No exception; the rewritten `.compaction.requested` plan is missing
operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
