vaquarkhan commented on issue #4055: URL: https://github.com/apache/amoro/issues/4055#issuecomment-3796127462
**Root Cause** The system currently experiences a "noisy neighbor" effect during minor compaction. In CommonPartitionEvaluator.isMinorNecessary(), the reachMinorInterval check depends on a table-level timestamp (lastMinorOptimizingTime). Since DefaultTableRuntime.completeProcess() updates this global timestamp whenever any partition completes an optimization, high-traffic partitions effectively "starve" quiet ones. The global clock is reset so frequently that the planTime - lastMinorOptimizingTime delta never reaches the required interval for partitions containing only a few small files. **Implementation Plan** We will shift from global tracking to per-partition tracking to ensure scheduling fairness. **State Updates:** We should add Map<String, Long> lastMinorOptimizingTimeByPartition to TableRuntimeOptimizingState. On MINOR run completion, update the map for each involved partition path using the planTime. Retain the table-level lastMinorOptimizingTime as a fallback/default. **Planner Logic:** During the planning phase (where partition paths for getFromSequence / getToSequence are resolved), the planner will perform the map lookup. The planner will resolve the specific lastMinorOptimizingTime for that partition (defaulting to the table-level value if missing). This resolved value is then passed directly into the CommonPartitionEvaluator constructor. **Backward Compatibility:** Missing map entries will automatically fall back to the table-level timestamp, ensuring no regressions for existing data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
