This is an automated email from the ASF dual-hosted git repository.
jshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new fc6e11ea99 [#10163] improvement(optimizer): improve recommender log
messaging (#10121)
fc6e11ea99 is described below
commit fc6e11ea993f9d5876cda1156ccf777d44031ca5
Author: Roman Horilyi <[email protected]>
AuthorDate: Wed Mar 4 09:58:56 2026 +0200
[#10163] improvement(optimizer): improve recommender log messaging (#10121)
### What changes were proposed in this pull request?
Improve log messaging for optimizer
[Recommender#recommendForStrategyType](https://github.com/apache/gravitino/blob/969f7be707334b0a11a64846fbf014ca062d3a4a/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/Recommender.java#L134C15-L134C39)
execution.
### Why are the changes needed?
Currently there're several issues when `recommendForStrategyType` method
is run:
1. it's not clear why there's no output for candidate tables that
doesn't match triggering criteria
2.
[CompactionJobContext](https://github.com/apache/gravitino/blob/main/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/handler/compaction/CompactionJobContext.java)
produces verbose output for cases w/ many partitions and large tables.
Here are the sample log messages when running `recommendForStrategyType`
w/ multiple test identifiers for `compaction` strategy type:
#### Without this patch:
```
2026-02-19 22:40:12.753 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:183)]
- Recommend strategy compactionSmallPartitionFiles for identifiers
[generic.ad.ads_hourly_onsite_insertion_data,
generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
generic.ad.ads_hourly_engagement_sampled_table_iceberg_insertion_id,
generic.ad.ads_hourly_l1_sampled_data_table,
generic.ad.ads_hourly_video_engagement_sampl [...]
2026-02-19 22:40:14.505 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:201)]
- Recommend strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid score: 45548
2026-02-19 22:40:15.859 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.job.NoopJobSubmitter.submitJob(NoopJobSubmitter.java:43)]
- NoopJobSubmitter submitJob: template=compaction,
identifier=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobExecuteContext=CompactionJobContext(name=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobOptions={}, jobTemplateName=compaction,
columns=[org.apache.gravitino.client.GenericColumn@f434c [...]
2026-02-19 22:40:15.875 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForStrategyType(Recommender.java:158)]
- Submit job for strategy compactionSmallPartitionFiles with context
CompactionJobContext(name=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobOptions={}, jobTemplateName=compaction,
columns=[org.apache.gravitino.client.GenericColumn@f434c029,
org.apache.gravitino.client.GenericColumn@510cb662, org.apache.graviti [...]
```
#### With this patch:
```
2026-02-19 22:26:16.679 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:183)]
- Recommend strategy compactionSmallPartitionFiles for identifiers
[generic.ad.ads_hourly_onsite_insertion_data,
generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
generic.ad.ads_hourly_engagement_sampled_table_iceberg_insertion_id,
generic.ad.ads_hourly_l1_sampled_data_table,
generic.ad.ads_hourly_video_engagement_sampl [...]
2026-02-19 22:26:18.154 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:191)]
- Skip strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_hourly_onsite_insertion_data because strategy handler trigger
condition is not met
2026-02-19 22:26:18.444 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:205)]
- Recommend strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid score: 45548
2026-02-19 22:26:19.010 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:191)]
- Skip strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_hourly_engagement_sampled_table_iceberg_insertion_id because
strategy handler trigger condition is not met
2026-02-19 22:26:19.181 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:191)]
- Skip strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_hourly_l1_sampled_data_table because strategy handler trigger
condition is not met
2026-02-19 22:26:19.736 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForOneStrategy(Recommender.java:191)]
- Skip strategy compactionSmallPartitionFiles for identifier
generic.ad.ads_hourly_video_engagement_sampled_table_iceberg because strategy
handler trigger condition is not met
2026-02-19 22:26:19.736 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.job.NoopJobSubmitter.submitJob(NoopJobSubmitter.java:43)]
- NoopJobSubmitter submitJob: template=compaction,
identifier=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobExecuteContext=CompactionJobContext{name=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobTemplateName='compaction', jobOptions={}, columnCount=5041,
partitioningCount=4, partitionCount=100}
2026-02-19 22:26:19.743 INFO [main]
[org.apache.gravitino.maintenance.optimizer.recommender.Recommender.recommendForStrategyType(Recommender.java:158)]
- Submit job for strategy compactionSmallPartitionFiles with context
CompactionJobContext{name=generic.ad.ads_daily_conversion_sampled_yellow_box_iceberg_userid,
jobTemplateName='compaction', jobOptions={}, columnCount=5041,
partitioningCount=4, partitionCount=100}
```
Fix: #10163
### Does this PR introduce _any_ user-facing change?
1. Adds extra INFO logs in `gravitino-optimizer` log file for not
matching candidates.
2. Removes redundant fields from `CompactionJobContext#toString` method
to improve readability.
### How was this patch tested?
- [x] Manual verification w/ `./gravitino-optimizer.sh -type
recommend_strategy_type` CLI command
Co-authored-by: Roman Horilyi <[email protected]>
---
.../optimizer/recommender/Recommender.java | 10 ++++++++++
.../handler/compaction/CompactionJobContext.java | 23 ++++++++++++++++++++--
2 files changed, 31 insertions(+), 2 deletions(-)
diff --git
a/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/Recommender.java
b/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/Recommender.java
index c6c66ffec6..8da6a52cf3 100644
---
a/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/Recommender.java
+++
b/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/Recommender.java
@@ -148,6 +148,12 @@ public class Recommender implements AutoCloseable {
String strategyName = entry.getKey();
List<StrategyEvaluation> evaluations =
recommendForOneStrategy(entry.getValue(), strategyName);
+
+ if (evaluations.isEmpty()) {
+ LOG.info("No evaluations for strategy {}", strategyName);
+ continue;
+ }
+
for (StrategyEvaluation evaluation : evaluations) {
results.add(toRecommendationResult(strategyName, evaluation, ""));
}
@@ -260,6 +266,10 @@ public class Recommender implements AutoCloseable {
for (NameIdentifier identifier : identifiers) {
StrategyHandler strategyHandler = loadStrategyHandler(strategy,
identifier);
if (!strategyHandler.shouldTrigger()) {
+ LOG.info(
+ "Skip strategy {} for identifier {} because strategy handler
trigger condition is not met",
+ strategyName,
+ identifier);
continue;
}
StrategyEvaluation evaluation = strategyHandler.evaluate();
diff --git
a/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/handler/compaction/CompactionJobContext.java
b/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/handler/compaction/CompactionJobContext.java
index f53000c455..4e36c4f1a3 100644
---
a/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/handler/compaction/CompactionJobContext.java
+++
b/maintenance/optimizer/src/main/java/org/apache/gravitino/maintenance/optimizer/recommender/handler/compaction/CompactionJobContext.java
@@ -23,7 +23,6 @@ import java.util.List;
import java.util.Map;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
-import lombok.ToString;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.maintenance.optimizer.api.common.PartitionPath;
import
org.apache.gravitino.maintenance.optimizer.api.recommender.JobExecutionContext;
@@ -31,7 +30,6 @@ import org.apache.gravitino.rel.Column;
import org.apache.gravitino.rel.expressions.transforms.Transform;
@RequiredArgsConstructor
-@ToString
public class CompactionJobContext implements JobExecutionContext {
private final NameIdentifier name;
private final Map<String, String> jobOptions;
@@ -54,4 +52,25 @@ public class CompactionJobContext implements
JobExecutionContext {
public String jobTemplateName() {
return jobTemplateName;
}
+
+ // `columns`, `partitioning` & `partitions` fields are not logged to make
toString() output more
+ // readable (especially for big tables). Only their length is logged instead.
+ @Override
+ public String toString() {
+ return "CompactionJobContext{"
+ + "name="
+ + name
+ + ", jobTemplateName='"
+ + jobTemplateName
+ + '\''
+ + ", jobOptions="
+ + jobOptions
+ + ", columnCount="
+ + (columns != null ? columns.length : 0)
+ + ", partitioningCount="
+ + (partitioning != null ? partitioning.length : 0)
+ + ", partitionCount="
+ + (partitions != null ? partitions.size() : 0)
+ + '}';
+ }
}
