Zouxxyy opened a new pull request, #9169:
URL: https://github.com/apache/paimon/pull/9169
### Purpose
`CALL sys.compact(...)` is hard to observe from the driver log. The log does
not
tell which compaction path was taken, how large the plan was, or whether the
procedure finished. Worse, `compactUnAwareBucketTable` and
`compactDataEvolutionTable` logged the exact same message
`Task plan is empty, no compact job to execute.`, so the two paths could not
be
told apart in a log file.
This PR improves the driver-side logging of `CompactProcedure`:
- Log one summary line when the procedure starts (table, bucket mode, compact
strategy, order type, order columns, whether a partition filter is applied
and
the partition idle time), and one when it finishes with the elapsed time.
- Make every "nothing to do" message unique per path and include the table
name,
so `compactAwareBucketTable`, `compactUnAwareBucketTable`,
`compactDataEvolutionTable`, `sortCompactUnAwareBucketTable` and
`clusterIncrementalUnAwareBucketTable` can be distinguished.
- Log the plan size and the read parallelism before submitting the Spark
job, so
the plan can be correlated with the stage shown in the Spark UI.
- Log the number of commit messages just before committing.
- For data evolution compaction, label each planning round. The previous
`Task plan is empty, no compact job to execute.` was misleading: an empty
plan
there only means the current manifest batch had nothing to compact, and
the loop
keeps scanning further batches until `EndOfScanException` is thrown.
- Log the previously silent early return taken when the table has no
snapshot.
One small non-logging adjustment: in `sortCompactUnAwareBucketTable` the
`TableSorter` is now created before the new emptiness check. Its constructor
validates the order columns, so keeping it in front of the early return
preserves
the existing fail-fast behaviour for an invalid `order_by` on an empty table.
### Tests
Logging only, no behaviour change, so no new tests.
`paimon-spark-common` compiles, `spotless:check` passes, and the existing
`org.apache.paimon.spark.procedure.CompactProcedureTest` suite (Spark 3.5)
runs
green over the affected paths (aware bucket, unaware bucket, sort compact,
incremental cluster, read parallelism).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
