LiJie20190102 opened a new pull request, #13114:
URL: https://github.com/apache/gravitino/pull/13114
### What changes were proposed in this pull request?
Add a reserved-config-key filter to
`IcebergJobUtils.parseCustomSparkConfigs` so that `--spark-conf` cannot
override critical Spark configurations such as `spark.sql.extensions`,
`spark.sql.catalog.*`, and `spark.master`. When a reserved key is detected, an
`IllegalArgumentException` with a clear message is thrown before the Spark
session is created.
Also adds `validateTableIdentifier` and `validateTimestamp` to
`IcebergJobUtils`, and wires `IcebergExpireSnapshotsJob` and
`IcebergRewriteDataFilesJob` to use them consistently.
### Why are the changes needed?
The `printUsage` of `IcebergExpireSnapshotsJob` and
`IcebergRewriteDataFilesJob` both state: "Cannot override catalog, extensions,
or app name configs". However, the code never enforces this — custom configs
from `--spark-conf` are applied directly without any filtering.
This is a correctness and safety gap: users can silently override
`spark.sql.extensions` (disabling Iceberg extensions, causing CALL procedures
to fail), or redirect `spark.sql.catalog.<name>` to an unintended catalog,
potentially running `expire_snapshots` / `rewrite_data_files` against the wrong
table set.
Additionally, `--older-than` timestamps are not validated before being
interpolated into SQL, and `--table` identifiers lack format validation in two
of the three jobs — both inconsistent with the existing `validateRetainLast` /
`validateStrategy` validators.
Fix: #13113
### Does this PR introduce _any_ user-facing change?
Yes. Passing reserved Spark config keys via `--spark-conf` now fails fast
with a clear error message instead of silently overriding them. Invalid
`--table` or `--older-than` values are also rejected earlier with actionable
messages.
### How was this patch tested?
- Added unit tests in `TestIcebergExpireSnapshotsJob` covering:
- Reserved key rejection (extensions, catalog prefix, spark.master)
- Valid custom configs still pass through
- `validateTableIdentifier` accepts valid `schema.table` and rejects
empty/multi-segment identifiers
- `validateTimestamp` accepts valid timestamps and rejects malformed
strings
- Existing `TestIcebergExpireSnapshotsJob` /
`TestIcebergRewriteDataFilesJob` / `TestIcebergUpdateStatsJob` continue to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
