roryqi commented on code in PR #13717: URL: https://github.com/apache/gravitino/pull/13717#discussion_r4270871326
########## docs/table-maintenance-service/optimizer-cli-reference.md: ########## @@ -282,6 +283,95 @@ table, so it must be asked for explicitly. Its `jobConf` is documented in [Configuration](./optimizer-configuration.md#job-submission-configuration). +## Update Manifest Statistics + +`builtin-iceberg-update-manifest-stats` is a separate Spark job that collects two table-level +statistics, including for partitioned tables: + +| Statistic | Object entry for each decimal spec ID | +| --- | --- | +| `custom-manifest-number-by-spec` | Long manifest count | +| `custom-avg-manifest-size-by-spec` | Double average manifest size in bytes | + +The job reads manifest metadata without scanning file statistics. It works after partition +evolution, including for specs whose partition fields are absent from older files. +`builtin-iceberg-update-stats` and the `submit-update-stats-job` CLI retain their existing behavior; +they do not collect manifest statistics. Submit the new template directly through the jobs API. + +Each table cycle pins one current snapshot, calls `allManifests` once, and groups data and +delete manifests by `partitionSpecId`. Every defined spec is included. A spec with no manifests, +including a table without a snapshot, produces count `0` and average `0.0`. +The job also publishes `custom-manifest-snapshot-id` in the same update (`-1` without a snapshot). +Snapshot IDs identify measurements; they are not monotonically increasing version numbers. + +The template accepts `catalog_name`, `table_identifier`, `coordination_directory`, +`updater_options`, and optional +`spark_conf`, alongside the standard Spark template settings. There is no `spec_id` filter: +collection always replaces the complete maps for the table. +`updater_options` is a JSON string containing `gravitino_uri` and `metalake`; it can also select +a `statistics_updater` implementing the existing table statistics update operation. +For example: + +```bash +curl -X POST http://localhost:8090/api/metalakes/test/jobs/runs \ + -H 'Content-Type: application/json' \ + -d '{ + "jobTemplateName": "builtin-iceberg-update-manifest-stats", + "jobConf": { + "catalog_name": "rest_catalog", + "table_identifier": "db.t1", + "coordination_directory": "/mnt/gravitino-maintenance-locks", + "updater_options": "{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\"}";, + "spark_master": "local[2]", + "spark_executor_instances": "1", + "spark_executor_cores": "1", + "spark_executor_memory": "1g", + "spark_driver_memory": "1g", + "catalog_type": "rest", + "catalog_uri": "http://localhost:9001/iceberg";, + "warehouse_location": "" + } + }' +``` + +For example, one collection can produce complete maps `{"0": 620, "1": 120}` and +`{"0": 10485760.0, "1": 4194304.0}`. The collector owns both statistic names and publishes +them together with the snapshot ID in one existing PUT/update call. Old map entries are +replaced, not merged. Other statistic names are unchanged. No PATCH or merge API is needed. + +### Shared coordination and recovery Review Comment: Now that the collector publishes complete maps for all specs in one PUT instead of merging per-spec updates, I think the mandatory coordination directory is over-designed. Concurrent publications cannot produce a partially merged result; at worst, a complete result from an older immutable snapshot wins. The payload already includes `custom-manifest-snapshot-id`, so consumers can reject stale statistics when it does not match the table's current snapshot. Could we make `coordination_directory` optional? Without it, the collector can use simple last-write-wins semantics; deployments that require stricter serialization can opt into the current shared-lock and recovery mechanism. Requiring every deployment to provide a shared POSIX filesystem and manually recover ambiguous writes seems disproportionate for rebuildable derived statistics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
