roryqi commented on code in PR #13717: URL: https://github.com/apache/gravitino/pull/13717#discussion_r4255282290
########## docs/table-maintenance-service/optimizer-cli-reference.md: ########## @@ -282,6 +283,95 @@ table, so it must be asked for explicitly. Its `jobConf` is documented in [Configuration](./optimizer-configuration.md#job-submission-configuration). +## Update Manifest Statistics + +`builtin-iceberg-update-manifest-stats` is a separate Spark job that collects two table-level +statistics, including for partitioned tables: + +| Statistic | Object entry for each decimal spec ID | +| --- | --- | +| `custom-manifest-number-by-spec` | Long manifest count | +| `custom-avg-manifest-size-by-spec` | Double average manifest size in bytes | + +The job reads manifest metadata without scanning file statistics. It works after partition +evolution, including for specs whose partition fields are absent from older files. +`builtin-iceberg-update-stats` and the `submit-update-stats-job` CLI retain their existing behavior; +they do not collect manifest statistics. Submit the new template directly through the jobs API. + +Each table cycle pins one current snapshot, calls `allManifests` once, and groups data and +delete manifests by `partitionSpecId`. Every defined spec is included. A spec with no manifests, +including a table without a snapshot, produces count `0` and average `0.0`. +The job also publishes `custom-manifest-snapshot-id` in the same update (`-1` without a snapshot). +Snapshot IDs identify measurements; they are not monotonically increasing version numbers. + +The template accepts `catalog_name`, `table_identifier`, `coordination_directory`, +`updater_options`, and optional +`spark_conf`, alongside the standard Spark template settings. There is no `spec_id` filter: +collection always replaces the complete maps for the table. +`updater_options` is a JSON string containing `gravitino_uri` and `metalake`; it can also select +a `statistics_updater` implementing the existing table statistics update operation. +For example: + +```bash +curl -X POST http://localhost:8090/api/metalakes/test/jobs/runs \ + -H 'Content-Type: application/json' \ + -d '{ + "jobTemplateName": "builtin-iceberg-update-manifest-stats", + "jobConf": { + "catalog_name": "rest_catalog", + "table_identifier": "db.t1", + "coordination_directory": "/mnt/gravitino-maintenance-locks", + "updater_options": "{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\"}";, + "spark_master": "local[2]", + "spark_executor_instances": "1", + "spark_executor_cores": "1", + "spark_executor_memory": "1g", + "spark_driver_memory": "1g", + "catalog_type": "rest", + "catalog_uri": "http://localhost:9001/iceberg";, + "warehouse_location": "" + } + }' +``` + +For example, one collection can produce complete maps `{"0": 620, "1": 120}` and +`{"0": 10485760.0, "1": 4194304.0}`. The collector owns both statistic names and publishes +them together with the snapshot ID in one existing PUT/update call. Old map entries are +replaced, not merged. Other statistic names are unchanged. No PATCH or merge API is needed. + +### Shared coordination and recovery Review Comment: Usually we just need to rerun the failed job instead of recovering a failure job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
