laserninja commented on code in PR #13717: URL: https://github.com/apache/gravitino/pull/13717#discussion_r4272966013
########## docs/table-maintenance-service/optimizer-cli-reference.md: ########## @@ -282,6 +283,95 @@ table, so it must be asked for explicitly. Its `jobConf` is documented in [Configuration](./optimizer-configuration.md#job-submission-configuration). +## Update Manifest Statistics + +`builtin-iceberg-update-manifest-stats` is a separate Spark job that collects two table-level +statistics, including for partitioned tables: + +| Statistic | Object entry for each decimal spec ID | +| --- | --- | +| `custom-manifest-number-by-spec` | Long manifest count | +| `custom-avg-manifest-size-by-spec` | Double average manifest size in bytes | + +The job reads manifest metadata without scanning file statistics. It works after partition +evolution, including for specs whose partition fields are absent from older files. +`builtin-iceberg-update-stats` and the `submit-update-stats-job` CLI retain their existing behavior; +they do not collect manifest statistics. Submit the new template directly through the jobs API. + +Each table cycle pins one current snapshot, calls `allManifests` once, and groups data and +delete manifests by `partitionSpecId`. Every defined spec is included. A spec with no manifests, +including a table without a snapshot, produces count `0` and average `0.0`. +The job also publishes `custom-manifest-snapshot-id` in the same update (`-1` without a snapshot). +Snapshot IDs identify measurements; they are not monotonically increasing version numbers. + +The template accepts `catalog_name`, `table_identifier`, `coordination_directory`, +`updater_options`, and optional +`spark_conf`, alongside the standard Spark template settings. There is no `spec_id` filter: +collection always replaces the complete maps for the table. +`updater_options` is a JSON string containing `gravitino_uri` and `metalake`; it can also select +a `statistics_updater` implementing the existing table statistics update operation. +For example: + +```bash +curl -X POST http://localhost:8090/api/metalakes/test/jobs/runs \ + -H 'Content-Type: application/json' \ + -d '{ + "jobTemplateName": "builtin-iceberg-update-manifest-stats", + "jobConf": { + "catalog_name": "rest_catalog", + "table_identifier": "db.t1", + "coordination_directory": "/mnt/gravitino-maintenance-locks", + "updater_options": "{\"gravitino_uri\":\"http://localhost:8090\",\"metalake\":\"test\"}";, + "spark_master": "local[2]", + "spark_executor_instances": "1", + "spark_executor_cores": "1", + "spark_executor_memory": "1g", + "spark_driver_memory": "1g", + "catalog_type": "rest", + "catalog_uri": "http://localhost:9001/iceberg";, + "warehouse_location": "" + } + }' +``` + +For example, one collection can produce complete maps `{"0": 620, "1": 120}` and +`{"0": 10485760.0, "1": 4194304.0}`. The collector owns both statistic names and publishes +them together with the snapshot ID in one existing PUT/update call. Old map entries are +replaced, not merged. Other statistic names are unchanged. No PATCH or merge API is needed. + +### Shared coordination and recovery Review Comment: Addressed in c7caccead. coordination_directory is now optional: omitted or blank uses last-write-wins publication of the complete maps plus snapshot ID, without lock files or manual recovery. Supplying a directory retains the existing shared-lock/recovery mechanism for deployments needing serialization. Updated the template default, CLI handling, examples, and docs. The docs explicitly require consumers needing fresh statistics to compare snapshot IDs for equality and skip/recollect on mismatch; this collector does not enforce that check. All writers must opt into the same coordination directory for strict mode to be effective. The full 221-test jobs suite passed, followed by both CLI startup cases (omitted and blank). Added Spark coverage for uncoordinated publication and rerunning a failed update; existing strict-mode tests still pass. Formatting, Javadoc, and docs checks passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
