laserninja commented on PR #12942: URL: https://github.com/apache/gravitino/pull/12942#issuecomment-5668251014
Updated in 02e0463c2: - Matched #11700's numbered sections and policy/handler/adapter/Spark-job layout, distinguishing the initial job from follow-up policy work. - Aligned every Markdown table. - Removed the general JobManager/placeholder discussion and its unrelated follow-up plan. - Limited initial statistics to `custom-manifest-number` and `custom-avg-manifest-size`. - Added spec discovery via `default-spec-id`, `partition-specs`, and `SELECT DISTINCT partition_spec_id FROM ...manifests`, with a day-to-hour example. The same guidance and corrected help text are pushed to #12937 in 3d6cf0985. One correction to both my earlier description and the suggested wording: in Iceberg 1.11.0, `spec_id` **does filter** the manifests selected for rewriting. `findMatchingManifests` checks `manifest.partitionSpecId() == spec.specId()`. It does not convert day-spec manifests/data files into hour-spec ones. Omitting the ID selects the current spec; supplying an older ID maintains that older spec. Source: https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java#L349-L360 I also corrected the caching default: the 1.11.0 action defaults to false, so the docs now defer to the installed runtime instead of promising true. Validation: license audit, docs build, formatting check, Markdown table alignment, and `git diff --check` passed. The dedicated Spark-backed test remains an explicit implementation follow-up; existing module integration checks are not proof of this job's spec-selection behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
