lasdf1234 commented on PR #12942:
URL: https://github.com/apache/gravitino/pull/12942#issuecomment-5676422792
@laserninja
1. Regarding the first open question, I looked into the solution of floe and
I think it can be handled based on this logic:
IF custom-manifest-number >= manifest_count_critical(500):
trigger
ELSE IF custom-manifest-number >= manifest_count_warning(100)
AND custom-avg-manifest-size < avg_manifest_size_threshold_bytes(8M):
trigger
ELSE:
do not trigger
2. For Open Question 2: collect manifest statistics for the same partition
spec that the rewrite targets. If spec_id is omitted, use the table’s default
spec (default-spec-id). Values written to statistic_meta must remain
distinguishable by spec. For manifest rewrite on a given table, store one row
per statistic name (not one row per spec). Design statistic_name and
statistic_value so that multiple specs can be represented in that single
row—for example, statistic_name = custom-manifest-number-by-spec and
statistic_value = an object/map keyed by spec_id (with a separate row for
custom-avg-manifest-size-by-spec if needed). On each collection run,
merge/update only the entry for the resolved spec. The trigger expression and
the submitted job must use the same resolved spec_id.
3. Regarding the third question. Align with the other built-in Iceberg jobs:
for the initial release, we only commit to and run CI against Spark 3.5 and
Iceberg 1.11.0.
I have no other questions regarding this document. Please add these points
to it. I will review let Rory review this design document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
