wombatu-kun commented on code in PR #19959: URL: https://github.com/apache/hudi/pull/19959#discussion_r4022056256
########## scripts/trino/check_dependency_drift.py: ########## @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Report version drift between hudi-trino's classpath and the Trino plugin's. + +hudi-trino's unit tests resolve dependency versions from Hudi's root pom, but +the plugin that ships is assembled under trino-root and bundles Trino's +versions. This script compares two `mvn dependency:list -DoutputFile=...` +outputs and prints a Markdown table of the libraries whose versions differ. + +Only dependencies present on both classpaths are compared: a library on one +side alone cannot run with a different version at runtime. org.apache.hudi and +io.trino artifacts are skipped because both sides take them from the same +source of truth. + +Exit codes: 0 no drift, 1 drift found, 2 usage or parse error. +""" + +import argparse +import re +import sys + +ANSI_ESCAPE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]") +SKIPPED_GROUPS = ("org.apache.hudi", "io.trino") + + +def parse_dependency_list(path): + """Returns {groupId:artifactId: set(versions)} parsed from a dependency:list file.""" + deps = {} + with open(path, encoding="utf-8") as handle: + for raw in handle: + line = ANSI_ESCAPE.sub("", raw).strip() + # Drop the JPMS " -- module ..." suffix and markers such as " (optional)". + coordinate = line.split(" -- ")[0].split()[0] if line else "" + fields = coordinate.split(":") + # groupId:artifactId:type:version:scope or + # groupId:artifactId:type:classifier:version:scope + if len(fields) not in (5, 6) or not all(fields): + continue + key = f"{fields[0]}:{fields[1]}" Review Comment: `parse_dependency_list` keys on groupId:artifactId only, so an artifact and its classifier siblings (a `tests` or `shaded` jar) merge into one row and the cell lists both versions with no way to tell which one drifted. Worth folding the classifier into the key so each artifact gets its own row - follow-up, not a blocker. ########## .github/workflows/hudi_trino_dependency_drift.yml: ########## @@ -0,0 +1,199 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Hudi Trino Dependency Drift + +# hudi-trino's tests resolve dependency versions from Hudi's root pom, but the shipped +# plugin is assembled under trino-root and bundles Trino's versions. This nightly job +# reports the libraries whose versions differ between the two classpaths (#19958). It is +# a report, not a gate: it files or updates an issue and is never a required check. +on: + schedule: + - cron: '47 5 * * *' + workflow_dispatch: Review Comment: Editing `check_dependency_drift.py` triggers the connector CI and the E2E pipeline through their `scripts/trino/**` filters, and neither runs the script, while the drift check itself now has no PR trigger at all. Worth narrowing those two filters to `bootstrap_trino.sh` - follow-up, not a blocker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
