imbajin commented on code in PR #3326: URL: https://github.com/apache/hugegraph/pull/3326#discussion_r4293620457
########## .github/workflows/submit-dependencies.yml: ########## @@ -0,0 +1,315 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +name: Dependency baseline + +on: + push: + branches: [master] + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: dependency-baseline-${{ github.repository }}-master + cancel-in-progress: false + +jobs: + validate: + if: >- + (github.repository == 'apache/hugegraph' || github.repository == 'hugegraph/hugegraph') && + github.ref == 'refs/heads/master' && + (github.event_name == 'push' || github.event_name == 'workflow_dispatch') + runs-on: ubuntu-latest + timeout-minutes: 10 + env: + PYTHONDONTWRITEBYTECODE: '1' + steps: + - name: Checkout event commit + uses: actions/checkout@v7 + with: + ref: ${{ github.sha }} + persist-credentials: false + - name: Test CI policy and dependency inventory + run: | + python3 -m unittest discover -s .github/scripts -p 'test_ci_policy.py' + python3 -m unittest discover -s .github/scripts -p 'test_dependency_snapshot.py' + python3 -m unittest discover -s install-dist/scripts/dependency -p 'test_*.py' + + generate: + needs: validate + if: >- + ((github.repository == 'apache/hugegraph' && github.event.repository.fork == false) || + github.repository == 'hugegraph/hugegraph') && + github.ref == 'refs/heads/master' && + (github.event_name == 'push' || github.event_name == 'workflow_dispatch') + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: read + env: + PYTHONDONTWRITEBYTECODE: '1' + INPUT_TOKEN: '' + GITHUB_TOKEN: '' + GH_TOKEN: '' + outputs: + artifact_id: ${{ steps.snapshot.outputs.artifact-id }} + steps: + - name: Checkout event commit + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + persist-credentials: false + - name: Set up project JDK + uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1 + with: + java-version-file: .github/workflows/.java-version + distribution: 'temurin' + - name: Verify checkout and current master + env: + EXPECTED_SHA: ${{ github.sha }} + run: | + test "$(git rev-parse HEAD)" = "$EXPECTED_SHA" + test "$(git ls-remote origin refs/heads/master | cut -f1)" = "$EXPECTED_SHA" Review Comment: This remains a known limitation of the current-tip producer. Superseded event SHAs are not backfilled by this PR. The updated `docs/ci.md` provides recovery by waiting for a successful current-master baseline, updating/pushing the PR branch to create a new event, and checking that event's exact base SHA. It does not repair the old event. Named historical-SHA submission remains deferred pending confirmation of the submission semantics. This is an explicit scope deferral, not a completed fix; this discussion remains open. The separate current-tip documentation alternative is addressed in https://github.com/apache/hugegraph/pull/3326#discussion_r4292238635. ########## .github/scripts/dependency-snapshot.py: ########## @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Serialize native Ferstl 4.0.3 graphs; no token or submission code. + +Uses maven-dependency-submission-action 6.0.1's Maven parser/toolkit conventions: +first classifier/type, shared PURL packages, direct precedence, child edge order +and cycle handling. Non-test scopes take precedence over test to match the +independent inventory. See docs/ci.md for the producer boundary. +""" + +import datetime +import json +import os +from pathlib import Path +import subprocess +import sys +from urllib.parse import quote + +REPORT = 'maven-dependency-submission-action-depgraph.json' +DETECTOR = {'name': 'maven-dependency-submission-action', 'version': '6.0.1', + 'url': 'https://github.com/advanced-security/maven-dependency-submission-action'} +CORRELATOR = '.github/workflows/submit-dependencies.yml:submit' +SERIALIZER = 'Ferstl 4.0.3; repository stdlib adapter for Maven parser 6.0.1' + + +def unique_object(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError(f'Duplicate JSON key: {key}') + result[key] = value + return result + + +def package_url(artifact): + # packageurl-js uses encodeURIComponent for components and URLSearchParams + # for qualifiers; notably '+' in versions and '/' in namespaces survive. + namespace = '/'.join(part for part in artifact['groupId'].split('/') if part) + name, version = artifact['artifactId'].strip(), artifact['version'].strip() + if not namespace or not name or not version: + raise ValueError('Incomplete Maven coordinates') + encoded_namespace = quote(namespace, safe="!~*'():/") + encoded_name = quote(name, safe="!~*'()") + encoded_version = quote(version, safe="!~*'():+") + purl = f'pkg:maven/{encoded_namespace}/{encoded_name}@{encoded_version}' + qualifiers = {} + for key, field in [('classifier', 'classifiers'), ('type', 'types')]: + values = artifact.get(field) or [] + if values and values[0].strip(): + qualifiers[key] = values[0].strip() + if qualifiers: + purl += '?' + '&'.join(f'{key}={quote(value, safe="*").replace("~", "%7E")}' + for key, value in sorted(qualifiers.items())) + return purl + + +def manifest(graph, pom): + if not isinstance(graph, dict) or not isinstance(graph.get('graphName'), str) or not graph['graphName'].strip(): + raise ValueError(f'Malformed graph name: {pom}') + artifacts, edges = graph.get('artifacts'), graph.get('dependencies') or [] + if not isinstance(artifacts, list) or not artifacts or not isinstance(edges, list): + raise ValueError(f'Empty or malformed graph: {pom}') + if graph.get('dependencies') is not None and not isinstance(graph['dependencies'], list): + raise ValueError(f'Malformed dependency list: {pom}') + ids, packages, scopes = {}, {}, {} + for artifact in artifacts: + if not isinstance(artifact, dict) or any(not isinstance(artifact.get(key), str) or not artifact[key].strip() + for key in ['id', 'groupId', 'artifactId', 'version']): + raise ValueError(f'Malformed artifact: {pom}') + for key in ['scopes', 'types', 'classifiers']: + values = artifact.get(key) + if values is not None and (not isinstance(values, list) or any(not isinstance(x, str) for x in values)): + raise ValueError(f'Malformed artifact {key}: {pom}') + identifier, purl = artifact['id'], package_url(artifact) + if identifier in ids: + raise ValueError(f'Duplicate artifact ID: {identifier}') + ids[identifier] = purl + packages.setdefault(purl, []) + scopes[purl] = 'development' if set(artifact.get('scopes') or []) == {'test'} else 'runtime' Review Comment: Update: the scenario below uses hand-built input. It has not been reproduced in unmodified output from the pinned Ferstl 4.0.3 exporter. The blocking classification was withdrawn in [the follow-up](https://github.com/apache/hugegraph/pull/3326#discussion_r4286306568). This discussion remains open for the requested minimal POM and native exporter output; no adapter change is currently justified. ⚠️ Unconfirmed. Blocking: no (classification withdrawn). Summary: `scopes[purl]` is overwritten for each artifact, even when distinct artifact IDs map to the same PURL. If a runtime-scoped alias is followed by a test-only alias, this records `development`, while `dependency-inventory.py` merges the same PURL with runtime precedence; the writer then rejects the otherwise valid baseline on inventory mismatch. Please merge scope across aliases with runtime precedence and add cases for both artifact orders. Evidence: the snapshot code merges aliased IDs by PURL below, the independent inventory preserves runtime when merging duplicate PURLs, and the writer requires exact scope equality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
