adoroszlai commented on code in PR #6916:
URL: https://github.com/apache/ozone/pull/6916#discussion_r1955777219
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
+ echo "Configurations.md has changed, proceeding to commit and
create PRs."
+ echo "hashes_differ=true" >> $GITHUB_ENV
+ else
+ echo "Configurations.md is unchanged, skipping commit and PR
creation."
+ echo "hashes_differ=false" >> $GITHUB_ENV
+ fi
+ - name: Commit and push changes to apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ run: |
+ git config --global user.name 'Github Actions'
+ git config --global user.email '[email protected]'
+ git checkout -b config-doc-update-from-$GITHUB_SHA
+ git add hadoop-hdds/docs/content/tools/Configurations.md
+ git commit -m "[Auto] Update Configurations.md from $GITHUB_SHA"
+ git push origin config-doc-update-from-$GITHUB_SHA
+ - name: Extract JIRA ID from the first commit of the branch
+ id: extract-jira
+ env:
+ BRANCH_NAME: ${{ github.ref_name }}
+ run: |
Review Comment:
`if: env.hashes_differ == 'true'` is missing here.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
Review Comment:
Please move the steps to a separate, [reusable
workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows#creating-a-reusable-workflow).
The workflow should take commit SHA as input.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
Review Comment:
The job has checked out the `ozone` repo and can run `git status` to decide
if `Configurations.md` has changed or not. Is there a benefit to checking
hashes instead?
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
Review Comment:
`ozone-bin` is not the "source" tarball, but the "binary" one.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
Review Comment:
These steps (setup python, download binary tarball, generate markdown file)
can be moved to a separate job (e.g. `generate-config-doc`). It should upload
the markdown file as an artifact.
The benefit is that we can run this separate job for any event and repo,
which makes it easier to test the python script. We can also download and
inspect the file.
The existing job should then depend on the new one, not `build`.
We can even split the remaining parts of the existing job (still in the same
workflow): one for updating `ozone` and one for `ozone-site`.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
+ echo "Configurations.md has changed, proceeding to commit and
create PRs."
+ echo "hashes_differ=true" >> $GITHUB_ENV
+ else
+ echo "Configurations.md is unchanged, skipping commit and PR
creation."
+ echo "hashes_differ=false" >> $GITHUB_ENV
+ fi
+ - name: Commit and push changes to apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ run: |
+ git config --global user.name 'Github Actions'
+ git config --global user.email '[email protected]'
+ git checkout -b config-doc-update-from-$GITHUB_SHA
+ git add hadoop-hdds/docs/content/tools/Configurations.md
+ git commit -m "[Auto] Update Configurations.md from $GITHUB_SHA"
+ git push origin config-doc-update-from-$GITHUB_SHA
+ - name: Extract JIRA ID from the first commit of the branch
+ id: extract-jira
+ env:
+ BRANCH_NAME: ${{ github.ref_name }}
+ run: |
+ REMOTE=$(git remote -v | grep -E 'github.com[:/]apache/ozone' | awk
'{print $1}' | head -1)
+ git fetch $REMOTE
+ JIRA_ID=$(git log ${REMOTE}/master..${REMOTE}/$(git rev-parse
--abbrev-ref HEAD) --oneline | tail -1 | sed -E 's/.*(HDDS-[[:digit:]]+).*/\1/')
+ echo "jira_id=$JIRA_ID" >> $GITHUB_ENV
+ - name: Create Pull Request in apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ WORKFLOW_NAME: ${{ github.workflow }}
+ WORKFLOW_RUN_ID: ${{ github.run_id }}
+ BRANCH_NAME: ${{ github.ref_name }}
+ JIRA_ID: ${{ env.jira_id }}
+ run: |
+ echo "## What changes were proposed in this pull request?" >
pr_body.txt
+ echo "This is an automated pull request triggered by the
[$WORKFLOW_NAME](https://github.com/apache/ozone/actions/runs/$WORKFLOW_RUN_ID)
workflow run from
[$BRANCH_NAME](https://github.com/apache/ozone/tree/$BRANCH_NAME). Please
delete the
[config-doc-update-from-$GITHUB_SHA](https://github.com/apache/ozone/tree/config-doc-update-from-$GITHUB_SHA)
branch after this PR is merged or closed." >> pr_body.txt
+ echo "" >> pr_body.txt
+ echo "## What is the link to the Apache JIRA?" >> pr_body.txt
+ echo "[$JIRA_ID](https://issues.apache.org/jira/browse/$JIRA_ID)" >>
pr_body.txt
+ echo "" >> pr_body.txt
+ echo "## How was this patch tested?" >> pr_body.txt
+ echo "Reviewers should manually verify the correctness of this
change." >> pr_body.txt
Review Comment:
This would be better placed in a shell script that prints the PR text to
stdout.
1. Easier to test and tweak locally.
2. Avoid the need for repeated `>> pr_body.txt`, output of the script can be
redirected in one step.
This applies to both `ozone` and `ozone-site` updates.
##########
dev-support/ci/xml_to_md.py:
##########
@@ -0,0 +1,131 @@
+#!/usr/bin/python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Python file to convert XML properties into Markdown
+import os
+import re
+import zipfile
+import xml.etree.ElementTree as ET
+from collections import namedtuple
+from pathlib import Path
+import sys
+
+Property = namedtuple('Property', ['name', 'value', 'tag', 'description'])
+
+def extract_xml_from_jar(jar_path, xml_filename):
+ xml_files = []
+ with zipfile.ZipFile(jar_path, 'r') as jar:
+ for file_info in jar.infolist():
+ if file_info.filename.endswith(xml_filename):
+ with jar.open(file_info.filename) as xml_file:
+ xml_files.append(xml_file.read())
+ return xml_files
+
+def wrap_config_keys_in_description(description, properties):
+ words = description.split()
+ wrapped_words = []
+ for word in words:
+ clean_word = word.strip('.,()')
+ if clean_word in properties:
+ word = f'`{word}`'
+ wrapped_words.append(word)
+ return ' '.join(wrapped_words)
+
+def parse_xml_file(xml_content, properties):
+ root = ET.fromstring(xml_content)
+ for prop in root.findall('property'):
+ name = prop.findtext('name')
+ if not name:
+ raise ValueError("Property 'name' is required but missing in XML.")
+ description = prop.findtext('description', '')
+ if not description:
+ raise ValueError(f"Property '{name}' is missing a description.")
+ tag = prop.findtext('tag')
+ properties[name] = Property(
+ name = name,
+ value = prop.findtext('value', ''),
+ tag = '<br/>'.join(f'`{t}`' for t in tag.split(', ')),
+ description = wrap_config_keys_in_description(
+ ' '.join(description.split()).strip(),
+ properties
+ )
+ )
+
+
+def generate_markdown(properties):
+ markdown = f"""---
+
+sidebar_label: Appendix
+---
+
+Configuration Key Appendix
+==========================
+
+This page provides a comprehensive overview of all configuration keys
available in Ozone.
Review Comment:
Can we add a note that this is auto-generated and should not be edited
manually?
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
+ echo "Configurations.md has changed, proceeding to commit and
create PRs."
+ echo "hashes_differ=true" >> $GITHUB_ENV
+ else
+ echo "Configurations.md is unchanged, skipping commit and PR
creation."
+ echo "hashes_differ=false" >> $GITHUB_ENV
+ fi
+ - name: Commit and push changes to apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ run: |
+ git config --global user.name 'Github Actions'
+ git config --global user.email '[email protected]'
+ git checkout -b config-doc-update-from-$GITHUB_SHA
+ git add hadoop-hdds/docs/content/tools/Configurations.md
+ git commit -m "[Auto] Update Configurations.md from $GITHUB_SHA"
+ git push origin config-doc-update-from-$GITHUB_SHA
+ - name: Extract JIRA ID from the first commit of the branch
+ id: extract-jira
+ env:
+ BRANCH_NAME: ${{ github.ref_name }}
+ run: |
+ REMOTE=$(git remote -v | grep -E 'github.com[:/]apache/ozone' | awk
'{print $1}' | head -1)
+ git fetch $REMOTE
+ JIRA_ID=$(git log ${REMOTE}/master..${REMOTE}/$(git rev-parse
--abbrev-ref HEAD) --oneline | tail -1 | sed -E 's/.*(HDDS-[[:digit:]]+).*/\1/')
+ echo "jira_id=$JIRA_ID" >> $GITHUB_ENV
Review Comment:
I don't understand the need for this complex logic. Why do we need to fetch
remote? Why get the "first" commit instead of the "last" one, i.e. the one
that is currently checked out?
The current commit's message we can get simply as: `git log -n 1
--format='%s'`.
What about feature branch merge commits, which will not have a Jira ID?
##########
dev-support/rat/rat-exclusions.txt:
##########
@@ -51,6 +51,7 @@ src/test/resources/test.db.ini
# hadoop-hdds/docs
**/themes/ozonedoc/**
static/slides/*
+content/tools/Configurations.md
Review Comment:
Can we add the license header instead of an exclusion?
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
+ echo "Configurations.md has changed, proceeding to commit and
create PRs."
+ echo "hashes_differ=true" >> $GITHUB_ENV
+ else
+ echo "Configurations.md is unchanged, skipping commit and PR
creation."
+ echo "hashes_differ=false" >> $GITHUB_ENV
+ fi
+ - name: Commit and push changes to apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ run: |
+ git config --global user.name 'Github Actions'
+ git config --global user.email '[email protected]'
+ git checkout -b config-doc-update-from-$GITHUB_SHA
Review Comment:
`Configurations.md` reflects the current state of config XMLs on the branch.
So if we have a PR for commit A, then another for commit B, the first one can
be abandoned.
We can think whether using a single branch is feasible. With that, if new
commit is added to the branch and the PR is still open, it will simply get the
latest change and we don't need to open a new PR.
Also, using a single branch reduces the burden of deleting all the
individual automated branches.
This applies to both `ozone` and `ozone-site` updates.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
Review Comment:
We can relax this condition to allow running for any branch in forks.
Hard-coded references to `apache` should be replaced with current repo's owner.
The job(s) should handle gracefully (skip without error) the case when the
owner does not have `ozone-site` repo. PRs should not need to be created in
forks. This would allow testing most of the steps without manually editing the
workflow.
##########
.github/workflows/ci.yml:
##########
@@ -166,6 +166,129 @@ jobs:
path: |
~/.m2/repository/org/apache/ozone
retention-days: 1
+ build-config-doc:
+ needs:
+ - build
+ if: ${{ github.repository == 'apache/ozone' && github.event_name == 'push'
&& github.ref_name == 'master' }}
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@v4
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - name: Download the source artifact
+ uses: actions/download-artifact@v4
+ with:
+ name: ozone-bin
+ path: ozone-bin
+ - name: Extract the source tarball
+ run: |
+ mkdir -p ozone-bin/extracted
+ tar -xzf ozone-bin/ozone-*.tar.gz -C ozone-bin/extracted
+ - name: Run the Python script to convert XML properties into Markdown
+ run: python3 dev-support/ci/xml_to_md.py ozone-bin/extracted >
hadoop-hdds/docs/content/tools/Configurations.md
+ - name: Check if Configurations.md file has changed
+ id: hash-check
+ run: |
+ HASH_FILES="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md') }}"
+ PREV_HASH="${{
hashFiles('hadoop-hdds/docs/content/tools/Configurations.md@{previous-commit}')
}}"
+ if [ "$HASH_FILES" != "$PREV_HASH" ]; then
+ echo "Configurations.md has changed, proceeding to commit and
create PRs."
+ echo "hashes_differ=true" >> $GITHUB_ENV
+ else
+ echo "Configurations.md is unchanged, skipping commit and PR
creation."
+ echo "hashes_differ=false" >> $GITHUB_ENV
+ fi
+ - name: Commit and push changes to apache/ozone
+ if: env.hashes_differ == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.OZONE_WEBSITE_BUILD }}
+ run: |
+ git config --global user.name 'Github Actions'
+ git config --global user.email '[email protected]'
+ git checkout -b config-doc-update-from-$GITHUB_SHA
+ git add hadoop-hdds/docs/content/tools/Configurations.md
+ git commit -m "[Auto] Update Configurations.md from $GITHUB_SHA"
+ git push origin config-doc-update-from-$GITHUB_SHA
Review Comment:
I hope this does not trigger regular CI workflow. If I remember correctly,
commits pushed from workflows do not, to avoid loops. Can you please confirm?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
