sadpandajoe commented on code in PR #45588:
URL: https://github.com/apache/superset/pull/45588#discussion_r4344772712
##########
.github/workflows/update-country-maps.yml:
##########
@@ -0,0 +1,178 @@
+name: Update Country Maps
+
+# Regenerates plugins/plugin-chart-country-map's GeoJSON files, countries.ts
+# and docs/data/countries.json from upstream Natural Earth + community geo
+# data, by running the same notebook `npm run update-maps` runs locally.
+#
+# Runs weekly and on workflow_dispatch, but only opens or updates a PR when
+# the regenerated output actually differs from what's on `master` - most
+# weeks this is a no-op. Unlike a translation-string catch-up, nothing here
+# auto-merges: a border/region change is the kind of thing that needs an
+# actual human look at the rendered maps. Community geo sources are pinned
+# to a commit where practical; the live WHO ArcGIS query can't be pinned
+# (it's a live service, not a repo), so a run can still legitimately pick up
+# an unreviewed upstream change there. The job itself fails (separately from
+# whether a PR was opened) if the notebook hits a cell error, so a broken
+# upstream source doesn't just quietly go green.
+on:
+ schedule:
+ - cron: "0 7 * * 1" # Mondays 07:00 UTC, an hour after
sync-translation-catalogs.yml
+ workflow_dispatch: {}
+
+permissions:
+ contents: read
+
+concurrency:
+ group: update-country-maps
+ cancel-in-progress: false
+
+jobs:
+ update-country-maps:
+ if: github.event_name != 'schedule' || github.repository ==
'apache/superset'
+ runs-on: ubuntu-26.04
+ permissions:
+ contents: write
+ pull-requests: write
+ steps:
+ - name: "Checkout master"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ ref: master
+ persist-credentials: true
+
+ - name: Setup Python
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 #
v7.0.0
+ with:
+ python-version: "3.11"
+
+ - name: Install notebook dependencies
+ run: |
+ pip install jupyter nbconvert ipykernel geopandas shapely matplotlib
requests
+
+ - name: Force speed_run (skip interactive preview rendering)
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF'
+ import json
+
+ path = "Country Map GeoJSON Generator.ipynb"
+ with open(path, encoding="utf-8") as f:
+ nb = json.load(f)
+
+ for cell in nb["cells"]:
+ if cell["cell_type"] != "code":
+ continue
+ if "".join(cell["source"]).strip().startswith("speed_run =
False"):
+ cell["source"] = [
+ "speed_run = True\n",
+ "# forced True for unattended CI regeneration; not
committed back\n",
+ ]
+ break
+ else:
+ raise SystemExit("Could not find the speed_run cell to patch")
+
+ with open(".ci-run.ipynb", "w", encoding="utf-8") as f:
+ json.dump(nb, f)
+ PYEOF
+
+ - name: Run notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ jupyter nbconvert --to notebook --execute --inplace --allow-errors \
+ --ExecutePreprocessor.timeout=1200 .ci-run.ipynb
+
+ - name: Check for cell errors and record a summary
+ id: notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF' >> "$GITHUB_OUTPUT"
+ import json
+
+ with open(".ci-run.ipynb", encoding="utf-8") as f:
+ nb = json.load(f)
+
+ errors = []
+ for i, cell in enumerate(nb["cells"]):
+ for output in cell.get("outputs", []):
Review Comment:
Only cells with an `error` output are counted here, but the Latvia cell
wraps `gpd.read_file` in `except Exception` and just prints the failure, so it
never produces an error output. If that download fails while everything else
succeeds, the notebook silently falls back to the Natural Earth boundaries,
`had_errors` stays `false`, and a PR with degraded Latvia data opens with no
warning and a green job. Should the check also scan stream output for these
printed failure messages, or should the notebook cell re-raise?
##########
.github/workflows/update-country-maps.yml:
##########
@@ -0,0 +1,178 @@
+name: Update Country Maps
+
+# Regenerates plugins/plugin-chart-country-map's GeoJSON files, countries.ts
+# and docs/data/countries.json from upstream Natural Earth + community geo
+# data, by running the same notebook `npm run update-maps` runs locally.
+#
+# Runs weekly and on workflow_dispatch, but only opens or updates a PR when
+# the regenerated output actually differs from what's on `master` - most
+# weeks this is a no-op. Unlike a translation-string catch-up, nothing here
+# auto-merges: a border/region change is the kind of thing that needs an
+# actual human look at the rendered maps. Community geo sources are pinned
+# to a commit where practical; the live WHO ArcGIS query can't be pinned
+# (it's a live service, not a repo), so a run can still legitimately pick up
+# an unreviewed upstream change there. The job itself fails (separately from
+# whether a PR was opened) if the notebook hits a cell error, so a broken
+# upstream source doesn't just quietly go green.
+on:
+ schedule:
+ - cron: "0 7 * * 1" # Mondays 07:00 UTC, an hour after
sync-translation-catalogs.yml
+ workflow_dispatch: {}
+
+permissions:
+ contents: read
+
+concurrency:
+ group: update-country-maps
+ cancel-in-progress: false
+
+jobs:
+ update-country-maps:
+ if: github.event_name != 'schedule' || github.repository ==
'apache/superset'
+ runs-on: ubuntu-26.04
+ permissions:
+ contents: write
+ pull-requests: write
+ steps:
+ - name: "Checkout master"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ ref: master
+ persist-credentials: true
+
+ - name: Setup Python
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 #
v7.0.0
+ with:
+ python-version: "3.11"
+
+ - name: Install notebook dependencies
+ run: |
+ pip install jupyter nbconvert ipykernel geopandas shapely matplotlib
requests
+
+ - name: Force speed_run (skip interactive preview rendering)
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF'
+ import json
+
+ path = "Country Map GeoJSON Generator.ipynb"
+ with open(path, encoding="utf-8") as f:
+ nb = json.load(f)
+
+ for cell in nb["cells"]:
+ if cell["cell_type"] != "code":
+ continue
+ if "".join(cell["source"]).strip().startswith("speed_run =
False"):
+ cell["source"] = [
+ "speed_run = True\n",
+ "# forced True for unattended CI regeneration; not
committed back\n",
+ ]
+ break
+ else:
+ raise SystemExit("Could not find the speed_run cell to patch")
+
+ with open(".ci-run.ipynb", "w", encoding="utf-8") as f:
+ json.dump(nb, f)
+ PYEOF
+
+ - name: Run notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ jupyter nbconvert --to notebook --execute --inplace --allow-errors \
+ --ExecutePreprocessor.timeout=1200 .ci-run.ipynb
+
+ - name: Check for cell errors and record a summary
+ id: notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF' >> "$GITHUB_OUTPUT"
+ import json
+
+ with open(".ci-run.ipynb", encoding="utf-8") as f:
+ nb = json.load(f)
+
+ errors = []
+ for i, cell in enumerate(nb["cells"]):
+ for output in cell.get("outputs", []):
+ if output.get("output_type") == "error":
+ errors.append(f"cell {i}: {output.get('ename')}:
{output.get('evalue')}")
+
+ if errors:
+ print("had_errors=true")
+ summary = "\n".join(f"- {e}" for e in errors[:20])
+ else:
+ print("had_errors=false")
+ summary = ""
+
+ with open("notebook-errors.txt", "w", encoding="utf-8") as f:
+ f.write(summary)
+ PYEOF
+ rm -f .ci-run.ipynb
+ if [ -s notebook-errors.txt ]; then
+ echo "::warning::Notebook hit cell errors; generated data may be
stale."
+ {
+ echo "### Country map notebook cell errors"
+ cat notebook-errors.txt
+ } >> "$GITHUB_STEP_SUMMARY"
+ fi
+
+ - name: Check for drift
+ id: diff
+ run: |
+
PATHS="superset-frontend/plugins/plugin-chart-country-map/src/countries
superset-frontend/plugins/plugin-chart-country-map/src/countries.ts
docs/data/countries.json"
+ if git diff --quiet -- $PATHS; then
+ echo "has_changes=false" >> "$GITHUB_OUTPUT"
+ else
+ echo "has_changes=true" >> "$GITHUB_OUTPUT"
+ fi
+
+ - name: Open or update sync PR
+ if: steps.diff.outputs.has_changes == 'true'
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ BRANCH: chore/update-country-maps-automated
+ HAD_ERRORS: ${{ steps.notebook.outputs.had_errors }}
+ run: |
+ git config user.name "GitHub Action"
+ git config user.email "[email protected]"
+ git checkout -b "$BRANCH"
+ git add
superset-frontend/plugins/plugin-chart-country-map/src/countries \
+
superset-frontend/plugins/plugin-chart-country-map/src/countries.ts \
+ docs/data/countries.json
+ git commit -m "chore(country-map): regenerate GeoJSON from upstream
sources" \
+ -m "Automated run of the Country Map GeoJSON Generator notebook
(npm run update-maps). Review the rendered diffs before merging - one source
(WHO ArcGIS) is a live query, not a pinned commit."
+ git push --force origin "$BRANCH"
+
+ BODY="Automated run of \`npm run update-maps\` against current
\`master\`. **Please review the rendered map diffs before merging** - this
pulls from several community geo sources plus a live WHO ArcGIS query that
can't be pinned to a commit, so an upstream change there hasn't had human eyes
on it yet."
+ if [ "$HAD_ERRORS" = "true" ]; then
+ BODY="$BODY
+
+ ⚠️ The notebook run hit cell errors (ran with --allow-errors, so it
continued past them - affected countries' data may be stale rather than
updated):
+ $(cat
superset-frontend/plugins/plugin-chart-country-map/scripts/notebook-errors.txt)"
+ fi
+
+ if gh pr list --repo "$GITHUB_REPOSITORY" --head "$BRANCH" --state
open --json number --jq 'length > 0' | grep -q true; then
+ gh pr edit "$BRANCH" --repo "$GITHUB_REPOSITORY" --body "$BODY"
+ else
+ gh pr create --repo "$GITHUB_REPOSITORY" \
+ --title "chore(country-map): regenerate GeoJSON from upstream
sources" \
+ --body "$BODY" \
+ --base master --head "$BRANCH"
+ fi
+
+ - name: Close stale sync PR when no drift remains
+ if: steps.diff.outputs.has_changes == 'false'
Review Comment:
This step only checks `has_changes`, but the notebook runs with
`--allow-errors`. If a Natural Earth download fails on a fresh runner, the
dependent cells error before writing any files, the drift check sees no diff,
and this step closes the open sync PR and deletes its branch. The `Fail if the
notebook hit cell errors` step only runs afterwards, so a valid pending update
is lost because of a failed run. Should this also require
`steps.notebook.outputs.had_errors != 'true'` so a failed regeneration never
closes the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
