sadpandajoe commented on code in PR #45588:
URL: https://github.com/apache/superset/pull/45588#discussion_r4343973216
##########
.github/workflows/update-country-maps.yml:
##########
@@ -0,0 +1,166 @@
+name: Update Country Maps
+
+# Regenerates plugins/plugin-chart-country-map's GeoJSON files, countries.ts
+# and docs/data/countries.json from upstream Natural Earth + community geo
+# data, by running the same notebook `npm run update-maps` runs locally.
+#
+# Runs weekly and on workflow_dispatch, but only opens or updates a PR when
+# the regenerated output actually differs from what's on `master` - most
+# weeks this is a no-op. Unlike a translation-string catch-up, nothing here
+# auto-merges: a border/region change is the kind of thing that needs an
+# actual human look at the rendered maps. Community geo sources are pinned
+# to a commit where practical; the live WHO ArcGIS query can't be pinned
+# (it's a live service, not a repo), so a run can still legitimately pick up
+# an unreviewed upstream change there. The job itself fails (separately from
+# whether a PR was opened) if the notebook hits a cell error, so a broken
+# upstream source doesn't just quietly go green.
+on:
+ schedule:
+ - cron: "0 7 * * 1" # Mondays 07:00 UTC, an hour after
sync-translation-catalogs.yml
+ workflow_dispatch: {}
+
+permissions:
+ contents: read
+
+concurrency:
+ group: update-country-maps
+ cancel-in-progress: false
+
+jobs:
+ update-country-maps:
+ runs-on: ubuntu-26.04
+ permissions:
+ contents: write
+ pull-requests: write
+ steps:
+ - name: "Checkout master"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ ref: master
+ persist-credentials: true
+
+ - name: Setup Python
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 #
v7.0.0
+ with:
+ python-version: "3.11"
+
+ - name: Install notebook dependencies
+ run: |
+ pip install jupyter nbconvert ipykernel geopandas shapely matplotlib
requests
+
+ - name: Force speed_run (skip interactive preview rendering)
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF'
+ import json
+
+ path = "Country Map GeoJSON Generator.ipynb"
+ with open(path, encoding="utf-8") as f:
+ nb = json.load(f)
+
+ for cell in nb["cells"]:
+ if cell["cell_type"] != "code":
+ continue
+ if "".join(cell["source"]).strip().startswith("speed_run =
False"):
+ cell["source"] = [
+ "speed_run = True\n",
+ "# forced True for unattended CI regeneration; not
committed back\n",
+ ]
+ break
+ else:
+ raise SystemExit("Could not find the speed_run cell to patch")
+
+ with open(".ci-run.ipynb", "w", encoding="utf-8") as f:
+ json.dump(nb, f)
+ PYEOF
+
+ - name: Run notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ jupyter nbconvert --to notebook --execute --inplace --allow-errors \
+ --ExecutePreprocessor.timeout=1200 .ci-run.ipynb
+
+ - name: Check for cell errors and record a summary
+ id: notebook
+ working-directory:
superset-frontend/plugins/plugin-chart-country-map/scripts
+ run: |
+ python3 - <<'PYEOF' >> "$GITHUB_OUTPUT"
+ import json
+
+ with open(".ci-run.ipynb", encoding="utf-8") as f:
+ nb = json.load(f)
+
+ errors = []
+ for i, cell in enumerate(nb["cells"]):
+ for output in cell.get("outputs", []):
+ if output.get("output_type") == "error":
+ errors.append(f"cell {i}: {output.get('ename')}:
{output.get('evalue')}")
+
+ if errors:
+ print("had_errors=true")
+ summary = "\n".join(f"- {e}" for e in errors[:20])
+ else:
+ print("had_errors=false")
+ summary = ""
+
+ with open("notebook-errors.txt", "w", encoding="utf-8") as f:
+ f.write(summary)
+ PYEOF
+ rm -f .ci-run.ipynb
+ if [ -s notebook-errors.txt ]; then
+ echo "::warning::Notebook hit cell errors; generated data may be
stale."
+ {
+ echo "### Country map notebook cell errors"
+ cat notebook-errors.txt
+ } >> "$GITHUB_STEP_SUMMARY"
+ fi
+
+ - name: Check for drift
+ id: diff
+ run: |
+
PATHS="superset-frontend/plugins/plugin-chart-country-map/src/countries
superset-frontend/plugins/plugin-chart-country-map/src/countries.ts
docs/data/countries.json"
+ if git diff --quiet -- $PATHS; then
Review Comment:
If a run opens the bot PR and the WHO source (or Natural Earth) later
reverts to what `master` has, the next run takes the `has_changes=false` path,
skips the PR step, and leaves the open PR proposing data that no longer matches
upstream. Reviewers merging it would be applying a stale change. Should the
no-drift path close or refresh an existing bot PR for
`chore/update-country-maps-automated`?
##########
.github/workflows/update-country-maps.yml:
##########
@@ -0,0 +1,166 @@
+name: Update Country Maps
+
+# Regenerates plugins/plugin-chart-country-map's GeoJSON files, countries.ts
+# and docs/data/countries.json from upstream Natural Earth + community geo
+# data, by running the same notebook `npm run update-maps` runs locally.
+#
+# Runs weekly and on workflow_dispatch, but only opens or updates a PR when
+# the regenerated output actually differs from what's on `master` - most
+# weeks this is a no-op. Unlike a translation-string catch-up, nothing here
+# auto-merges: a border/region change is the kind of thing that needs an
+# actual human look at the rendered maps. Community geo sources are pinned
+# to a commit where practical; the live WHO ArcGIS query can't be pinned
+# (it's a live service, not a repo), so a run can still legitimately pick up
+# an unreviewed upstream change there. The job itself fails (separately from
+# whether a PR was opened) if the notebook hits a cell error, so a broken
+# upstream source doesn't just quietly go green.
+on:
+ schedule:
+ - cron: "0 7 * * 1" # Mondays 07:00 UTC, an hour after
sync-translation-catalogs.yml
+ workflow_dispatch: {}
+
+permissions:
+ contents: read
+
+concurrency:
+ group: update-country-maps
+ cancel-in-progress: false
+
+jobs:
+ update-country-maps:
+ runs-on: ubuntu-26.04
Review Comment:
This job also runs on the weekly schedule in any fork with Actions enabled,
where it will download the sources, regenerate maps, force-push a bot branch
and try to open a PR against the fork. The other scheduled workflows here gate
on the upstream repository. Should this one do the same while keeping manual
`workflow_dispatch` runs working?
```suggestion
if: github.event_name != 'schedule' || github.repository ==
'apache/superset'
runs-on: ubuntu-26.04
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
