This is an automated email from the ASF dual-hosted git repository. krickert pushed a commit to branch gazetteer-api in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit e4bad3e6549e27da6d4439aca8725ac41abeccdb Author: Kristian Rickert <[email protected]> AuthorDate: Thu Jul 16 01:13:29 2026 -0400 OPENNLP-1879: Record the gazetteer data licenses per LEGAL-732 and fix the Overture divisions license to ODbL --- .../opennlp-geo/dev/README-gazetteer-data.md | 73 ++++++++++++++++++++++ .../opennlp-geo/dev/derive-overture-divisions.py | 13 ++-- .../main/java/opennlp/geo/OvertureGazetteer.java | 9 +-- 3 files changed, 86 insertions(+), 9 deletions(-) diff --git a/opennlp-extensions/opennlp-geo/dev/README-gazetteer-data.md b/opennlp-extensions/opennlp-geo/dev/README-gazetteer-data.md new file mode 100644 index 000000000..11188428e --- /dev/null +++ b/opennlp-extensions/opennlp-geo/dev/README-gazetteer-data.md @@ -0,0 +1,73 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# Gazetteer data: where it comes from and how to regenerate it + +The `opennlp-geo` module reads plain text tables instead of parsing upstream distribution formats. This directory holds the scripts that derive those tables. None of the scripts are part of the build; they exist so every bundled or derivable table can be regenerated from its public source and reviewed as a plain diff. + +## The three gazetteers and their data + +| Gazetteer | Data source | License of the data | Shipped in the jar? | +|---|---|---|---| +| `opennlp.geo.BundledGazetteer` | Natural Earth, Populated Places theme | Public domain | Yes: `naturalearth-populated-places.txt` | +| `opennlp.geo.GeoNamesGazetteer` | GeoNames main-table extracts, downloaded by the user | CC-BY 4.0 (see the `readme.txt` alongside the downloads) | No. The user downloads the file at their own explicit action; attribution and license compliance are the caller's responsibility, as the class javadoc states. | +| `opennlp.geo.OvertureGazetteer` | Overture Maps, divisions theme, flattened by a script in this directory | ODbL 1.0. Attribution and database share-alike terms apply, and they follow the derived table, which is built and owned by the user. | No. Nothing ODbL-licensed is distributed by the project in any form. | + +Only public-domain data is bundled; every other dataset is acquired by the user on the user's own infrastructure, so the project never redistributes it. The license classifications behind this split are on record in LEGAL-732: CDLA-Permissive-2.0 was judged Category A, ODbL Category X, public-domain/CC0 data needs only a LICENSE section, and data fetched at runtime on the user's machine creates no ASF licensing obligation at all because nothing is redistributed. Note the theme distinctio [...] + +## `derive-populated-places.py` + +Regenerates the bundled table from the Natural Earth Populated Places GeoJSON. + +``` +derive-populated-places.py <ne_10m_populated_places.geojson> [output.txt] +``` + +1. Download `ne_10m_populated_places.geojson` from the `nvkelso/natural-earth-vector` GitHub repository, the mirror of naturalearthdata.com. The script header records the mirror commit the current table was extracted from. +2. Run the script. It needs only the Python 3 standard library. +3. The output is `naturalearth-populated-places.txt`, the semicolon-separated, pure-ASCII, LF-terminated table read by `BundledGazetteer`. Copy it over `src/main/resources/opennlp/geo/naturalearth-populated-places.txt` and review the diff. + +The upstream GeoJSON carries encoding damage in some name fields (an accented letter replaced by `?` or by a wrong ASCII letter). The script detects damage by character-class artifacts and repairs it through explicit, audited replacement tables in the script source; values it cannot verify are omitted and counted, never carried. It hard-fails on any anomaly it does not recognize, so refreshing to a newer Natural Earth release forces a reviewed decision instead of silently importing new c [...] + +## `derive-overture-divisions.py` + +Derives a division table from an Overture Maps release, for `OvertureGazetteer`. + +``` +derive-overture-divisions.py <release> [output.txt] [min_locality_population] +``` + +for example: + +``` +derive-overture-divisions.py 2026-06-18.0 overture-divisions.txt 10000 +``` + +1. Requires the `duckdb` Python package with the `httpfs` and `spatial` extensions, and network access; it reads the divisions theme (Point features of type `division`) as Parquet directly from the Overture release bucket. +2. The output is a tab-separated table, one row per division: id, primary name, comma-separated alternate names, latitude, longitude, ISO 3166-1 alpha-2 country code, subtype, population. A `#` header line carries the derivation record (release, date, filters), and rows are ordered by id so a regeneration diffs cleanly against the previous run. +3. Divisions include countries and regions, not only settlements, which is why a derived table also resolves mentions like `Australia` or `Bavaria` that place-only gazetteers miss. + +## GeoNames data (no script needed) + +`GeoNamesGazetteer` reads the GeoNames main table format directly, so there is nothing to derive. Download a filtered city extract (for example `cities500.zip`) from the GeoNames export dump area, unzip it, and load the `.txt` with `GeoNamesGazetteer.load(...)`. The loader indexes the whole table in memory, so use the filtered extracts rather than the full `allCountries` dump; memory grows with row and alternate-name count. + +## Regeneration checklist + +1. Record where the input came from: release id, mirror commit, or download date. The scripts embed this in their output headers or carry it in constants at the top of the script; update those constants when you refresh. +2. Run the script and review the whole diff of the regenerated table before committing anything. +3. If a script fails on an anomaly, that is the intended behavior: inspect the new upstream value, extend the audited repair tables only with values you have verified, and rerun. +4. Never commit downloaded upstream files, and never move an attribution-required or share-alike dataset into `src/main/resources`. diff --git a/opennlp-extensions/opennlp-geo/dev/derive-overture-divisions.py b/opennlp-extensions/opennlp-geo/dev/derive-overture-divisions.py index b5f0ad3e8..721c5ea72 100644 --- a/opennlp-extensions/opennlp-geo/dev/derive-overture-divisions.py +++ b/opennlp-extensions/opennlp-geo/dev/derive-overture-divisions.py @@ -20,9 +20,11 @@ Usage: derive-overture-divisions.py <release> [output.txt] [min_locality_popula Input: the divisions theme, type=division (Point features), read as Parquet directly from the Overture release bucket. Requires the duckdb Python package with the - httpfs and spatial extensions, and network access. The data is published under - a permissive data license (CDLA-Permissive-2.0); check and record the license - of the release you pull. + httpfs and spatial extensions, and network access. The divisions theme is published + under the Open Database License (ODbL), which requires attribution and applies + share-alike terms to derivative databases, so the derived table carries those + terms too; check and record the license and attribution requirements of the + release you pull. Output: the tab-separated table read by opennlp.geo.OvertureGazetteer, one row per division: id, primary name, comma-separated alternate names, latitude, longitude, ISO 3166-1 alpha-2 country code, subtype, population. A '#' header @@ -85,8 +87,9 @@ def main() -> None: out.write("# derivation of Overture Maps divisions (read by opennlp.geo.OvertureGazetteer).\n") out.write(f"# Derivation record: release {release}, subtypes {','.join(KEPT_SUBTYPES)},\n") out.write(f"# locality population floor {min_population}, derived {datetime.date.today()}\n") - out.write(f"# by dev/derive-overture-divisions.py. Upstream license: CDLA-Permissive-2.0\n") - out.write("# (verify against the release notes of the release named above).\n") + out.write("# by dev/derive-overture-divisions.py. Upstream license: ODbL, attribution and\n") + out.write("# database share-alike terms apply to this derived table (verify against the\n") + out.write("# licensing documentation of the release named above).\n") for row in rows: (division_id, name, alternates, lat, lon, country, subtype, population) = row if not name or lat is None or lon is None: diff --git a/opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/OvertureGazetteer.java b/opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/OvertureGazetteer.java index f5adb7839..bf22bb224 100644 --- a/opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/OvertureGazetteer.java +++ b/opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/OvertureGazetteer.java @@ -40,10 +40,11 @@ import opennlp.tools.geo.GeoPoint; * per division with id, primary name, comma-separated alternate names, coordinates, * country code, Overture subtype, and population. * - * <p>The upstream data is published under a permissive data license and distributed as - * partitioned Parquet, which this module deliberately does not parse; the derivation - * script flattens the division features into this plain table, and the script's output - * header carries the derivation record. Because divisions include countries and + * <p>The upstream divisions theme is published under the Open Database License, whose + * attribution and database share-alike terms follow the derived table; it is + * distributed as partitioned Parquet, which this module deliberately does not parse. + * The derivation script flattens the division features into this plain table, and the + * script's output header carries the derivation record. Because divisions include countries and * regions, not only settlements, a derived table also resolves mentions like * {@code Australia} or {@code Bavaria} that place-only gazetteers miss.</p> *
