kosiew opened a new issue, #26803: URL: https://github.com/apache/datafusion/issues/26803
## Context Follow up to this [comment](https://github.com/apache/datafusion/pull/26025#issuecomment-6368643837) ## Problem The "Known Users" active list in `docs/source/user-guide/introduction.md` uses one criterion, written only as an HTML comment: ```md <!-- "Active" means github repositories that had at least one commit in the last 6 months --> ``` That criterion alone has two gaps: - It is not enforced. As of 2026-10-27 (cutoff 2026-04-27), five listed entries already fail it. - It says nothing about adoption. A brand-new, single-author repo qualifies as soon as it has a commit. KalamDB (0 stars, 1 contributor, created 2026-06-23) is listed alongside InfluxDB (31.8k stars, ~700 contributors). ## Why it matters The list reads as an endorsement of production-grade DataFusion users. Stale or very small entries weaken that signal and make it harder to judge PRs that add links. Without objective thresholds, every addition becomes a judgement call for reviewers. ## Invariant / desired behavior An entry in the active list must link to a public GitHub repo that meets **all** of the following: 1. At least 1 commit in the last 6 months (existing rule). 2. At least **N stars** (proposed: 100). 3. At least **M contributors** (proposed: 5). The criteria are stated in visible doc text, not only in an HTML comment. Entries without a public repo follow an explicitly documented rule (see open questions). ## Current data (GitHub API, 2026-10-27) Contributors = commit authors from the GitHub `contributors` endpoint (`anon=1`); counts are approximate and capped for very large repos. | Project | Repo | Stars | Contributors | Last commit | Active (6 mo) | | --- | --- | ---: | ---: | --- | :-: | | InfluxDB | influxdata/influxdb | 31,786 | 696 | 2026-10-22 | ✅ | | OpenObserve | openobserve/openobserve | 22,502 | 166 | 2026-10-27 | ✅ | | Cube Store | cube-js/cube | 21,084 | 393 | 2026-10-26 | ✅ | | rerun.io | rerun-io/rerun | 11,650 | 195 | 2026-10-26 | ✅ | | ParadeDB | paradedb/paradedb | 9,454 | 151 | 2026-10-27 | ✅ | | Lance | lance-format/lance (moved) | 7,216 | 325 | 2026-10-27 | ✅ | | GreptimeDB | GreptimeTeam/greptimedb | 6,767 | 167 | 2026-10-27 | ✅ | | Arroyo | ArroyoSystems/arroyo | 5,068 | 44 | 2026-10-26 | ✅ | | Restate | restatedev/restate | 4,610 | 55 | 2026-10-26 | ✅ | | Sail | lakehq/sail | 3,474 | 52 | 2026-10-27 | ✅ | | ROAPI | roapi/roapi | 3,435 | 40 | 2026-03-25 | ❌ | | delta-rs | delta-io/delta-rs | 3,350 | 322 | 2026-10-25 | ✅ | | Vortex | vortex-data/vortex | 3,284 | 92 | 2026-10-26 | ✅ | | LakeSoul | lakesoul-io/LakeSoul | 3,260 | 32 | 2026-10-27 | ✅ | | Spice.ai | spiceai/spiceai | 3,109 | 63 | 2026-10-26 | ✅ | | HoraeDB | apache/horaedb | 2,823 | 45 | 2025-12-11 | ❌ (archived) | | Parseable | parseablehq/parseable | 2,564 | 63 | 2026-10-26 | ✅ | | Kubeflow Trainer | kubeflow/trainer | 2,258 | 315 | 2026-10-27 | ✅ | | Ballista | apache/datafusion-ballista | 2,157 | 397 | 2026-10-26 | ✅ | | Feldera | feldera/feldera | 2,139 | 79 | 2026-10-27 | ✅ | | Auron | apache/auron | 1,837 | 69 | 2026-10-26 | ✅ | | CnosDB | cnosdb/cnosdb | 1,754 | 43 | 2026-10-23 | ✅ | | Iceberg-rust | apache/iceberg-rust | 1,433 | 213 | 2026-10-27 | ✅ | | ArkFlow | arkflow-rs/arkflow | 1,315 | 5 | 2026-10-25 | ✅ | | Comet | apache/datafusion-comet | 1,304 | 160 | 2026-10-26 | ✅ | | GlareDB | GlareDB/glaredb | 1,024 | 5 | 2025-11-05 | ❌ | | dbt Fusion | dbt-labs/dbt-fusion | 729 | 89 | 2026-06-04 | ✅ (barely) | | Xorq | xorq-labs/xorq | 556 | 19 | 2026-10-27 | ✅ | | SedonaDB | apache/sedona-db | 520 | 41 | 2026-10-27 | ✅ | | Seafowl (EDB Postgres Lakehouse) | splitgraph/seafowl | 514 | 11 | 2025-01-08 | ❌ | | VegaFusion | vega/vegafusion | 424 | 7 | 2026-03-13 | ❌ | | qv | timvw/qv | 372 | 7 | 2026-09-23 | ✅ | | Kamu | kamu-data/kamu-cli | 351 | 15 | 2026-10-26 | ✅ | | datafusion-dft | datafusion-contrib/datafusion-dft | 205 | 7 | 2026-10-26 | ✅ | | Infino | infino-ai/infino | 138 | 14 | 2026-10-27 | ✅ | | Sleeper | gchq/sleeper | 107 | 23 | 2026-10-26 | ✅ | | KalamDB | jamals86/KalamDB | 0 | 1 | 2026-06-23 | ✅ | **No public GitHub repo (stars/contributors not measurable):** datapress, Funnel, hiop, Hotdata, IceGate, Massive.com, R2 Query Engine, Streamling, Supermetal, Synnada, Telemetry. ### Observations - **Already fail the 6-month rule:** ROAPI, HoraeDB (archived), GlareDB, Seafowl (backs the EDB Postgres Lakehouse entry), VegaFusion. - **About to fail:** dbt Fusion (last commit 2026-06-04, about 5 weeks from the cutoff). - **Clear outlier:** KalamDB (0 stars, 1 contributor, repo created 2026-06-23). - Stars among active repos are mostly in the hundreds to thousands; the lowest are Sleeper (107), Infino (138), datafusion-dft (205). - Low contributor counts (≤7): ArkFlow (5), datafusion-dft (7), qv (7) are active; GlareDB (5) and VegaFusion (7) are already inactive. ### Threshold impact | Thresholds | Removed from active list (beyond 6-month failures) | | --- | --- | | ≥100 stars, ≥5 contributors (proposed) | KalamDB | | ≥100 stars, ≥10 contributors | KalamDB, ArkFlow, datafusion-dft, qv | ### Stale links found - Lance: `lancedb/lance` → `lance-format/lance` - HoraeDB: `apache/incubator-horaedb` → `apache/horaedb` (archived) - Restate: links to the org `restatedev`, not the repo `restatedev/restate` ## Proposed direction 1. Replace the HTML comment with a short visible paragraph that lists the three criteria and the exemption rule for entries without a public repo. 2. Apply the criteria to the current list: move failing entries to "less active projects", and remove or move entries below the thresholds. 3. Fix the stale links above. 4. Optional: commit a small script (for example under `dev/`) that prints stars, contributors and last commit for every GitHub link in the section, so the list can be re-audited periodically. ## Scope ### In - Inclusion criteria text in `docs/source/user-guide/introduction.md`. - Re-categorizing current "Known Users" entries against the criteria. - Fixing the stale repo links listed above. ### Out - The "Integrations and Extensions" / "Language Bindings" lists, unless maintainers want the same criteria there (follow-up). - Automated CI enforcement of the thresholds. - Judging projects on anything beyond the stated criteria. ## Open questions - Final values for N (stars) and M (contributors). Proposed: 100 / 5. - Entries without a public repo (11 today): exempt them (e.g. as "commercial products using DataFusion"), list them separately, or require some other evidence of use (a blog post or docs page). - Should entries that fail the thresholds go to "less active projects" or be removed entirely? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
