edmondop opened a new issue, #24569:
URL: https://github.com/apache/datafusion/issues/24569
A **cookbook** is a collection of short, self-contained, task-framed recipes
("I want to do X → here's how, here's why, here's the gotchas") — as opposed to
reference docs (API surface) or a linear tutorial (one path through the
basics). It's a proven format once a technology has enough surface area that no
single tutorial covers it well.
**Why it'd help DataFusion**
DataFusion is powerful enough that the hard part for new adopters isn't
"does this feature exist" but "how do I wire it together for my use case" —
custom `TableProvider`, which UDF/UDAF/UDWF/UDTF variant to use, pushdown,
catalog integration, remote object stores. A cookbook is the standard way
projects close that gap.
**Examples of cookbooks elsewhere**
Published books:
- AWS Cookbook (O'Reilly) —
https://www.amazon.com/AWS-Cookbook-Recipes-Success/dp/1492092606
- Terraform Cookbook (O'Reilly) —
https://www.amazon.com/Terraform-Cookbook-Kerim-Satirli/dp/1804613477
- Django Web Development Cookbook (Packt, 4th ed.) —
https://www.amazon.com/Django-Web-Development-Cookbook-development/dp/1838987428
- Polars Cookbook (Packt) —
https://www.amazon.com/Polars-Cookbook-practical-transform-manipulate/dp/1805121154
Same-ecosystem, in-project precedents:
- Rust Cookbook — https://github.com/rust-lang-nursery/rust-cookbook
- ADBC Python Cookbook, embedded in the Arrow docs —
https://arrow.apache.org/adbc/current/python/recipe/index.html
**Relationship with examples**
`datafusion-examples`
(https://github.com/apache/datafusion/blob/main/datafusion-examples/README.md)
already has ~50 runnable, tested examples across 11 groups: UDF/UDAF/UDWF/UDTF,
custom `TableProvider`, custom file formats, Parquet encryption (incl. KMS),
custom optimizer/analyzer rules, Flight SQL, querying S3, remote catalogs, and
more.
These examples can be turned into recipe for a cookbook by rewriting each as
a short step-by-step recipe — problem statement, code, explanation, pitfalls.
Probably 50 recipes are excessive, a good number would be in the 15-20, at
least to start.
**Organization options**
Existing cookbooks adopt both the colocated and the separate repo approach,
so we would need to decide between these two options:
- As a new section of the existing docs site, next to
`datafusion-examples`, versioned with the code (closest precedent: ADBC's
cookbook, which lives directly in the Sphinx docs).
- A separate repo, lower barrier for outside contributors, easier to
restructure freely, but risks drifting from releases (precedent: Rust Cookbook,
Spice.ai's cookbook repo).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
