Hi all, Quick update: the PR to bring the guides to the web site has been merged.
This allowed me to set PR [3553], which adds CI for the guides to "ready for review." Cheers, Robert [3553] https://github.com/apache/polaris/pull/3553 On Tue, Feb 3, 2026 at 3:02 PM Robert Stupp <[email protected]> wrote: > Hi all, > > The intermediate PRs have all been merged now and the PR [3550] to move > the guides to the web site is ready now. > > Robert > > [3550] https://github.com/apache/polaris/pull/3550 > > On Thu, Jan 29, 2026 at 11:39 AM Robert Stupp <[email protected]> wrote: > >> Some PRs are already up. But digging more into docker-compose and "all >> the things" around it discovered a couple of more things. >> I've summarized the findings and came up with some recommendations here >> https://github.com/snazy/polaris/blob/guides-ci/site/content/guides/_index.md#authoring-guides >> (on the WIP branch for "guides-CI"). >> It covers Docker Compose, the usage of 'curl', health-checks, service >> dependencies and "final setup services". >> >> I'll come up with a few more PRs to fix the existing guides after the >> existing PRs are in and do not conflict with the PRs for the new findings. >> >> >> On Wed, Jan 28, 2026 at 11:57 PM Adnan Hemani via dev < >> [email protected]> wrote: >> >>> Robert, this is super cool! Looks great, and I'm looking forward to the >>> separate PRs to get us into shape for getting the CIs working. >>> >>> Personally, I really like that we are not having to create separate CI >>> scripts as compared to just using the markdown file directly - this will >>> be >>> helpful to ensure that the CI scripts and the guide don't accidentally >>> drift. >>> >>> Best, >>> Adnan Hemani >>> >>> On Wed, Jan 28, 2026 at 4:24 AM Robert Stupp <[email protected]> wrote: >>> >>> > Hi all, >>> > >>> > And here's a working version for "CI on the guides" : >>> > https://github.com/apache/polaris/pull/3553. It builds upon PR #3550 >>> > >>> > A CI run, takes about 10 minutes for all guides, looks like this one: >>> > https://github.com/apache/polaris/actions/runs/21437618287?pr=3553 >>> > >>> > The TL;DR of how it works: >>> > - It exercises for all getting-started guides Markdown files in CI >>> > - Testing can be run locally as well, either for all guides or a single >>> > guide >>> > - For each Markdown file, the `shell` and `sql` code blocks are >>> extracted >>> > - Supports "docker compose" and Spark SQL Shell invocations >>> > - Custom assertions can be added (as a `shell` code block in an HTML >>> > comment, so the assertions aren't rendered on the web site) >>> > >>> > While working on this, I had to fix a couple of things in the >>> > docker-compose files. Some are related to docker-compose service >>> > dependencies and timing, others due to the guides just not working >>> anymore. >>> > I'll come up with separate PRs to address the findings individually. >>> > >>> > Robert >>> > >>> > >>> > On Mon, Jan 26, 2026 at 3:29 PM Robert Stupp <[email protected]> wrote: >>> > >>> > > Hi all, >>> > > >>> > > Here's a prototype as a PR >>> https://github.com/apache/polaris/pull/3550 - >>> > > please try it out and let me know what you think. >>> > > >>> > > On Tue, Jan 20, 2026 at 9:12 PM Dmitri Bourlatchkov < >>> [email protected]> >>> > > wrote: >>> > > >>> > >> Hi All, >>> > >> >>> > >> Building CI for getting-started guides sounds useful to me. I >>> suppose >>> > we'd >>> > >> have to formalize the format of the related `.md` files somehow to >>> make >>> > >> automated execution possible. >>> > >> >>> > >> I wonder about the reliability of these tests too. If CI is flaky >>> (e.g. >>> > >> containers not starting properly), it might be an irritation more >>> than >>> > an >>> > >> aid. It's worth a try in any case. >>> > >> >>> > >> Cheers, >>> > >> Dmitri. >>> > >> >>> > >> On Tue, Jan 20, 2026 at 2:48 PM Yong Zheng <[email protected]> >>> wrote: >>> > >> >>> > >> > 100%. There are so many open source projects with outdated >>> > >> getting-started >>> > >> > examples and it will be nice to have these in our CI pipelines. >>> The >>> > only >>> > >> > concern on my end is how do we defined coverage for >>> getting-started >>> > >> > example? Currently most of them have simple examples to do >>> following: >>> > >> > 1. use catalog >>> > >> > 2. create namespace >>> > >> > 3. create table under namespace >>> > >> > 4. create some dummy data >>> > >> > >>> > >> > Will these be sufficient for CI? With these, we will only know the >>> > basic >>> > >> > stuff work but if users tried to more complex things, we can't >>> really >>> > >> > guarantee it will still work. But will this be sufficient? >>> > >> > >>> > >> > Thanks, >>> > >> > Yong Zheng >>> > >> > >>> > >> > On 2026/01/20 10:55:30 Robert Stupp wrote: >>> > >> > > Hi all, >>> > >> > > >>> > >> > > We have a nice collection of getting started guides in the >>> source >>> > >> > > repository [1]. >>> > >> > > The user-targeting description of each guide is in a README.md >>> file. >>> > >> > > >>> > >> > > I would like to start a discussion and gather feedback about two >>> > >> > > topics regarding the getting-started guides: >>> > >> > > >>> > >> > > 1. Website: >>> > >> > > The user facing getting-started guides are well written but not >>> very >>> > >> > > visible to users, because those are not on the web site. >>> > >> > > What are your thoughts of moving the getting-started guides to >>> the >>> > >> > website? >>> > >> > > >>> > >> > > 2. CI coverage: >>> > >> > > Most, actually all, getting-started guides include code snippets >>> > >> > > referencing Docker compose files. >>> > >> > > Manually verifying these code snippets and Docker compose files, >>> > >> > > during initial contribution or when those are being updated, is >>> > quite >>> > >> > > some work. >>> > >> > > I _think_ we can automate the verification of the code >>> snippets, and >>> > >> > > with those the Docker compose files, in CI. >>> > >> > > The overall idea is to parse the getting-started guide markdown >>> and >>> > >> > > let a workflow execute the code blocks for shell/bash. >>> > >> > > I am not sure whether all guides can actually be verified, >>> because >>> > >> > > some of those Docker compose files start a couple of containers, >>> > which >>> > >> > > can be a resource (RAM/CPU) issue in GitHub's hosted runners. >>> > >> > > The alternatives would be: >>> > >> > > - Never update the getting-started guides with the risk that >>> those >>> > >> > > become stale and outdated. >>> > >> > > - Keep the manual verification process. >>> > >> > > Any thoughts on this? >>> > >> > > >>> > >> > > Robert >>> > >> > > >>> > >> > > >>> > >> > > [1] https://github.com/apache/polaris/tree/main/getting-started >>> > >> > > >>> > >> > >>> > >> >>> > > >>> > >>> >>
