On: ASF Admin's don't understand project health due to repo structure -- i disagree with the idea that itshould at all inform/dictate how work gets structured and completed by the Flagon project.
I am not sure I understand how documentation improves by having a monorepo. I guess if pitched about a single README section. Open Source projects commonly have docs that get autogenerated and published to a webpage, and can be done so from a number of disparate repos to a single location. Off handed, it seems just as straightforward to invest in the ability to "treat a number of highly inter-related projects as" coupled, than to refactor repos and invest in alternative testing infra. Either way, the problem outlined appears actually to surface the urgency for increasingly robust CI? The rest is docs and education, which can be handled in a number of ways. I don't buy the necessity of a monorepo, but *I accept the preferences of those that will choose to do the work*. Wondering: has anyone done the equivalent planning work to actually surface a pros/cons; ex: what would the tooling need to look like to address the problems and maintain the current repo structure? On Tue, Jan 30, 2024 at 2:18 PM Joshua Poore <poor...@apache.org> wrote: > The idea is growing on me—I’m pretty sure we can generate Pypi and NPM > releases from the same repo, different folders. > > Agree that now is the time to do it before user base on Distill grows. > > I do think we should run a consensus/community VOTE on this. Let me > refresh myself on which ruleset we should use for this. It’s a big change, > so may need to fall under the same guidelines as RELEASE. > > Josh > > > On Jan 30, 2024, at 4:05 PM, Evan Jones <evan.a.jon...@gmail.com> wrote: > > > > *When should this process begin? *In terms of where UserALE and currently > > are.** > > I want to get all *non-breaking* PRs merged into UserALE/test and > > Distill/main and then do a new release of each of them, assuming there > are > > changes to release. UserALE has been queued up for a 2.4.0 for quite some > > time. I don't think we have anything new for Distill. That way the > monorepo > > consolidation would happen on top of clean releases. > > > > I want to start the monorepo migration ASAP which means getting these > > releases in progress ASAP. > > > > *The scope and sequence of the changes? Roadmap for the migration?* > > If by scope you mean what aspects of the code base does this touch and > > who's impacted, then the answer to the former is all aspects -- albeit > > purely in a structural way -- and the answer to the latter is > > contributors/devs only. Nothing about our individual projects will change > > to start except they'll all be moved into the Flagon repo as > > sub-directories. Within those directories, everything will be identical > to > > how they are currently. The only real repo that will see changes is the > > Flagon repo. > > > > Because of this consolidation, the way developers contribute to the > > projects will change. People will fork the Flagon Suite repo and merge > back > > into it regardless of the flagon product (distill, userale) they're > > contributing to. Each project will still be released the same way. > > > > On the ASF side, we would have a single release process for the entire > > Flagon project, rather than different releases for different > sub-projects. > > My hope is this vastly lowers overhead. > > > > The steps proposed above are the roadmap, more or less. As far as > timeline > > goes, I think the biggest bottleneck will be getting the current releases > > out. I believe I can do the consolidation into a single repo in one > weekend. > > > > *For documentation, do we intend to use ReadTheDocs and Sphinx? Any > others > > that you might know of?* > > > > All documentation and website files would be stored in a single directory > > in the repo. It sits alongside userale, distill, etc. The API > documentation > > for each project should leverage automated doc generators: > > > > - Distill = Sphinx + PyDoc --> ReadTheDocs > > - UserALE = Sphinx + JsDoc --> ReadTheDocs > > > > The main flagon website should be the primary source of documentation for > > things like tutorials, getting started, concept definitions, etc. It can > > then point to the ReadTheDocs pages for each of packages' API documents. > > > > The overhauling of the documentation is not really couple to the monorepo > > migration, though. The monorepo migration merely lowers to burden of the > > workflow I describe above by centralizing everything. > > > > *I think we should set up a robust CI/CD pipeline to handle the monorepo > to > > ensure that all projects are integrated seamlessly and that automated > tests > > are in place for the monorepo.* > > > > Yes. > > > > Best > > > > Evan Jones > > Website: www.ea-jones.com > > > > > > On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <agha...@umd.edu> wrote: > > > >> Hi Evan, > >> > >> I support this monorepo structure - as you stated, there are > >> several benefits of consolidating the Apache Flagon projects. Just a few > >> questions/suggestions: > >> > >> - Do you have a timeline in mind for: > >> > >> > >> - When should this process begin? *In terms of where UserALE and > Distill > >> currently are.* > >> - The scope and sequence of the changes? Roadmap for the migration? > >> > >> > >> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any > >> others that you might know of? > >> - I think we should set up a robust CI/CD pipeline to handle the > >> monorepo to ensure that all projects are integrated seamlessly and > that > >> automated tests are in place for the monorepo. > >> > >> > >> Best, > >> *Amir M. Ghaemi* > >> > >> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <evan.a.jon...@gmail.com> > >> wrote: > >> > >>> There are likely a few more steps: > >>> > >>> 1. Send a do not merge email for all repos > >>> 2. Add userale/distill code, git tree, and github actions to the flagon > >>> repo > >>> 3. *Consolidate documents into a single repo* > >>> 4. *Migrate to a build tool for monorepos like Bazel* (example: > >>> > https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli; > >>> Beam uses gradle) > >>> 5. Mark userale/distill repos as archived and update read me with a > link > >>> 6. Send a merge freely email > >>> 7. Update flagon readme and flagon site links > >>> > >>> The bolded things could happen after the structural changes depending > on > >>> our priorities. If we implement those changes as part of the migration > it > >>> would simplify contribution conventions post migration but slow things > >> down > >>> in the short-term. If we implement them after, we'll open back up to > >> merges > >>> faster but the repo will be in a "dirty" state for longer as we > >> transition > >>> to monorepo best practices. > >>> > >>>> I don't think there's anything besides the flagon site that points to > >> any > >>> github repo, but I'm not sure. > >>> > >>> We definitely want to check our p's and q's though, especially on the > >>> Apache side. Are the Distill, UserALE and Flagon repos treated as > >> separate > >>> projects / releases in the eyes of Apache or are their releases all > >> bundled > >>> into Flagon releases? > >>> > >>> Best > >>> > >>> Evan Jones > >>> Website: www.ea-jones.com > >>> > >>> > >>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <j...@apache.org> wrote: > >>> > >>>> I'm all for a monorepo. Especially because it's hard to get visibility > >>>> into a single project when its spread across multiple repos in the sea > >> of > >>>> the Apache GitHub org. > >>>> > >>>> Are these what the steps look like for moving to a monorepo? > >>>> > >>>> 1. Send a do not merge email for all repos > >>>> 2. Add userale/distill code, git tree, and github actions to the > flagon > >>>> repo > >>>> 3. Mark userale/distill repos as archived and update read me with a > >> link > >>>> 4. Send a merge freely email > >>>> 5. Update flagon readme and flagon site links > >>>> > >>>> I don't think there's anything besides the flagon site that points to > >> any > >>>> github repo, but I'm not sure. > >>>> > >>>> On 2024/01/17 00:36:41 Evan Jones wrote: > >>>>> Hi all, > >>>>> > >>>>> I've recently had more bandwidth to contribute to Flagon. Recently, I > >>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up > >> the > >>>>> instrumentation team at ARLIS. My goal is for us to ramp up our > >>>>> contribution to the Flagon suite. > >>>>> > >>>>> After spending quite a bit of time thinking about Flagon, I've come > >> to > >>>>> believe the best way for Flagon to deliver truly differentiated value > >>> to > >>>>> end users is to deliver on the original vision of all the > >> sub-projects: > >>>>> > >>>>> - UserALE for instrumentation > >>>>> - Distill for analytics > >>>>> - STOUT as a UI for the analytics, metrics, and experiment > >>>>> - TAP (re-envisioned) as the infrastructure platform to connect > >> all > >>>>> those pieces > >>>>> > >>>>> By eventually building all these pieces, Flagon could be a very > >> strong > >>>>> open-source alternative to services like Google Analytics, LogRocket, > >>> and > >>>>> others. I intend to socialize the above vision more in the future. > >> For > >>>> now > >>>>> I digress since that's not the point of this email. > >>>>> > >>>>> The point of this email is to share an RFC > >>>>> < > >>>> > >>> > >> > https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing > >>>>> > >>>>> for *an overhaul I believe needs to happen as a precursor to > >> achieving > >>>> the > >>>>> above vision*. > >>>>> > >>>>> *Please share feedback, comments and concerns right on the google > >> doc*. > >>>>> I've also opened a GH issue < > >>> https://github.com/apache/flagon/issues/44> > >>>> if > >>>>> folks would like to contribute to the conversation there, though I > >>> prefer > >>>>> to keep as much commentary within the document itself to simplify > >>>> collation > >>>>> later. > >>>>> > >>>>> Many thanks! > >>>>> > >>>>> Best > >>>>> > >>>>> Evan Jones > >>>>> > >>>> > >>> > >> > >
