> > has anyone done the > equivalent planning work to actually surface a pros/cons; ex: what would > the tooling need to look like to address the problems and maintain the > current repo structure?
Not that I know of, but it's worth exploring first. I am not sure I understand how documentation improves by having a monorepo. Documentation doesn't magically improve. You still have to do the work. it seems just as straightforward to invest in the ability to "treat > a number of highly inter-related projects as" coupled, Guidance on this more straightforward way? This RFC contends a monorepo is the straight-forward ability. Best Evan Jones Website: www.ea-jones.com On Tue, Jan 30, 2024 at 8:16 PM Austin Bennett <aus...@apache.org> wrote: > On: ASF Admin's don't understand project health due to repo structure -- > i disagree with the idea that itshould at all inform/dictate how work gets > structured and completed by the Flagon project. > > I am not sure I understand how documentation improves by having a > monorepo. I guess if pitched about a single README section. Open Source > projects commonly have docs that get autogenerated and published to a > webpage, and can be done so from a number of disparate repos to a single > location. > > Off handed, it seems just as straightforward to invest in the ability to > "treat > a number of highly inter-related projects as" coupled, than to refactor > repos and invest in alternative testing infra. Either way, the problem > outlined appears actually to surface the urgency for increasingly robust > CI? > > The rest is docs and education, which can be handled in a number of ways. > > I don't buy the necessity of a monorepo, but *I accept the preferences of > those that will choose to do the work*. Wondering: has anyone done the > equivalent planning work to actually surface a pros/cons; ex: what would > the tooling need to look like to address the problems and maintain the > current repo structure? > > On Tue, Jan 30, 2024 at 2:18 PM Joshua Poore <poor...@apache.org> wrote: > > > The idea is growing on me—I’m pretty sure we can generate Pypi and NPM > > releases from the same repo, different folders. > > > > Agree that now is the time to do it before user base on Distill grows. > > > > I do think we should run a consensus/community VOTE on this. Let me > > refresh myself on which ruleset we should use for this. It’s a big > change, > > so may need to fall under the same guidelines as RELEASE. > > > > Josh > > > > > On Jan 30, 2024, at 4:05 PM, Evan Jones <evan.a.jon...@gmail.com> > wrote: > > > > > > *When should this process begin? *In terms of where UserALE and > currently > > > are.** > > > I want to get all *non-breaking* PRs merged into UserALE/test and > > > Distill/main and then do a new release of each of them, assuming there > > are > > > changes to release. UserALE has been queued up for a 2.4.0 for quite > some > > > time. I don't think we have anything new for Distill. That way the > > monorepo > > > consolidation would happen on top of clean releases. > > > > > > I want to start the monorepo migration ASAP which means getting these > > > releases in progress ASAP. > > > > > > *The scope and sequence of the changes? Roadmap for the migration?* > > > If by scope you mean what aspects of the code base does this touch and > > > who's impacted, then the answer to the former is all aspects -- albeit > > > purely in a structural way -- and the answer to the latter is > > > contributors/devs only. Nothing about our individual projects will > change > > > to start except they'll all be moved into the Flagon repo as > > > sub-directories. Within those directories, everything will be identical > > to > > > how they are currently. The only real repo that will see changes is the > > > Flagon repo. > > > > > > Because of this consolidation, the way developers contribute to the > > > projects will change. People will fork the Flagon Suite repo and merge > > back > > > into it regardless of the flagon product (distill, userale) they're > > > contributing to. Each project will still be released the same way. > > > > > > On the ASF side, we would have a single release process for the entire > > > Flagon project, rather than different releases for different > > sub-projects. > > > My hope is this vastly lowers overhead. > > > > > > The steps proposed above are the roadmap, more or less. As far as > > timeline > > > goes, I think the biggest bottleneck will be getting the current > releases > > > out. I believe I can do the consolidation into a single repo in one > > weekend. > > > > > > *For documentation, do we intend to use ReadTheDocs and Sphinx? Any > > others > > > that you might know of?* > > > > > > All documentation and website files would be stored in a single > directory > > > in the repo. It sits alongside userale, distill, etc. The API > > documentation > > > for each project should leverage automated doc generators: > > > > > > - Distill = Sphinx + PyDoc --> ReadTheDocs > > > - UserALE = Sphinx + JsDoc --> ReadTheDocs > > > > > > The main flagon website should be the primary source of documentation > for > > > things like tutorials, getting started, concept definitions, etc. It > can > > > then point to the ReadTheDocs pages for each of packages' API > documents. > > > > > > The overhauling of the documentation is not really couple to the > monorepo > > > migration, though. The monorepo migration merely lowers to burden of > the > > > workflow I describe above by centralizing everything. > > > > > > *I think we should set up a robust CI/CD pipeline to handle the > monorepo > > to > > > ensure that all projects are integrated seamlessly and that automated > > tests > > > are in place for the monorepo.* > > > > > > Yes. > > > > > > Best > > > > > > Evan Jones > > > Website: www.ea-jones.com > > > > > > > > > On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <agha...@umd.edu> wrote: > > > > > >> Hi Evan, > > >> > > >> I support this monorepo structure - as you stated, there are > > >> several benefits of consolidating the Apache Flagon projects. Just a > few > > >> questions/suggestions: > > >> > > >> - Do you have a timeline in mind for: > > >> > > >> > > >> - When should this process begin? *In terms of where UserALE and > > Distill > > >> currently are.* > > >> - The scope and sequence of the changes? Roadmap for the > migration? > > >> > > >> > > >> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any > > >> others that you might know of? > > >> - I think we should set up a robust CI/CD pipeline to handle the > > >> monorepo to ensure that all projects are integrated seamlessly and > > that > > >> automated tests are in place for the monorepo. > > >> > > >> > > >> Best, > > >> *Amir M. Ghaemi* > > >> > > >> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <evan.a.jon...@gmail.com> > > >> wrote: > > >> > > >>> There are likely a few more steps: > > >>> > > >>> 1. Send a do not merge email for all repos > > >>> 2. Add userale/distill code, git tree, and github actions to the > flagon > > >>> repo > > >>> 3. *Consolidate documents into a single repo* > > >>> 4. *Migrate to a build tool for monorepos like Bazel* (example: > > >>> > > https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli; > > >>> Beam uses gradle) > > >>> 5. Mark userale/distill repos as archived and update read me with a > > link > > >>> 6. Send a merge freely email > > >>> 7. Update flagon readme and flagon site links > > >>> > > >>> The bolded things could happen after the structural changes depending > > on > > >>> our priorities. If we implement those changes as part of the > migration > > it > > >>> would simplify contribution conventions post migration but slow > things > > >> down > > >>> in the short-term. If we implement them after, we'll open back up to > > >> merges > > >>> faster but the repo will be in a "dirty" state for longer as we > > >> transition > > >>> to monorepo best practices. > > >>> > > >>>> I don't think there's anything besides the flagon site that points > to > > >> any > > >>> github repo, but I'm not sure. > > >>> > > >>> We definitely want to check our p's and q's though, especially on the > > >>> Apache side. Are the Distill, UserALE and Flagon repos treated as > > >> separate > > >>> projects / releases in the eyes of Apache or are their releases all > > >> bundled > > >>> into Flagon releases? > > >>> > > >>> Best > > >>> > > >>> Evan Jones > > >>> Website: www.ea-jones.com > > >>> > > >>> > > >>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <j...@apache.org> wrote: > > >>> > > >>>> I'm all for a monorepo. Especially because it's hard to get > visibility > > >>>> into a single project when its spread across multiple repos in the > sea > > >> of > > >>>> the Apache GitHub org. > > >>>> > > >>>> Are these what the steps look like for moving to a monorepo? > > >>>> > > >>>> 1. Send a do not merge email for all repos > > >>>> 2. Add userale/distill code, git tree, and github actions to the > > flagon > > >>>> repo > > >>>> 3. Mark userale/distill repos as archived and update read me with a > > >> link > > >>>> 4. Send a merge freely email > > >>>> 5. Update flagon readme and flagon site links > > >>>> > > >>>> I don't think there's anything besides the flagon site that points > to > > >> any > > >>>> github repo, but I'm not sure. > > >>>> > > >>>> On 2024/01/17 00:36:41 Evan Jones wrote: > > >>>>> Hi all, > > >>>>> > > >>>>> I've recently had more bandwidth to contribute to Flagon. > Recently, I > > >>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up > > >> the > > >>>>> instrumentation team at ARLIS. My goal is for us to ramp up our > > >>>>> contribution to the Flagon suite. > > >>>>> > > >>>>> After spending quite a bit of time thinking about Flagon, I've come > > >> to > > >>>>> believe the best way for Flagon to deliver truly differentiated > value > > >>> to > > >>>>> end users is to deliver on the original vision of all the > > >> sub-projects: > > >>>>> > > >>>>> - UserALE for instrumentation > > >>>>> - Distill for analytics > > >>>>> - STOUT as a UI for the analytics, metrics, and experiment > > >>>>> - TAP (re-envisioned) as the infrastructure platform to connect > > >> all > > >>>>> those pieces > > >>>>> > > >>>>> By eventually building all these pieces, Flagon could be a very > > >> strong > > >>>>> open-source alternative to services like Google Analytics, > LogRocket, > > >>> and > > >>>>> others. I intend to socialize the above vision more in the future. > > >> For > > >>>> now > > >>>>> I digress since that's not the point of this email. > > >>>>> > > >>>>> The point of this email is to share an RFC > > >>>>> < > > >>>> > > >>> > > >> > > > https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing > > >>>>> > > >>>>> for *an overhaul I believe needs to happen as a precursor to > > >> achieving > > >>>> the > > >>>>> above vision*. > > >>>>> > > >>>>> *Please share feedback, comments and concerns right on the google > > >> doc*. > > >>>>> I've also opened a GH issue < > > >>> https://github.com/apache/flagon/issues/44> > > >>>> if > > >>>>> folks would like to contribute to the conversation there, though I > > >>> prefer > > >>>>> to keep as much commentary within the document itself to simplify > > >>>> collation > > >>>>> later. > > >>>>> > > >>>>> Many thanks! > > >>>>> > > >>>>> Best > > >>>>> > > >>>>> Evan Jones > > >>>>> > > >>>> > > >>> > > >> > > > > >
