Since maven snapshots appear to be the norm - my view is that if we can get the maven snapshot working then we can switch off the nightlies (after we see the snapshots are publishing ok).
On Tue, 24 Jan 2023 at 13:22, Matthew Benedict de Detrich <[email protected]> wrote: > > As a result of the ongoing work that’s been happening to get Pekko ready > for release, more concretely > https://github.com/apache/incubator-pekko/pull/105#discussion_r1071912938 I > came to the conclusion that I think its a smart idea to clarify what our > process is going to be in regards to snapshots/nightlies. > > The current state of affairs is that we have a nightly job which manually > packages Pekko and then pushes the jar to the Apache nightlies repository ( > https://nightlies.apache.org/pekko/) via rsync, see > https://github.com/apache/incubator-pekko/blob/main/.github/actions/sync-nightlies/action.yml > and > https://github.com/apache/incubator-pekko/blob/main/.github/workflows/nightly-builds.yml > for more details. > > While it’s fantastic that we have a nightly system setup (especially since > it unblocks other Pekko modules from being able to update the package to > org.apache.pekko) the current implementation of it raises a couple of > questions. The first more technical one is that we are using the nightlies > directory as a pseudo maven repository even though Apache has its own Nexus > repo that supports snapshots (see > https://infra.apache.org/publishing-maven-artifacts.html and > https://repository.apache.org/content/groups/snapshots/). In fact, after > manually checking each Apache project directory in the nightlies > directories at https://nightlies.apache.org there is no single other > project that is publishing library jars here. The content that appears to > be published there roughly falls under one of the following > > > * Build results, either executables, fatjars or some other output of a build > > * Documentation > > * Microsite > > > This means from what I can tell, as of now we are the only project that is > publishing snapshots this way. I asked some various related questions in > the #asfinfra slack channel and they also said that any kind of JVM > snapshot/nightlies should be published to the maven repo, not the nightlies > directory (see > https://the-asf.slack.com/archives/CBX4TSBQ8/p1674497939153149 for thread > trail). The only other critical point here is regarding snapshot expiry > (current nightlies setup has a 30 day expiry) but they also confirmed that > there already exists a job on the Apache Nexus repo which performs snapshot > expiry with the following relevant settings > > > * Minimum snapshot count: 2 > > * Snapshot retention (days): 2 > > * Recurrence: Weekly > > > The main thing that was also mentioned in that Slack channel was when using > the Apache snapshot Nexus repo we should keep snapshots to minimum required > which then leads to the second point of how we should handle snapshot > creation that is useful/brings value (note that to me snapshots are > analogous to nightlies, it's just the latter describes a specific way of > creating snapshots). Since Pekko (and its related modules) is a JVM > library, not a package/application/executable/database I think that nightly > snapshot creation is excessive. Nightlies make sense for projects that need > to be run because often the process of packaging/building it is non-trivial > and a large portion of users are not developers. On the other hand with > Pekko being a library, almost all of our users are developers which use > Pekko by adding it as a dependency to their build tool (maven, gradle, sbt > etc etc). > > Due to this, one of the main problems that nightlies are solving (i.e. > testing a change which hasn’t been released yet) can be trivially done by > just pointing a checked out Pekko git repo to a specific point in time and > doing publishLocal which publishes the entire Pekko modules to local ivy > repo which can then be picked up by other JVM build tools on that local > machine (note that publishLocal was also added to the useful sbt commands, > see https://github.com/apache/incubator-pekko/pull/83). The other typical > use case that needs to be handled is when someone needs to test a specific > version/revision of Pekko with a feature in a pre-production/production > setting (i.e. not a local machine) to see if its working as intended, > snapshots in a remote repository are helpful here. Snapshots are also handy > when having to deal with Pekko modules that require Pekko (some Pekko > modules such as pekko-http happen to have custom sbt code to lookup the > latest snapshot, see > https://github.com/apache/incubator-pekko-http/blob/main/project/AkkaDependency.scala > and > https://github.com/apache/incubator-pekko-http/blob/main/project/VersionGenerator.scala, > although modifying that custom snapshot lookup code to also check local ivy > repositories also seems sensible). > > Given this, what to me personally makes sense is that snapshots should be > published when a PR is merged to main, at least for now since we don’t have > that many PR’s being merged, the amount of snapshots being created is > likely going to be less than nightly (on average) which should satisfy the > “minimum necessary”. However I am perfectly happy with keeping the current > nightly build because there are other considerations as well(i.e. should we > deploy a snapshot only when a nightly build + test runs?) and in the future > when Pekko picks up steam this can change. > > > On a similar note, a proposed ideal nice to have in the future would be to > implement a GitHub bot where a maintainer can trigger the bot via a GitHub > comment on a PR to trigger a snapshot of that specific PR before it gets > merged, this can then handle the use case of “I want to test these changes > on actual use case scenario before merging the PR”. Since Pekko build > currently distinguishes between clean and unclean artifacts via a timestamp > (see https://github.com/apache/incubator-pekko/pull/125 for a recent PR > that fixes this) this also means that we can also separate snapshots > generated from PR’s from snapshots generated by the main job (whether that > be nightly or after merge into main). > > What are everyones thoughts on this? > > -- > > Matthew de Detrich > > *Aiven Deutschland GmbH* > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > *m:* +491603708037 > > *w:* aiven.io *e:* [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
