As a result of the ongoing work that’s been happening to get Pekko ready for release, more concretely https://github.com/apache/incubator-pekko/pull/105#discussion_r1071912938 I came to the conclusion that I think its a smart idea to clarify what our process is going to be in regards to snapshots/nightlies.
The current state of affairs is that we have a nightly job which manually packages Pekko and then pushes the jar to the Apache nightlies repository ( https://nightlies.apache.org/pekko/) via rsync, see https://github.com/apache/incubator-pekko/blob/main/.github/actions/sync-nightlies/action.yml and https://github.com/apache/incubator-pekko/blob/main/.github/workflows/nightly-builds.yml for more details. While it’s fantastic that we have a nightly system setup (especially since it unblocks other Pekko modules from being able to update the package to org.apache.pekko) the current implementation of it raises a couple of questions. The first more technical one is that we are using the nightlies directory as a pseudo maven repository even though Apache has its own Nexus repo that supports snapshots (see https://infra.apache.org/publishing-maven-artifacts.html and https://repository.apache.org/content/groups/snapshots/). In fact, after manually checking each Apache project directory in the nightlies directories at https://nightlies.apache.org there is no single other project that is publishing library jars here. The content that appears to be published there roughly falls under one of the following * Build results, either executables, fatjars or some other output of a build * Documentation * Microsite This means from what I can tell, as of now we are the only project that is publishing snapshots this way. I asked some various related questions in the #asfinfra slack channel and they also said that any kind of JVM snapshot/nightlies should be published to the maven repo, not the nightlies directory (see https://the-asf.slack.com/archives/CBX4TSBQ8/p1674497939153149 for thread trail). The only other critical point here is regarding snapshot expiry (current nightlies setup has a 30 day expiry) but they also confirmed that there already exists a job on the Apache Nexus repo which performs snapshot expiry with the following relevant settings * Minimum snapshot count: 2 * Snapshot retention (days): 2 * Recurrence: Weekly The main thing that was also mentioned in that Slack channel was when using the Apache snapshot Nexus repo we should keep snapshots to minimum required which then leads to the second point of how we should handle snapshot creation that is useful/brings value (note that to me snapshots are analogous to nightlies, it's just the latter describes a specific way of creating snapshots). Since Pekko (and its related modules) is a JVM library, not a package/application/executable/database I think that nightly snapshot creation is excessive. Nightlies make sense for projects that need to be run because often the process of packaging/building it is non-trivial and a large portion of users are not developers. On the other hand with Pekko being a library, almost all of our users are developers which use Pekko by adding it as a dependency to their build tool (maven, gradle, sbt etc etc). Due to this, one of the main problems that nightlies are solving (i.e. testing a change which hasn’t been released yet) can be trivially done by just pointing a checked out Pekko git repo to a specific point in time and doing publishLocal which publishes the entire Pekko modules to local ivy repo which can then be picked up by other JVM build tools on that local machine (note that publishLocal was also added to the useful sbt commands, see https://github.com/apache/incubator-pekko/pull/83). The other typical use case that needs to be handled is when someone needs to test a specific version/revision of Pekko with a feature in a pre-production/production setting (i.e. not a local machine) to see if its working as intended, snapshots in a remote repository are helpful here. Snapshots are also handy when having to deal with Pekko modules that require Pekko (some Pekko modules such as pekko-http happen to have custom sbt code to lookup the latest snapshot, see https://github.com/apache/incubator-pekko-http/blob/main/project/AkkaDependency.scala and https://github.com/apache/incubator-pekko-http/blob/main/project/VersionGenerator.scala, although modifying that custom snapshot lookup code to also check local ivy repositories also seems sensible). Given this, what to me personally makes sense is that snapshots should be published when a PR is merged to main, at least for now since we don’t have that many PR’s being merged, the amount of snapshots being created is likely going to be less than nightly (on average) which should satisfy the “minimum necessary”. However I am perfectly happy with keeping the current nightly build because there are other considerations as well(i.e. should we deploy a snapshot only when a nightly build + test runs?) and in the future when Pekko picks up steam this can change. On a similar note, a proposed ideal nice to have in the future would be to implement a GitHub bot where a maintainer can trigger the bot via a GitHub comment on a PR to trigger a snapshot of that specific PR before it gets merged, this can then handle the use case of “I want to test these changes on actual use case scenario before merging the PR”. Since Pekko build currently distinguishes between clean and unclean artifacts via a timestamp (see https://github.com/apache/incubator-pekko/pull/125 for a recent PR that fixes this) this also means that we can also separate snapshots generated from PR’s from snapshots generated by the main job (whether that be nightly or after merge into main). What are everyones thoughts on this? -- Matthew de Detrich *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491603708037 *w:* aiven.io *e:* [email protected]
