My main concern is that our testing is not correct. And this becomes more urgent when we have the scala 2 inlining affecting our releases.
Take for example these pekko-connectors integration tests: https://github.com/apache/incubator-pekko-connectors/blob/main/.github/workflows/check-build-test.yml This build recompiles the pekko-connectors classes with the pekko-core jars that we want to test with. What we need is to compile the pekko-connectors jars with the pekko-core jars that we intend to use in the release. Then we can run the integration tests with these pekko-connectors jars and different pekko-core jars. We can't trust that the process of compiling the pekko-connectors code does not affect our ability to use those pekko-connectors jars with different pekko core jars. On Tue, 19 Mar 2024 at 13:54, Matthew de Detrich <matthew.dedetr...@aiven.io.invalid> wrote: > > > I don't think we can have full releases that depend on milestone > releases of transitive dependencies. > > Technically speaking process wise there is nothing wrong with this (it > doesn't break anything ASF, ASF does not care about the > distinction between milestone or non milestone) although it obviously > will raise some eyebrows because of what it semantically implies. > > > Essentially, we are blocked from > doing 1.1 releases of anything until Pekko 1.1.0 is released (other > than milestone releases). > > > I think we should have a full discussion and vote about the idea of > changing all of our builds to use Pekko 1.1.0(-M1). It was always my > view that I didn't want to have to do months of releases of all the > modules after Pekko 1.1 was released > > But we are already in this situation (albeit not fully) anyways. For example > pekko-project will require pekko-persistence-jdbc 1.1.x because > of a breaking update from Slick 3.3 to Slick 3.5 in > pekko-persistence-jdbc (which > pekko-projection has a dependency on). Transitively this will also mean > pekko-connectors should also depend on pekko core 1.1.x series because it > also has slick integration which got updated to 3.5 > > We will also need to do this kind of testing matrix due to differences > in Pekko Core > i.e. see logback 1 vs 2 (which means we now have different versions of > pekko-testkit) > and there are other cases such as Jackson. > > Even with pekko grpc, if I understand correctly i.e. from your original OP > > > With the 2 gRPC based > Pekko Connectors that we maintain, we are already blocked from > upgrading some Google libs - because they are incompatible with the > Pekko gRPC jars. > > Pekko Connectors will have to depend on pekko-grpc 1.1.x There is also > probably more stuff we are missing, i.e. we also wanted to do cross JDK > testing[1] whose solution is the same as what we are talking about (even if > the problem is different). > > The takeway I am trying to make here is that we have to deal with this > even when completely ignoring the inliner changes since we have enough of > these fundamental incompatibilities (often, but not always due to updated > dependencies in the 1.1.x series not being compatible with those same > dependencies 1.0.x series). Ontop of that there is an argument that with > such fundamental big version changes in these dependencies (which gRPC > is one of, it seems) that it warrants an -M1 (again completely > ignoring inliner). > Ontop of that, another point to boot is that pekko-connectors is likely > not going to have a non milestone release a while[2] which means that it > will be stuck in milestone (-M1, -M2 etc etc) for that time and I don't think > pekko-connectors-M1 depending on pekko-grpc-M1 is going to be controversial. > > Also finally, our bottleneck when making Pekko Release has historically been > the incubator itself, the PPMC on the other hand has almost always made > the vote within 3 days (I think there were 2-3 exceptions to that). > > 1: https://github.com/apache/incubator-pekko/issues/1118 > 2: https://github.com/apache/incubator-pekko-connectors/discussions/443 > > On Tue, Mar 19, 2024 at 1:24 PM PJ Fanning <fannin...@apache.org> wrote: > > > > If we go the route of releasing pekko grpc 1.1.x by building with > > pekko 1.1.0-M1 then the grpc release will need to be a milestone too. > > I don't think we can have full releases that depend on milestone > > releases of transitive dependencies. Essentially, we are blocked from > > doing 1.1 releases of anything until Pekko 1.1.0 is released (other > > than milestone releases). > > > > I think we should have a full discussion and vote about the idea of > > changing all of our builds to use Pekko 1.1.0(-M1). It was always my > > view that I didn't want to have to do months of releases of all the > > modules after Pekko 1.1 was released. I think we are going to get a > > lot of frustrated users wondering why they are getting version > > conflicts when they inadvertently end up with a mix of pekko > > dependencies, some 1.0, some 1.1. > > > > I, for one, am -1 on going down this route. > > > > > > On Tue, 19 Mar 2024 at 13:15, Matthew de Detrich > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > pekko 1.1.0 to run pekko (which is what > > > will appear in the pom)* > > > > > > pekko 1.1.0 to run pekko-grpc (which is what > > > will appear in the pom) > > > > > > On Tue, Mar 19, 2024 at 1:08 PM Matthew de Detrich > > > <matthew.dedetr...@aiven.io> wrote: > > > > > > > > > I disagree. I want to see a full description of what the new release > > > > pipeline looks like. I want to know exactly what appears in our poms. > > > > > > > > I really don't see what's problematic about setting the pekko core > > > > dependency > > > > as 1.1.0[1]. This doesn't mean you need pekko 1.1.0 to run pekko (which > > > > is what > > > > will appear in the pom), it's just what gets resolved by default and > > > > what it's built with. > > > > > > > > Akka-Http has done this multiple times, they bumped the core Akka > > > > version within > > > > Akka-Http for various reasons. > > > > > > > > > I stayed out of the discussion because I thought it had a low impact. > > > > Now it is blocking our ability to do releases. Up until today, there > > > > was no indication that Pekko 1.1 was going to require major changes to > > > > our builds and release pipelines. > > > > > > > > This is untrue, it was discussed multiple times which is not the same > > > > as it not being > > > > noticed. I just spent 5 minutes looking for the first instance and tbh > > > > I am not > > > > exactly motivated to trawl through all of the discussions to find all > > > > instances of this > > > > as its not a productive use of time > > > > > > > > 1: > > > > https://github.com/apache/incubator-pekko-grpc/blob/792f886381d5a5539901c2da4f99917b57389209/project/PekkoCoreDependency.scala#L25 > > > > > > > > On Tue, Mar 19, 2024 at 12:59 PM PJ Fanning <fannin...@gmail.com> wrote: > > > > > > > > > > I disagree. I want to see a full description of what the new release > > > > > pipeline looks like. I want to know exactly what appears in our poms. > > > > > > > > > > Of all the features in the Pekko 1.1.0-M1, the scala 2 inliner is of > > > > > the least interest to me. Johannes Rudolph has said on a number of > > > > > times that Akka team did not prioritise it because the Hotspot > > > > > compiler would be expected to make most of the optimisations anyway. > > > > > > > > > > I stayed out of the discussion because I thought it had a low impact. > > > > > Now it is blocking our ability to do releases. Up until today, there > > > > > was no indication that Pekko 1.1 was going to require major changes to > > > > > our builds and release pipelines. > > > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 12:53, Matthew de Detrich > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > I agree, no release of a module should depend on an unreleased > > > > > > > version of > > > > > > Pekko. > > > > > > > > > > > > This is incorrect, -M1 is technically an ASF release. Having > > > > > > pekko-grpc rely > > > > > > on a non released version of Pekko is not what is being discussed > > > > > > and > > > > > > never was on the table. > > > > > > > > > > > > > This Pekko 1.1 build set up has never been agreed on. I'm against > > > > > > > it > > > > > > due to its complexity and unproven benefits. > > > > > > > > > > > > This is the same build setup that is required to cross test pekko > > > > > > against > > > > > > multiple pekko versions which was also discussed in multiple other > > > > > > places. > > > > > > > > > > > > I have a suspicion that you have an inaccurate impression of whats > > > > > > being > > > > > > discussed, its really not that complex. The minimum build version > > > > > > will be set > > > > > > to Pekko 1.1.0 or Pekko 1.1.0-M1 and another pipeline will be added > > > > > > to run > > > > > > it against Pekko 1.0.x which we were planning to do anyways. > > > > > > > > > > > > On Tue, Mar 19, 2024 at 12:46 PM Nicolas Vollmar > > > > > > <nvoll...@gmail.com> wrote: > > > > > > > > > > > > > > I agree, no release of a module should depend on an unreleased > > > > > > > version of > > > > > > > Pekko. > > > > > > > People would end up with unintended milestones in their class > > > > > > > path. > > > > > > > > > > > > > > If gRPC 1.1 needs Pekko 1.1 we should push for that release first. > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 12:01, PJ Fanning <fannin...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > This Pekko 1.1 build set up has never been agreed on. I'm > > > > > > > > against it > > > > > > > > due to its complexity and unproven benefits. > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 11:54, Matthew de Detrich > > > > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > > > > This was discussed previously multiple times, in order for the > > > > > > > > > inlining to be fully > > > > > > > > > enabled it needs to be built with Pekko 1.1.0-M1 because > > > > > > > > > that's the > > > > > > > > > version of Pekko > > > > > > > > > with the fixed inlining. This does not mean that a pekko gRPC > > > > > > > > > built with Pekko 1.1.0-M1 won't work with Pekko 1.0.x, it > > > > > > > > > just means it > > > > > > > > needs to > > > > > > > > > be built with it. > > > > > > > > > > > > > > > > > > Note that this also implies we need to add a test to make > > > > > > > > > sure a pekko > > > > > > > > > gRPC built > > > > > > > > > with Pekko 1.1.0-M1 works when linked with Pekko 1.0.x at > > > > > > > > > runtime > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2024 at 10:36 AM PJ Fanning > > > > > > > > > <fannin...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > I don't agree that we want to build Pekko gRPC 1.1 off of > > > > > > > > > > Pekko > > > > > > > > > > 1.1.0-M1. Pekko 1.1.0-M1 is > > > > > > > > > > * a milestone - not a proper release > > > > > > > > > > * I thought we were trying to maintain it so that our > > > > > > > > > > modules would be > > > > > > > > > > compatible with Pekko 1.0 > > > > > > > > > > > > > > > > > > > > I don't see the need for a milestone release of Pekko gRPC. > > > > > > > > > > There is > > > > > > > > > > nothing experimental in the main branch. The only reason to > > > > > > > > > > make this > > > > > > > > > > a 1.1 release instead of 1.0.3 release is that the changes > > > > > > > > > > in our > > > > > > > > > > Protobuf dependencies are not trivial. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 07:12, Matthew de Detrich > > > > > > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > I would wait till Pekko Core 1.1.0-M1 release so that > > > > > > > > > > > gRPC 1.1.0 can > > > > > > > > > > > build off of Pekko Core 1.1.0-M1 > > > > > > > > > > > as this will enable the full cross inlining[1]. > > > > > > > > > > > > > > > > > > > > > > Also not sure if we want to do -M1 first before full > > > > > > > > > > > release (or > > > > > > > > not). > > > > > > > > > > > The reasons here are weaker than > > > > > > > > > > > other projects but I may be missing something. > > > > > > > > > > > > > > > > > > > > > > 1: > > > > > > > > https://github.com/pjfanning/sbt-pekko-build/blob/main/src/main/scala/com/github/pjfanning/pekkobuild/PekkoInlinePlugin.scala#L49-L56 > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 18, 2024 at 9:26 PM PJ Fanning > > > > > > > > > > > <fannin...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > The gRPC code is heavily dependent on protobuf and > > > > > > > > > > > > there are > > > > > > > > upgrades > > > > > > > > > > > > to grpc-protobuf and protoc in the main branch. With > > > > > > > > > > > > the 2 gRPC > > > > > > > > based > > > > > > > > > > > > Pekko Connectors that we maintain, we are already > > > > > > > > > > > > blocked from > > > > > > > > > > > > upgrading some Google libs - because they are > > > > > > > > > > > > incompatible with > > > > > > > > the > > > > > > > > > > > > Pekko gRPC jars. These connectors will compile with the > > > > > > > > > > > > latest > > > > > > > > Google > > > > > > > > > > > > jars if Pekko gRPC 1.1 snapshots are used. > > > > > > > > > > > > > > > > > > > > > > > > These projects demo this: > > > > > > > > > > > > * > > > > > > > > https://github.com/pjfanning/pekko-connectors-google-cloud-pub-sub-grpc/ > > > > > > > > > > > > * > > > > > > > > https://github.com/pjfanning/pekko-connectors-google-cloud-bigquery-storage > > > > > > > > > > > > > > > > > > > > > > > > There has recently been a new protobuf-java 4.0.26 > > > > > > > > > > > > release but the > > > > > > > > > > > > gRPC and other libs have been changed to support this > > > > > > > > > > > > incompatible > > > > > > > > > > > > version. I suspect that we might need a Pekko gRPC > > > > > > > > > > > > release for > > > > > > > > these > > > > > > > > > > > > new protobuf-java 4.0,x releases at some point also. > > > > > > > > > > > > > > > > > > > > > > > > Does anyone have any thoughts about this? > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > PJ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > -- > > > > > > > > Matthew de Detrich > > > > > > > > Aiven Deutschland GmbH > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > m: +491603708037 > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > -- > > > > > > Matthew de Detrich > > > > > > Aiven Deutschland GmbH > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > m: +491603708037 > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > -- > > Matthew de Detrich > > Aiven Deutschland GmbH > > Immanuelkirchstraße 26, 10405 Berlin > > Alexanderufer 3-7, 10117 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491603708037 > > w: aiven.io e: matthew.dedetr...@aiven.io > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > For additional commands, e-mail: dev-h...@pekko.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org For additional commands, e-mail: dev-h...@pekko.apache.org
