> My main concern is that our testing is not correct. And this becomes more urgent when we have the scala 2 inlining affecting our releases.
Oh agreed, but there is a difference between urgency and requirement and while the inlining changes may push that urgency a bit forward it doesn't necessitate it (even though as mentioned earlier[1] the inliner changes by themselves won't break anything at runtime with different pekko-core jars because inlining with pekko-core 1.1.x literally means inline bytecode from pekko-core into pekko-connectors as part of building which can't break anything at runtime by definition). What is actually far more worrying is not inliner vs non inliner but rather all of the behavioural changes that have accompanied such major dependency updates, i.e. the runtime breakage of pekko-grpc going from protobuf v3 to v4 (even though technically speaking it doesn't break MiMA, it is breaking runtime behaviour which inliner does not do). If we are talking about urgency, Pekko 1.0.0-M1 is going to soon be released anyway so there isn't any debate on that. Pekko-connectors seems like it's going to require Pekko gRPC 1.1.x anyways, so that part isn't urgent. The only hint of urgency I am getting is whether to do pekko-grpc 1.1.0 vs pekko-grpc 1.1.0-M1 but that urgency is coming from pekko-connectors requiring pekko-grpc 1.1.x, not the community (this is unlike the pekko-persistence which has gotten numerous questions from release from **actual users**) Maybe I wasn't being clear earlier, but the main underlying intentions of the -M1 release is also as a stop gap for not having this full testing which you describe, i.e. so the community/users can run the various pekko libraries with these massive dependency updates and report back any issues. For example to figure out if there are no problems with pekko-connectors 1.1.x using pekko-grpc 1.1.x is to actually release pekko-grpc-1.1.0 as an -M1 and then release pekko-connectors as an -M1 (using pekko-grpc 1.1.0-M1 as a dependency) and see if there are reported problems. The more comprehensive way is to build with one version of pekko and run the tests against another version of Pekko, this isn't too hard (I have some local code that does this, which I used to both verify the inliner even though it's unnecessary and with a pekko-connector change[2]). > We can't trust that the process of compiling the pekko-connectors code does not affect our ability to use those pekko-connectors jars with different pekko core jars. The ironic thing here is that for the inliner specifically we can trust it won't break anything because of how it fundamentally works, on the other hand the other hand bumps in critical dependencies of the various pekko modules is what we don't have a guarantee with. 1: https://lists.apache.org/thread/lk7snnxv2kb9qghgydwm52fpgrfyqpmr 2: https://github.com/apache/incubator-pekko-connectors/pull/280#issuecomment-1807748739 On Tue, Mar 19, 2024 at 2:26 PM PJ Fanning <fannin...@apache.org> wrote: > > My main concern is that our testing is not correct. And this becomes > more urgent when we have the scala 2 inlining affecting our releases. > > Take for example these pekko-connectors integration tests: > https://github.com/apache/incubator-pekko-connectors/blob/main/.github/workflows/check-build-test.yml > > This build recompiles the pekko-connectors classes with the pekko-core > jars that we want to test with. What we need is to compile the > pekko-connectors jars with the pekko-core jars that we intend to use > in the release. Then we can run the integration tests with these > pekko-connectors jars and different pekko-core jars. > > We can't trust that the process of compiling the pekko-connectors code > does not affect our ability to use those pekko-connectors jars with > different pekko core jars. > > On Tue, 19 Mar 2024 at 13:54, Matthew de Detrich > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > I don't think we can have full releases that depend on milestone > > releases of transitive dependencies. > > > > Technically speaking process wise there is nothing wrong with this (it > > doesn't break anything ASF, ASF does not care about the > > distinction between milestone or non milestone) although it obviously > > will raise some eyebrows because of what it semantically implies. > > > > > Essentially, we are blocked from > > doing 1.1 releases of anything until Pekko 1.1.0 is released (other > > than milestone releases). > > > > > I think we should have a full discussion and vote about the idea of > > changing all of our builds to use Pekko 1.1.0(-M1). It was always my > > view that I didn't want to have to do months of releases of all the > > modules after Pekko 1.1 was released > > > > But we are already in this situation (albeit not fully) anyways. For example > > pekko-project will require pekko-persistence-jdbc 1.1.x because > > of a breaking update from Slick 3.3 to Slick 3.5 in > > pekko-persistence-jdbc (which > > pekko-projection has a dependency on). Transitively this will also mean > > pekko-connectors should also depend on pekko core 1.1.x series because it > > also has slick integration which got updated to 3.5 > > > > We will also need to do this kind of testing matrix due to differences > > in Pekko Core > > i.e. see logback 1 vs 2 (which means we now have different versions of > > pekko-testkit) > > and there are other cases such as Jackson. > > > > Even with pekko grpc, if I understand correctly i.e. from your original OP > > > > > With the 2 gRPC based > > Pekko Connectors that we maintain, we are already blocked from > > upgrading some Google libs - because they are incompatible with the > > Pekko gRPC jars. > > > > Pekko Connectors will have to depend on pekko-grpc 1.1.x There is also > > probably more stuff we are missing, i.e. we also wanted to do cross JDK > > testing[1] whose solution is the same as what we are talking about (even if > > the problem is different). > > > > The takeway I am trying to make here is that we have to deal with this > > even when completely ignoring the inliner changes since we have enough of > > these fundamental incompatibilities (often, but not always due to updated > > dependencies in the 1.1.x series not being compatible with those same > > dependencies 1.0.x series). Ontop of that there is an argument that with > > such fundamental big version changes in these dependencies (which gRPC > > is one of, it seems) that it warrants an -M1 (again completely > > ignoring inliner). > > Ontop of that, another point to boot is that pekko-connectors is likely > > not going to have a non milestone release a while[2] which means that it > > will be stuck in milestone (-M1, -M2 etc etc) for that time and I don't > > think > > pekko-connectors-M1 depending on pekko-grpc-M1 is going to be controversial. > > > > Also finally, our bottleneck when making Pekko Release has historically been > > the incubator itself, the PPMC on the other hand has almost always made > > the vote within 3 days (I think there were 2-3 exceptions to that). > > > > 1: https://github.com/apache/incubator-pekko/issues/1118 > > 2: https://github.com/apache/incubator-pekko-connectors/discussions/443 > > > > On Tue, Mar 19, 2024 at 1:24 PM PJ Fanning <fannin...@apache.org> wrote: > > > > > > If we go the route of releasing pekko grpc 1.1.x by building with > > > pekko 1.1.0-M1 then the grpc release will need to be a milestone too. > > > I don't think we can have full releases that depend on milestone > > > releases of transitive dependencies. Essentially, we are blocked from > > > doing 1.1 releases of anything until Pekko 1.1.0 is released (other > > > than milestone releases). > > > > > > I think we should have a full discussion and vote about the idea of > > > changing all of our builds to use Pekko 1.1.0(-M1). It was always my > > > view that I didn't want to have to do months of releases of all the > > > modules after Pekko 1.1 was released. I think we are going to get a > > > lot of frustrated users wondering why they are getting version > > > conflicts when they inadvertently end up with a mix of pekko > > > dependencies, some 1.0, some 1.1. > > > > > > I, for one, am -1 on going down this route. > > > > > > > > > On Tue, 19 Mar 2024 at 13:15, Matthew de Detrich > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > pekko 1.1.0 to run pekko (which is what > > > > will appear in the pom)* > > > > > > > > pekko 1.1.0 to run pekko-grpc (which is what > > > > will appear in the pom) > > > > > > > > On Tue, Mar 19, 2024 at 1:08 PM Matthew de Detrich > > > > <matthew.dedetr...@aiven.io> wrote: > > > > > > > > > > > I disagree. I want to see a full description of what the new release > > > > > pipeline looks like. I want to know exactly what appears in our poms. > > > > > > > > > > I really don't see what's problematic about setting the pekko core > > > > > dependency > > > > > as 1.1.0[1]. This doesn't mean you need pekko 1.1.0 to run pekko > > > > > (which is what > > > > > will appear in the pom), it's just what gets resolved by default and > > > > > what it's built with. > > > > > > > > > > Akka-Http has done this multiple times, they bumped the core Akka > > > > > version within > > > > > Akka-Http for various reasons. > > > > > > > > > > > I stayed out of the discussion because I thought it had a low > > > > > > impact. > > > > > Now it is blocking our ability to do releases. Up until today, there > > > > > was no indication that Pekko 1.1 was going to require major changes to > > > > > our builds and release pipelines. > > > > > > > > > > This is untrue, it was discussed multiple times which is not the same > > > > > as it not being > > > > > noticed. I just spent 5 minutes looking for the first instance and > > > > > tbh I am not > > > > > exactly motivated to trawl through all of the discussions to find all > > > > > instances of this > > > > > as its not a productive use of time > > > > > > > > > > 1: > > > > > https://github.com/apache/incubator-pekko-grpc/blob/792f886381d5a5539901c2da4f99917b57389209/project/PekkoCoreDependency.scala#L25 > > > > > > > > > > On Tue, Mar 19, 2024 at 12:59 PM PJ Fanning <fannin...@gmail.com> > > > > > wrote: > > > > > > > > > > > > I disagree. I want to see a full description of what the new release > > > > > > pipeline looks like. I want to know exactly what appears in our > > > > > > poms. > > > > > > > > > > > > Of all the features in the Pekko 1.1.0-M1, the scala 2 inliner is of > > > > > > the least interest to me. Johannes Rudolph has said on a number of > > > > > > times that Akka team did not prioritise it because the Hotspot > > > > > > compiler would be expected to make most of the optimisations anyway. > > > > > > > > > > > > I stayed out of the discussion because I thought it had a low > > > > > > impact. > > > > > > Now it is blocking our ability to do releases. Up until today, there > > > > > > was no indication that Pekko 1.1 was going to require major changes > > > > > > to > > > > > > our builds and release pipelines. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 12:53, Matthew de Detrich > > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > I agree, no release of a module should depend on an unreleased > > > > > > > > version of > > > > > > > Pekko. > > > > > > > > > > > > > > This is incorrect, -M1 is technically an ASF release. Having > > > > > > > pekko-grpc rely > > > > > > > on a non released version of Pekko is not what is being discussed > > > > > > > and > > > > > > > never was on the table. > > > > > > > > > > > > > > > This Pekko 1.1 build set up has never been agreed on. I'm > > > > > > > > against it > > > > > > > due to its complexity and unproven benefits. > > > > > > > > > > > > > > This is the same build setup that is required to cross test pekko > > > > > > > against > > > > > > > multiple pekko versions which was also discussed in multiple > > > > > > > other places. > > > > > > > > > > > > > > I have a suspicion that you have an inaccurate impression of > > > > > > > whats being > > > > > > > discussed, its really not that complex. The minimum build version > > > > > > > will be set > > > > > > > to Pekko 1.1.0 or Pekko 1.1.0-M1 and another pipeline will be > > > > > > > added to run > > > > > > > it against Pekko 1.0.x which we were planning to do anyways. > > > > > > > > > > > > > > On Tue, Mar 19, 2024 at 12:46 PM Nicolas Vollmar > > > > > > > <nvoll...@gmail.com> wrote: > > > > > > > > > > > > > > > > I agree, no release of a module should depend on an unreleased > > > > > > > > version of > > > > > > > > Pekko. > > > > > > > > People would end up with unintended milestones in their class > > > > > > > > path. > > > > > > > > > > > > > > > > If gRPC 1.1 needs Pekko 1.1 we should push for that release > > > > > > > > first. > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 12:01, PJ Fanning <fannin...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > This Pekko 1.1 build set up has never been agreed on. I'm > > > > > > > > > against it > > > > > > > > > due to its complexity and unproven benefits. > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 11:54, Matthew de Detrich > > > > > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > This was discussed previously multiple times, in order for > > > > > > > > > > the > > > > > > > > > > inlining to be fully > > > > > > > > > > enabled it needs to be built with Pekko 1.1.0-M1 because > > > > > > > > > > that's the > > > > > > > > > > version of Pekko > > > > > > > > > > with the fixed inlining. This does not mean that a pekko > > > > > > > > > > gRPC > > > > > > > > > > built with Pekko 1.1.0-M1 won't work with Pekko 1.0.x, it > > > > > > > > > > just means it > > > > > > > > > needs to > > > > > > > > > > be built with it. > > > > > > > > > > > > > > > > > > > > Note that this also implies we need to add a test to make > > > > > > > > > > sure a pekko > > > > > > > > > > gRPC built > > > > > > > > > > with Pekko 1.1.0-M1 works when linked with Pekko 1.0.x at > > > > > > > > > > runtime > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2024 at 10:36 AM PJ Fanning > > > > > > > > > > <fannin...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > I don't agree that we want to build Pekko gRPC 1.1 off of > > > > > > > > > > > Pekko > > > > > > > > > > > 1.1.0-M1. Pekko 1.1.0-M1 is > > > > > > > > > > > * a milestone - not a proper release > > > > > > > > > > > * I thought we were trying to maintain it so that our > > > > > > > > > > > modules would be > > > > > > > > > > > compatible with Pekko 1.0 > > > > > > > > > > > > > > > > > > > > > > I don't see the need for a milestone release of Pekko > > > > > > > > > > > gRPC. There is > > > > > > > > > > > nothing experimental in the main branch. The only reason > > > > > > > > > > > to make this > > > > > > > > > > > a 1.1 release instead of 1.0.3 release is that the > > > > > > > > > > > changes in our > > > > > > > > > > > Protobuf dependencies are not trivial. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2024 at 07:12, Matthew de Detrich > > > > > > > > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > I would wait till Pekko Core 1.1.0-M1 release so that > > > > > > > > > > > > gRPC 1.1.0 can > > > > > > > > > > > > build off of Pekko Core 1.1.0-M1 > > > > > > > > > > > > as this will enable the full cross inlining[1]. > > > > > > > > > > > > > > > > > > > > > > > > Also not sure if we want to do -M1 first before full > > > > > > > > > > > > release (or > > > > > > > > > not). > > > > > > > > > > > > The reasons here are weaker than > > > > > > > > > > > > other projects but I may be missing something. > > > > > > > > > > > > > > > > > > > > > > > > 1: > > > > > > > > > https://github.com/pjfanning/sbt-pekko-build/blob/main/src/main/scala/com/github/pjfanning/pekkobuild/PekkoInlinePlugin.scala#L49-L56 > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 18, 2024 at 9:26 PM PJ Fanning > > > > > > > > > > > > <fannin...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > The gRPC code is heavily dependent on protobuf and > > > > > > > > > > > > > there are > > > > > > > > > upgrades > > > > > > > > > > > > > to grpc-protobuf and protoc in the main branch. With > > > > > > > > > > > > > the 2 gRPC > > > > > > > > > based > > > > > > > > > > > > > Pekko Connectors that we maintain, we are already > > > > > > > > > > > > > blocked from > > > > > > > > > > > > > upgrading some Google libs - because they are > > > > > > > > > > > > > incompatible with > > > > > > > > > the > > > > > > > > > > > > > Pekko gRPC jars. These connectors will compile with > > > > > > > > > > > > > the latest > > > > > > > > > Google > > > > > > > > > > > > > jars if Pekko gRPC 1.1 snapshots are used. > > > > > > > > > > > > > > > > > > > > > > > > > > These projects demo this: > > > > > > > > > > > > > * > > > > > > > > > https://github.com/pjfanning/pekko-connectors-google-cloud-pub-sub-grpc/ > > > > > > > > > > > > > * > > > > > > > > > https://github.com/pjfanning/pekko-connectors-google-cloud-bigquery-storage > > > > > > > > > > > > > > > > > > > > > > > > > > There has recently been a new protobuf-java 4.0.26 > > > > > > > > > > > > > release but the > > > > > > > > > > > > > gRPC and other libs have been changed to support this > > > > > > > > > > > > > incompatible > > > > > > > > > > > > > version. I suspect that we might need a Pekko gRPC > > > > > > > > > > > > > release for > > > > > > > > > these > > > > > > > > > > > > > new protobuf-java 4.0,x releases at some point also. > > > > > > > > > > > > > > > > > > > > > > > > > > Does anyone have any thoughts about this? > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > PJ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > dev-unsubscr...@pekko.apache.org > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > > dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > > > > > dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Matthew de Detrich > > > > > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > > > > > m: +491603708037 > > > > > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Matthew de Detrich > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > m: +491603708037 > > > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > > > > > > > > > -- > > > > > > > > Matthew de Detrich > > > > > > > > Aiven Deutschland GmbH > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > m: +491603708037 > > > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > > > > > > -- > > > > Matthew de Detrich > > > > Aiven Deutschland GmbH > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Alexanderufer 3-7, 10117 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > m: +491603708037 > > > > w: aiven.io e: matthew.dedetr...@aiven.io > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > > For additional commands, e-mail: dev-h...@pekko.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org > For additional commands, e-mail: dev-h...@pekko.apache.org > -- Matthew de Detrich Aiven Deutschland GmbH Immanuelkirchstraße 26, 10405 Berlin Alexanderufer 3-7, 10117 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen m: +491603708037 w: aiven.io e: matthew.dedetr...@aiven.io --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pekko.apache.org For additional commands, e-mail: dev-h...@pekko.apache.org
