When voting on RC candidates, PMC members "are required to download the signed source code package, compile it as provided, and test the resulting executable on their own platform”.
If geode-benchmarks is included, that implies that an RC cannot be approved until reviewers can successfully run the benchmark suite from the geode-benchmarks source distribution. Is that what we want? Similarly, if CI is included, that seems to imply that an RC cannot be approved until reviewers can stand up their own pipeline from the geode/ci source distribution. Is that what we want? So far there doesn’t seem to be consensus on what to include in a Geode source release, but let’s keep in mind that anything we add to the release becomes an Act Of The Foundation and is held to a higher standard. Apache makes a clear distinction between between development activity and official releases to the public. Development activity is anything that should stay within the dev list. Deploying CI pipelines and running Benchmarks seems like a prime example of things we’d be happy to help others in the community with on the dev list — but not something we would expect questions about on the user list. > On Jan 16, 2020, at 10:23 AM, Dan Smith <dsm...@pivotal.io> wrote: > > We are supposed to be including all of the source necessary to test Geode > in the source release [1] - I think that would include benchmarks as well. > > I don't really see any compelling reason *not* to include the benchmarks, > let's go ahead and get them into our source release! > > [1] > http://www.apache.org/legal/release-policy.html#what-must-every-release-contain > > On Wed, Jan 15, 2020 at 10:26 PM Owen Nichols <onich...@pivotal.io> wrote: > >> +1 for no changes >> >> On Wed, Jan 15, 2020 at 5:57 PM Jacob Barrett <jbarr...@pivotal.io> wrote: >> >>> We can live in areas of gray that don’t require any changes. Nobody is >>> asking for benchmarks so let’s not do work to add them. Nobody is >>> complaining they CI is included so let’s not do work to remove them. Is >> it >>> ideal, meh... >>> >>>> On Jan 15, 2020, at 5:50 PM, Mark Hanson <mhan...@pivotal.io> wrote: >>>> >>>> Just my two cents. >>>> >>>> I think that we should probably strip CI into a separate repo. The key >>> indicator is that if something were wrong in the CI yaml, would I hold a >>> release for that? I think no. So that suggests to me it is a separate >>> thing. Same goes for benchmarks. If we were failing a benchmark I would >> be >>> concerned, but if the script were broken, would I hold the release? I >> think >>> no as well. >>>> >>>> I think that says that the CI code should also be a separate repo. >>>> >>>> Thanks, >>>> Mark >>>> >>>>> On Jan 14, 2020, at 10:21 PM, Jacob Barrett <jbarr...@pivotal.io> >>> wrote: >>>>> >>>>> Until someone outside of the geode ci community is asking for it I >> just >>> don’t see utility in going through the motions of making a release for >> it. >>>>> >>>>>>> On Jan 14, 2020, at 10:13 PM, Owen Nichols <onich...@pivotal.io> >>> wrote: >>>>>> >>>>>> The source is already public, so on some level a source release is >> no >>> different from a git tag. Benchmarks has matured enough that I think it >>> makes sense to at least start branching and tagging the geode-benchmarks >>> repo to capture exactly what was used in that Geode release. >>>>>> >>>>>> Others in the dev and user community may find the benchmarks useful >> in >>> other ways than we use them. While our focus for CI is on tuning for >>> repeatability, someone else might just want a load generator to break in >> a >>> new cluster or get some rough numbers. Some might want to get under the >>> hood and tinker and tune, or contribute their own benchmarks, with the >>> understanding that it’s not a turnkey or standalone product, but a tool >>> that requires getting your hands dirty. >>>>>> >>>>>> Would a “1 page” readme with a few tips on “how to run on a laptop” >> be >>> enough to let other interested contributors help get geode-benchmarks to >> a >>> “better state”? >>>>>> >>>>>>> On Jan 14, 2020, at 9:38 PM, Jacob Barrett <jbarr...@pivotal.io> >>> wrote: >>>>>>> >>>>>>> I don’t think the benchmarks provide any material benefit to a user >>> in their current state. They are heavily tuned for our CI process which >>> relies on very beefy machines. Usage on other hardware will require more >>> tuning. I don’t think it’s worth putting the source in the release until >>> they are in a better state. >>>>>>> >>>>>>> -Jake >>>>>>> >>>>>>> >>>>>>>>> On Jan 14, 2020, at 4:14 PM, Dan Smith <dsm...@pivotal.io> wrote: >>>>>>>> >>>>>>>> On Tue, Jan 14, 2020 at 4:11 PM Owen Nichols <onich...@pivotal.io >>> >>> wrote: >>>>>>>> >>>>>>>>> I believe the desire is to include the source code for >>> geode-benchmarks as >>>>>>>>> part of the official geode release, much like how we include >>> geode-examples. >>>>>>>>> >>>>>>>> >>>>>>>> Oh! I thought you meant running the benchmarks in the release >>> pipeline - I >>>>>>>> think last release we were running them but decided they were too >>> flaky to >>>>>>>> use. >>>>>>>> >>>>>>>> +1 to including the benchmark source in the source release. >>>>>>>> >>>>>>>> -Dan >>>>>> >>>> >>> >>
