Geode PMC has 52 members. If this were a vote, it looks like the results would have been: +1: 2 (Anthony, Dan) -1: 1 (Jake)
If the next release manager were to go ahead and put geode-benchmarks in the Geode 1.12.0 source release, at least 3 PMC members would need to be willing to vote +1. So it sounds like we need a few more of the other 49 PMC members to weigh in on this discussion. To summarize so far: Proposal: - add a geode-benchmarks-n.n.n-src.tgz artifact to all Geode releases going forward, starting with 1.12.0 Arguments in favor: - why not? - it’s already public - we should default to including all things - it might be of interest to the user community - it might encourage contributions back to further improve it - it is required by CI, which is already included - Apache mandates that source releases must include test code too Arguments against: - doing nothing is less work - it will burden PMC members with additional work to validate and vote on RCs - nobody outside the dev community has asked for it to be included - maybe it’s not ready - maybe it’s not documented well enough - it’s not needed to use Geode - Apache's legal separation between dev stuff and public release stuff - legal or license review may be not have been conducted yet > On Jan 16, 2020, at 4:48 PM, Dan Smith <dsm...@pivotal.io> wrote: > >> If geode-benchmarks is included, that implies that an RC cannot be > approved until reviewers can successfully run the benchmark suite from the > geode-benchmarks source distribution. Is that what we want? > > I think it would be sufficient to run the tests of the benchmarks, eg > ./gradlew test > >> Deploying CI pipelines and running Benchmarks seems like a prime example > of things we’d be happy to help others in the community with on the dev > list — but not something we would expect questions about on the user list. > > I think it would be valuable to share our benchmarks with the geode user > community. The benchmark framework itself (the harness module) is a fairly > generic benchmarking framework than can be used to benchmark anything that > can be spun up using java. The geode-benchmark module has geode benchmarks > that could be used for testing specific hardware, for example. > > -Dan > > On Thu, Jan 16, 2020 at 12:37 PM Owen Nichols <onich...@pivotal.io> wrote: > >> When voting on RC candidates, PMC members "are required to download the >> signed source code package, compile it as provided, and test the resulting >> executable on their own platform”. >> >> If geode-benchmarks is included, that implies that an RC cannot be >> approved until reviewers can successfully run the benchmark suite from the >> geode-benchmarks source distribution. Is that what we want? >> >> Similarly, if CI is included, that seems to imply that an RC cannot be >> approved until reviewers can stand up their own pipeline from the geode/ci >> source distribution. Is that what we want? >> >> So far there doesn’t seem to be consensus on what to include in a Geode >> source release, but let’s keep in mind that anything we add to the release >> becomes an Act Of The Foundation and is held to a higher standard. Apache >> makes a clear distinction between between development activity and official >> releases to the public. Development activity is anything that should stay >> within the dev list. Deploying CI pipelines and running Benchmarks seems >> like a prime example of things we’d be happy to help others in the >> community with on the dev list — but not something we would expect >> questions about on the user list. >> >>> On Jan 16, 2020, at 10:23 AM, Dan Smith <dsm...@pivotal.io> wrote: >>> >>> We are supposed to be including all of the source necessary to test Geode >>> in the source release [1] - I think that would include benchmarks as >> well. >>> >>> I don't really see any compelling reason *not* to include the benchmarks, >>> let's go ahead and get them into our source release! >>> >>> [1] >>> >> http://www.apache.org/legal/release-policy.html#what-must-every-release-contain >>> >>> On Wed, Jan 15, 2020 at 10:26 PM Owen Nichols <onich...@pivotal.io> >> wrote: >>> >>>> +1 for no changes >>>> >>>> On Wed, Jan 15, 2020 at 5:57 PM Jacob Barrett <jbarr...@pivotal.io> >> wrote: >>>> >>>>> We can live in areas of gray that don’t require any changes. Nobody is >>>>> asking for benchmarks so let’s not do work to add them. Nobody is >>>>> complaining they CI is included so let’s not do work to remove them. Is >>>> it >>>>> ideal, meh... >>>>> >>>>>> On Jan 15, 2020, at 5:50 PM, Mark Hanson <mhan...@pivotal.io> wrote: >>>>>> >>>>>> Just my two cents. >>>>>> >>>>>> I think that we should probably strip CI into a separate repo. The key >>>>> indicator is that if something were wrong in the CI yaml, would I hold >> a >>>>> release for that? I think no. So that suggests to me it is a separate >>>>> thing. Same goes for benchmarks. If we were failing a benchmark I would >>>> be >>>>> concerned, but if the script were broken, would I hold the release? I >>>> think >>>>> no as well. >>>>>> >>>>>> I think that says that the CI code should also be a separate repo. >>>>>> >>>>>> Thanks, >>>>>> Mark >>>>>> >>>>>>> On Jan 14, 2020, at 10:21 PM, Jacob Barrett <jbarr...@pivotal.io> >>>>> wrote: >>>>>>> >>>>>>> Until someone outside of the geode ci community is asking for it I >>>> just >>>>> don’t see utility in going through the motions of making a release for >>>> it. >>>>>>> >>>>>>>>> On Jan 14, 2020, at 10:13 PM, Owen Nichols <onich...@pivotal.io> >>>>> wrote: >>>>>>>> >>>>>>>> The source is already public, so on some level a source release is >>>> no >>>>> different from a git tag. Benchmarks has matured enough that I think >> it >>>>> makes sense to at least start branching and tagging the >> geode-benchmarks >>>>> repo to capture exactly what was used in that Geode release. >>>>>>>> >>>>>>>> Others in the dev and user community may find the benchmarks useful >>>> in >>>>> other ways than we use them. While our focus for CI is on tuning for >>>>> repeatability, someone else might just want a load generator to break >> in >>>> a >>>>> new cluster or get some rough numbers. Some might want to get under >> the >>>>> hood and tinker and tune, or contribute their own benchmarks, with the >>>>> understanding that it’s not a turnkey or standalone product, but a tool >>>>> that requires getting your hands dirty. >>>>>>>> >>>>>>>> Would a “1 page” readme with a few tips on “how to run on a laptop” >>>> be >>>>> enough to let other interested contributors help get geode-benchmarks >> to >>>> a >>>>> “better state”? >>>>>>>> >>>>>>>>> On Jan 14, 2020, at 9:38 PM, Jacob Barrett <jbarr...@pivotal.io> >>>>> wrote: >>>>>>>>> >>>>>>>>> I don’t think the benchmarks provide any material benefit to a user >>>>> in their current state. They are heavily tuned for our CI process which >>>>> relies on very beefy machines. Usage on other hardware will require >> more >>>>> tuning. I don’t think it’s worth putting the source in the release >> until >>>>> they are in a better state. >>>>>>>>> >>>>>>>>> -Jake >>>>>>>>> >>>>>>>>> >>>>>>>>>>> On Jan 14, 2020, at 4:14 PM, Dan Smith <dsm...@pivotal.io> >> wrote: >>>>>>>>>> >>>>>>>>>> On Tue, Jan 14, 2020 at 4:11 PM Owen Nichols < >> onich...@pivotal.io >>>>> >>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I believe the desire is to include the source code for >>>>> geode-benchmarks as >>>>>>>>>>> part of the official geode release, much like how we include >>>>> geode-examples. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Oh! I thought you meant running the benchmarks in the release >>>>> pipeline - I >>>>>>>>>> think last release we were running them but decided they were too >>>>> flaky to >>>>>>>>>> use. >>>>>>>>>> >>>>>>>>>> +1 to including the benchmark source in the source release. >>>>>>>>>> >>>>>>>>>> -Dan >>>>>>>> >>>>>> >>>>> >>>> >> >>
