> Do you suggest test all OS, all Guava versions, all timezones in CI?
> That is hard to maintain and it takes CI time
No I don’t suggest that. That seemed to be what you were suggesting!
My suggestion is to test a ‘diagonal’. Suppose we need to test 5 JDK versions,
5 Guava versions, and 5 operating systems. The full matrix is 5 ^ 3 = 125
entries. But if we just test the diagonal, {(jdk #0, guava #0, os #0), … (jdk
#4, guava #4, os #4)} then each parameter gets tested once. That’s not perfect,
but it’s a huge improvement with a minimal amount of CI resources. We missed
some bugs because we were testing only one Guava version and only one time zone.
Infrequently - say once a month or once per release - someone can run the whole
matrix.
Julian
> On Oct 1, 2021, at 10:02 AM, Vladimir Sitnikov <sitnikov.vladi...@gmail.com>
> wrote:
>
>> CI should be deterministic
>
> That is an interesting view.
> One of the approaches that I have not considered earlier is "seed matrix
> randomization from PR index and the branch name".
> That would make the set of jobs consistent for each execution of the same
> PR.
>
> However, based on my testng/testng experience, it is just fine if the set
> of CI jobs varies.
> The CI job description does clarify what was special (e.g. timezone, JVM,
> OS, etc).
>
>> And also tests with different Guava versions
>
> Do you suggest test all OS, all Guava versions, all timezones in CI?
> That is hard to maintain and it takes CI time
>
>> CI should be deterministic. So that if it breaks, the person to fix it is
> clear
>
> CI resources are limited, and we can't infinitely scale the build matrix.
> I do not really see why Guava versions are more important than operating
> systems.
> The users should prefer recent Guava versions, and the idea of supporting
> (and testing!) ancient Guava versions is doubtful.
>
> On the other hand, we might want to verify different cases like
> -XX:hashCode=2 that makes every Object#hashCode() to return 1 always.
>
> Vladimir
