+1 to adding a doc for this, along with some PR conventions about when to
use these. Some questions that I would love to see documented:
* What tests are run automatically and when (pre-commits, all languages, on
every commit)
* What other test suites exist, and how should they be used (post-commits
and performance tests run on a schedule and validate already-merged code;
if a merge breaks post-commits we should prefer rollback over rollforward
to keep master in a healthy state; if you think your change could affect
one of these suites you can invoke them manually on your PR)
* How do reviewers use Jenkins signals (i.e. do reviewers expect
pre-commits to be green before reviewing PR's?)
* What should you do if you think a flaky test caused a failure on your PR?
(re-run tests using magical Jenkins incantation)
I also like the idea of automatically scraping the string from the
source-of-truth rather than manually maintaining. However, I suspect it
might be more trouble than it's worth (the groovy files are in a different
git repo than beam-site; the site is currently built from simple markdown
files which get compiled to HTML and checked-in). If sourcing the strings
from the source proves difficult, I'm in favor of going the simple route
and manually maintaining the list; I think there's sufficient value to
justify the maintenance cost of.
Another idea would be to see if we could query Jenkins via javascript on
page load for the set of beam jobs and trigger phrases. This could be a
simpler integration point and has the benefit of always being up-to-date
regardless of which commit was used to generate the jobs.
On Fri, May 11, 2018 at 7:11 AM Alexey Romanenko <aromanenko....@gmail.com>
wrote:
> +1 to add such reference guide of Jenkins commands to Testing guide. It
> should be extremely useful, especially for those who were not aware about
> this before.
>
> WBR,
> Alexey
>
>
> On 11 May 2018, at 02:44, Ankur Goenka <goe...@google.com> wrote:
>
> In my experience affect of white space in commit is inconsistent for
> certain commands they do matter while for others they don't.
>
> On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev <valen...@google.com>
> wrote:
>
>> +1 to writing a Beam Jenkins spellbook.
>> I have observed that Jenkins commands sometimes don't work for the first
>> time, why could this be? Do end of lines at the end of command matter?
>>
>>
>> On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud <apill...@google.com>
>> wrote:
>>
>>> It would be great to have the set of "Run {Java,Python,Go} PreCommit"
>>> documented in the contributors guide as well. Those match up to the
>>> jobs auto run on every PR and are the ones I use most. There is no
>>> security, anyone can run them including 'Run Seed Job'. That one seems like
>>> a good one to document in testing, because it is the one that loads changes
>>> to the rest.
>>>
>>> Andrew
>>>
>>> On Thu, May 10, 2018 at 12:27 PM Huygaa Batsaikhan <bat...@google.com>
>>> wrote:
>>>
>>>> Hi devs,
>>>>
>>>> We can run various jenkins commands (precommit, postcommit,
>>>> performance tests) directly from Github Pull Request UI by commenting
>>>> phrases such as "retest this please". Unfortunately, this tool is not
>>>> documented. I am adding a brief documentation in
>>>> https://beam.apache.org/contribute/testing/ and I need some help.
>>>>
>>>> 1. What are the most common phrases used?
>>>> 2. Can anyone run these commands? Are there any permission issues?
>>>> 3. Does it make sense to categorize the commands as Performance
>>>> tests, Precommit, Postcommit, and Release Validation?
>>>>
>>>> Let me know what you think,
>>>>
>>>> Thanks,
>>>> Huygaa
>>>>
>>>
>>
>
