This proposal looks good to me. On Fri, Jul 13, 2018 at 10:49 AM, Jason Kuster <jasonkus...@google.com> wrote:
> I like the idea of auto-creating bugs. I think that if we do this we > should make sure to set a time period after which we will evaluate whether > it has succeeded, i.e. whether bugs filed as a result of this have been > triaged, owned, fixed, and closed. > > On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin <mig...@google.com> > wrote: > >> Can we get some PMC opinions on this? >> >> According to https://www.apache.org/dev/infra-contact#request-checklist, >> it is highly recommended to have your opinions when filing a request. >> > Do you need a specific comment or a LGTM is good enough? > >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> >> >> On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin <ke...@google.com> wrote: >> >>> I think this looks good, we should enable the plugin and try it out. >>> Concrete details of the follow-up tasks (auto-assignment, triage, and >>> dashboarding) will probably depend on how functional the plugin is and what >>> the test failures data looks like. >>> >>> Regards, >>> Anton >>> >>> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin <mig...@google.com> >>> wrote: >>> >>>> @Yifan Zou <yifan...@google.com> >>>> >>>> I believe that we should test-drive the system with tickets + PR first >>>> and decide on email notification later. We already have tests failure >>>> emails sent to commits@, I believe most people filter out or not >>>> signed up for that list though. >>>> >>>> It creates only one ticket, and keeps it for recurring test failures. >>>> >>>> @Andrew Pilloud <apill...@google.com> >>>> Thank you for the suggestion. I'll add it to design doc. >>>> >>>> --Mikhail >>>> >>>> >>>> >>>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou <yifan...@google.com> wrote: >>>> >>>>> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the >>>>> ticket being ignored and no actions being taken. >>>>> >>>>> I can see the challenges on ticket assignment. Like Mikhail mentioned, >>>>> the plugin does not support dynamic assignments. We have to implement >>>>> custom script to determine the assignees and do some tricks to the jenkins >>>>> job. Also, the post-commits tests usually cover tons of stuffs that it is >>>>> difficult to find which part was broken and ask the right person to look >>>>> into within the Auto JIRA process. Some naive thoughts: Are we able to >>>>> send >>>>> emails to the dev@ to ask people to take care of the JIRA issues? Are >>>>> we able to find component leads and ask them triage the test failure >>>>> tickets? >>>>> >>>>> Another nitpick comment. Does the jenkins job file the JIRA issue in >>>>> every test failure? Sometimes the test continuously fails in a time period >>>>> due to the same reason. In this case, we will get some duplicate issues >>>>> filed by Jenkins. I think it could be better if we can avoid filing issues >>>>> if the previous one has not been resolved. >>>>> >>>>> Thanks. >>>>> Yifan >>>>> >>>>> >>>>> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud <apill...@google.com> >>>>> wrote: >>>>> >>>>>> That sounds great. You should add this detail to the doc. >>>>>> >>>>>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin <mig...@google.com> >>>>>> wrote: >>>>>> >>>>>>> We already have component for this purpose: "test-failures". All >>>>>>> tickets created will go to that component. As an option, we can add >>>>>>> link to >>>>>>> view list of open JIRA tickets to PR template. >>>>>>> >>>>>>> We also would want to create graph on dashboard with amount of >>>>>>> unassigned and assigned bugs. >>>>>>> >>>>>>> I believe that we can also add counter of unassigned bugs to PR >>>>>>> template. This way it will be easier for everyone to know when there's >>>>>>> some >>>>>>> tests issue not attended. >>>>>>> >>>>>>> --Mikhail >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud <apill...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> So it sounds like you will want to create a component for untriaged >>>>>>>> issues so they are easy to find. I like the idea of distributing the >>>>>>>> work >>>>>>>> of triaging post commit failures to new PR authors as a condition of >>>>>>>> merging. I feel like we will just be filling JIRA with spam if the >>>>>>>> issues >>>>>>>> are automatically created without a plan for triage. >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang <ruw...@google.com> wrote: >>>>>>>> >>>>>>>>> Maybe this is also a good thread to start the discussion that if >>>>>>>>> we want to enforce postcommit test for every PR. >>>>>>>>> >>>>>>>>> Can we afford the cost of longer waiting time to catch potential >>>>>>>>> bugs? >>>>>>>>> >>>>>>>>> -Rui >>>>>>>>> >>>>>>>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin < >>>>>>>>> mig...@google.com> wrote: >>>>>>>>> >>>>>>>>>> That's a valid point. >>>>>>>>>> >>>>>>>>>> Unfortunately, the JiraTestResultReporter plugin does not have >>>>>>>>>> features to dynamically assign owners. Additionally, I don't think >>>>>>>>>> it is >>>>>>>>>> always easy to find proper owner for post-commit tests at first >>>>>>>>>> glance, >>>>>>>>>> since they usually cover broad specter of issues. >>>>>>>>>> >>>>>>>>>> My assumption is that we need someone to triage new issues. >>>>>>>>>> >>>>>>>>>> Ideally, any contributor, who sees failing test, should check >>>>>>>>>> unassigned tickets and either do triage, or assign them to someone >>>>>>>>>> who can. >>>>>>>>>> I strongly encourage this approach. >>>>>>>>>> >>>>>>>>>> We have couple other ready-made options to consider: >>>>>>>>>> 1. We can configure JIRA component owner who would be assigned to >>>>>>>>>> created tickets. >>>>>>>>>> 2. JiraTestReporterPlugin can assign tickets to specific user. >>>>>>>>>> This is configured per Jenkins job. We can utilize this if someone >>>>>>>>>> volunteers. >>>>>>>>>> 3. Dynamic assignment will most likely require custom solution. >>>>>>>>>> >>>>>>>>>> --Mikhail >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jul 11, 2018 at 3:34 PM Andrew Pilloud < >>>>>>>>>> apill...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Mikhail, >>>>>>>>>>> >>>>>>>>>>> I like the proposal! Hopefully this can replace the constant >>>>>>>>>>> stream of build failure emails. I noticed one detail seems to be >>>>>>>>>>> missing: >>>>>>>>>>> How will new issues be assigned to the proper owner? Will the tool >>>>>>>>>>> do this >>>>>>>>>>> automatically or will we need someone to triage new issues? >>>>>>>>>>> >>>>>>>>>>> Andrew >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 11, 2018 at 3:07 PM Mikhail Gryzykhin < >>>>>>>>>>> mig...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> I want to add an automatic JIRA tickets creation for failing >>>>>>>>>>>> post-commit tests. >>>>>>>>>>>> >>>>>>>>>>>> I wrote up design proposal doc with more details on this: >>>>>>>>>>>> https://docs.google.com/document/d/ >>>>>>>>>>>> 1kpsTy0sYJkLqlZvkPalDkqzBbpu-Wug0z-oWIVPo6UI >>>>>>>>>>>> >>>>>>>>>>>> Quick summary: >>>>>>>>>>>> I suggest to utilize JiraTestResultReporter plugin. >>>>>>>>>>>> Since this plugin is not installed on our Jenkins yet, we have >>>>>>>>>>>> to request to Infra team to add it. >>>>>>>>>>>> >>>>>>>>>>>> Please, comment if this approach sounds good to you. >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> --Mikhail >>>>>>>>>>>> >>>>>>>>>>>> > > -- > ------- > Jason Kuster > Apache Beam / Google Cloud Dataflow > > See something? Say something. go/jasonkuster-feedback > <https://goto.google.com/jasonkuster-feedback> >
