Thanks Kiley, it seems that the failures are not related to the *Ubuntu 20.04.3* upgrade.
In case the *Python Release Candidate* job works, we can proceed with the upgrade of the remaining Jenkins workers. Thank you. Regards, On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <[email protected]> wrote: > XVR failures: https://issues.apache.org/jira/browse/BEAM-13778 > > On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín < > [email protected]> wrote: > >> Hi everyone, >> >> Thank you for the information and details. >> >> We reviewed the tests and we are taking a look at the following jobs that >> started failing close to the date the Jenkins instance was upgraded. We are >> not sure if it’s because of the Ubuntu 20.04.3 upgrade or for another >> reason. >> >> >> - >> >> beam_PostCommit_XVR_Direct >> >> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/> >> - >> >> beam_PostCommit_XVR_Flink >> >> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/> >> - >> >> beam_PostCommit_XVR_Spark3 >> >> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/> >> >> >> We will continue testing them. Any information regarding these jobs would >> be greatly appreciated. >> >> Emily, could you please re-run the Python Release Candidate >> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/> job >> in your PR #16632 <https://github.com/apache/beam/pull/16632>? >> >> >> Thank you very much. >> >> Regards, >> >> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <[email protected]> >> wrote: >> >>> Hey Daniela and Giomar (& all), do you see any concerns with the current >>> image so far? If not, let's upgrade the remaining workers 9-15 to the new >>> image. Thanks! >>> >>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev < >>> [email protected]> wrote: >>> >>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep >>>> the pool at 50% capacity with the benched workers still on ubuntu 16 as >>>> backup and either upgrade all of the workers to ubuntu 20 or roll back >>>> again to ubuntu 16 next week. >>>> >>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <[email protected]> wrote: >>>> >>>>> Nexmark failures are from an unrelated change, fyi >>>>> >>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev < >>>>> [email protected]> wrote: >>>>> >>>>>> It looks like something is resetting the *Restrict where this >>>>>> project can be run* label setting on >>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure >>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20. >>>>>> I'll try to rollback all workers to ubuntu 16. >>>>>> >>>>>> Daniela & Giomar: Here are other builds that have likely got >>>>>> broken by the update: >>>>>> >>>>>> >>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console >>>>>> >>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/ >>>>>> >>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console >>>>>> >>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/ >>>>>> >>>>>> I found them by looking at https://ci-beam.apache.org/ and sorting >>>>>> by status. Looks like we also have a lot of permared Jenkins suites... >>>>>> >>>>>> Also, we should maybe think of some sustainable rolling-upgrades for >>>>>> Jenkins + autorollback to a healthy state as babysitting these upgrades >>>>>> becomes quite toilsome. Admittedly, they are not very frequent though. >>>>>> >>>>>> >>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <[email protected]> wrote: >>>>>> >>>>>>> I'm also running into these errors on my PR for testing the release >>>>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem >>>>>>> to help. Do we have to force the machine per-job/task? >>>>>>> >>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Thinking some more, the correct course of action may also be to fix >>>>>>>> the job to install necessary Python packages if they are missing. This >>>>>>>> would make it self-contained, and the job would install them in the >>>>>>>> virtual environment, instead of relying on the global python >>>>>>>> environment. >>>>>>>> >>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected >>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure >>>>>>>>> why my >>>>>>>>> changes were not saved initially, and restarted the test on your PR: >>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/ >>>>>>>>> >>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it >>>>>>>>>> failed again with the same error: >>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/ >>>>>>>>>> >>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> We are investigating an issue affecting >>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed >>>>>>>>>>> for you, >>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run >>>>>>>>>>> on >>>>>>>>>>> ubuntu16 nodes for now. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the >>>>>>>>>>> build queue, which may have been triggered by your PRs, you many >>>>>>>>>>> need to >>>>>>>>>>> rerun them as well. Sorry about this inconvenience. >>>>>>>>>>> >>>>>>>>>>> As a reminder, trigger phrases are in >>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md >>>>>>>>>>> (which is linked in PR description template). >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> We are updating the Jenkins workers with a new image that was >>>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see >>>>>>>>>>>> more >>>>>>>>>>>> details on: BEAM-12621 >>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621> >>>>>>>>>>>> >>>>>>>>>>>> If something comes up with the workers or the tests please >>>>>>>>>>>> reach out to us. >>>>>>>>>>>> >>>>>>>>>>>> Please let us know if you have any questions or comments. >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/> >>>>>>>>>>>> >>>>>>>>>>>> Site Reliability Engineer III >>>>>>>>>>>> >>>>>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 >>>>>>>>>>>> Zapopan, Jal. >>>>>>>>>>>> >>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | >>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> | Instagram >>>>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn >>>>>>>>>>>> <https://www.linkedin.com/company/wizeline> >>>>>>>>>>>> >>>>>>>>>>>> Share feedback on Clutch >>>>>>>>>>>> <https://clutch.co/review/submit/375119> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *This email and its contents (including any attachments) are >>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be >>>>>>>>>>>> protected >>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the >>>>>>>>>>>> intended >>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, >>>>>>>>>>>> please >>>>>>>>>>>> immediatelynotify the sender by replying to this message and >>>>>>>>>>>> delete the >>>>>>>>>>>> materialimmediately from your system. Any further use, >>>>>>>>>>>> dissemination, >>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. >>>>>>>>>>>> Further, >>>>>>>>>>>> norepresentation is made with respect to any content contained in >>>>>>>>>>>> this >>>>>>>>>>>> email.* >>>>>>>>>>> >>>>>>>>>>> >> >> -- >> >> Daniela Martín (She/Her) | <https://www.wizeline.com/> >> >> Site Reliability Engineer III >> >> [email protected] >> >> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. >> >> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook >> <https://www.facebook.com/WizelineGlobal> | Instagram >> <https://www.instagram.com/wizelineglobal/> | LinkedIn >> <https://www.linkedin.com/company/wizeline> >> >> Share feedback on Clutch <https://clutch.co/review/submit/375119> >> >> >> >> >> >> >> >> >> *This email and its contents (including any attachments) are being sent >> toyou on the condition of confidentiality and may be protected by >> legalprivilege. Access to this email by anyone other than the intended >> recipientis unauthorized. If you are not the intended recipient, please >> immediatelynotify the sender by replying to this message and delete the >> materialimmediately from your system. Any further use, dissemination, >> distributionor reproduction of this email is strictly prohibited. Further, >> norepresentation is made with respect to any content contained in this >> email.* > > -- Daniela Martín (She/Her) | <https://www.wizeline.com/> Site Reliability Engineer III [email protected] Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook <https://www.facebook.com/WizelineGlobal> | Instagram <https://www.instagram.com/wizelineglobal/> | LinkedIn <https://www.linkedin.com/company/wizeline> Share feedback on Clutch <https://clutch.co/review/submit/375119> -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
