Hi Valentyn, Giomar and I are working on the upgrade of Jenkins VMs to modern Ubuntu version (BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>). We are very close to finishing it, we will reach out for the review.
Thank you. Regards, On Wed, Jan 5, 2022 at 7:57 PM Valentyn Tymofieiev <valen...@google.com> wrote: > Heads-up, I am planning a Jenkins image upgrade with a minor change to > clean up some unwanted log4j artifacts from gradle caches to silence some > alerts I received. Hopefully noone else is currently doing an upgrade, > otherwise - please reach out. > > I will take care of of the PATH issue discussed here (if it is still an > issue). > > > On Tue, Nov 2, 2021 at 4:16 PM Robert Burke <rob...@frantil.com> wrote: > >> TIL as well. Sounds like the right location. Thanks Valentyn! >> >> On Tue, Nov 2, 2021, 11:00 AM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Yeah, .profile is only sourced by login shells. Adding the PATH in >>> .bashrc can be a workaround, but since .bashrc is executed every time a new >>> shell runs, PATH variable will be growing with every shell subprocess, so >>> several sources recommend .profile instead, which does not always work. >>> We should be able to fix this by updating /etc/environment instead >>> (TIL). >>> >>> This is the current content: >>> cat /etc/environment >>> >>> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" >>> >>> >>> >>> >>> On Mon, Nov 1, 2021 at 10:50 AM Robert Burke <rob...@frantil.com> wrote: >>> >>>> Looks like while .profile was edited to add in a PATH section pointing >>>> to /snap/bin (where go is now installed), it doesn't seem like .profile is >>>> executed by the jenkins login shells. >>>> >>>> >>>> >>>> On Fri, Oct 29, 2021, 6:23 PM Valentyn Tymofieiev <valen...@google.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Oct 20, 2021 at 11:16 AM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 20, 2021 at 11:12 AM Pablo Estrada <pabl...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Thanks everyone for investigating and documenting this. I'll use it >>>>>>> today : ) >>>>>>> >>>>>> Dan may be also in the middle of doing this, please coordinate. >>>>>> >>>>>>> >>>>>>> ahem - maybe we should rename the image name/image family names >>>>>>> to jenkins-worker-boot-image ? Does anyone foresee issues if we do that? >>>>>>> Does jenkins depend on these names in some undocumented way? >>>>>>> >>>>>> +1. it should 'just work', need to update the wiki after the change. >>>>>> Jenkins also did a terminology adjustment. >>>>>> >>>>> I had to reimage Jenkins workers again, took care of the rename and >>>>> changed the instructions. >>>>> >>>>> I am not sure what is the status of Go Postcommit problem, but noticed >>>>> that jenkins worker #1 had a different boot disk. I reimaged all workers >>>>> building on top of the latest image from the image family. If Go tests >>>>> start failing, we may need to get help from Dan again. >>>>> >>>>> >>>>>> >>>>>>> On Tue, Oct 19, 2021 at 1:43 PM Daniel Oliveira < >>>>>>> danolive...@google.com> wrote: >>>>>>> >>>>>>>> I'm ok with deciding to avoid the "lite" update option, feel free >>>>>>>> to revise the instructions as it seems appropriate. As for the issue, I >>>>>>>> fixed it with a workaround that should work until we need to add a new >>>>>>>> image to the agents, and I'm currently investigating the root cause and >>>>>>>> prepare a fixed image. >>>>>>>> >>>>>>>> That said, I think this issue would have still happened even if we >>>>>>>> didn't perform the "lite" update. I'm still trying to figure out the >>>>>>>> exact >>>>>>>> problem, but it looks to be a PATH issue that wasn't effectively >>>>>>>> caught by >>>>>>>> the current process. I won't get into details too much in this thread >>>>>>>> (see >>>>>>>> the Jira for that), but essentially everything works in my environment >>>>>>>> when >>>>>>>> I SSH into the VMs, but because the location of the "go" command >>>>>>>> changed in >>>>>>>> the PATH, it seems to have stopped working for every other user, >>>>>>>> including >>>>>>>> the Jenkins agents. I actually did notice that would happen when I was >>>>>>>> working on the image, but the solution seemed to be to reboot the >>>>>>>> machine, >>>>>>>> which I assumed happened already since I shut down the VM to image it. >>>>>>>> >>>>>>>> On Tue, Oct 19, 2021 at 12:09 PM Robert Burke <rob...@frantil.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 to only having one way to do things. The Lite option seems >>>>>>>>> liable to cause more problems since it means it's changes can be >>>>>>>>> blown away >>>>>>>>> if a new image isn't prepared anyway. >>>>>>>>> I don't think we are changing the images often enough for it. >>>>>>>>> Perhaps call it the option to test changes if anything? >>>>>>>>> >>>>>>>>> On Tue, Oct 19, 2021, 11:55 AM Valentyn Tymofieiev < >>>>>>>>> valen...@google.com> wrote: >>>>>>>>> >>>>>>>>>> All workers were updated to >>>>>>>>>> use jenkins-slave-boot-image-20211011, which should have had a go >>>>>>>>>> command, >>>>>>>>>> but it appears slightly misconfigured. I reopened BEAM-13037 [1] and >>>>>>>>>> added >>>>>>>>>> some details there. >>>>>>>>>> >>>>>>>>>> I also added instructions to wiki [2] on how to perform an image >>>>>>>>>> swap and it is actually very straightforward. I think a lesson here >>>>>>>>>> is that >>>>>>>>>> making 'lite' upgrades is brittle as misconfigurations could >>>>>>>>>> resurface down >>>>>>>>>> the road when the context of the lite upgrade is no longer fresh in >>>>>>>>>> our >>>>>>>>>> memory. >>>>>>>>>> >>>>>>>>>> I suggest we revise the instructions to keep only image swap >>>>>>>>>> commands and remove the 'lite' update option. +Daniel Oliveira >>>>>>>>>> <danolive...@google.com>, WDYT? In the meantime, we should also >>>>>>>>>> prepare an image that fixes the misconfiguration. Would you be able >>>>>>>>>> to help >>>>>>>>>> with that? Thank you. >>>>>>>>>> >>>>>>>>>> [1] https://issues.apache.org/jira/browse/BEAM-13037 >>>>>>>>>> [2] >>>>>>>>>> https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-HowtoinstallandupgradesoftwareonJenkinsworkers >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 19, 2021 at 8:46 AM Robert Burke <rob...@frantil.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> FYI it looks like all the Go tests are now failing because it >>>>>>>>>>> can't find the Go command at all. >>>>>>>>>>> Did a Jenkins image without Go (v1.16+) pre-installed get pushed? >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 18, 2021, 1:45 PM Valentyn Tymofieiev < >>>>>>>>>>> valen...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Daniel, >>>>>>>>>>>> >>>>>>>>>>>> I can recreate the VMs on new disks. >>>>>>>>>>>> >>>>>>>>>>>> We currently have a set of stopped jenkins workers (named: >>>>>>>>>>>> apache-beam-jenkins-##) and running workers (named: >>>>>>>>>>>> apache-ci-beam-jenkins-##) >>>>>>>>>>>> >>>>>>>>>>>> Are there any concerns about deleting the stopped group of >>>>>>>>>>>> workers? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Oct 18, 2021 at 11:19 AM Ahmet Altay <al...@google.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thank you Daniel, Valentyn! >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 18, 2021 at 8:02 AM Daniel Oliveira < >>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I performed a light update of both Go and Python (from >>>>>>>>>>>>>> Valentyn's update) on each worker VM over the weekend. I also >>>>>>>>>>>>>> added >>>>>>>>>>>>>> additional instructions for the light update to Confluence (as an >>>>>>>>>>>>>> alternative to the current instructions). >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is still reason to perform a full update at some point: >>>>>>>>>>>>>> Valentyn updated the VM image from 500 GB to 1000 GB of storage, >>>>>>>>>>>>>> which >>>>>>>>>>>>>> requires a full update to actually take effect. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Oct 12, 2021 at 10:32 AM Valentyn Tymofieiev < >>>>>>>>>>>>>> valen...@google.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > 3. SSH into the agent and perform the update. >>>>>>>>>>>>>>> So, this would be a 'lite' version of the update, where we >>>>>>>>>>>>>>> make changes to the live worker without recreating worker VM >>>>>>>>>>>>>>> with a new >>>>>>>>>>>>>>> image? We could perhaps document both options, and also make it >>>>>>>>>>>>>>> clear that >>>>>>>>>>>>>>> producing a VM image that has necessary updates is mandatory >>>>>>>>>>>>>>> even if we >>>>>>>>>>>>>>> perform 'lite' updates without recreating the worker. >>>>>>>>>>>>>>> Also, for a lite update, marking the Jenkins offer offline >>>>>>>>>>>>>>> may be optional, as some updates might not be disruptive (such >>>>>>>>>>>>>>> as >>>>>>>>>>>>>>> installing some software that will not be used immediately). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Oct 11, 2021 at 7:53 PM Robert Burke < >>>>>>>>>>>>>>> rob...@frantil.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> SGTM. Thank you very much Daniel! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Oct 11, 2021, 7:51 PM Ahmet Altay <al...@google.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you Daniel. Could you please update the wiki once >>>>>>>>>>>>>>>>> you are done with the process? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Oct 11, 2021 at 6:22 PM Daniel Oliveira < >>>>>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Took me a bit to get to this, sorry. I finally figured >>>>>>>>>>>>>>>>>> out an approach for updating Go and did so and will be >>>>>>>>>>>>>>>>>> updating the image >>>>>>>>>>>>>>>>>> momentarily. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I think a more important note is that I tried what >>>>>>>>>>>>>>>>>> Valentyn was considering, which is SSHing into workers and >>>>>>>>>>>>>>>>>> updating the >>>>>>>>>>>>>>>>>> dependency. I'll describe the process below, but the summary >>>>>>>>>>>>>>>>>> is that I did >>>>>>>>>>>>>>>>>> it on one worker with Go so far, saw no problems over the >>>>>>>>>>>>>>>>>> weekend, and >>>>>>>>>>>>>>>>>> would like to continue updating the rest of the workers if >>>>>>>>>>>>>>>>>> there are no >>>>>>>>>>>>>>>>>> objections. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Here's a step-by-step of what I did. If we decide to >>>>>>>>>>>>>>>>>> stick with this approach, these instructions can be added to >>>>>>>>>>>>>>>>>> Confluence: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. Go to the page for the Jenkins agent you want to >>>>>>>>>>>>>>>>>> update [1] and click "Mark this node temporarily offline", >>>>>>>>>>>>>>>>>> leaving a reason >>>>>>>>>>>>>>>>>> such as "Updating X dependency." >>>>>>>>>>>>>>>>>> 2. Wait until there are no more tests running in that >>>>>>>>>>>>>>>>>> agent (under "Build Executor Status" on the left of the >>>>>>>>>>>>>>>>>> page). >>>>>>>>>>>>>>>>>> 3. SSH into the agent and perform the update. >>>>>>>>>>>>>>>>>> 4. Mark the node as online again. >>>>>>>>>>>>>>>>>> 5. Repeat for every worker. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> And these are some additional steps if you want to >>>>>>>>>>>>>>>>>> immediately run a test suite to check that the update worked >>>>>>>>>>>>>>>>>> correctly. For >>>>>>>>>>>>>>>>>> example in my case, I wanted to check against the Go >>>>>>>>>>>>>>>>>> Postcommit, and it was >>>>>>>>>>>>>>>>>> a good thing I did, because it actually failed the first >>>>>>>>>>>>>>>>>> time and I had to >>>>>>>>>>>>>>>>>> go back in to fix a small oversight I made. So doing this >>>>>>>>>>>>>>>>>> after you update >>>>>>>>>>>>>>>>>> your first worker is probably a good idea before updating >>>>>>>>>>>>>>>>>> the rest: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. Go to the page for the job you want to run (for >>>>>>>>>>>>>>>>>> example: [2]). >>>>>>>>>>>>>>>>>> 2. Click "Configure" on the left menu. >>>>>>>>>>>>>>>>>> 3. Find the checkmark "Restrict where this project can be >>>>>>>>>>>>>>>>>> run" and change the restriction from "beam" to the specific >>>>>>>>>>>>>>>>>> name of the >>>>>>>>>>>>>>>>>> agent (ex. "apache-beam-jenkins-1"). >>>>>>>>>>>>>>>>>> 4. Save and apply that change. >>>>>>>>>>>>>>>>>> 5. Back on the page for the job, click "Build with >>>>>>>>>>>>>>>>>> Parameters" on the left menu. >>>>>>>>>>>>>>>>>> 6. Run the build on "master". >>>>>>>>>>>>>>>>>> 7. Once you're done checking the results, change >>>>>>>>>>>>>>>>>> the restriction for the job back to "beam". (This also gets >>>>>>>>>>>>>>>>>> reset once >>>>>>>>>>>>>>>>>> every 24 hours in case you forget.) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I did that on one agent (apache-beam-jenkins-2) on Friday >>>>>>>>>>>>>>>>>> evening when it wasn't too busy, and got Go updated and >>>>>>>>>>>>>>>>>> working. I checked >>>>>>>>>>>>>>>>>> that agent's execution history again today just in case, and >>>>>>>>>>>>>>>>>> it was healthy >>>>>>>>>>>>>>>>>> over the weekend, with no Go-related problems as far as I >>>>>>>>>>>>>>>>>> could see. If >>>>>>>>>>>>>>>>>> there's no objections I'd like to go ahead and continue >>>>>>>>>>>>>>>>>> updating the rest >>>>>>>>>>>>>>>>>> of the workers (I'll do this late at night or over the >>>>>>>>>>>>>>>>>> weekend to avoid >>>>>>>>>>>>>>>>>> disrupting dev work). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>> https://ci-beam.apache.org/computer/apache-beam-jenkins-1/ >>>>>>>>>>>>>>>>>> [2] https://ci-beam.apache.org/job/beam_PostCommit_Go/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Oct 4, 2021 at 6:14 PM Valentyn Tymofieiev < >>>>>>>>>>>>>>>>>> valen...@google.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I updated the image in [1], but did not change the >>>>>>>>>>>>>>>>>>> workers yet to pick up the new image yet. We can do this >>>>>>>>>>>>>>>>>>> once we add Go >>>>>>>>>>>>>>>>>>> changes on top of it. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am also considering to SSH into every worker and run a >>>>>>>>>>>>>>>>>>> one-line command that adds the dependency that was missing. >>>>>>>>>>>>>>>>>>> It seems to be >>>>>>>>>>>>>>>>>>> low risk, and there is a fall-back plan to re-start the >>>>>>>>>>>>>>>>>>> worker using the >>>>>>>>>>>>>>>>>>> saved image - both new and old images are saved and >>>>>>>>>>>>>>>>>>> available in Cloud >>>>>>>>>>>>>>>>>>> Console. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Ideally, we should find a way to do a rolling upgrade >>>>>>>>>>>>>>>>>>> that a PMC or committer could trigger without logging into >>>>>>>>>>>>>>>>>>> every machine. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424228#comment-17424228 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, Sep 22, 2021 at 3:28 PM Daniel Oliveira < >>>>>>>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> @Brian Hulette <bhule...@google.com> That button seems >>>>>>>>>>>>>>>>>>>> like exactly what we'd need. Doing it manually would be a >>>>>>>>>>>>>>>>>>>> pain, but it's >>>>>>>>>>>>>>>>>>>> probably still preferable to causing a bunch of aborted >>>>>>>>>>>>>>>>>>>> tests. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> @Valentyn Tymofieiev <valen...@google.com> Collaborating >>>>>>>>>>>>>>>>>>>> to do both updates at once is a great idea! I'll message >>>>>>>>>>>>>>>>>>>> you directly about >>>>>>>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 22, 2021 at 2:44 PM Valentyn Tymofieiev < >>>>>>>>>>>>>>>>>>>> valen...@google.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I am also interested in this updating version of >>>>>>>>>>>>>>>>>>>>> Python on VMs, I need to install Python 3.9. Thanks for >>>>>>>>>>>>>>>>>>>>> looking into this. >>>>>>>>>>>>>>>>>>>>> We can coordinate together to make one update instead of >>>>>>>>>>>>>>>>>>>>> two. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Wed, Sep 22, 2021 at 2:40 PM Brian Hulette < >>>>>>>>>>>>>>>>>>>>> bhule...@google.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I'm not sure about best practices here. Out of >>>>>>>>>>>>>>>>>>>>>> curiosity I just poked around in the Jenkins UI (e.g. >>>>>>>>>>>>>>>>>>>>>> [1]) and it looks >>>>>>>>>>>>>>>>>>>>>> like you can manually "Mark node temporarily offline" >>>>>>>>>>>>>>>>>>>>>> when logged in (if >>>>>>>>>>>>>>>>>>>>>> you're a committer). According to [2] this will prevent >>>>>>>>>>>>>>>>>>>>>> it from picking up >>>>>>>>>>>>>>>>>>>>>> new jobs after it's finished the currently executing >>>>>>>>>>>>>>>>>>>>>> ones. Doing that >>>>>>>>>>>>>>>>>>>>>> manually for every worker could be a pain though. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Brian >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>> https://ci-beam.apache.org/computer/apache-beam-jenkins-13/ >>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>> https://stackoverflow.com/questions/26553612/how-do-i-disable-a-node-in-jenkins-ui-after-it-has-completed-its-currently-runni >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 22, 2021 at 1:03 PM Daniel Oliveira < >>>>>>>>>>>>>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hey everyone, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I'm aiming at upgrading the version of Go on our >>>>>>>>>>>>>>>>>>>>>>> Jenkins VMs, and I found these instructions on >>>>>>>>>>>>>>>>>>>>>>> upgrading software on Jenkins >>>>>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Jenkins+Tips#JenkinsTips-HowtoinstallandupgradesoftwareonJenkinsworkers> >>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>> our cwiki. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I haven't started going through it yet, but I was >>>>>>>>>>>>>>>>>>>>>>> wondering about the last few steps that involve >>>>>>>>>>>>>>>>>>>>>>> stopping VMs, deleting boot >>>>>>>>>>>>>>>>>>>>>>> disks, and restarting executors. Is there some best >>>>>>>>>>>>>>>>>>>>>>> practice for >>>>>>>>>>>>>>>>>>>>>>> that section to avoid causing interruptions in our >>>>>>>>>>>>>>>>>>>>>>> automated testing? >>>>>>>>>>>>>>>>>>>>>>> Should I be trying to do this outside of peak dev >>>>>>>>>>>>>>>>>>>>>>> hours, or going one VM at >>>>>>>>>>>>>>>>>>>>>>> a time so others can pick up extra load, or anything >>>>>>>>>>>>>>>>>>>>>>> like that? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> Daniel Oliveira >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- Daniela Martín (She/Her) | <https://www.wizeline.com/> Site Reliability Engineer daniela.mar...@wizeline.com Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook <https://www.facebook.com/WizelineGlobal> | Instagram <https://www.instagram.com/wizelineglobal/> | LinkedIn <https://www.linkedin.com/company/wizeline> Share feedback on Clutch <https://clutch.co/review/submit/375119> -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
