[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Daniel Zahn]

> This means codesearch will be affected (and won't get updated) and possibly 
> even will be down during that time.

We, at least in my team, would like to switch codesearch (and other
clients) back to just use gerrit.wikimedia.org and not the replica
directly.

Just today we agreed to make a new ticket for specifically this,
because soon we have to reimage the replica to bullseye and add more
downtime.

The reason we did the split in the past was to reduce load on the main
gerrit server but meanwhile first the issue has been fixed in newer
Gerrit
versions and then also just a few days ago we switched to brand new hardware.

So now if anything it should be beefier than before and even without
that it seemed already a thing of the past.

And we pay for this with this issue that the replica becomes a second
production system, with the need for downtimes. It complicates
fail-over scenarios
too and in a way means there is never a passive host when we do DC switch-over.

So yea, I suggest we change the config of codesearch now to use the
main gerrit unless you have concerns about that.

On Mon, May 15, 2023 at 1:18 PM Amir Sarabadani  wrote:
>
> This means codesearch will be affected (and won't get updated) and possibly 
> even will be down during that time.
>
> Best
>
> Am Mo., 15. Mai 2023 um 22:03 Uhr schrieb Tyler Cipriani 
> :
>>
>> Hello
>>
>> The read-only Gerrit replica[0] will be down for 30 minutes tomorrow (Tue, 
>> 16 May 2023) between 13:00–15:00 UTC[1] due to network switch upgrades in 
>> codfw row D[2].
>>
>> During this window, git reads from the replica will not work.
>>
>> To my knowledge, this affects bots which rely on the replica for git read 
>> operations.
>>
>> Apologies for any inconvenience.
>>
>> Tyler Cipriani (he/him)
>> Engineering Manager, Release Engineering
>> Wikimedia Foundation
>>
>> [0]: 
>> [1]: 
>> [2]: 
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Amir (he/him)
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Daniel Zahn 
Site Reliability Engineer
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Daniel Zahn]

I have used codesearch to search for the config of codesearch with things like

https://codesearch.wmcloud.org/search/?q=codesearch===

I did find the puppet module codesearch and a hound config file in there.

But somehow I have not found yet where the "gerrit-replica" URL s configured.

Do you see it? Could that be in Horizon Hiera instead of the repos maybe?
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: send mail in python

[Bryan Davis]

On Mon, May 15, 2023 at 2:57 PM Tim Moody  wrote:
>
> I'd like to send an email from a python3 process on a wmcs VPS to report 
> errors.
>
> I looked at https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS but 
> could use some help.
>
> sudo echo "Subject: sendmail test2" | /usr/sbin/sendmail -v  works.

The `sudo` here does nothing useful. It is bound to the `echo`
invocation and not the `sendmail` one.

> When I try to send the equivalent from python smtplib I get a 221 error 
> message.

221 is the SMTP status code for closing connection/goodbye. This isn't
specifically an error, instead it means that the SMTP server has
decided to end the session.

* Are there other status codes you see from your attempted python code
prior to the 221?
* What SMTP server are you connecting to?
* Is the python code available somewhere for review?

Here is a quick example of sending email using Python 3.9 and smtplib
from inside Toolforge:

  $ become bd808-test -- webservice python3.9 shell -- python3.9
  >>> import smtplib
  >>> import ssl
  >>> context = ssl.create_default_context()
  >>> server = smtplib.SMTP("mail.tools.wmcloud.org", 587)
  >>> server.starttls(context=context)
  (220, b'TLS go ahead')
  >>> server.sendmail("bd808-test.maintain...@toolforge.org",
"bd...@wikimedia.org", """Subject: smtplib example from Toolforge
  ...
  ... Hello world.
  ... ---
  ... Bryan""")
  {}
  >>> server.quit()
  (221, b'mail.tools.wmcloud.org closing connection')

Things would typically look similar from a Cloud VPS project. The
major change would be to use mx-out03.wmcloud.org or
mx-out04.wmcloud.org as your outbound SMTP service. There is a bit
more complication in using TLS as well due to the x509 certificate
being a bit of a mess (bad subject and expired):

  $ ssh devportal-demo01.devportal.eqiad1.wikimedia.cloud
  $ python3.9
  >>> import smtplib
  >>> import ssl
  >>> context = ssl.create_default_context()
  >>> context.check_hostname = False
  >>> context.verify_mode = ssl.CERT_NONE
  >>> server = smtplib.SMTP("mx-out03.wmcloud.org", 25)
  >>> server.starttls(context=context)
  (220, b'TLS go ahead')
  >>> server.sendmail("bd...@wikimedia.org", "bd...@wikimedia.org",
"""Subject: smtplib example from Cloud VPS
  ...
  ... Hello world.
  ... ---
  ... Bryan""")
  {}
  >>> server.close()

I hope that helps a bit.

Bryan
-- 
Bryan Davis  Technical Engagement  Wikimedia Foundation
Principal Software Engineer   Boise, ID USA
[[m:User:BDavis_(WMF)]]  irc: bd808
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Amir Sarabadani]

This means codesearch will be affected (and won't get updated) and possibly
even will be down during that time.

Best

Am Mo., 15. Mai 2023 um 22:03 Uhr schrieb Tyler Cipriani <
tcipri...@wikimedia.org>:

> Hello
>
> The read-only Gerrit replica[0] will be down for 30 minutes tomorrow (Tue,
> 16 May 2023) between 13:00–15:00 UTC[1] due to network switch upgrades in
> codfw row D[2].
>
> During this window, git reads from the replica will not work.
>
> To my knowledge, this affects bots which rely on the replica for git read
> operations.
>
> Apologies for any inconvenience.
>
> Tyler Cipriani (he/him)
> Engineering Manager, Release Engineering
> Wikimedia Foundation
>
> [0]: 
> [1]: 
> [2]: 
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Daniel Zahn]

Thank you Amir!  Also here:

https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919925

Sorry for duplicate work, merge / abandon either at will from my side.

But I used commit message for reasoning and needed a ticket, so linked to new:

https://phabricator.wikimedia.org/T336710

Just today in our team meeting it came up and that we should create a
dedicated ticket for it.

So the patch would either be temporary or simply never be reverted,
even after the maintenance.

On Mon, May 15, 2023 at 3:27 PM Amir Sarabadani  wrote:
>
> I made the patch: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919924 I 
> can merge and deploy it soon.
>
> Am Di., 16. Mai 2023 um 00:23 Uhr schrieb Daniel Zahn :
>>
>> I have used codesearch to search for the config of codesearch with things 
>> like
>>
>> https://codesearch.wmcloud.org/search/?q=codesearch===
>>
>> I did find the puppet module codesearch and a hound config file in there.
>>
>> But somehow I have not found yet where the "gerrit-replica" URL s configured.
>>
>> Do you see it? Could that be in Horizon Hiera instead of the repos maybe?
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Amir (he/him)
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Daniel Zahn 
Site Reliability Engineer
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Amir Sarabadani]

I made the patch: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919924
I can merge and deploy it soon.

Am Di., 16. Mai 2023 um 00:23 Uhr schrieb Daniel Zahn :

> I have used codesearch to search for the config of codesearch with things
> like
>
>
> https://codesearch.wmcloud.org/search/?q=codesearch===
>
> I did find the puppet module codesearch and a hound config file in there.
>
> But somehow I have not found yet where the "gerrit-replica" URL s
> configured.
>
> Do you see it? Could that be in Horizon Hiera instead of the repos maybe?
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>


-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Tyler Cipriani]

Hello

The read-only Gerrit replica[0] will be down for 30 minutes tomorrow (Tue,
16 May 2023) between 13:00–15:00 UTC[1] due to network switch upgrades in
codfw row D[2].

During this window, git reads from the replica will not work.

To my knowledge, this affects bots which rely on the replica for git read
operations.

Apologies for any inconvenience.

Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation

[0]: 
[1]: 
[2]: 
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] send mail in python

[Tim Moody]

I'd like to send an email from a python3 process on a wmcs VPS to report
errors.

I looked at https://wikitech.wikimedia.org/wiki/Help:Email_in_Cloud_VPS but
could use some help.

sudo echo "Subject: sendmail test2" | /usr/sbin/sendmail -v  works.

When I try to send the equivalent from python smtplib I get a 221 error
message.

Thanks,

Tim
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Brian Wolff]

Does this include other uses of gerrit replica? Should extension
distributor be switched to main gerrit?

--
Brian

On Tuesday, May 16, 2023, Daniel Zahn  wrote:

> > This means codesearch will be affected (and won't get updated) and
> possibly even will be down during that time.
>
> We, at least in my team, would like to switch codesearch (and other
> clients) back to just use gerrit.wikimedia.org and not the replica
> directly.
>
> Just today we agreed to make a new ticket for specifically this,
> because soon we have to reimage the replica to bullseye and add more
> downtime.
>
> The reason we did the split in the past was to reduce load on the main
> gerrit server but meanwhile first the issue has been fixed in newer
> Gerrit
> versions and then also just a few days ago we switched to brand new
> hardware.
>
> So now if anything it should be beefier than before and even without
> that it seemed already a thing of the past.
>
> And we pay for this with this issue that the replica becomes a second
> production system, with the need for downtimes. It complicates
> fail-over scenarios
> too and in a way means there is never a passive host when we do DC
> switch-over.
>
> So yea, I suggest we change the config of codesearch now to use the
> main gerrit unless you have concerns about that.
>
> On Mon, May 15, 2023 at 1:18 PM Amir Sarabadani 
> wrote:
> >
> > This means codesearch will be affected (and won't get updated) and
> possibly even will be down during that time.
> >
> > Best
> >
> > Am Mo., 15. Mai 2023 um 22:03 Uhr schrieb Tyler Cipriani <
> tcipri...@wikimedia.org>:
> >>
> >> Hello
> >>
> >> The read-only Gerrit replica[0] will be down for 30 minutes tomorrow
> (Tue, 16 May 2023) between 13:00–15:00 UTC[1] due to network switch
> upgrades in codfw row D[2].
> >>
> >> During this window, git reads from the replica will not work.
> >>
> >> To my knowledge, this affects bots which rely on the replica for git
> read operations.
> >>
> >> Apologies for any inconvenience.
> >>
> >> Tyler Cipriani (he/him)
> >> Engineering Manager, Release Engineering
> >> Wikimedia Foundation
> >>
> >> [0]: 
> >> [1]: 
> >> [2]: 
> >> ___
> >> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> >> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> >> https://lists.wikimedia.org/postorius/lists/wikitech-l.
> lists.wikimedia.org/
> >
> >
> >
> > --
> > Amir (he/him)
> >
> > ___
> > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> > https://lists.wikimedia.org/postorius/lists/wikitech-l.
> lists.wikimedia.org/
>
>
>
> --
> Daniel Zahn 
> Site Reliability Engineer
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.
> lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Daniel Zahn]

> Does this include other uses of gerrit replica? Should extension distributor 
> be switched to main gerrit?

Personally I think so, yes.  We can make a list of users and discuss
on https://phabricator.wikimedia.org/T336710
I also don't know yet which of the users is causing the highest load
compared to others.

On Mon, May 15, 2023 at 9:12 PM Brian Wolff  wrote:
>
> Does this include other uses of gerrit replica? Should extension distributor 
> be switched to main gerrit?
>
> --
> Brian
>
> On Tuesday, May 16, 2023, Daniel Zahn  wrote:
>>
>> > This means codesearch will be affected (and won't get updated) and 
>> > possibly even will be down during that time.
>>
>> We, at least in my team, would like to switch codesearch (and other
>> clients) back to just use gerrit.wikimedia.org and not the replica
>> directly.
>>
>> Just today we agreed to make a new ticket for specifically this,
>> because soon we have to reimage the replica to bullseye and add more
>> downtime.
>>
>> The reason we did the split in the past was to reduce load on the main
>> gerrit server but meanwhile first the issue has been fixed in newer
>> Gerrit
>> versions and then also just a few days ago we switched to brand new hardware.
>>
>> So now if anything it should be beefier than before and even without
>> that it seemed already a thing of the past.
>>
>> And we pay for this with this issue that the replica becomes a second
>> production system, with the need for downtimes. It complicates
>> fail-over scenarios
>> too and in a way means there is never a passive host when we do DC 
>> switch-over.
>>
>> So yea, I suggest we change the config of codesearch now to use the
>> main gerrit unless you have concerns about that.
>>
>> On Mon, May 15, 2023 at 1:18 PM Amir Sarabadani  wrote:
>> >
>> > This means codesearch will be affected (and won't get updated) and 
>> > possibly even will be down during that time.
>> >
>> > Best
>> >
>> > Am Mo., 15. Mai 2023 um 22:03 Uhr schrieb Tyler Cipriani 
>> > :
>> >>
>> >> Hello
>> >>
>> >> The read-only Gerrit replica[0] will be down for 30 minutes tomorrow 
>> >> (Tue, 16 May 2023) between 13:00–15:00 UTC[1] due to network switch 
>> >> upgrades in codfw row D[2].
>> >>
>> >> During this window, git reads from the replica will not work.
>> >>
>> >> To my knowledge, this affects bots which rely on the replica for git read 
>> >> operations.
>> >>
>> >> Apologies for any inconvenience.
>> >>
>> >> Tyler Cipriani (he/him)
>> >> Engineering Manager, Release Engineering
>> >> Wikimedia Foundation
>> >>
>> >> [0]: 
>> >> [1]: 
>> >> [2]: 
>> >> ___
>> >> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> >> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> >> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>> >
>> >
>> >
>> > --
>> > Amir (he/him)
>> >
>> > ___
>> > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>>
>> --
>> Daniel Zahn 
>> Site Reliability Engineer
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Daniel Zahn 
Site Reliability Engineer
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/