Re: [jenkins-infra] Jenkins Core Release automation

[Kohsuke Kawaguchi]

Wow, congratulations!!

After 15 years and ~900 releases, this process has finally left my hands...


On Fri, Apr 17, 2020 at 2:32 AM Olblak  wrote:

> Hi Everybody
>
> Yesterday we released a weekly version from the new release environment
> and while we managed to ship our weekly version, we also faced unplanned
> issues that we had to solve so here a quick retro.
>
> The process was done in two stages the release and then the packaging
>
> Stage 1, Release:
>
> The release part, was the easiest one. We only add two issues.
> The first issue was due to the password used to authenticate on Maven
> repository, containing special characters that couldn't be parsed by Maven
> release plugin. We just changed the password.
> The second problem was a permission issue on repo.jenkins-ci.org which
> was easily fixed.
> In the result of this stage, as expected every artifact was pushed to
> repo.jenkins-ci.org
>
> Stage 2, Package:
>
> Well, this stage was more tricky as we heavily refactored that part while
> also heavily relying on existing service. We discovered multiple side cases
> that we didn't catch during the testing period and I will describe here the
> most important ones.
>
> *  GPG: While this was something that we discussed a long time ago we
> forgot to communicate in advance but the GPG key used to sign releases
> changed for weekly release in order to not use Kohsuke gpg key. In a result
> of this, people who use RedHat/Debian/Suse repository need to reimport the
> new key. Remark: this is something that will also have to be done for Lts
> once we also release Lts with the current process
>
> New
> Key:
> pub   rsa4096 2020-03-30 [SC] [expires: 2023-03-30]
>   62A9756BFD780C377CF24BA8FCEF32E745F2C3D5
> uid   Jenkins Project 
> sub   rsa4096 2020-03-30 [E] [expires: 2023-03-30].
>
> Old Key:
> pub   dsa1024 2009-02-01 [SC]
>   150FDE3F7787E7D11EF4E12A9B7D32F2D50582E6
> uid   Kohsuke Kawaguchi 
> uid   [jpeg image of size 3704]
> uid   Kohsuke Kawaguchi 
> uid   Kohsuke Kawaguchi 
> uid   Kohsuke Kawaguchi 
> sub   elg2048 2009-02-01 [E]
> sub   rsa4096 2016-11-01 [S]
>
> * mirror.jenkins-ci.org: This service has now additional Html files
> `HEADER.html` and `FOOTER.html` per directory. Those two files are used by
> apache directory listing to prepend and append Html information to the
> directory listing. The main advantage of this is to provide information
> from mirrors about how to install those packages from your operating system
> from every mirror. At least apache with the correct configuration, an
> example of this is
>
> http://mirrors.jenkins-ci.org/debian/ or
> http://archives.jenkins-ci.org/debian/
>
> Please note that the instruction to configure os package manager is
> referencing the wrong URL and it should be pkg.jenkins.io. A fix needs to
> be provided, other remark CSS needs to align with other Jenkins website if
> someone is willing to work on this, feel free to do, it's defined here
> https://github.com/jenkinsci/packaging/blob/INFRA-910-core-release-automation/templates/base.html
>
> * pkg.jenkins.io: this service is used by RedHat/Debian/Suse package
> manager to know which version can be installed, previously a list of
> packages that could be download from pkg.jenkins.io was generated for
> each release, this won't be the case anymore instead a link to mirrors
> should be provided. As a result of this, the latest weekly version is not
> listed even though you can easily craft an URL.
>
> We still have improvement coming and the next release can only be smoother.
>
> Thanks to Tim Jacob, Marky Jackson, Oleg Nenashev, Alex Earl, Mark Wait,
> Daniel Beck, Kohsuke, Oliver Ogondza who helped me to ship this first
> release yesterday but also to all the people who helped me, over the last
> two years to make this project a thing.
>
> Have a good day,
>
> Cheers
>
> On Wed, Apr 15, 2020, at 10:53 PM, Olblak wrote:
>
> After some discussion with the people involved in the process, I'll start
> at 11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned
> issues.
>
> On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote:
>
> I confirm my availability for tomorrow. Will also spend some time today to
> merge changes into the Jenkins core so that the release includes some real
> enhancements and fixes for Jenkins users.
>
> Regarding the schedule, it would be great to start the release process a
> bit earlier so that we have more time to fix issues if any. E.g. we could
> start the release Pipeline early so that artifacts are ready to packaging
> when US wakes up.
>
> Best regards,
> Oleg
>
>
>
> On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote:
>
> Today, during the infra meeting, we discussed creating a new weekly
> release from the new process this Thursday the 16th at 3PM UTC.
> While there won't be major features, the purpose of this release is to
> validate that everything is working as expected in a 

Re: [jenkins-infra] Jenkins Core Release automation

[Olblak]

Hi Everybody

Yesterday we released a weekly version from the new release environment and 
while we managed to ship our weekly version, we also faced unplanned issues 
that we had to solve so here a quick retro.

The process was done in two stages the release and then the packaging

Stage 1, Release:

The release part, was the easiest one. We only add two issues.
The first issue was due to the password used to authenticate on Maven 
repository, containing special characters that couldn't be parsed by Maven 
release plugin. We just changed the password.
The second problem was a permission issue on repo.jenkins-ci.org which was 
easily fixed.
In the result of this stage, as expected every artifact was pushed to 
repo.jenkins-ci.org

Stage 2, Package:

Well, this stage was more tricky as we heavily refactored that part while also 
heavily relying on existing service. We discovered multiple side cases that we 
didn't catch during the testing period and I will describe here the most 
important ones.

* GPG: While this was something that we discussed a long time ago we forgot to 
communicate in advance but the GPG key used to sign releases changed for weekly 
release in order to not use Kohsuke gpg key. In a result of this, people who 
use RedHat/Debian/Suse repository need to reimport the new key. Remark: this is 
something that will also have to be done for Lts once we also release Lts with 
the current process

New Key: 
pub rsa4096 2020-03-30 [SC] [expires: 2023-03-30]
 62A9756BFD780C377CF24BA8FCEF32E745F2C3D5
uid Jenkins Project 
sub rsa4096 2020-03-30 [E] [expires: 2023-03-30].

Old Key:
pub dsa1024 2009-02-01 [SC]
 150FDE3F7787E7D11EF4E12A9B7D32F2D50582E6
uid Kohsuke Kawaguchi 
uid [jpeg image of size 3704]
uid Kohsuke Kawaguchi 
uid Kohsuke Kawaguchi 
uid Kohsuke Kawaguchi 
sub elg2048 2009-02-01 [E]
sub rsa4096 2016-11-01 [S]

* mirror.jenkins-ci.org: This service has now additional Html files 
`HEADER.html` and `FOOTER.html` per directory. Those two files are used by 
apache directory listing to prepend and append Html information to the 
directory listing. The main advantage of this is to provide information from 
mirrors about how to install those packages from your operating system from 
every mirror. At least apache with the correct configuration, an example of 
this is

http://mirrors.jenkins-ci.org/debian/ or http://archives.jenkins-ci.org/debian/

Please note that the instruction to configure os package manager is referencing 
the wrong URL and it should be pkg.jenkins.io. A fix needs to be provided, 
other remark CSS needs to align with other Jenkins website if someone is 
willing to work on this, feel free to do, it's defined here 
https://github.com/jenkinsci/packaging/blob/INFRA-910-core-release-automation/templates/base.html

* pkg.jenkins.io: this service is used by RedHat/Debian/Suse package manager to 
know which version can be installed, previously a list of packages that could 
be download from pkg.jenkins.io was generated for each release, this won't be 
the case anymore instead a link to mirrors should be provided. As a result of 
this, the latest weekly version is not listed even though you can easily craft 
an URL.

We still have improvement coming and the next release can only be smoother.

Thanks to Tim Jacob, Marky Jackson, Oleg Nenashev, Alex Earl, Mark Wait, Daniel 
Beck, Kohsuke, Oliver Ogondza who helped me to ship this first release 
yesterday but also to all the people who helped me, over the last two years to 
make this project a thing.

Have a good day,

Cheers

On Wed, Apr 15, 2020, at 10:53 PM, Olblak wrote:
> After some discussion with the people involved in the process, I'll start at 
> 11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned issues.
> 
> On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote:
>> I confirm my availability for tomorrow. Will also spend some time today to 
>> merge changes into the Jenkins core so that the release includes some real 
>> enhancements and fixes for Jenkins users.
>> 
>> Regarding the schedule, it would be great to start the release process a bit 
>> earlier so that we have more time to fix issues if any. E.g. we could start 
>> the release Pipeline early so that artifacts are ready to packaging when US 
>> wakes up.
>> 
>> Best regards,
>> Oleg
>> 
>> 
>> 
>> On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote:
>>> Today, during the infra meeting, we discussed creating a new weekly release 
>>> from the new process this Thursday the 16th at 3PM UTC.
>>> While there won't be major features, the purpose of this release is to 
>>> validate that everything is working as expected in a real scenario.
>>> 
>>> 
>>> I am proposing the following agenda
>>> 
>>> 
>>> * 3PM(UTC), I start the release job and we release directly to the master 
>>> branch and directly on repo.jenkins-ci.org/releases.
>>> Currently the release job take approximatively 1h30
>>> 
>>> 
>>> * 4:30PM(UTC), or early if ready, I start 

Re: [jenkins-infra] Jenkins Core Release automation

[Olblak]

After some discussion with the people involved in the process, I'll start at 
11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned issues.

On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote:
> I confirm my availability for tomorrow. Will also spend some time today to 
> merge changes into the Jenkins core so that the release includes some real 
> enhancements and fixes for Jenkins users.
> 
> Regarding the schedule, it would be great to start the release process a bit 
> earlier so that we have more time to fix issues if any. E.g. we could start 
> the release Pipeline early so that artifacts are ready to packaging when US 
> wakes up.
> 
> Best regards,
> Oleg
> 
> 
> 
> On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote:
>> Today, during the infra meeting, we discussed creating a new weekly release 
>> from the new process this Thursday the 16th at 3PM UTC.
>> While there won't be major features, the purpose of this release is to 
>> validate that everything is working as expected in a real scenario.
>> 
>> 
>> I am proposing the following agenda
>> 
>> 
>> * 3PM(UTC), I start the release job and we release directly to the master 
>> branch and directly on repo.jenkins-ci.org/releases.
>> Currently the release job take approximatively 1h30
>> 
>> 
>> * 4:30PM(UTC), or early if ready, I start the packaging process in order to 
>> publish artifacts to pkg.jenkins.io
>> Currently, the packaging job take approximatively 15min to run and I 
>> estimate the script sync.sh to take 15min to run
>> 
>> * 5PM(UTC), we verify that the release is working correctly.
>> 
>> During the whole all process we'll be on IRC, #jenkins-infra channel
>> 
>> Again feel free to raise any concerns you may have.
>> 
>> Cheers
>> 
>> On Tue, Apr 14, 2020, at 4:44 PM, Oleg Nenashev wrote:
>>> Thanks a lot to Olivier and all other contributors who invested lots of 
>>> time to get it done! I am looking forward to finally get it over the line, 
>>> at least for weeklies.
>>> 
>>> FWIW, the Sunday's weekly got delayed (due to Easter). We could make an 
>>> attempt to cut this release using the new flow without waiting for the next 
>>> weekly.
>>> 
>>> Best regards,
>>> Oleg
>>> 
>>> 
>>> On Tue, Apr 14, 2020 at 4:32 PM Olblak  wrote:
 __
 Hi Everybody,
 
 This project is getting to the end and I am looking for a last round of 
 reviewers and testers before planning a weekly release with the new 
 process.
 I opened a pull request (link 
 ) with 
 instructions about how to trigger a release but currently, it only covers 
 weekly releases.
 Our first objective is to successfully release weekly versions from 
 https://release.ci.jenkins.io, then we'll focus on stable and security 
 releases.
 
 For security releases, the way I see it:
 1) We add two security profile, one for weekly and the second one for the 
 stable in order to trigger a release from jenkinsci-cert/jenkins:master to 
 https://repo.jenkins-ci.org/releases/
 2) Trigger packaging job(weekly or stable profile), 
 3) We manually merge from jenkinsci-cert/jenkins:master to 
 jenkinsci/jenkins:master
 
 For stable releases:
 1) We update the stable 
  
 profile branch to 'stable-', variable JENKINS_GIT_BRANCH)
 2) We trigger stable release job
 3) We trigger stable packaging job
 
 Another topic that needs clarification is who should be able to trigger a 
 release from release.ci.jenkins.io and who will trigger the different 
 releases?
 
 Cheers
 

 --
 You received this message because you are subscribed to the Google Groups 
 "Jenkins Infrastructure" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to jenkin...@googlegroups.com.
 To view this discussion on the web, visit 
 https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com
  
 .
>>> 

>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "Jenkins Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to jenkin...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com
>>>  
>>> .
>> 
> 

> --
>  You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
>  To 

Re: [jenkins-infra] Jenkins Core Release automation

[Oleg Nenashev]

I confirm my availability for tomorrow. Will also spend some time today to 
merge changes into the Jenkins core so that the release includes some real 
enhancements and fixes for Jenkins users.

Regarding the schedule, it would be great to start the release process a 
bit earlier so that we have more time to fix issues if any. E.g. we could 
start the release Pipeline early so that artifacts are ready to packaging 
when US wakes up.

Best regards,
Oleg



On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote:
>
> Today, during the infra meeting, we discussed creating a new weekly 
> release from the new process this Thursday the 16th at 3PM UTC.
> While there won't be major features, the purpose of this release is to 
> validate that everything is working as expected in a real scenario.
>
>
> I am proposing the following agenda
>
>
> * 3PM(UTC), I start the release job and we release directly to the master 
> branch and directly on repo.jenkins-ci.org/releases.
> Currently the release job take approximatively 1h30
>
>
> * 4:30PM(UTC), or early if ready, I start the packaging process in order 
> to publish artifacts to pkg.jenkins.io
> Currently, the packaging job take approximatively 15min to run and I 
> estimate the script sync.sh to take 15min to run
>
> * 5PM(UTC), we verify that the release is working correctly.
>
> During the whole all process we'll be on IRC, #jenkins-infra channel
>
> Again feel free to raise any concerns you may have.
>
> Cheers
>
> On Tue, Apr 14, 2020, at 4:44 PM, Oleg Nenashev wrote:
>
> Thanks a lot to Olivier and all other contributors who invested lots of 
> time to get it done! I am looking forward to finally get it over the line, 
> at least for weeklies.
>
> FWIW, the Sunday's weekly got delayed (due to Easter). We could make an 
> attempt to cut this release using the new flow without waiting for the next 
> weekly.
>
> Best regards,
> Oleg
>  
>
> On Tue, Apr 14, 2020 at 4:32 PM Olblak > 
> wrote:
>
>
> Hi Everybody,
>
> This project is getting to the end and I am looking for a last round of 
> reviewers and testers before planning a weekly release with the new process.
> I opened a pull request (link 
> ) with 
> instructions about how to trigger a release but currently, it only covers 
> weekly releases.
> Our first objective is to successfully release weekly versions from 
> https://release.ci.jenkins.io,  then we'll 
> focus on stable and security releases.
>
> For security releases, the way I see it:
> 1) We add two security profile, one for weekly and the second one for the 
> stable in order to trigger a release from jenkinsci-cert/jenkins:master  to 
> https://repo.jenkins-ci.org/releases/
> 2) Trigger packaging job(weekly or stable profile), 
> 3) We manually merge from jenkinsci-cert/jenkins:master to 
> jenkinsci/jenkins:master
>
> For stable releases:
> 1) We update the stable 
>  
> profile branch to 'stable-', variable JENKINS_GIT_BRANCH)
> 2) We trigger stable release job
> 3) We trigger stable packaging job
>
> Another topic that needs clarification is who should be able to trigger a 
> release from release.ci.jenkins.io and who will trigger the different 
> releases?
>
> Cheers
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Infrastructure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkin...@googlegroups.com .
> To view this discussion on the web, visit 
> https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com
>  
> 
> .
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkin...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/cb175d2d-b8f3-403c-a0d1-5c2ebfdbfea6%40googlegroups.com.

Re: [jenkins-infra] Jenkins Core Release automation

[Olblak]

Today, during the infra meeting, we discussed creating a new weekly release 
from the new process this Thursday the 16th at 3PM UTC.
While there won't be major features, the purpose of this release is to validate 
that everything is working as expected in a real scenario.


I am proposing the following agenda


* 3PM(UTC), I start the release job and we release directly to the master 
branch and directly on repo.jenkins-ci.org/releases.
Currently the release job take approximatively 1h30


* 4:30PM(UTC), or early if ready, I start the packaging process in order to 
publish artifacts to pkg.jenkins.io
Currently, the packaging job take approximatively 15min to run and I estimate 
the script sync.sh to take 15min to run

* 5PM(UTC), we verify that the release is working correctly.

During the whole all process we'll be on IRC, #jenkins-infra channel

Again feel free to raise any concerns you may have.

Cheers

On Tue, Apr 14, 2020, at 4:44 PM, Oleg Nenashev wrote:
> Thanks a lot to Olivier and all other contributors who invested lots of time 
> to get it done! I am looking forward to finally get it over the line, at 
> least for weeklies.
> 
> FWIW, the Sunday's weekly got delayed (due to Easter). We could make an 
> attempt to cut this release using the new flow without waiting for the next 
> weekly.
> 
> Best regards,
> Oleg
> 
> 
> On Tue, Apr 14, 2020 at 4:32 PM Olblak  wrote:
>> __
>> Hi Everybody,
>> 
>> This project is getting to the end and I am looking for a last round of 
>> reviewers and testers before planning a weekly release with the new process.
>> I opened a pull request (link 
>> ) with instructions 
>> about how to trigger a release but currently, it only covers weekly releases.
>> Our first objective is to successfully release weekly versions from 
>> https://release.ci.jenkins.io, then we'll focus on stable and security 
>> releases.
>> 
>> For security releases, the way I see it:
>> 1) We add two security profile, one for weekly and the second one for the 
>> stable in order to trigger a release from jenkinsci-cert/jenkins:master to 
>> https://repo.jenkins-ci.org/releases/
>> 2) Trigger packaging job(weekly or stable profile), 
>> 3) We manually merge from jenkinsci-cert/jenkins:master to 
>> jenkinsci/jenkins:master
>> 
>> For stable releases:
>> 1) We update the stable 
>>  
>> profile branch to 'stable-', variable JENKINS_GIT_BRANCH)
>> 2) We trigger stable release job
>> 3) We trigger stable packaging job
>> 
>> Another topic that needs clarification is who should be able to trigger a 
>> release from release.ci.jenkins.io and who will trigger the different 
>> releases?
>> 
>> Cheers
>> 

>> --
>>  You received this message because you are subscribed to the Google Groups 
>> "Jenkins Infrastructure" group.
>>  To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkins-infra+unsubscr...@googlegroups.com.
>>  To view this discussion on the web, visit 
>> https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com
>>  
>> .
> 

> --
>  You received this message because you are subscribed to the Google Groups 
> "Jenkins Developers" group.
>  To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
>  To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/90698ce9-9e17-4d62-be65-6cd13618ed24%40www.fastmail.com.

Re: [jenkins-infra] Jenkins Core Release automation

[Oleg Nenashev]

Thanks a lot to Olivier and all other contributors who invested lots of
time to get it done! I am looking forward to finally get it over the line,
at least for weeklies.

FWIW, the Sunday's weekly got delayed (due to Easter). We could make an
attempt to cut this release using the new flow without waiting for the next
weekly.

Best regards,
Oleg


On Tue, Apr 14, 2020 at 4:32 PM Olblak  wrote:

> Hi Everybody,
>
> This project is getting to the end and I am looking for a last round of
> reviewers and testers before planning a weekly release with the new process.
> I opened a pull request (link
> ) with
> instructions about how to trigger a release but currently, it only covers
> weekly releases.
> Our first objective is to successfully release weekly versions from
> https://release.ci.jenkins.io,  then we'll
> focus on stable and security releases.
>
> For security releases, the way I see it:
> 1) We add two security profile, one for weekly and the second one for the
> stable in order to trigger a release from jenkinsci-cert/jenkins:master  to
> https://repo.jenkins-ci.org/releases/
> 2) Trigger packaging job(weekly or stable profile),
> 3) We manually merge from jenkinsci-cert/jenkins:master to
> jenkinsci/jenkins:master
>
> For stable releases:
> 1) We update the stable
> 
> profile branch to 'stable-', variable JENKINS_GIT_BRANCH)
> 2) We trigger stable release job
> 3) We trigger stable packaging job
>
> Another topic that needs clarification is who should be able to trigger a
> release from release.ci.jenkins.io and who will trigger the different
> releases?
>
> Cheers
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Infrastructure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkins-infra+unsubscr...@googlegroups.com.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com.