Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-09-25 Thread Tony Breeds 
On Fri, Jun 16, 2017 at 12:06:47PM +1000, Tony Breeds wrote:
> Hi All,
>   I just push a review [1] to bump the minimum etcd version to
> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
> cycle to be making changes like this but releasing pike with a dependacy
> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
> 
> Yours Tony.
> 
> [1] https://review.openstack.org/474825

So this came up at the PTG and the current plan is:

Interim solution:
 1. Get the mirroring tool to the point it can be consumed by infra.
 2. setup a new zuulv3 job to run this and do the mirroring.

Middle term solution:
 1. Get etcd3 packages updated in debian/ubuntu create a PPA (or
similar) for infra to consume.

Both of these are intended to be done during Queens.

Long term plan:
 1. Ensue the packages above are just there for 18.04 so we can put this
behind us.

With hindsight I think we need to add something like "Current packages
can be consumed in our CI" as a requirement for anything added as a base
service.



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-19 Thread Tony Breeds 
On Mon, Jun 19, 2017 at 08:17:53AM -0400, Davanum Srinivas wrote:
> Tony,
> 
> 
> On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds  wrote:
> > On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:
> >
> >> Awesome! thanks Tony, some kolla jobs do that for example, but i think
> >> this job is a better one to key off of:
> >> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
> >>
> >> Outline of the work is - check if there are any new releases in github
> >> downloads, if so download them using wget and then delegate to the scp
> >> publisher (with keep-hierarchy) to create the new directories and
> >> upload the file(s).
> >
> > So perhaps I'm dense but I can't see an easy way to get a list of
> > release artefacts from github in a form that wget can consume.  The best
> > I can see is via the API.  I've knocked up a quick'n'dirty mirror
> > script[1] but I really feel like I've gone off into the weeds.
> >
> > You basically need to do:
> >
> > git clone  && cd
> > virtualenv .venv
> > .venv/bin/pip install -U pip setuptools wheel
> > .venv/bin/pip install -r ./requirements.txt   # [2]
> > .venv/bin/python ./mirror-github-releases.py \
> > 'coreos/etcd::.*linux.*gz:etcd' \
> > 'coreos/etcd:6225411:.*linux.*gz:etcd'
> 
> Works for me!

Okay I'll put something more compleet together for infra review.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-19 Thread Davanum Srinivas 
Tony,


On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds  wrote:
> On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:
>
>> Awesome! thanks Tony, some kolla jobs do that for example, but i think
>> this job is a better one to key off of:
>> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
>>
>> Outline of the work is - check if there are any new releases in github
>> downloads, if so download them using wget and then delegate to the scp
>> publisher (with keep-hierarchy) to create the new directories and
>> upload the file(s).
>
> So perhaps I'm dense but I can't see an easy way to get a list of
> release artefacts from github in a form that wget can consume.  The best
> I can see is via the API.  I've knocked up a quick'n'dirty mirror
> script[1] but I really feel like I've gone off into the weeds.
>
> You basically need to do:
>
> git clone  && cd
> virtualenv .venv
> .venv/bin/pip install -U pip setuptools wheel
> .venv/bin/pip install -r ./requirements.txt   # [2]
> .venv/bin/python ./mirror-github-releases.py \
> 'coreos/etcd::.*linux.*gz:etcd' \
> 'coreos/etcd:6225411:.*linux.*gz:etcd'

Works for me!

> This will in theory from the 3.2.0 (latest) release and look at the
> 3.1.7 release, see that it's already publically mirrored and move on.
>
> It wouldn't be too hard to incorporate into a job.  Thoughts?
>
> Yours Tony.
>
> [1]  https://github.com/tbreeds/mirror-github-releases
> [2] Yes of course I could publish it on pypi if we want to go down this
> path
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-18 Thread Tony Breeds 
On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:

> Awesome! thanks Tony, some kolla jobs do that for example, but i think
> this job is a better one to key off of:
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
> 
> Outline of the work is - check if there are any new releases in github
> downloads, if so download them using wget and then delegate to the scp
> publisher (with keep-hierarchy) to create the new directories and
> upload the file(s).

So perhaps I'm dense but I can't see an easy way to get a list of
release artefacts from github in a form that wget can consume.  The best
I can see is via the API.  I've knocked up a quick'n'dirty mirror
script[1] but I really feel like I've gone off into the weeds.

You basically need to do:

git clone  && cd
virtualenv .venv
.venv/bin/pip install -U pip setuptools wheel
.venv/bin/pip install -r ./requirements.txt   # [2]
.venv/bin/python ./mirror-github-releases.py \
'coreos/etcd::.*linux.*gz:etcd' \
'coreos/etcd:6225411:.*linux.*gz:etcd'

This will in theory from the 3.2.0 (latest) release and look at the
3.1.7 release, see that it's already publically mirrored and move on.

It wouldn't be too hard to incorporate into a job.  Thoughts?

Yours Tony.

[1]  https://github.com/tbreeds/mirror-github-releases
[2] Yes of course I could publish it on pypi if we want to go down this
path


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-18 Thread Davanum Srinivas 
On Sun, Jun 18, 2017 at 7:36 PM, Tony Breeds  wrote:
> On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:
>> Mikhail,
>>
>> I have a TODO on my list - " adding a job that looks for new releases
>> and uploads them to tarballs periodically "
>
> If you point me to how things are added to that mirror I can work
> towards that.

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

Thanks,
Dims

>
> Tony.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-18 Thread Tony Breeds 
On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:
> Mikhail,
> 
> I have a TODO on my list - " adding a job that looks for new releases
> and uploads them to tarballs periodically "

If you point me to how things are added to that mirror I can work
towards that.

Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-16 Thread Davanum Srinivas 
Mikhail,

I have a TODO on my list - " adding a job that looks for new releases
and uploads them to tarballs periodically "

Thanks,
-- Dims

On Fri, Jun 16, 2017 at 3:32 PM, Mikhail Medvedev  wrote:
> On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague  wrote:
>> On 06/15/2017 10:06 PM, Tony Breeds wrote:
>>> Hi All,
>>>   I just push a review [1] to bump the minimum etcd version to
>>> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
>>> cycle to be making changes like this but releasing pike with a dependacy
>>> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>>>
>>> Yours Tony.
>>>
>>> [1] https://review.openstack.org/474825
>>
>> It should be fine, no one is really using these much at this point.
>> However it looks like mirroring is not happening automatically? The
>> patch fails on not existing in the infra mirror.
>>
>> -Sean
>>
>
> It appears so. Also, IIRC, infra mirror would only host x86 binaries.
> Right now PowerKVM CI works by patching devstack-gate to override
> infra etcd download url. The fix [2] still needs to get merged to make
> it a bit easier to use d-g with your own etcd mirror.
>
> [2] https://review.openstack.org/#/c/467437/
>
> ---
> Mikhail Medvedev
> IBM OpenStack CI for KVM on Power
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-16 Thread Mikhail Medvedev 
On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague  wrote:
> On 06/15/2017 10:06 PM, Tony Breeds wrote:
>> Hi All,
>>   I just push a review [1] to bump the minimum etcd version to
>> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
>> cycle to be making changes like this but releasing pike with a dependacy
>> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>>
>> Yours Tony.
>>
>> [1] https://review.openstack.org/474825
>
> It should be fine, no one is really using these much at this point.
> However it looks like mirroring is not happening automatically? The
> patch fails on not existing in the infra mirror.
>
> -Sean
>

It appears so. Also, IIRC, infra mirror would only host x86 binaries.
Right now PowerKVM CI works by patching devstack-gate to override
infra etcd download url. The fix [2] still needs to get merged to make
it a bit easier to use d-g with your own etcd mirror.

[2] https://review.openstack.org/#/c/467437/

---
Mikhail Medvedev
IBM OpenStack CI for KVM on Power

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] etcd v3.2.0?

2017-06-16 Thread Sean Dague 
On 06/15/2017 10:06 PM, Tony Breeds wrote:
> Hi All,
>   I just push a review [1] to bump the minimum etcd version to
> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
> cycle to be making changes like this but releasing pike with a dependacy
> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
> 
> Yours Tony.
> 
> [1] https://review.openstack.org/474825

It should be fine, no one is really using these much at this point.
However it looks like mirroring is not happening automatically? The
patch fails on not existing in the infra mirror.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev