Re: [openstack-dev] [Magnum] Bug 1541105 options

[Steven Dake (stdake)]

Sorry for 1 day delay in response - busy prepping for Kolla midcycle next
week.  Responses inline.

On 2/5/16, 12:14 PM, "Steve Gordon"  wrote:

>- Original Message -
>> From: "Steven Dake (stdake)" 
>> To: "OpenStack Development Mailing List (not for usage questions)"
>>
>> 
>> Steve,
>> 
>> Comments inline
>> 
>> On 2/3/16, 3:08 PM, "Steve Gordon"  wrote:
>> 
>> >- Original Message -
>> >> From: "Hongbin Lu" 
>> >> To: "OpenStack Development Mailing List (not for usage questions)"
>> >>
>> >> 
>> >> I would vote for a quick fix + a blueprint.
>> >> 
>> >> BTW, I think it is a general consensus that we should move away from
>> >>Atomic
>> >> for various reasons (painful image building, lack of document, hard
>>to
>> >>use,
>> >> etc.). We are working on fixing the CoreOS templates which could
>>replace
>> >> Atomic in the future.
>> >> 
>> >> Best regards,
>> >> Hongbin
>> >
>> >Hi Hongbin,
>> >
>> >I had heard this previously in Tokyo and again when I was asking around
>> >about the image support on IRC last week, is there a list of the exact
>> >issues with image building etc. with regards to Atomic? When I was
>> >following up on this it seemed like the main issue is that the docs in
>> >the magnum repo are quite out of date (versus the upstream fedora
>>atomic
>> >docs) both with regards to the content of the image and the process
>>used
>> >to (re)build it - there didn't seem to be anything quantifiable that's
>> >wrong with the current Atomic images but perhaps I was asking the wrong
>> >folks. I was able to rebuild fairly trivially using the Fedora built
>> >artefacts [1][2].
>> 
>> Steve,
>> 
>> I hope you can forgive my directness and lack of diplomacy in this
>> message. :)
>> 
>> At least when I was heavily involved with Magnum, building atomic images
>> resulted in a situation in which the binaries built did not work
>>properly.
>>  I begged on the irc channels for help and begged on the mailing list
>>for
>> help for _ months _ on end and nobody listened.  It is almost as if
>>nobody
>> is actually working on Atomic.  If there are people, they do not
>>maintain
>> any kind of support footprint upstream to make Atomic a viable platform
>> for Magnum.
>
>Hi Steve,
>
>No worries about directness, it's actually what I am trying to elicit so
>I can understand and hopefully help address the issues. I did come across
>a couple of mailing list posts from you mainly asking about newer
>k8s/etcd/flannel versions and also etcdctl inclusion (there was a
>workaround posted at the time but I note it's now in the image) [1][2].
>These threads are actually what prompted me to ask "what else?" to try
>and determine if Magnum had other needs here, e.g. the rpm-ostree issue
>you refer to below is the type of hint I'm after to try and ensure
>everything is tracked somewhere.
>
>> I taught Tango how to build the images, who wrote the instructions down
>>in
>> the Magnum documentation.  That documentation ends up producing images
>> that randomly don't always work. The binaries return some weird system
>> call error, ebadlink I think but not sure.  Tango may remember.
>
>I was playing around with this earlier in the week and had some more
>success via a slightly different path, I'm in the process of attempting
>to re-produce on a clean system to ensure that (a) I didn't skip anything
>in the notes I took and (b) weed out any other setup that was peculiar to
>my system before I propose something against the docs.
>
>> Perhaps the rpm-ostree defect has been resolved now.  I have to be clear
>> that I was told "please wait 6 months for us to fix the build system and
>> bugs" while Atomic was our only distro implemented.  It was very
>> maddening.  I was so frustrated with Atomic, at the start of Mitaka I
>>was
>> going to propose deprecating Atomic because of a complete lack of
>>upstream
>> responsiveness.  I decided to let other folks make the call about what
>> they wanted to do with Atomic since I was myself unresponsive with the
>> Magnum upstream because of my full-time Kolla commitment.
>>
>> I am pretty sure a bug was filed about this issue in the Red Hat
>>bugzilla,
>> but I can't find it.
>
>Thanks, the hint that it regarded rpm-ostree was enough for me to find
>this:
>
>https://bugzilla.redhat.com/show_bug.cgi?id=1177989

This is the bug that ailed us :)

>
>Does that seem like the right issue? That in turn led me to this bug
>which appears fixed in F21, but I need to follow up to work out what if
>anything happened w.r.t. the second half of Colin's comment in the above
>bug (or whether it is still relevant):
>
>https://bugzilla.redhat.com/show_bug.cgi?id=1178208

I think the speculation around selinux may or may not be correct.  We run
Magnum without Selinux, but building the magnum images with rpm-ostree may
run with selinux, so 

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Steve Gordon]

- Original Message -
> From: "Steven Dake (stdake)" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> 
> Steve,
> 
> Comments inline
> 
> On 2/3/16, 3:08 PM, "Steve Gordon"  wrote:
> 
> >- Original Message -
> >> From: "Hongbin Lu" 
> >> To: "OpenStack Development Mailing List (not for usage questions)"
> >>
> >> 
> >> I would vote for a quick fix + a blueprint.
> >> 
> >> BTW, I think it is a general consensus that we should move away from
> >>Atomic
> >> for various reasons (painful image building, lack of document, hard to
> >>use,
> >> etc.). We are working on fixing the CoreOS templates which could replace
> >> Atomic in the future.
> >> 
> >> Best regards,
> >> Hongbin
> >
> >Hi Hongbin,
> >
> >I had heard this previously in Tokyo and again when I was asking around
> >about the image support on IRC last week, is there a list of the exact
> >issues with image building etc. with regards to Atomic? When I was
> >following up on this it seemed like the main issue is that the docs in
> >the magnum repo are quite out of date (versus the upstream fedora atomic
> >docs) both with regards to the content of the image and the process used
> >to (re)build it - there didn't seem to be anything quantifiable that's
> >wrong with the current Atomic images but perhaps I was asking the wrong
> >folks. I was able to rebuild fairly trivially using the Fedora built
> >artefacts [1][2].
> 
> Steve,
> 
> I hope you can forgive my directness and lack of diplomacy in this
> message. :)
> 
> At least when I was heavily involved with Magnum, building atomic images
> resulted in a situation in which the binaries built did not work properly.
>  I begged on the irc channels for help and begged on the mailing list for
> help for _ months _ on end and nobody listened.  It is almost as if nobody
> is actually working on Atomic.  If there are people, they do not maintain
> any kind of support footprint upstream to make Atomic a viable platform
> for Magnum.

Hi Steve,

No worries about directness, it's actually what I am trying to elicit so I can 
understand and hopefully help address the issues. I did come across a couple of 
mailing list posts from you mainly asking about newer k8s/etcd/flannel versions 
and also etcdctl inclusion (there was a workaround posted at the time but I 
note it's now in the image) [1][2]. These threads are actually what prompted me 
to ask "what else?" to try and determine if Magnum had other needs here, e.g. 
the rpm-ostree issue you refer to below is the type of hint I'm after to try 
and ensure everything is tracked somewhere.

> I taught Tango how to build the images, who wrote the instructions down in
> the Magnum documentation.  That documentation ends up producing images
> that randomly don't always work. The binaries return some weird system
> call error, ebadlink I think but not sure.  Tango may remember.

I was playing around with this earlier in the week and had some more success 
via a slightly different path, I'm in the process of attempting to re-produce 
on a clean system to ensure that (a) I didn't skip anything in the notes I took 
and (b) weed out any other setup that was peculiar to my system before I 
propose something against the docs.

> Perhaps the rpm-ostree defect has been resolved now.  I have to be clear
> that I was told "please wait 6 months for us to fix the build system and
> bugs" while Atomic was our only distro implemented.  It was very
> maddening.  I was so frustrated with Atomic, at the start of Mitaka I was
> going to propose deprecating Atomic because of a complete lack of upstream
> responsiveness.  I decided to let other folks make the call about what
> they wanted to do with Atomic since I was myself unresponsive with the
> Magnum upstream because of my full-time Kolla commitment.
>
> I am pretty sure a bug was filed about this issue in the Red Hat bugzilla,
> but I can't find it.

Thanks, the hint that it regarded rpm-ostree was enough for me to find this:

https://bugzilla.redhat.com/show_bug.cgi?id=1177989

Does that seem like the right issue? That in turn led me to this bug which 
appears fixed in F21, but I need to follow up to work out what if anything 
happened w.r.t. the second half of Colin's comment in the above bug (or whether 
it is still relevant):

https://bugzilla.redhat.com/show_bug.cgi?id=1178208

Somewhat tangentially I was also wondering if there are specific version 
combinations of components (be they images, or the versions of specific 
components they contain) that are currently recommended/preferred for a given 
release. Obviously there is the custom image that is currently provided but I 
am thinking more for folks trying to roll their own etc. I didn't spot anything 
while poking around in the docs as yet but I may be looking in the wrong 
places. What I am 

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Steve Gordon]

- Original Message -
> From: "Hongbin Lu" <hongbin...@huawei.com>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> 
> I would vote for a quick fix + a blueprint.
> 
> BTW, I think it is a general consensus that we should move away from Atomic
> for various reasons (painful image building, lack of document, hard to use,
> etc.). We are working on fixing the CoreOS templates which could replace
> Atomic in the future.
> 
> Best regards,
> Hongbin

Hi Hongbin,

I had heard this previously in Tokyo and again when I was asking around about 
the image support on IRC last week, is there a list of the exact issues with 
image building etc. with regards to Atomic? When I was following up on this it 
seemed like the main issue is that the docs in the magnum repo are quite out of 
date (versus the upstream fedora atomic docs) both with regards to the content 
of the image and the process used to (re)build it - there didn't seem to be 
anything quantifiable that's wrong with the current Atomic images but perhaps I 
was asking the wrong folks. I was able to rebuild fairly trivially using the 
Fedora built artefacts [1][2].

So are the exact requirements of Magnum w.r.t. the image and how they aren't 
currently met listed somewhere? If there are quantifiable issues then I can get 
them in front of the atomic folks to address them.

Thanks,

Steve

[1] https://git.fedorahosted.org/git/spin-kickstarts.git
[2] https://git.fedorahosted.org/git/fedora-atomic.git


> From: Corey O'Brien [mailto:coreypobr...@gmail.com]
> Sent: February-03-16 2:53 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Magnum] Bug 1541105 options
> 
> As long as configurations for 2.2 and 2.0 are compatible we shouldn't have an
> issue I wouldn't think. I just don't know enough about etcd deployment to be
> sure about that.
> 
> If we want to quickly improve the gate, I can patch the problematic areas in
> the templates and then we can make a blueprint for upgrading to Atomic 23.
> 
> Corey
> 
> On Wed, Feb 3, 2016 at 1:47 PM Vilobh Meshram
> <vilobhmeshram.openst...@gmail.com<mailto:vilobhmeshram.openst...@gmail.com>>
> wrote:
> Hi Corey,
> 
> This is slowing down our merge rate and needs to be fixed IMHO.
> 
> What risk are you talking about when using newer version of etcd ? Is it
> documented somewhere for the team to have a look ?
> 
> -Vilobh
> 
> On Wed, Feb 3, 2016 at 8:11 AM, Corey O'Brien
> <coreypobr...@gmail.com<mailto:coreypobr...@gmail.com>> wrote:
> Hey team,
> 
> I've been looking into https://bugs.launchpad.net/magnum/+bug/1541105 which
> covers a bug with etcdctl, and I wanted opinions on how best to fix it.
> 
> Should we update the image to include the latest version of etcd? Or, should
> we temporarily install the latest version as a part of notify-heat (see bug
> for patch)?
> 
> I'm personally in favor of updating the image, but there is presumably some
> small risk with using a newer version of etcd.
> 
> Thanks,
> Corey O'Brien

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Hongbin Lu]

I would vote for a quick fix + a blueprint.

BTW, I think it is a general consensus that we should move away from Atomic for 
various reasons (painful image building, lack of document, hard to use, etc.). 
We are working on fixing the CoreOS templates which could replace Atomic in the 
future.

Best regards,
Hongbin

From: Corey O'Brien [mailto:coreypobr...@gmail.com]
Sent: February-03-16 2:53 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Magnum] Bug 1541105 options

As long as configurations for 2.2 and 2.0 are compatible we shouldn't have an 
issue I wouldn't think. I just don't know enough about etcd deployment to be 
sure about that.

If we want to quickly improve the gate, I can patch the problematic areas in 
the templates and then we can make a blueprint for upgrading to Atomic 23.

Corey

On Wed, Feb 3, 2016 at 1:47 PM Vilobh Meshram 
<vilobhmeshram.openst...@gmail.com<mailto:vilobhmeshram.openst...@gmail.com>> 
wrote:
Hi Corey,

This is slowing down our merge rate and needs to be fixed IMHO.

What risk are you talking about when using newer version of etcd ? Is it 
documented somewhere for the team to have a look ?

-Vilobh

On Wed, Feb 3, 2016 at 8:11 AM, Corey O'Brien 
<coreypobr...@gmail.com<mailto:coreypobr...@gmail.com>> wrote:
Hey team,

I've been looking into https://bugs.launchpad.net/magnum/+bug/1541105 which 
covers a bug with etcdctl, and I wanted opinions on how best to fix it.

Should we update the image to include the latest version of etcd? Or, should we 
temporarily install the latest version as a part of notify-heat (see bug for 
patch)?

I'm personally in favor of updating the image, but there is presumably some 
small risk with using a newer version of etcd.

Thanks,
Corey O'Brien

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Steven Dake (stdake)]

Steve,

Comments inline

On 2/3/16, 3:08 PM, "Steve Gordon" <sgor...@redhat.com> wrote:

>- Original Message -
>> From: "Hongbin Lu" <hongbin...@huawei.com>
>> To: "OpenStack Development Mailing List (not for usage questions)"
>><openstack-dev@lists.openstack.org>
>> 
>> I would vote for a quick fix + a blueprint.
>> 
>> BTW, I think it is a general consensus that we should move away from
>>Atomic
>> for various reasons (painful image building, lack of document, hard to
>>use,
>> etc.). We are working on fixing the CoreOS templates which could replace
>> Atomic in the future.
>> 
>> Best regards,
>> Hongbin
>
>Hi Hongbin,
>
>I had heard this previously in Tokyo and again when I was asking around
>about the image support on IRC last week, is there a list of the exact
>issues with image building etc. with regards to Atomic? When I was
>following up on this it seemed like the main issue is that the docs in
>the magnum repo are quite out of date (versus the upstream fedora atomic
>docs) both with regards to the content of the image and the process used
>to (re)build it - there didn't seem to be anything quantifiable that's
>wrong with the current Atomic images but perhaps I was asking the wrong
>folks. I was able to rebuild fairly trivially using the Fedora built
>artefacts [1][2].

Steve,

I hope you can forgive my directness and lack of diplomacy in this
message. :)

At least when I was heavily involved with Magnum, building atomic images
resulted in a situation in which the binaries built did not work properly.
 I begged on the irc channels for help and begged on the mailing list for
help for _ months _ on end and nobody listened.  It is almost as if nobody
is actually working on Atomic.  If there are people, they do not maintain
any kind of support footprint upstream to make Atomic a viable platform
for Magnum.

I taught Tango how to build the images, who wrote the instructions down in
the Magnum documentation.  That documentation ends up producing images
that randomly don't always work. The binaries return some weird system
call error, ebadlink I think but not sure.  Tango may remember.

Perhaps the rpm-ostree defect has been resolved now.  I have to be clear
that I was told "please wait 6 months for us to fix the build system and
bugs" while Atomic was our only distro implemented.  It was very
maddening.  I was so frustrated with Atomic, at the start of Mitaka I was
going to propose deprecating Atomic because of a complete lack of upstream
responsiveness.  I decided to let other folks make the call about what
they wanted to do with Atomic since I was myself unresponsive with the
Magnum upstream because of my full-time Kolla commitment.

I am pretty sure a bug was filed about this issue in the Red Hat bugzilla,
but I can't find it.

Personally if I was running the Atomic project, I'd containerize all of
the etcd/flannel/kubernetes and only leave docker in the base image.  Next
up I'd get the distro being produced in a CI pipeline with atleast some
basic dead chicken testing.  I am pretty sure this would fit Magnum's use
case well.  This would make interacting with the 6 month release cycle of
Fedora much more viable.  But alas I'm not running Atomic, and I don't
have the bandwidth to make a contribution here other then to say Atomic
needs more attention from it's upstream if it is to have any hope.

Warm regards,
-steve


>
>So are the exact requirements of Magnum w.r.t. the image and how they
>aren't currently met listed somewhere? If there are quantifiable issues
>then I can get them in front of the atomic folks to address them.
>
>Thanks,
>
>Steve
>
>[1] https://git.fedorahosted.org/git/spin-kickstarts.git
>[2] https://git.fedorahosted.org/git/fedora-atomic.git
>
>
>> From: Corey O'Brien [mailto:coreypobr...@gmail.com]
>> Sent: February-03-16 2:53 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [Magnum] Bug 1541105 options
>> 
>> As long as configurations for 2.2 and 2.0 are compatible we shouldn't
>>have an
>> issue I wouldn't think. I just don't know enough about etcd deployment
>>to be
>> sure about that.
>> 
>> If we want to quickly improve the gate, I can patch the problematic
>>areas in
>> the templates and then we can make a blueprint for upgrading to Atomic
>>23.
>> 
>> Corey
>> 
>> On Wed, Feb 3, 2016 at 1:47 PM Vilobh Meshram
>> 
>><vilobhmeshram.openst...@gmail.com<mailto:vilobhmeshram.openstack@gmail.c
>>om>>
>> wrote:
>> Hi Corey,
>> 
>> This is slowing down our merge rate and needs to be fixed IMHO.
>> 
>> What risk are

[openstack-dev] [Magnum] Bug 1541105 options

[Corey O'Brien]

Hey team,

I've been looking into https://bugs.launchpad.net/magnum/+bug/1541105 which
covers a bug with etcdctl, and I wanted opinions on how best to fix it.

Should we update the image to include the latest version of etcd? Or,
should we temporarily install the latest version as a part of notify-heat
(see bug for patch)?

I'm personally in favor of updating the image, but there is presumably some
small risk with using a newer version of etcd.

Thanks,
Corey O'Brien
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Vilobh Meshram]

Hi Corey,

This is slowing down our merge rate and needs to be fixed IMHO.

What risk are you talking about when using newer version of etcd ? Is it
documented somewhere for the team to have a look ?

-Vilobh

On Wed, Feb 3, 2016 at 8:11 AM, Corey O'Brien 
wrote:

> Hey team,
>
> I've been looking into https://bugs.launchpad.net/magnum/+bug/1541105 which
> covers a bug with etcdctl, and I wanted opinions on how best to fix it.
>
> Should we update the image to include the latest version of etcd? Or,
> should we temporarily install the latest version as a part of notify-heat
> (see bug for patch)?
>
> I'm personally in favor of updating the image, but there is presumably
> some small risk with using a newer version of etcd.
>
> Thanks,
> Corey O'Brien
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Magnum] Bug 1541105 options

[Corey O'Brien]

As long as configurations for 2.2 and 2.0 are compatible we shouldn't have
an issue I wouldn't think. I just don't know enough about etcd deployment
to be sure about that.

If we want to quickly improve the gate, I can patch the problematic areas
in the templates and then we can make a blueprint for upgrading to Atomic
23.

Corey

On Wed, Feb 3, 2016 at 1:47 PM Vilobh Meshram <
vilobhmeshram.openst...@gmail.com> wrote:

> Hi Corey,
>
> This is slowing down our merge rate and needs to be fixed IMHO.
>
> What risk are you talking about when using newer version of etcd ? Is it
> documented somewhere for the team to have a look ?
>
> -Vilobh
>
> On Wed, Feb 3, 2016 at 8:11 AM, Corey O'Brien 
> wrote:
>
>> Hey team,
>>
>> I've been looking into https://bugs.launchpad.net/magnum/+bug/1541105 which
>> covers a bug with etcdctl, and I wanted opinions on how best to fix it.
>>
>> Should we update the image to include the latest version of etcd? Or,
>> should we temporarily install the latest version as a part of notify-heat
>> (see bug for patch)?
>>
>> I'm personally in favor of updating the image, but there is presumably
>> some small risk with using a newer version of etcd.
>>
>> Thanks,
>> Corey O'Brien
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev