Solved! TL;DR - You must deploy quantum-gateway or neutron-gateway to bare
metal. It will not work in an LXC container.
It appears as though you cannot deploy the {quantum,neutron}-gateway charm
to an LXC. I eventually tracked down the stuck config-changed state due to
the openvswitch vswitchd daemon not being able to start because it cannot
load the kernel module properly, which causes a loop whereby it keeps
attempting to add flows to the br-int OVS but fails because the daemon is
not running.
I removed the service from the LXC and deployed it to a bare metal machine
(co-hosted with Ceph services that don’t seem to do anything fancy with
bridges or OVS) and everything worked just fine.
Thanks,
Jeff
On Fri, Sep 18, 2015 at 4:29 PM, Jeff McLamb <[email protected]> wrote:
> Thanks for the info, Mike! I will provide a bit more detail as to what’s
> going on in case any Juju experts have seen something similar.
>
> Where I last left off, I had deleted all units of anything
> {neutron,quantum}-gateway related from juju’s perspective. However, the
> original bare metal node was still providing the neutron gateway (i.e.
> network node) services, so I manually issued a stop on all of those (e.g.
> stop neutron-dhcp, stop neutron-metadata-server, stop neutron-metering,
> etc.) The only thing I left running on that server was openvswitch because
> it is also a compute node, so it needs to have the openvswitch plugin
> running, as I understand it.
>
> I then issued a `juju deploy neutron-gateway —to lxc:3` in order to
> install the gateway in an lxc container.
>
> The lxc container is provisioned and everything seems to be going OK until
> it gets stuck in agent status -> message: running config-changed hook
>
> If I ssh into the container itself and look at processes, I can see it is
> in a seemingly infinite loop where the neutron rootwrapper is constantly
> issuing ovs-vsctl add-br br-int calls, along with adding and removing
> flows. It’s as though it keeps trying to add and remove the br-int ovs, and
> just sits there doing that forever.
>
> I manually killed the config-changed hook and then issued a `juju
> debug-hooks neutron-gateway/0 config-changed` followed by `juju resolved
> neutron-gateway/0 —retry`, which has dumped me into a config-changed tmux
> session where I manually issued ./hooks/config-changed.
>
> There is not much other than the following output, after which it hangs:
>
> root@juju-machine-3-lxc-11:/var/lib/juju/agents/unit-neutron-gateway-0/charm#
> ./hooks/config-changed
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> python-dbus is already the newest version.
> The following packages were automatically installed and are no longer
> required:
> libfreetype6 os-prober
> Use 'apt-get autoremove' to remove them.
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> At this point everything hangs. I have not yet Ctrl-C’d the process yet.
>
> In the /var/log/juju/unit-neutron-gateway-0.log it looks pretty standard,
> finishing all log output with Creating bridge br-int, Creating bridge
> br-ex, Creating bridge br-tun.
>
> At this point there is no more streaming log output either.
>
> Has anyone else seen this or have an idea what might be going on?
>
> Thanks,
>
> Jeff
>
> On Fri, Sep 18, 2015 at 2:22 PM, Mike McCracken <
> [email protected]> wrote:
>
>> Hi Jeff!
>>
>> On Fri, Sep 18, 2015 at 11:12 AM, Jeff McLamb <[email protected]> wrote:
>>
>>> Hi all -
>>>
>>> * Sorry for the cross-post on openstack-installer, but I think this is
>>> more of a juju question than an openstack-installer issue*
>>>
>>> As I originally deployed my current test setup via openstack-installer,
>>> I am wondering how I am supposed to use it and/or juju directly for future
>>> changes/additions.
>>>
>>
>> Yes, the openstack-installer is intended to be a first step and not a
>> full purpose juju front-end. For future changes of the sort you're
>> describing, you'll want to get familiar with using juju directly. Note that
>> the official Juju web GUI is one of the services that you can install via
>> the manual placement UI in the installer.
>>
>> I am trying to understand neutron networking, investigating all of the
>>> OVS and Linux bridges, how traffic flows between them and the network node
>>> etc. In my initial deployment I co-located a compute node and the network
>>> node on bare metal via manual placement in openstack-installer.
>>>
>>> Now, in order to better see the separation of stuff (i.e. not conflate
>>> what the compute node is doing with the network node) I decided to `juju
>>> remove-service quantum-gateway` followed by a `juju add-service
>>> quantum-gateway —to lxc:3` in order to remove it from the original node and
>>> install it into its own container on the controller node that hosts lots of
>>> container-based services.
>>>
>>> I have received lots of errors in the process during which various hooks
>>> get stuck. I have been able to get around it by manually killing hooks and
>>> then issuing a `juju resolved quantum-gateway` until finally the service
>>> goes away. Not to mention the fact that a new deployment seems to be in a
>>> “blocked” state saying I should upgrade to neutron-gateway. I tried that
>>> with similar results.
>>>
>>> At this point I managed to get rid of everything — i.e. I do not have a
>>> quantum-gateway or a neutron-gateway as reported by juju.
>>>
>>> However, I have noticed that the original server (colocated compute and
>>> quantum-gateway) still has all of the neutron services running. Issuing a
>>> `neutron agent-list` shows everything connected and happy as it ever was.
>>> It seems as though my juju fiddling has basically been just manipulating
>>> the juju database without actually removing the underlying services. I have
>>> noticed that when removing services on an lxc container it does indeed kill
>>> the container (so obviously the service too)… however, maybe because this
>>> original service was deployed on bare metal with another service, it
>>> doesn’t bother to actually stop the services and remove their /etc/init
>>> files?
>>>
>>
>> This does sound unexpected, hopefully someone more directly involved with
>> Juju can comment here.
>>
>>
>>> I have tried to redeploy via openstack-status, but it seems like it is
>>> useless for future deployment even though it has that nice curses-looking
>>> GUI from which I can deploy. It’s understanding of where everything is
>>> placed is completely broken, and it does not seem to honor any requests to
>>> place stuff in the first place.
>>>
>>
>> I'll follow up with you about the openstack-status tool on the installer
>> list, to keep this thread juju-specific.
>>
>>
>>> Any ideas?
>>>
>>> Thanks,
>>>
>>> Jeff
>>>
>>> --
>>> Juju mailing list
>>> [email protected]
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>>
>>
>
--
Juju mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
