[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
** Changed in: charm-neutron-gateway Milestone: None => 18.05 ** Summary changed: - lbaas load balancer does not forward traffic unless agent restarted + apparmor profile blocks operation of haproxy loadbalancer updates -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: apparmor profile blocks operation of haproxy loadbalancer updates To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
** Changed in: charm-neutron-gateway Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
The apparmor profile would appear to be the issue here. I'll look at a fix but as a workaround please disable for gateway applications. ** Changed in: neutron-lbaas (Ubuntu) Status: Incomplete => Invalid ** Changed in: charm-neutron-gateway Importance: Undecided => Medium ** Changed in: charm-neutron-gateway Status: Incomplete => New ** Changed in: neutron-lbaas (Ubuntu) Assignee: James Page (james-page) => (unassigned) ** Changed in: charm-neutron-gateway Assignee: (unassigned) => James Page (james-page) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
This was reproduced with a heat template, but just running the steps at the start of the case from horizon are enough. Note that neutron- gateway was deployed with aa-profile-mode set to complain, not the default setting. Changing this to 'disable' seems to have fixed the problem, more testing is in progress. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
So, /var/log/neutron/neutron-lbaasv2-agent.log had: "WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES: error: [Errno 13] EACCES" with aa-profile-mode=complain. After setting aa-profile-mode=disabled (juju config --reset), it seems working now (the customer is still in testing though). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
When you restart "After restarting the service, all the traffic passes perfectly." this issues a GARP which re advertiser the location of the floating IP. In our case the floating IP could be one any of the compute 6 nodes (if used by nova) Or on the 2 neutron servers (used by LBaasS) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
James, I add some comments. LBaaS not serving traffic with Floating IP (DVR) https://answers.launchpad.net/ubuntu/+question/668889 I came across this bug which sort of touches on a few items, but I assume this would have already be fix is pike. https://bugs.launchpad.net/neutron/+bug/1583694 "Distributed Virtual Routers are created on each Compute node dynamically on demand and removed when not required. Distributed Virtual Routers heavily depend on the port binding to identify the requirement of a DVR service on a particular node." "This would create an issue because we will be seeing the same FloatingIP being advertised(GARP) from all nodes, and so the users on the external network will get confused on where the actual "ACTIVE" port is" -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Apparmor is in 'complain' mode, the logs show the same entries but allowed rather than denied. Worth trying that change first, then installing -proposed if that makes no difference. This is a production site after all. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
I may be completely wrong, but one possible reason to cause 503 from haproxy is AppArmor. @Xav, what happens if you disable apparmor, i.e. aa-disable /usr/bin /neutron-lbaasv2-agent? As you see in an unrelated bug[1], the apparmor profile installed by neutron-gateway charm blocks lbaasv2 if it's set in enforced mode. [kernel log] Sep 21 19:46:44 HOSTNAME kernel: audit: type=1400 audit(1506023204.857:304): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0 [/var/log/neutron/neutron-lbaasv2-agent.log] 2017-09-21 19:44:44.850 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES In complain mode, if you see "ALLOWED" message for operation="connect" and info="Failed name lookup - disconnected path", but still see EACCES from lbaasv2 log. It may be hit by a bug in apparmor which blocks operation="connect" even in complain mode[2][3]. [1] https://bugs.launchpad.net/charm-neutron-gateway/+bug/1718768 [2] https://bugs.launchpad.net/apparmor/+bug/1624497 [3] https://bugs.launchpad.net/apparmor/+bug/1624300 ** Changed in: charm-neutron-gateway Status: Invalid => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
I need logs from neutron-gateway and neutron-api units, as well as the exact commands the end-user is using to create the loadbalancers. ** Changed in: neutron-lbaas (Ubuntu) Status: Confirmed => Incomplete ** Changed in: neutron-lbaas (Ubuntu) Assignee: (unassigned) => James Page (james-page) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
(just to be clear that's logs from /var/log/neutron on the neutron-* units). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
I'm not able to reproduce following the lbaas v2 docs: https://docs.openstack.org/mitaka/networking-guide/config-lbaas.html haproxy stats reports both backend server are in the configuration indicating that haproxy has been reloaded as the pool was updated. echo 'show stat;show table' | sudo socat stdio /var/lib/neutron/lbaas/v2/aa689d45-6853-44ba-8b46-a40da8663e9a/haproxy_stats.sock # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, ea3b4ef0-2cad-40b0-8051-1247c6c99bc0,FRONTEND,,,0,2,2000,4,308,848,0,0,0,OPEN,1,2,00,0,0,10,0,0,0,4,0,,0,1,4,,,0,0,0,0 66153c41-10d7-4f22-a63d-6ab276a0244a,57cc336c-cca9-4c8e-8fd1-680ca7379eff,0,0,0,2,,8,77,212,,0,,1,0,0,7,no check,1,1,0,,1,3,1,,8,,2,0,,20,0,0,0,0,0,00,0,29,,,0,0,0,0, 66153c41-10d7-4f22-a63d-6ab276a0244a,995b2445-ca83-4de4-93d0-fe106501265a,0,0,0,2,,8,231,636,,0,,3,0,0,5,no check,1,1,0,,1,3,2,,8,,2,0,,20,0,0,0,0,0,00,0,33,,,0,0,0,0, 66153c41-10d7-4f22-a63d-6ab276a0244a,BACKEND,0,0,0,2,200,4,308,848,0,0,,4,0,0,12,UP,2,2,0,,0,170,0,,1,3,0,,16,,1,0,,10,0,0,0,4,0,0,0,0,0,0,0,29,,,0,0,0,0, -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Thanks Paul There is a 11.0.3 update in pike-proposed - I can't see anything definitive but it would be good to test with that (both on neutron- gateway and neutron-api units) to see if that resolves the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Attempting to reproduce. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Please note that this affects customers as follows; - customer creates a lbaas, no backends come up - we restart the service, and backends come to life - customer creates another lbaas, the running one is fine but the new one has no backends - we restart... etc This means for every new load balancer, we need to restart the service to get it actually forwarding traffic. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
Due to customer impact, have subscribed field-high. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1770040] Re: lbaas load balancer does not forward traffic unless agent restarted
The customer cloud where we're seeing this is running pike on xenial from the Ubuntu Cloud Archive. Package version 2:11.0.2-0ubuntu1~cloud0 is what's installed on both neutron-gateway units. ** Changed in: neutron-lbaas (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1770040 Title: lbaas load balancer does not forward traffic unless agent restarted To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-gateway/+bug/1770040/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
