Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-04-11 Thread Fujimoto Satoshi 

Hi,
Sorry for delay.

When we shutdown the BGPSpeaker,
threads associated with each neighbors don't seem to be stopped.
I think this causes the problem.

I wrote the patch for this:

diff --git a/ryu/services/protocols/bgp/base.py 
b/ryu/services/protocols/bgp/base.py

index 8b69aa3..37d42a9 100644
--- a/ryu/services/protocols/bgp/base.py
+++ b/ryu/services/protocols/bgp/base.py
@@ -285,11 +285,12 @@ class Activity(object):
 """Stops all threads spawn by this activity.
 """
 for thread_name, thread in list(self._child_thread_map.items()):
-    if name is not None and thread_name is name:
-    LOG.debug('%s: Stopping child thread %s',
-  self.name, thread_name)
-    thread.kill()
-    self._child_thread_map.pop(thread_name, None)
+    if name is not None and thread_name != name:
+    continue
+    LOG.debug('%s: Stopping child thread %s',
+  self.name, thread_name)
+    thread.kill()
+    self._child_thread_map.pop(thread_name, None)

 def _close_asso_sockets(self):
 """Closes all the sockets linked to this activity.

Could you try this?

Thanks,
Fujimoto

On 2018年04月05日 18:58, Jens Harbott wrote:

2018-04-05 0:12 GMT+00:00 Fujimoto Satoshi :

Hi, Jens

Could you check whether you have reinstalled Ryu after applying the patch.

# At Ryu top directory:
$ python setup.py install

I did patch the installed version, but forgot to remove the matching
.pyc file, so your patch was indeed not used, sorry for that. The
situation when the BGP session is not established is still the same.
When the session is established at shutdown time, I'm now seeing
slightly different tracebacks:

Shutting down
('Peer down:', '10.42.0.4', 65001)
hub: uncaught exception: Traceback (most recent call last):
   File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
59, in _launch
 return func(*args, **kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/peer.py",
line 1305, in _connect_loop
 password=password)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 437, in _connect_tcp
 self._spawn(conn_name, conn_handler, sock)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 219, in _spawn
 self._validate_callable(callable_)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 215, in _validate_callable
 raise ActivityException(desc='Tried to spawn a child thread '
ActivityException: 100.1 - Tried to spawn a child thread before this
Activity was started.

Sent notification to ('10.42.0.12', '51898') >>
BGPNotification(data='',error_code=4,error_subcode=1,len=21,type=3)
hub: uncaught exception: Traceback (most recent call last):
   File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
80, in _launch
 return func(*args, **kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/utils/evtlet.py",
line 63, in __call__
 self._funct(*self._args, **self._kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 545, in _expired
 self.stop()
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 405, in stop
 Activity.stop(self)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 314, in stop
 raise ActivityException(desc='Cannot call stop when activity is '
ActivityException: 100.1 - Cannot call stop when activity is not
started or has been stopped already.


Yours,
Jens

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel

Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-04-05 Thread Jens Harbott 
2018-04-05 0:12 GMT+00:00 Fujimoto Satoshi :
> Hi, Jens
>
> Could you check whether you have reinstalled Ryu after applying the patch.
>
> # At Ryu top directory:
> $ python setup.py install

I did patch the installed version, but forgot to remove the matching
.pyc file, so your patch was indeed not used, sorry for that. The
situation when the BGP session is not established is still the same.
When the session is established at shutdown time, I'm now seeing
slightly different tracebacks:

Shutting down
('Peer down:', '10.42.0.4', 65001)
hub: uncaught exception: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
59, in _launch
return func(*args, **kwargs)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/peer.py",
line 1305, in _connect_loop
password=password)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 437, in _connect_tcp
self._spawn(conn_name, conn_handler, sock)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 219, in _spawn
self._validate_callable(callable_)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 215, in _validate_callable
raise ActivityException(desc='Tried to spawn a child thread '
ActivityException: 100.1 - Tried to spawn a child thread before this
Activity was started.

Sent notification to ('10.42.0.12', '51898') >>
BGPNotification(data='',error_code=4,error_subcode=1,len=21,type=3)
hub: uncaught exception: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
80, in _launch
return func(*args, **kwargs)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/utils/evtlet.py",
line 63, in __call__
self._funct(*self._args, **self._kwargs)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 545, in _expired
self.stop()
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 405, in stop
Activity.stop(self)
  File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 314, in stop
raise ActivityException(desc='Cannot call stop when activity is '
ActivityException: 100.1 - Cannot call stop when activity is not
started or has been stopped already.


Yours,
Jens

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel

Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-04-04 Thread Fujimoto Satoshi 

Hi, Jens

Could you check whether you have reinstalled Ryu after applying the patch.

# At Ryu top directory:
$ python setup.py install

Thanks,
Fujimoto

On 2018年04月04日 20:14, Jens Harbott wrote:

2018-04-03 5:22 GMT+00:00 Fujimoto Satoshi :

Hi,

I just sent the patch for your issue:

subject: [PATCH] bgp/peer: Don't stop BGPProtocol instance which already
stops

Currently, when a connection is lost, BGP Speaker tries to
stop BGPProtocol instance, even if it already stops.
This commit fixes this by checking the status of BGPProtocol
instance before trying to stop it.

Signed-off-by: Satoshi Fujimoto 
---
  ryu/services/protocols/bgp/peer.py | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ryu/services/protocols/bgp/peer.py
b/ryu/services/protocols/bgp/peer.py
index 70b486c..1e26bfd 100644
--- a/ryu/services/protocols/bgp/peer.py
+++ b/ryu/services/protocols/bgp/peer.py
@@ -2325,7 +2325,8 @@ class Peer(Source, Sink, NeighborConfListener,
Activity):
  )
  self.state.bgp_state = const.BGP_FSM_IDLE
  if self._protocol:
-self._protocol.stop()
+if self._protocol._started:
+self._protocol.stop()
  self._protocol = None
  # Create new collection for initial RT NLRIs
  self._init_rtc_nlri_path = []
--
2.7.4

Could you try this?

Hi Fujimoto,

thanks for looking at my issue. Sadly in my test I do not see any
difference for either of my issues with your patch applied.

Yours,
Jens

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel

Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-04-04 Thread Jens Harbott 
2018-04-03 5:22 GMT+00:00 Fujimoto Satoshi :
> Hi,
>
> I just sent the patch for your issue:
>
> subject: [PATCH] bgp/peer: Don't stop BGPProtocol instance which already
> stops
>
> Currently, when a connection is lost, BGP Speaker tries to
> stop BGPProtocol instance, even if it already stops.
> This commit fixes this by checking the status of BGPProtocol
> instance before trying to stop it.
>
> Signed-off-by: Satoshi Fujimoto 
> ---
>  ryu/services/protocols/bgp/peer.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/ryu/services/protocols/bgp/peer.py
> b/ryu/services/protocols/bgp/peer.py
> index 70b486c..1e26bfd 100644
> --- a/ryu/services/protocols/bgp/peer.py
> +++ b/ryu/services/protocols/bgp/peer.py
> @@ -2325,7 +2325,8 @@ class Peer(Source, Sink, NeighborConfListener,
> Activity):
>  )
>  self.state.bgp_state = const.BGP_FSM_IDLE
>  if self._protocol:
> -self._protocol.stop()
> +if self._protocol._started:
> +self._protocol.stop()
>  self._protocol = None
>  # Create new collection for initial RT NLRIs
>  self._init_rtc_nlri_path = []
> --
> 2.7.4
>
> Could you try this?

Hi Fujimoto,

thanks for looking at my issue. Sadly in my test I do not see any
difference for either of my issues with your patch applied.

Yours,
Jens

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel

Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-04-02 Thread Fujimoto Satoshi 

Hi,

I just sent the patch for your issue:

subject: [PATCH] bgp/peer: Don't stop BGPProtocol instance which already 
stops

Currently, when a connection is lost, BGP Speaker tries to
stop BGPProtocol instance, even if it already stops.
This commit fixes this by checking the status of BGPProtocol
instance before trying to stop it.

Signed-off-by: Satoshi Fujimoto
---
  ryu/services/protocols/bgp/peer.py | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ryu/services/protocols/bgp/peer.py 
b/ryu/services/protocols/bgp/peer.py
index 70b486c..1e26bfd 100644
--- a/ryu/services/protocols/bgp/peer.py
+++ b/ryu/services/protocols/bgp/peer.py
@@ -2325,7 +2325,8 @@ class Peer(Source, Sink, NeighborConfListener, Activity):
  )
  self.state.bgp_state = const.BGP_FSM_IDLE
  if self._protocol:
-self._protocol.stop()
+if self._protocol._started:
+self._protocol.stop()
  self._protocol = None
  # Create new collection for initial RT NLRIs
  self._init_rtc_nlri_path = []
-- 2.7.4

Could you try this?

Thanks,
Fujimoto

On 2018年03月23日 10:03, Fujimoto Satoshi wrote:

Hi,
Sorry for delay.

I reproduced the situation.
The cause may be that the disconnection is incomplete
when the speaker is being shutdown while the connection is existing.

Let me investigate this problem.
In the meantime, please delete all peers before shutting down.

Thanks,
Fujimoto

On 2018年03月14日 00:58, Jens Harbott wrote:

These issues were found while I was testing the
neutron-dynamic-routing service in OpenStack, see
https://bugs.launchpad.net/neutron/+bug/1750383 and
https://bugs.launchpad.net/neutron/+bug/1750129

But using a modified version of the sample program shown at
http://ryu.readthedocs.io/en/latest/library_bgp_speaker.html I can
reproduce them with only ryu:


---<--->---<--->---snip---<--->---<--->---snap---<--->---<--->---
import eventlet

# BGPSpeaker needs sockets patched
eventlet.monkey_patch()

# initialize a log handler
# this is not strictly necessary but useful if you get messages like:
#    No handlers could be found for logger "ryu.lib.hub"
import logging
import sys
log = logging.getLogger()
log.addHandler(logging.StreamHandler(sys.stderr))

from ryu.services.protocols.bgp.bgpspeaker import BGPSpeaker

def dump_remote_best_path_change(event):
 print ('the best path changed:', event.remote_as, event.prefix,\
 event.nexthop, event.is_withdraw)

def detect_peer_down(remote_ip, remote_as):
 print ('Peer down:', remote_ip, remote_as)

if __name__ == "__main__":
 speaker = BGPSpeaker(as_number=65001, router_id='10.0.0.1',
bgp_server_port=0,
best_path_change_handler=dump_remote_best_path_change,
  peer_down_handler=detect_peer_down)

 speaker.neighbor_add('10.42.0.4', 65001)
 print ("BGP started")
 count = 1
 while True:
 eventlet.sleep(10)
 prefix = '10.20.' + str(count) + '.0/24'
 print ("add a new prefix", prefix)
 speaker.prefix_add(prefix)
 count += 1
 if count == 3:
 break

 # eventlet.sleep(10)
 # print ("Remove neighbor")
 # speaker.neighbor_del('10.42.0.4')
 eventlet.sleep(10)
 print ("Shutting down")
 speaker.shutdown()
 eventlet.sleep(300)
---<--->---<--->---snip---<--->---<--->---snap---<--->---<--->---

The result differs depending on whether there is actually a BGP peer
listening at the remote end, leading to the scenarios described in the
two bugs above. When the BGP session is connected at the time of the
shutdown, these tracebacks are being shown repeatedly:

hub: uncaught exception: Traceback (most recent call last):
   File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
80, in _launch
 return func(*args, **kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/utils/evtlet.py",

line 63, in __call__
 self._funct(*self._args, **self._kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",

line 542, in _expired
 self.send_notification(code, subcode)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",

line 374, in send_notification
 self._send_with_lock(notification)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",

line 386, in _send_with_lock
 self.connection_lost('failed to write to socket')
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",

line 596, in connection_lost
 self._peer.connection_lost(reason)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/peer.py",

line 2328, in connection_lost
 self._protocol.stop()
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",

line 405, in stop
 Activity.stop(self)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/se

Re: [Ryu-devel] Problems when a BGP speaker is being shutdown

2018-03-22 Thread Fujimoto Satoshi 

Hi,
Sorry for delay.

I reproduced the situation.
The cause may be that the disconnection is incomplete
when the speaker is being shutdown while the connection is existing.

Let me investigate this problem.
In the meantime, please delete all peers before shutting down.

Thanks,
Fujimoto

On 2018年03月14日 00:58, Jens Harbott wrote:

These issues were found while I was testing the
neutron-dynamic-routing service in OpenStack, see
https://bugs.launchpad.net/neutron/+bug/1750383 and
https://bugs.launchpad.net/neutron/+bug/1750129

But using a modified version of the sample program shown at
http://ryu.readthedocs.io/en/latest/library_bgp_speaker.html I can
reproduce them with only ryu:


---<--->---<--->---snip---<--->---<--->---snap---<--->---<--->---
import eventlet

# BGPSpeaker needs sockets patched
eventlet.monkey_patch()

# initialize a log handler
# this is not strictly necessary but useful if you get messages like:
#No handlers could be found for logger "ryu.lib.hub"
import logging
import sys
log = logging.getLogger()
log.addHandler(logging.StreamHandler(sys.stderr))

from ryu.services.protocols.bgp.bgpspeaker import BGPSpeaker

def dump_remote_best_path_change(event):
 print ('the best path changed:', event.remote_as, event.prefix,\
 event.nexthop, event.is_withdraw)

def detect_peer_down(remote_ip, remote_as):
 print ('Peer down:', remote_ip, remote_as)

if __name__ == "__main__":
 speaker = BGPSpeaker(as_number=65001, router_id='10.0.0.1',
bgp_server_port=0,
  best_path_change_handler=dump_remote_best_path_change,
  peer_down_handler=detect_peer_down)

 speaker.neighbor_add('10.42.0.4', 65001)
 print ("BGP started")
 count = 1
 while True:
 eventlet.sleep(10)
 prefix = '10.20.' + str(count) + '.0/24'
 print ("add a new prefix", prefix)
 speaker.prefix_add(prefix)
 count += 1
 if count == 3:
 break

 # eventlet.sleep(10)
 # print ("Remove neighbor")
 # speaker.neighbor_del('10.42.0.4')
 eventlet.sleep(10)
 print ("Shutting down")
 speaker.shutdown()
 eventlet.sleep(300)
---<--->---<--->---snip---<--->---<--->---snap---<--->---<--->---

The result differs depending on whether there is actually a BGP peer
listening at the remote end, leading to the scenarios described in the
two bugs above. When the BGP session is connected at the time of the
shutdown, these tracebacks are being shown repeatedly:

hub: uncaught exception: Traceback (most recent call last):
   File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line
80, in _launch
 return func(*args, **kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/utils/evtlet.py",
line 63, in __call__
 self._funct(*self._args, **self._kwargs)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 542, in _expired
 self.send_notification(code, subcode)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 374, in send_notification
 self._send_with_lock(notification)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 386, in _send_with_lock
 self.connection_lost('failed to write to socket')
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 596, in connection_lost
 self._peer.connection_lost(reason)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/peer.py",
line 2328, in connection_lost
 self._protocol.stop()
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/speaker.py",
line 405, in stop
 Activity.stop(self)
   File 
"/usr/local/lib/python2.7/dist-packages/ryu/services/protocols/bgp/base.py",
line 314, in stop
 raise ActivityException(desc='Cannot call stop when activity is '
ActivityException: 100.1 - Cannot call stop when activity is not
started or has been stopped already.

When the BGP session is not connected, there is no error, but using
tcpdump one can see connection attempts continuing after the shutdown.

As a workaround, removing all neighbors works, if I do that (currently
commented in the sample code), both issues disappear. I do think
however that ryu should be doing this automatically. Fixes welcome. ;)

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Ryu-devel mailing list
Ryu-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ryu-devel



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_