[vpp-dev] Master branch l2bd test perf dop

[Zhang Yuwei]

Hi All,
 In our recent test, I found a performance drop in master branch. I 
execute the l2bd test case in a 2.5GHZ CPU and found almost 35% drop compared 
to 18.07 release. My test is set two NIC ports to a same bridge domain and send 
traffic to test the l2 forward performance. I found in the master branch, 
l2flood function consume much more CPU cycles than 18.07 which means any test 
use the l2flood function will also have a performance drop. Can anybody kindly 
help to check this issue? Thanks a lot.

Regards,
Yuwei

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10678): https://lists.fd.io/g/vpp-dev/message/10678
Mute This Topic: https://lists.fd.io/mt/26289209/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Unable to build router plugin

[carlito nueno]

Hi all,

I am trying to build the router-plugin:
make V=0 PLATFORM=vpp TAG=vpp_debug install-deb netlink-install router-install

I am using the Vagrantfile present in vpp repo and am pulling the
current master (as of September 26 2018). I am also pulling the
current master of vppsb.

But I am getting this error:

 Building router in /vpp/build-root/build-vpp_debug-native/router 
make[1]: Entering directory '/vpp/build-root/build-vpp_debug-native/router'
  CC   router/tap_inject.lo
  CC   router/tap_inject_netlink.lo
/vpp/build-data/../router/router/tap_inject_netlink.c: In function
‘add_del_neigh’:
/vpp/build-data/../router/router/tap_inject_netlink.c:140:9: error:
too many arguments to function ‘vnet_unset_ip6_ethernet_neighbor’
 vnet_unset_ip6_ethernet_neighbor (vm, sw_if_index,
 ^~~~
In file included from
/vpp/build-data/../router/router/tap_inject_netlink.c:19:0:
/vpp/build-root/install-vpp_debug-native/vpp/include/vnet/ip/ip6_neighbor.h:84:12:
note: declared here
 extern int vnet_unset_ip6_ethernet_neighbor (vlib_main_t * vm,
^~~~
Makefile:483: recipe for target 'router/tap_inject_netlink.lo' failed
make[1]: *** [router/tap_inject_netlink.lo] Error 1
make[1]: *** Waiting for unfinished jobs
make[1]: Leaving directory '/vpp/build-root/build-vpp_debug-native/router'
Makefile:691: recipe for target 'router-build' failed
make: *** [router-build] Error 2

Any advice?

Thanks
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10677): https://lists.fd.io/g/vpp-dev/message/10677
Mute This Topic: https://lists.fd.io/mt/26280661/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Odd problem in adding new session on vpp when its session table is full

[Andrew Yourtchenko]

You are welcome and thank you for the usage/testing ! :)

And thank you for the bug report too - I made a fix for the stateful
ICMP handling in the master, and added a few testcases to catch it in
the future:

https://gerrit.fd.io/r/#/c/15004/

--a

On 9/26/18, Rubina Bianchi  wrote:
> Thanks for your following up. 
> 
> From: Andrew  Yourtchenko 
> Sent: Wednesday, September 26, 2018 12:31 AM
> To: Rubina Bianchi
> Cc: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its
> session table is full
>
> Dear Rubina,
>
> On 9/25/18, Rubina Bianchi  wrote:
>> Dear Andrew,
>>
>> Actually it's not a RFC standard , or known attack, we extracted the
>> restriction using experiment on linux netfiltter and tried to implement
>> its
>> behavior.
>
> hmm okay. so my position about implementing it still stands then :-)
>
> Also, another point - if you send two echo requests at once with the
> same code (which is fine, because they are differentiated by ICMP ID
> and sequence number), then the second request will overwrite bihash
> values on activation (which is fine) and then the first reply that
> comes will erase those upon deactivation, so the second request will
> be dropped. This can make the troubleshooting harder in some cases
> since this is confusing.  One way to deal with that is to keep some
> sort of reference counter, or extend the bihash key, but again in the
> absence of real-world threat model this looks like shooting pigeons
> from ICBMs... (the only attack I know for this scenario is smurf and
> the existing implementation covers it fine)
>
>>
>> I checked your patch, throughput is fine. After about 2000 seconds my
>> throughput was still maximum and there was no drop-rate on trex. Also,
>> rx-miss was very little in this patch.
>
> Excellent, thanks a lot!
>
>> However, one thing is attracting for
>> me is that, in the same scenario in previous discussion, vpp served this
>> traffic with 3.6 million sessions, but in current patch it serves with
>> 3.9
>> million sessions. Is this related to your last changes?
>
> Yeah, it could be...  the new change  make the expiry somewhat longer:
> the session in any given timeout list is checked twice per timeout,
> rather than every X seconds (X being the shortest list timeout). So if
> e.g. a UDP session timeout is 100 seconds, and there is a packet on it
> 10 seconds before the session is checked, the next check will be 50
> seconds later, at which point the idle time on the session (60
> seconds) is still smaller than 100, so the session will be expunged
> only 50 seconds later, with the idle time of 110 seconds.
>
> Contrast this with the previous behavior where all the lists might be
> checked as often as 2 seconds - this would have resulted in a more
> precise timing out, but at an expense of a lot of CPU usage.
>
> But then all of the TCP sessions will transition into the tcp
> transient state upon closure, so the only impact in real world will be
> with the UDP connections, which should be relatively small.
>
> All that said: I am considering  to move the timeout infra to the
> tw_timers (which weren't available back when I was writing the code) -
> then in theory we can get both the precise expiry and the efficiency.
> But that is a larger change, and I am not sure of it yet, so I wanted
> to get this simpler solution in first.
>
> --a
>
>>
>> Thanks,
>> Sincerely
>> 
>> From: Andrew  Yourtchenko 
>> Sent: Monday, September 24, 2018 6:49 PM
>> To: Rubina Bianchi
>> Cc: vpp-dev@lists.fd.io
>> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its
>> session table is full
>>
>> Dear Rubina,
>>
>> On 9/24/18, Rubina Bianchi  wrote:
>>> Dear Andrew,
>>>
>>> It's hardcoded as it was simple and fast solution for our default
>>> scenario
>>> implementation.
>>> As you correctly mentioned in previous email one of the bug fixes was
>>> the
>>> restriction. Also another one is preventing reply packets pass through
>>> even
>>
>> ok. that's more a "featurette" - I deliberately did not attempt to
>> implement the "strict checking" because I had difficult time finding
>> the attack vector. (rather than maybe some kind of "compliance" checks
>> for the reasons of compliance ?)
>>
>>> if those packets are matched with an acl rule. In another word these
>>> reply
>>> packets are not belong to any echo request in reverse direction.
>>
>> hmm so you are sort-of making a "protocol inspection engine" there ? :-)
>>
>> Anyway, so far I haven't managed to recreate this condition - though
>> if you were running the 18.07 rather than 18.07.1 code, then the bug
>> related to hash acl manipulation on ACL changes might have caused that
>> effect... I will experiment a bit more, though.
>>
>> Also, remember the other thread we discussed a while ago about the
>> throughput getting lower over time.. I have made
>> https://gerrit.fd.io/r/#/c/14821/ which 

Re: [vpp-dev] PCI domain should be 32 bit

[Damjan Marion via Lists.Fd.Io]

-- 
Damjan

> On 26 Sep 2018, at 15:59, Stephen Hemminger  
> wrote:
> 
> On Tue, 25 Sep 2018 15:16:04 +0200
> Damjan Marion mailto:dmar...@me.com>> wrote:
> 
>>> On 25 Sep 2018, at 15:03, Stephen Hemminger  
>>> wrote:
>>> 
>>> I noticed that the PCI domain in VPP is limited to 16 bits.
>>> This is a problem on Azure/Hyper-V and other virtual environments.
>>> In these environments, the host will generate a 32 bit synthetic value
>>> when passing a PCI device through. The Linux kernel has 32 bit PCI 
>>> domains, and libpciaccess used by X now has 32 bit PCI domains.
>>> The PCI specification says domain is supposed to be 32 bit so this is
>>> not just Linux/Hyper-V issue.
>>> 
>>> Unfortunately, VPP has 16 bit PCI domain hard coded. This is not a show
>>> stopper now because the main use of PCI is for SR-IOV passthrough. In the
>>> passthrough case, the VF is taken over by other devices and the actual PCI
>>> value is not something VPP has to worry about directly. With DPDK today,
>>> this is done via TAP/FAILSAFE/VDEV_NETVSC devices. The issue would only be
>>> visible with direct device assignment.
>>> 
>>> Fixing it in not too hard (only in pci.h) but it would cause the device
>>> union to get larger and all the places as_u32 is used would have to become
>>> as_u64. Mostly mechanical, but always a possibility of errors.  
>> 
>> Feel free to submit patch which moves to u64, i don't expect major issues.
>> This is purely historic
>> 
> 
> I started on this, but the one sticking point is that there are places using
> pci address as a key into hash_get. The key is a uword which could be only
> 32 bits on some architectures. It shouldn't be a complete blocker but
> it does create potential issues.

That can be easily fixed my using *_mem variant of hash functions. 
hash_create_mem,
hash_get_mem, hash_set_mem There are many examples in the code...


> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#10674): https://lists.fd.io/g/vpp-dev/message/10674 
> 
> Mute This Topic: https://lists.fd.io/mt/26219976/675642 
> 
> Group Owner: vpp-dev+ow...@lists.fd.io 
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub 
>   [dmar...@me.com 
> ]
> -=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10675): https://lists.fd.io/g/vpp-dev/message/10675
Mute This Topic: https://lists.fd.io/mt/26219976/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] PCI domain should be 32 bit

[Stephen Hemminger]

On Tue, 25 Sep 2018 15:16:04 +0200
Damjan Marion  wrote:

> > On 25 Sep 2018, at 15:03, Stephen Hemminger  
> > wrote:
> > 
> > I noticed that the PCI domain in VPP is limited to 16 bits.
> > This is a problem on Azure/Hyper-V and other virtual environments.
> > In these environments, the host will generate a 32 bit synthetic value
> > when passing a PCI device through. The Linux kernel has 32 bit PCI 
> > domains, and libpciaccess used by X now has 32 bit PCI domains.
> > The PCI specification says domain is supposed to be 32 bit so this is
> > not just Linux/Hyper-V issue.
> > 
> > Unfortunately, VPP has 16 bit PCI domain hard coded. This is not a show
> > stopper now because the main use of PCI is for SR-IOV passthrough. In the
> > passthrough case, the VF is taken over by other devices and the actual PCI
> > value is not something VPP has to worry about directly. With DPDK today,
> > this is done via TAP/FAILSAFE/VDEV_NETVSC devices. The issue would only be
> > visible with direct device assignment.
> > 
> > Fixing it in not too hard (only in pci.h) but it would cause the device
> > union to get larger and all the places as_u32 is used would have to become
> > as_u64. Mostly mechanical, but always a possibility of errors.  
> 
> Feel free to submit patch which moves to u64, i don't expect major issues.
> This is purely historic
> 

I started on this, but the one sticking point is that there are places using
pci address as a key into hash_get. The key is a uword which could be only
32 bits on some architectures. It shouldn't be a complete blocker but
it does create potential issues.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10674): https://lists.fd.io/g/vpp-dev/message/10674
Mute This Topic: https://lists.fd.io/mt/26219976/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] one question about IP fragment

[Ole Troan]

Zhiyong,

> When I use vxlan in both VM and physical machine , packet 
> drop issue is come across, after setting VPP MTU = 1600,  this issue will 
> disappear.  As we all know, some of routes on the network doesn’t support 
> more than 1518 bytes packet.
> Should I try to use IP fragment in this case? Or any other better solution?  
> Does VPP IP fragment work now? If yes, Could you show me how to configure? 
> Thank you very much.

Fragmentation now works. Currently both for IPv4 and IPv6 packets are 
fragmented in ip{4,6}_rewrite. Note that only locally originated IPv6 packets 
can be fragmented, and only IPv4 packets with DF = 0.
Assuming the above two restrictions are adhered to, if the VXLAN node sent 
packets larger than the outgoing interface MTU they should be fragmented now.
There is no configuration for fragmentation. (Although I am thinking of adding 
a knob, with default disabled).

For reassembly, VPP has a short-coming, It can only reassemble as an input 
feature, meaning all fragments, even though not destined for the VPP instance 
itself are reassembled.
I think Juraj is working on a fix for that.

The next steps for tunnels, to help avoid fragmentation is to add some sort of 
tunnel path MTU discovery.

But in short, you are much better off with a well managed MTU, than you are 
with fragmentation.
See our draft in intarea for a list: 
https://tools.ietf.org/html/draft-ietf-intarea-frag-fragile-00

Cheers,
Ole-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10673): https://lists.fd.io/g/vpp-dev/message/10673
Mute This Topic: https://lists.fd.io/mt/26229382/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] one question about IP fragment

[Zhiyong Yang]

Hi  Ole, VPP maintainers,

When I use vxlan in both VM and physical machine , packet drop 
issue is come across, after setting VPP MTU = 1600,  this issue will disappear. 
 As we all know, some of routes on the network doesn't support more than 1518 
bytes packet.
Should I try to use IP fragment in this case? Or any other better solution?  
Does VPP IP fragment work now? If yes, Could you show me how to configure? 
Thank you very much.

Thanks
Zhiyong


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10672): https://lists.fd.io/g/vpp-dev/message/10672
Mute This Topic: https://lists.fd.io/mt/26229382/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] CSIT IPsec AES-GCM 128 tests failing

[Peter Mikus via Lists.Fd.Io]

Peter Mikus
Engineer - Software
Cisco Systems Limited

From: Nicolau, Radu 
Sent: Tuesday, September 25, 2018 8:38 PM
To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) 
Cc: Kinsella, Ray 
Subject: RE: CSIT IPsec AES-GCM 128 tests failing

Hi Peter,

I'm traveling this week so I won't be able to have a look, but, can you help 
with some more info?


-  Is it happening consistently?

Yes


-  What is the output of show crypto device mapping [verbose]

$ sudo vppctl show int
  Name   IdxState  MTU (L3/IP4/IP6/MPLS) 
Counter  Count
FortyGigabitEthernet88/0/01  up  9200/0/0/0
FortyGigabitEthernet88/0/12  up  9200/0/0/0 rx packets  
   173678340
rx bytes
 10420700400
drops   
   173678340
ip4 
   173678340
rx-miss 
 737
ipsec03  up   0/0/0/0   tx-error
   173678340
local00 down  0/0/0/0

$ sudo vppctl show err
   CountNode  Reason
 167158808  ipsec0-output interface is down

It is not clear to me as why ipsec0 is up but error says interface down.

$ sudo vppctl show crypto
show: unknown input `crypto'

$ sudo vppctl show ipsec
tunnel interfaces
  ipsec0 seq
   seq 0 seq-hi 0 esn 0 anti-replay 0 udp-encap 0
   local-spi 1 local-ip 172.168.1.1
   local-crypto aes-gcm-128 70394d61596b657635793046705443744c45786c
   local-integrity none
   last-seq 0 last-seq-hi 0 esn 0 anti-replay 0 window 

   remote-spi 2 remote-ip 172.168.1.2
   remote-crypto aes-gcm-128 70394d61596b657635793046705443744c45786c
   remote-integrity none



-  What is the vpp version/commit id causing the issue?

Currently using vpp v18.10-rc0~465-gb7020d6~b5194 but I did not bisect, For 
sure it is broken for several weeks now.


Regards,
Radu

From: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) 
[mailto:pmi...@cisco.com]
Sent: Tuesday, September 25, 2018 3:32 PM
To: vpp-dev@lists.fd.io; Nicolau, Radu 
mailto:radu.nico...@intel.com>>
Cc: csit-...@lists.fd.io
Subject: CSIT IPsec AES-GCM 128 tests failing

Hello devs,

We are observing IPSec failing tests in CSIT-perf, specially and only 
combination with aes-gcm (both interface AND tunnel mode, Integ Alg AES GCM 
128).
Note: Integ Alg SHA1 96, is working.

Can you please help decrypt the error message below?

Thank you.


VAT error message (on both DUTs)
-
sw_interface_set_flags error: Unspecified Error


-Note: This error is happening when trying to get interface UP.

DUT1 config
---

sw_interface_set_flags sw_if_index 2 admin-up link-up
sw_interface_set_flags sw_if_index 1 admin-up link-up
sw_interface_dump
hw_interface_set_mtu sw_if_index 2 mtu 9200
hw_interface_set_mtu sw_if_index 1 mtu 9200
sw_interface_dump
sw_interface_add_del_address sw_if_index 2 192.168.10.1/24
sw_interface_add_del_address sw_if_index 1 172.168.1.1/24
ip_neighbor_add_del sw_if_index 2 dst 192.168.10.2 mac 68:05:ca:35:79:1c
ip_neighbor_add_del sw_if_index 1 dst 172.168.1.2 mac 68:05:ca:35:76:b1
ip_add_del_route 10.0.0.0/8 via 192.168.10.2  sw_if_index 2 resolve-attempts 10 
count 1
ipsec_tunnel_if_add_del local_spi 1 remote_spi 2 crypto_alg aes-gcm-128 
local_crypto_key 4e4d444178336b4958744a374365356235643545 remote_crypto_key 
4e4d444178336b4958744a374365356235643545  local_ip 172.168.1.1 remote_ip 
172.168.1.2
ip_add_del_route 20.0.0.0/32 via 172.168.1.2 ipsec0
exec set interface unnumbered ipsec0 use FortyGigabitEthernet88/0/0
sw_interface_set_flags ipsec0 admin-up


DUT2 config
---
sw_interface_set_flags sw_if_index 2 admin-up link-up
sw_interface_set_flags sw_if_index 1 admin-up link-up
sw_interface_dump
hw_interface_set_mtu sw_if_index 2 mtu 9200
hw_interface_set_mtu sw_if_index 1 mtu 9200
sw_interface_dump
sw_interface_add_del_address sw_if_index 2 172.168.1.2/24
sw_interface_add_del_address sw_if_index 1 192.168.20.1/24
ip_neighbor_add_del sw_if_index 1 dst 192.168.20.2 mac 68:05:ca:35:79:19
ip_neighbor_add_del sw_if_index 2 dst 172.168.1.1 mac 68:05:ca:37:25:18
ip_add_del_route 20.0.0.0/8 via 192.168.20.2  sw_if_index 1 resolve-attempts 10 
count 1
ipsec_tunnel_if_add_del local_spi 2 remote_spi 1 crypto_alg aes-gcm-128 
local_crypto_key 4e4d444178336b4958744a374365356235643545 remote_crypto_key 
4e4d444178336b4958744a374365356235643545  local_ip 172.168.1.2 remote_ip 
172.168.1.1
ip_add_del_route 10.0.0.0/32 via 

Re: [vpp-dev] continuous decline in throughput with acl

[emma sdi]

I test latest master, throughput slowdown is fixed, Although I see growing
memory usage.
I will post a new thread and describe how to reproduce the situation.

Regards

On Tue, Sep 25, 2018 at 3:14 PM emma sdi  wrote:

> Excellent, I'm going out of office. I will rerun my test with latest
> version of master tomorrow.
>
>
> On Tue, Sep 25, 2018 at 3:03 PM Andrew  Yourtchenko 
> wrote:
>
>> Hmm this output does look exactly like what I had before commit ed435504
>> - the IPv6 table bucket area gets corrupted by the overruns from IPv4 table
>> arena.
>>
>> And then I spotted the leak which was happening during the rehash which I
>> fixed in 11521387.
>>
>> Your (very few) freelists look like you don’t have that fix either -
>> normally you would see something
>> like in http://paste.ubuntu.com/p/y43MsrdHSr/ - notice how many short
>> elements are in the freelists.
>>
>> And given your tests confirm the session cleaner optimization work as I
>> intended, I have it now in master.
>>
>> So you can just pull your fresh tree from master and recheck :)
>>
>> --a
>>
>> On 25 Sep 2018, at 13:10, khers  wrote:
>>
>> I checked out from gerrrit, I think  it's using latest master. ( but i
>> make another version to make sure )
>> let me explain how I produce this situation,
>>
>> while true  :)
>> 1- run trex
>> ./t-rex-64 --cfg cfg/trex_config.yaml  -f cap2/sfr.yaml -m 7 -c 2
>> 2-stop trex after ~120 seconds
>> 3- wait until all session deleted from 'sh acl-plugin session'
>>
>> As you see I waited until all session is deleted so all bucket must be
>> completely free,
>> I send you part of  'show acl-plugin sessions verbose 1'  in this link
>> .
>>
>>
>>
>> On Tue, Sep 25, 2018 at 1:52 PM Andrew  Yourtchenko 
>> wrote:
>>
>>> Are you using latest master ? I fixed a couple of issues in bihash last
>>> week related to memory usage... if it’s the latest master, the output of
>>> used vs available looks weird... - so please let me know...
>>>
>>> As for the “general” growth - basically what happens is bihash doubles
>>> each bucket size whenever there is a collision on insert, and then converts
>>> the bucket into linear lookup whenever there is still a collision after
>>> that growth.
>>>
>>> Then the only time the shrinkage/reset is happening is when the bucket
>>> is completely free - which with long living sessions with overlapping
>>> lifetimes might mean never.
>>>
>>> So one approach to this is to increase the number of buckets. Then they
>>> will be smaller and have higher probability of being freed.
>>>
>>> This is assuming there is nothing else “funny” going on. You can do
>>> “show acl-plugin sessions verbose 1” via vppctl (It will take forever to
>>> complete and needs pager disabled since it dumps the entire bihash) to
>>> inspect the way the buckets are filled...
>>>
>>> --a
>>>
>>> On 25 Sep 2018, at 12:12, khers  wrote:
>>>
>>> It's amazing!!!
>>>
>>> IPv4 Session lookup hash table:
>>> Hash table ACL plugin FA IPv4 session bihash
>>> 968086 active elements 65536 active buckets
>>> 13 free lists
>>>[len 16] 1 free elts
>>>[len 32] 1 free elts
>>>[len 256] 10669 free elts
>>>[len 512] 36768 free elts
>>>[len 1024] 4110 free elts
>>>[len 2048] 156 free elts
>>>[len 4096] 4 free elts
>>> 844 linear search buckets
>>> arena: base 7fe91232, next 2680ca780
>>> *   used 10335594368 b (9856 Mbytes) of 100 b (9536
>>> Mbytes)*
>>>
>>>
>>> On Tue, Sep 25, 2018 at 1:39 PM khers  wrote:
>>>
 Yes, that's right. I think is completely another issue from the patch
 you sent

 On Tue, Sep 25, 2018 at 1:35 PM Andrew  Yourtchenko <
 ayour...@gmail.com> wrote:

> Excellent, thanks!
>
> Memory usage - you mean in bihash arena ?
>
> --a
>
> On 25 Sep 2018, at 11:38, khers  wrote:
>
> Throughput and session add/del is stable as rock. The only danger i
> see is growing memory usage.
> look at this 
>
>
> On Tue, Sep 25, 2018 at 11:31 AM khers  wrote:
>
>> Of course, I test your patch, there is no slowdown with my scenario.
>> I need more time to test other
>> scenarios and make sure.
>>
>>
>> On Mon, Sep 24, 2018 at 3:11 PM Andrew  Yourtchenko <
>> ayour...@gmail.com> wrote:
>>
>>> Cool. Then it is probably indeed the session requeues that are not
>>> yet efficient... I have been looking at optimizing that.
>>>
>>> I have a draft in the works which should have less session requeues
>>> - I have just added you to it, could you give it a shot and see if it 
>>> makes
>>> things better ?
>>>
>>> --a
>>>
>>> On 24 Sep 2018, at 12:55, khers  wrote:
>>>
>>> yes, I confirm
>>>
>>> On Mon, Sep 24, 2018 at 2:08 PM Andrew  Yourtchenko <
>>> ayour...@gmail.com> wrote:
>>>
 

Re: [vpp-dev] unformat_check_input crashed during multiarch function initialization

[Lijian Zhang]

Thank you Damjan for your confirmation.
I'll restore to previous code.
Thanks again.

From: Damjan Marion (damarion) 
Sent: Wednesday, September 26, 2018 5:52 PM
To: Lijian Zhang (Arm Technology China) 
Cc: d...@barachs.net; vpp-dev@lists.fd.io
Subject: Re: unformat_check_input crashed during multiarch function 
initialization


Ah, i forgot that you are executing this from constructor before clib_mem_init 
happens.
So it is expected not to work. You will need to stay on your old code, sorry 
for wasting your time.. :(

--
Damjan


On 26 Sep 2018, at 11:28, Lijian Zhang (Arm Technology China) 
mailto:lijian.zh...@arm.com>> wrote:

Hi Damjan,
I tried to rewrite the function reading CPU info from /proc/cpuinfo, using VPP 
style functions, for multi-arch dynamic function selection.
But function unformat_check_input failed and crashed.
I pasted the /proc/cpuinfo reading function and gdb process as below.
Could you suggest on this issue?

static inline int
clib_cpu_supports_implements (const u32 implid, const u32 partid)
{
   u32 implementer = -1, part = -1;
   int fd;
   unformat_input_t input;

   fd = open ("/proc/cpuinfo", 0);
   if (fd < 0)
 return 0;

   unformat_init_clib_file (, fd);

   while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
 {
   if (unformat (, "CPU implementer : %x", ))
 ;
   else if (unformat (, "CPU part: %x", ))
 ;
   else
 unformat_skip_line ();
   if ((-1 != implementer) && (-1 != part))
 break;
 }

   unformat_free ();

   close (fd);

   printf("++ %x %x\n", implementer, part);
   if ((implid == implementer) && (partid == part))
 return 1;
   else
 return 0;
}


lijian@net-arm-thunderx2:~/tasks/vpp.upstream/vpp$
 sudo gdb ./build-root/build-vpp_debug-native/vpp/bin/vpp -c 
/home/lijian/startup.conf
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build-root/build-vpp_debug-native/vpp/bin/vpp...done.
"/home/lijian/startup.conf" is not a core dump: File format not recognized
(gdb) b clib_cpu_supports_implements
Function "clib_cpu_supports_implements" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (clib_cpu_supports_implements) pending.
(gdb) run -c /home/lijian/startup.conf
Starting program: 
/home/lijian/tasks/vpp.upstream/vpp/build-root/build-vpp_debug-native/vpp/bin/vpp
 -c /home/lijian/startup.conf
warning: Unable to determine the number of hardware watchpoints available.
warning: Unable to determine the number of hardware breakpoints available.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
28  {
(gdb) bt
#0  clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
#1  0xbeac4294 in clib_cpu_march_priority_qdf24xx () at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:248
#2  0xbeac5c04 in svm_fifo_enqueue_nowait_march_constructor_qdf24xx () 
at /home/lijian/tasks/vpp.upstream/vpp/src/svm/svm_fifo.c:457
#3  0xbf6dfb60 in call_init (l=, argc=argc@entry=3, 
argv=argv@entry=0xf5f8, env=env@entry=0xf618) at dl-init.c:72
#4  0xbf6dfca8 in call_init (env=0xf618, argv=0xf5f8, 
argc=3, l=) at dl-init.c:30
#5  _dl_init (main_map=0xbf700190, argc=3, argv=0xf5f8, 
env=0xf618) at dl-init.c:120
#6  0xbf6d2d44 in _dl_start_user () from /lib/ld-linux-aarch64.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) n
29u32 implementer = -1, part = -1;
(gdb)
33fd = open ("/proc/cpuinfo", 0);
(gdb)
34if (fd < 0)
(gdb)
37unformat_init_clib_file (, fd);
(gdb)
39while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
(gdb) p input
$1 = {buffer = 0x0, index = 0, buffer_marks = 0x0, fill_buffer = 0xbea41188 
, fill_buffer_arg = 0x3}
(gdb) s
unformat_check_input (i=0xf4f0) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/format.h:171
171 {
(gdb) n
175   if 

Re: [vpp-dev] unformat_check_input crashed during multiarch function initialization

[Damjan Marion via Lists.Fd.Io]

Ah, i forgot that you are executing this from constructor before clib_mem_init 
happens.
So it is expected not to work. You will need to stay on your old code, sorry 
for wasting your time.. :(

--
Damjan

On 26 Sep 2018, at 11:28, Lijian Zhang (Arm Technology China) 
mailto:lijian.zh...@arm.com>> wrote:

Hi Damjan,
I tried to rewrite the function reading CPU info from /proc/cpuinfo, using VPP 
style functions, for multi-arch dynamic function selection.
But function unformat_check_input failed and crashed.
I pasted the /proc/cpuinfo reading function and gdb process as below.
Could you suggest on this issue?

static inline int
clib_cpu_supports_implements (const u32 implid, const u32 partid)
{
   u32 implementer = -1, part = -1;
   int fd;
   unformat_input_t input;

   fd = open ("/proc/cpuinfo", 0);
   if (fd < 0)
 return 0;

   unformat_init_clib_file (, fd);

   while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
 {
   if (unformat (, "CPU implementer : %x", ))
 ;
   else if (unformat (, "CPU part: %x", ))
 ;
   else
 unformat_skip_line ();
   if ((-1 != implementer) && (-1 != part))
 break;
 }

   unformat_free ();

   close (fd);

   printf("++ %x %x\n", implementer, part);
   if ((implid == implementer) && (partid == part))
 return 1;
   else
 return 0;
}


lijian@net-arm-thunderx2:~/tasks/vpp.upstream/vpp$ sudo gdb 
./build-root/build-vpp_debug-native/vpp/bin/vpp -c /home/lijian/startup.conf
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build-root/build-vpp_debug-native/vpp/bin/vpp...done.
"/home/lijian/startup.conf" is not a core dump: File format not recognized
(gdb) b clib_cpu_supports_implements
Function "clib_cpu_supports_implements" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (clib_cpu_supports_implements) pending.
(gdb) run -c /home/lijian/startup.conf
Starting program: 
/home/lijian/tasks/vpp.upstream/vpp/build-root/build-vpp_debug-native/vpp/bin/vpp
 -c /home/lijian/startup.conf
warning: Unable to determine the number of hardware watchpoints available.
warning: Unable to determine the number of hardware breakpoints available.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
28  {
(gdb) bt
#0  clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
#1  0xbeac4294 in clib_cpu_march_priority_qdf24xx () at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:248
#2  0xbeac5c04 in svm_fifo_enqueue_nowait_march_constructor_qdf24xx () 
at /home/lijian/tasks/vpp.upstream/vpp/src/svm/svm_fifo.c:457
#3  0xbf6dfb60 in call_init (l=, argc=argc@entry=3, 
argv=argv@entry=0xf5f8, env=env@entry=0xf618) at dl-init.c:72
#4  0xbf6dfca8 in call_init (env=0xf618, argv=0xf5f8, 
argc=3, l=) at dl-init.c:30
#5  _dl_init (main_map=0xbf700190, argc=3, argv=0xf5f8, 
env=0xf618) at dl-init.c:120
#6  0xbf6d2d44 in _dl_start_user () from /lib/ld-linux-aarch64.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) n
29u32 implementer = -1, part = -1;
(gdb)
33fd = open ("/proc/cpuinfo", 0);
(gdb)
34if (fd < 0)
(gdb)
37unformat_init_clib_file (, fd);
(gdb)
39while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
(gdb) p input
$1 = {buffer = 0x0, index = 0, buffer_marks = 0x0, fill_buffer = 0xbea41188 
, fill_buffer_arg = 0x3}
(gdb) s
unformat_check_input (i=0xf4f0) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/format.h:171
171 {
(gdb) n
175   if (i->index >= vec_len (i->buffer) && i->index != 
UNFORMAT_END_OF_INPUT)
(gdb)
179 _unformat_fill_input (i);
(gdb) s
_unformat_fill_input (i=0xf4f0) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/unformat.c:43
43  {
(gdb) n
46if (i->index == UNFORMAT_END_OF_INPUT)
(gdb)
49first_mark = l = vec_len (i->buffer);
(gdb)
50if (vec_len 

[vpp-dev] unformat_check_input crashed during multiarch function initialization

[Lijian Zhang]

Hi Damjan,
I tried to rewrite the function reading CPU info from /proc/cpuinfo, using VPP 
style functions, for multi-arch dynamic function selection.
But function unformat_check_input failed and crashed.
I pasted the /proc/cpuinfo reading function and gdb process as below.
Could you suggest on this issue?

static inline int
clib_cpu_supports_implements (const u32 implid, const u32 partid)
{
   u32 implementer = -1, part = -1;
   int fd;
   unformat_input_t input;

   fd = open ("/proc/cpuinfo", 0);
   if (fd < 0)
 return 0;

   unformat_init_clib_file (, fd);

   while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
 {
   if (unformat (, "CPU implementer : %x", ))
 ;
   else if (unformat (, "CPU part: %x", ))
 ;
   else
 unformat_skip_line ();
   if ((-1 != implementer) && (-1 != part))
 break;
 }

   unformat_free ();

   close (fd);

   printf("++ %x %x\n", implementer, part);
   if ((implid == implementer) && (partid == part))
 return 1;
   else
 return 0;
}


lijian@net-arm-thunderx2:~/tasks/vpp.upstream/vpp$ sudo gdb 
./build-root/build-vpp_debug-native/vpp/bin/vpp -c /home/lijian/startup.conf
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build-root/build-vpp_debug-native/vpp/bin/vpp...done.
"/home/lijian/startup.conf" is not a core dump: File format not recognized
(gdb) b clib_cpu_supports_implements
Function "clib_cpu_supports_implements" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (clib_cpu_supports_implements) pending.
(gdb) run -c /home/lijian/startup.conf
Starting program: 
/home/lijian/tasks/vpp.upstream/vpp/build-root/build-vpp_debug-native/vpp/bin/vpp
 -c /home/lijian/startup.conf
warning: Unable to determine the number of hardware watchpoints available.
warning: Unable to determine the number of hardware breakpoints available.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Breakpoint 1, clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
28  {
(gdb) bt
#0  clib_cpu_supports_implements (implid=81, partid=3072) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:28
#1  0xbeac4294 in clib_cpu_march_priority_qdf24xx () at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/cpu.h:248
#2  0xbeac5c04 in svm_fifo_enqueue_nowait_march_constructor_qdf24xx () 
at /home/lijian/tasks/vpp.upstream/vpp/src/svm/svm_fifo.c:457
#3  0xbf6dfb60 in call_init (l=, argc=argc@entry=3, 
argv=argv@entry=0xf5f8, env=env@entry=0xf618) at dl-init.c:72
#4  0xbf6dfca8 in call_init (env=0xf618, argv=0xf5f8, 
argc=3, l=) at dl-init.c:30
#5  _dl_init (main_map=0xbf700190, argc=3, argv=0xf5f8, 
env=0xf618) at dl-init.c:120
#6  0xbf6d2d44 in _dl_start_user () from /lib/ld-linux-aarch64.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) n
29u32 implementer = -1, part = -1;
(gdb)
33fd = open ("/proc/cpuinfo", 0);
(gdb)
34if (fd < 0)
(gdb)
37unformat_init_clib_file (, fd);
(gdb)
39while (unformat_check_input () != UNFORMAT_END_OF_INPUT)
(gdb) p input
$1 = {buffer = 0x0, index = 0, buffer_marks = 0x0, fill_buffer = 0xbea41188 
, fill_buffer_arg = 0x3}
(gdb) s
unformat_check_input (i=0xf4f0) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/format.h:171
171 {
(gdb) n
175   if (i->index >= vec_len (i->buffer) && i->index != 
UNFORMAT_END_OF_INPUT)
(gdb)
179 _unformat_fill_input (i);
(gdb) s
_unformat_fill_input (i=0xf4f0) at 
/home/lijian/tasks/vpp.upstream/vpp/src/vppinfra/unformat.c:43
43  {
(gdb) n
46if (i->index == UNFORMAT_END_OF_INPUT)
(gdb)
49first_mark = l = vec_len (i->buffer);
(gdb)
50if (vec_len (i->buffer_marks) > 0)
(gdb)
54if (first_mark > 0)
(gdb)
57i->index = vec_len (i->buffer);
(gdb)
58for (l = 0; l < vec_len (i->buffer_marks); l++)
(gdb)
62if (i->fill_buffer)
(gdb)
63  i->index = i->fill_buffer (i);
(gdb) s
clib_file_fill_buffer (input=0xf4f0) at 

Re: [vpp-dev] IPsec+ikev2 #vpp

[Ray Kinsella]

Hi Ken,

I believe it is IPEV2 only at the moment.

Please see

https://software.intel.com/en-us/articles/get-started-with-ipsec-acceleration-in-the-fdio-vpp-project

Ray K



On 26/09/2018 03:33, yang@itibia.com wrote:

Hi all,

   I'm trying to setup ipsec tunnel in VPP and my gateway, but it always 
fails. After check I found my gateway is used ikev1 but VPP used ikev2, 
so if ikev1 and ikev2 can interactive support ? How to configure ? If 
can not interactive support, I want to explant the openswan ikev1 to 
VPP, is it be come true ?


Thanks,
Best regards.

Ken


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10663): https://lists.fd.io/g/vpp-dev/message/10663
Mute This Topic: https://lists.fd.io/mt/26226994/675355
Mute #vpp: https://lists.fd.io/mk?hashtag=vpp=1480675
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [m...@ashroe.eu]
-=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#10666): https://lists.fd.io/g/vpp-dev/message/10666
Mute This Topic: https://lists.fd.io/mt/26226994/21656
Mute #vpp: https://lists.fd.io/mk?hashtag=vpp=1480452
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Odd problem in adding new session on vpp when its session table is full

[Rubina Bianchi]

Thanks for your following up. 

From: Andrew  Yourtchenko 
Sent: Wednesday, September 26, 2018 12:31 AM
To: Rubina Bianchi
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its 
session table is full

Dear Rubina,

On 9/25/18, Rubina Bianchi  wrote:
> Dear Andrew,
>
> Actually it's not a RFC standard , or known attack, we extracted the
> restriction using experiment on linux netfiltter and tried to implement its
> behavior.

hmm okay. so my position about implementing it still stands then :-)

Also, another point - if you send two echo requests at once with the
same code (which is fine, because they are differentiated by ICMP ID
and sequence number), then the second request will overwrite bihash
values on activation (which is fine) and then the first reply that
comes will erase those upon deactivation, so the second request will
be dropped. This can make the troubleshooting harder in some cases
since this is confusing.  One way to deal with that is to keep some
sort of reference counter, or extend the bihash key, but again in the
absence of real-world threat model this looks like shooting pigeons
from ICBMs... (the only attack I know for this scenario is smurf and
the existing implementation covers it fine)

>
> I checked your patch, throughput is fine. After about 2000 seconds my
> throughput was still maximum and there was no drop-rate on trex. Also,
> rx-miss was very little in this patch.

Excellent, thanks a lot!

> However, one thing is attracting for
> me is that, in the same scenario in previous discussion, vpp served this
> traffic with 3.6 million sessions, but in current patch it serves with 3.9
> million sessions. Is this related to your last changes?

Yeah, it could be...  the new change  make the expiry somewhat longer:
the session in any given timeout list is checked twice per timeout,
rather than every X seconds (X being the shortest list timeout). So if
e.g. a UDP session timeout is 100 seconds, and there is a packet on it
10 seconds before the session is checked, the next check will be 50
seconds later, at which point the idle time on the session (60
seconds) is still smaller than 100, so the session will be expunged
only 50 seconds later, with the idle time of 110 seconds.

Contrast this with the previous behavior where all the lists might be
checked as often as 2 seconds - this would have resulted in a more
precise timing out, but at an expense of a lot of CPU usage.

But then all of the TCP sessions will transition into the tcp
transient state upon closure, so the only impact in real world will be
with the UDP connections, which should be relatively small.

All that said: I am considering  to move the timeout infra to the
tw_timers (which weren't available back when I was writing the code) -
then in theory we can get both the precise expiry and the efficiency.
But that is a larger change, and I am not sure of it yet, so I wanted
to get this simpler solution in first.

--a

>
> Thanks,
> Sincerely
> 
> From: Andrew  Yourtchenko 
> Sent: Monday, September 24, 2018 6:49 PM
> To: Rubina Bianchi
> Cc: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] Odd problem in adding new session on vpp when its
> session table is full
>
> Dear Rubina,
>
> On 9/24/18, Rubina Bianchi  wrote:
>> Dear Andrew,
>>
>> It's hardcoded as it was simple and fast solution for our default
>> scenario
>> implementation.
>> As you correctly mentioned in previous email one of the bug fixes was the
>> restriction. Also another one is preventing reply packets pass through
>> even
>
> ok. that's more a "featurette" - I deliberately did not attempt to
> implement the "strict checking" because I had difficult time finding
> the attack vector. (rather than maybe some kind of "compliance" checks
> for the reasons of compliance ?)
>
>> if those packets are matched with an acl rule. In another word these
>> reply
>> packets are not belong to any echo request in reverse direction.
>
> hmm so you are sort-of making a "protocol inspection engine" there ? :-)
>
> Anyway, so far I haven't managed to recreate this condition - though
> if you were running the 18.07 rather than 18.07.1 code, then the bug
> related to hash acl manipulation on ACL changes might have caused that
> effect... I will experiment a bit more, though.
>
> Also, remember the other thread we discussed a while ago about the
> throughput getting lower over time.. I have made
> https://gerrit.fd.io/r/#/c/14821/ which should significantly reduce
> the amount of session list shuffling work in normal case scenarios.
> Before I commit it, could you give it a shot to see if it indeed
> behaves as I would expect it to behave ?  Thanks a lot!
>
> --a
>
>> Thanks,
>> Sincerely
>> 
>> From: Andrew  Yourtchenko 
>> Sent: Tuesday, September 18, 2018 4:06 PM
>> To: Rubina Bianchi
>> Cc: vpp-dev@lists.fd.io
>> Subject: Re: [vpp-dev] Odd