Re: [vpp-dev] VPP - DPDK - No ARP learning on VPP and no ARP reply sent.

[Mrityunjay Kumar]

This issue, we have seen on centos as well. Just for work around,

ip link set promisc on eth0   #[eth0 ---  PF interface]
ip link set eth00 vf 0 spoofchk off   #[give your vf id ]
ip link set eth0 vf 0 trust on,

, But you will see other packet on vpp VF interface as well.

Can you try with IPv6, you will see more surprises. :)
//MJ

*Regards*,
Mrityunjay Kumar.
Mobile: +91 - 9731528504



On Fri, May 15, 2020 at 7:55 AM Laurent Dumont 
wrote:

> Hi!
>
> I was doing some initial experimentation with VPP/SRIOV/DPDK and I just
> wanted to see if some of the things I was experiencing we're expected. I
> was looking at understanding the baseline behavior for something like VPP +
> DPDK.
>
> I have a small test POC with the following topology.
>
> VM Ubuntu 18.04( with the "linux-image-extra-virtual" package)
> --->  (VPP + DPDK + IP address on the VPP interface) ---> VF ---> PF
> (compute with SRIOV on X710 cards) ---> TOR interface - untagged.
>
> What I am seeing :
>
>1. VPP is telling me I have both RX and TX packets.
>2. That said, no ARP entry for the TOR is seen on VPP
>3. The interface on the TOR is also not seeing any ARP entry for the
>VPP interface. The TOR is also indicating that RX and TX packets are
>present.
>
> Are there any inherent limitations to the ARP with VPP/DPDK?
>
> One weird thing is that trying to show the ARP from VPP doesn't seem to be
> a valid command :
> vpp# sh ip arp
> show ip: unknown input `arp'
>
> Here are some more debugging output.
>
> vpp# show version
> vpp v20.01-release built by root on 980ae64453f3 at 2020-01-29T22:13:47
>
> vpp# show int address
> VirtualFunctionEthernet0/5/0 (up):
>   L3 1.2.3.4/31
>
>
> vpp# show hardware-interfaces
>   NameIdx   Link  Hardware
> VirtualFunctionEthernet0/5/0   1 up   VirtualFunctionEthernet0/5/0
>   Link speed: 10 Gbps
>   Ethernet address fa:16:3e:92:30:f1
>   Intel X710/XL710 Family VF
> carrier up full duplex mtu 9206
> flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum
> rx-ip4-cksum
> Devargs:
> rx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
> tx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
> pci: device 8086:154c subsystem 103c: address :00:05.00 numa 0
> max rx packet len: 9728
> promiscuous: unicast off all-multicast on
> vlan offload: strip off filter off qinq off
> rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum
> qinq-strip
>outer-ipv4-cksum vlan-filter jumbo-frame scatter
> rx offload active: ipv4-cksum jumbo-frame scatter
> tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum
> sctp-cksum
>tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso
>gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs
> tx offload active: udp-cksum tcp-cksum multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other
> ipv6-frag
>ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload
> rss active:none
> tx burst function: i40e_xmit_pkts
> rx burst function: i40e_recv_scattered_pkts_vec_avx2
>
> tx frames ok  36
> tx bytes ok 1512
> rx frames ok  26
> rx bytes ok 3014
> extended stats:
>   rx good packets 26
>   tx good packets 36
>   rx good bytes 3014
>   tx good bytes 1512
>   rx bytes  3014
>   rx broadcast packets26
>   tx bytes  1512
>   tx broadcast packets36
> local0 0down  local0
>   Link speed: unknown
>   local
>
>
> vpp# show interface
>   Name   IdxState  MTU (L3/IP4/IP6/MPLS)
> Counter  Count
> VirtualFunctionEthernet0/5/0  1  up  9000/0/0/0 rx
> packets26
> rx
> bytes2910
> tx
> packets36
> tx
> bytes1512
> drops
> 14
> ip4
> 25
> local00 down  0/0/0/0   drops
> 25
> vpp#
>
> 

Re: [vpp-dev] VPP - DPDK - No ARP learning on VPP and no ARP reply sent.

[John Lo (loj) via lists.fd.io]

The new vpp cli to show ip4-arp and ip6-neighbor entries is “show ip neighbors”:

DBGvpp# sho ip neighbor
Time   IPFlags  Ethernet
  Interface
  8.436410.0.3.3   D00:50:56:88:00:ac 
GigabitEthernet1b/0/0
 17.0595 7.0.0.2   D00:50:56:88:ca:7e 
GigabitEthernetb/0/0
114.1346   fc00:0:a00:6500::30 D00:50:56:88:ca:7e 
GigabitEthernetb/0/0

Regards,
John

From: vpp-dev@lists.fd.io  On Behalf Of Laurent Dumont
Sent: Thursday, May 14, 2020 10:25 PM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] VPP - DPDK - No ARP learning on VPP and no ARP reply sent.

Hi!

I was doing some initial experimentation with VPP/SRIOV/DPDK and I just wanted 
to see if some of the things I was experiencing we're expected. I was looking 
at understanding the baseline behavior for something like VPP + DPDK.

I have a small test POC with the following topology.

VM Ubuntu 18.04( with the "linux-image-extra-virtual" package) --->  (VPP + 
DPDK + IP address on the VPP interface) ---> VF ---> PF (compute with SRIOV on 
X710 cards) ---> TOR interface - untagged.

What I am seeing :

  1.  VPP is telling me I have both RX and TX packets.
  2.  That said, no ARP entry for the TOR is seen on VPP
  3.  The interface on the TOR is also not seeing any ARP entry for the VPP 
interface. The TOR is also indicating that RX and TX packets are present.
Are there any inherent limitations to the ARP with VPP/DPDK?

One weird thing is that trying to show the ARP from VPP doesn't seem to be a 
valid command :
vpp# sh ip arp
show ip: unknown input `arp'

Here are some more debugging output.

vpp# show version
vpp v20.01-release built by root on 980ae64453f3 at 2020-01-29T22:13:47

vpp# show int address
VirtualFunctionEthernet0/5/0 (up):
  L3 1.2.3.4/31


vpp# show hardware-interfaces
  NameIdx   Link  Hardware
VirtualFunctionEthernet0/5/0   1 up   VirtualFunctionEthernet0/5/0
  Link speed: 10 Gbps
  Ethernet address fa:16:3e:92:30:f1
  Intel X710/XL710 Family VF
carrier up full duplex mtu 9206
flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
Devargs:
rx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
tx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
pci: device 8086:154c subsystem 103c: address :00:05.00 numa 0
max rx packet len: 9728
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip
   outer-ipv4-cksum vlan-filter jumbo-frame scatter
rx offload active: ipv4-cksum jumbo-frame scatter
tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
   tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso
   gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs
tx offload active: udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other 
ipv6-frag
   ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload
rss active:none
tx burst function: i40e_xmit_pkts
rx burst function: i40e_recv_scattered_pkts_vec_avx2

tx frames ok  36
tx bytes ok 1512
rx frames ok  26
rx bytes ok 3014
extended stats:
  rx good packets 26
  tx good packets 36
  rx good bytes 3014
  tx good bytes 1512
  rx bytes  3014
  rx broadcast packets26
  tx bytes  1512
  tx broadcast packets36
local0 0down  local0
  Link speed: unknown
  local


vpp# show interface
  Name   IdxState  MTU (L3/IP4/IP6/MPLS) 
Counter  Count
VirtualFunctionEthernet0/5/0  1  up  9000/0/0/0 rx packets  
  26
rx bytes
2910
tx packets  
  36
tx bytes
1512
drops   
  14
ip4 
  25
local00 down  

[vpp-dev] VPP - DPDK - No ARP learning on VPP and no ARP reply sent.

[Laurent Dumont]

Hi!

I was doing some initial experimentation with VPP/SRIOV/DPDK and I just
wanted to see if some of the things I was experiencing we're expected. I
was looking at understanding the baseline behavior for something like VPP +
DPDK.

I have a small test POC with the following topology.

VM Ubuntu 18.04( with the "linux-image-extra-virtual" package) --->  (VPP +
DPDK + IP address on the VPP interface) ---> VF ---> PF (compute with SRIOV
on X710 cards) ---> TOR interface - untagged.

What I am seeing :

   1. VPP is telling me I have both RX and TX packets.
   2. That said, no ARP entry for the TOR is seen on VPP
   3. The interface on the TOR is also not seeing any ARP entry for the VPP
   interface. The TOR is also indicating that RX and TX packets are present.

Are there any inherent limitations to the ARP with VPP/DPDK?

One weird thing is that trying to show the ARP from VPP doesn't seem to be
a valid command :
vpp# sh ip arp
show ip: unknown input `arp'

Here are some more debugging output.

vpp# show version
vpp v20.01-release built by root on 980ae64453f3 at 2020-01-29T22:13:47

vpp# show int address
VirtualFunctionEthernet0/5/0 (up):
  L3 1.2.3.4/31


vpp# show hardware-interfaces
  NameIdx   Link  Hardware
VirtualFunctionEthernet0/5/0   1 up   VirtualFunctionEthernet0/5/0
  Link speed: 10 Gbps
  Ethernet address fa:16:3e:92:30:f1
  Intel X710/XL710 Family VF
carrier up full duplex mtu 9206
flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum
rx-ip4-cksum
Devargs:
rx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
tx: queues 1 (max 16), desc 1024 (min 64 max 4096 align 32)
pci: device 8086:154c subsystem 103c: address :00:05.00 numa 0
max rx packet len: 9728
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip
   outer-ipv4-cksum vlan-filter jumbo-frame scatter
rx offload active: ipv4-cksum jumbo-frame scatter
tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum
sctp-cksum
   tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso
   gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs
tx offload active: udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other
ipv6-frag
   ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload
rss active:none
tx burst function: i40e_xmit_pkts
rx burst function: i40e_recv_scattered_pkts_vec_avx2

tx frames ok  36
tx bytes ok 1512
rx frames ok  26
rx bytes ok 3014
extended stats:
  rx good packets 26
  tx good packets 36
  rx good bytes 3014
  tx good bytes 1512
  rx bytes  3014
  rx broadcast packets26
  tx bytes  1512
  tx broadcast packets36
local0 0down  local0
  Link speed: unknown
  local


vpp# show interface
  Name   IdxState  MTU (L3/IP4/IP6/MPLS)
Counter  Count
VirtualFunctionEthernet0/5/0  1  up  9000/0/0/0 rx
packets26
rx
bytes2910
tx
packets36
tx
bytes1512
drops
  14
ip4
  25
local00 down  0/0/0/0   drops
  25
vpp#

RP/0/RP0/CPU0:tor-4.qa-site1#show run int tenGigE 0/0/0/37
Fri May 15 02:22:57.640 UTC
interface TenGigE0/0/0/37
 mtu 9216
 ipv4 address 1.2.3.5 255.255.255.252
 load-interval 30
!
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16397): https://lists.fd.io/g/vpp-dev/message/16397
Mute This Topic: https://lists.fd.io/mt/74220020/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Govindarajan Mohandoss]

Packet size is 64B.

From: Mrityunjay Kumar 
Sent: Thursday, May 14, 2020 2:18 PM
To: Christian Hopps 
Cc: Govindarajan Mohandoss ; s...@tkos.co.il; 
vpp-dev@lists.fd.io; Nitin Saxena ; nd 
Subject: Re: [vpp-dev] Performance on the Macchiatobin


  >  We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
Master branch) and got 5.77 MPPS with single core @ 1.6 GHz
---
Friends, what was the packet size with this MPPS. Good if you share some more 
details.

Regards,
Mrityunjay Kumar.
Mobile: +91 - 9731528504


On Fri, May 15, 2020 at 12:20 AM Christian Hopps 
mailto:cho...@chopps.org>> wrote:
We've also seen good numbers from these boxes; however, we have had issues with 
connecting them back-to-back (they don't work that way the interfaces flap), 
they seem to work when connected through a *non-marvel* based switch. IOW 
there's something weird going on with the marvel chip talking to itself (or the 
parts they use).

Then we experienced problems with the HW encryption offload, getting into 
non-recoverable failures. This could be sw bugs either in DPDK or in the 
marvell mvsam support code, or perhaps in the HW, who knows.

-> can you dump the details of issue over the group? if bug with dpdk, will try 
to address.


Not seeing any recent development and being unable to get support from anyone 
for the boxes caused us to stop using them. It's a shame the performance was 
fantastic with the encryption HW so it was a hard choice to make.

If you have time to debug these issues then they might be good HW, otherwise ...

Thanks,
Chris.



> On May 14, 2020, at 2:07 PM, Govindarajan Mohandoss 
> mailto:govindarajan.mohand...@arm.com>> wrote:
>
> Hi Shmuel,
>  We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
> Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used Marvell 
> I/O Plugin (PMD) and not DPDK. We don’t have the latest performance numbers.  
> Currently, our Macchiatobin boards are not functional.  Please contact Nitin 
> Saxena for latest updates.
>
> Thanks
> Govind
>
>> -Original Message-
>> From: vpp-dev@lists.fd.io 
>> mailto:vpp-dev@lists.fd.io>> On Behalf Of Shmuel H. via
>> lists.fd.io
>> Sent: Thursday, May 14, 2020 9:53 AM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Performance on the Macchiatobin
>>
>> Hi,
>>
>> I have noticed that there has been some work on the Macchiatobin on vpp
>> (as a part of the aarach64 project).
>>
>> However, I have not managed to find any performance results for the
>> Macchiatobin as it is not a part of CSIT.
>>
>> I have found this related(?) [1] JIRA task from 2019/2018, but with no actual
>> results.
>>
>> If someone has some information about what performance I should expect
>> from the Macchiatobin, I would appreciate it a lot.
>>
>> [1]: https://jira.fd.io/browse/VPP-1267
>>
>> Regards,
>> --
>> - Shmuel Hazan
>>
>> mailto:s...@tkos.co.il | tel:+972-523-746-435 | 
>> http://tkos.co.il
>
>


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16396): https://lists.fd.io/g/vpp-dev/message/16396
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev]: Buffer Memory Allocation on 2 NUMA socket platform

[Damjan Marion via lists.fd.io]

> On 14 May 2020, at 18:43, Mrityunjay Kumar  wrote:
> 
> Hi Team
> Please correct me if I understood wrongly.
> I believe, socket-mem inside dpdk { .. } section has been deprecated. Now in 
> latest VPP release , there is no option to specifying socket-mem manually.

it is removed to avoid confusion, as we are not using dpdk buffer manager 
anymore and modifying that parameter will not help much when it comes to buffer 
memory allocation.


> VPP has taken forcefully ownership on platform and allocating memory from 
> both NUMA socket.
> -
> What if I don’t want to share memory from both NUMA socket.  I have one use 
> case, where  we are running our docker container on different NUMA socket. 
> VPP is failing when we are trying to bring up on NUMA 1.
> -
> With few lines of fixes, we are able to run VPP successfully. If someone can 
> point me, how to bring up VPP on specific NUMA socket without any changes in 
> VPP 20.01, it will really appreciated.

New buffer manager supports that kind of setup, but configuration knob is not 
exposed.
contributions are welcome….

— 
Damjan-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16395): https://lists.fd.io/g/vpp-dev/message/16395
Mute This Topic: https://lists.fd.io/mt/74209249/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Mrityunjay Kumar]

  >  We measured IPv4 forwarding performance in the beginning of 2019 (with
VPP Master branch) and got 5.77 MPPS with single core @ 1.6 GHz
---
Friends, what was the packet size with this MPPS. Good if you share some
more details.

*Regards*,
Mrityunjay Kumar.
Mobile: +91 - 9731528504



On Fri, May 15, 2020 at 12:20 AM Christian Hopps  wrote:

> We've also seen good numbers from these boxes; however, we have had issues
> with connecting them back-to-back (they don't work that way the interfaces
> flap), they seem to work when connected through a *non-marvel* based
> switch. IOW there's something weird going on with the marvel chip talking
> to itself (or the parts they use).


> Then we experienced problems with the HW encryption offload, getting into
> non-recoverable failures. This could be sw bugs either in DPDK or in the
> marvell mvsam support code, or perhaps in the HW, who knows.
>

-> can you dump the details of issue over the group? if bug with dpdk, will
try to address.


>
> Not seeing any recent development and being unable to get support from
> anyone for the boxes caused us to stop using them. It's a shame the
> performance was fantastic with the encryption HW so it was a hard choice to
> make.
>
> If you have time to debug these issues then they might be good HW,
> otherwise ...
>
> Thanks,
> Chris.
>
>
>
> > On May 14, 2020, at 2:07 PM, Govindarajan Mohandoss <
> govindarajan.mohand...@arm.com> wrote:
> >
> > Hi Shmuel,
> >  We measured IPv4 forwarding performance in the beginning of 2019 (with
> VPP Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used
> Marvell I/O Plugin (PMD) and not DPDK. We don’t have the latest performance
> numbers.  Currently, our Macchiatobin boards are not functional.  Please
> contact Nitin Saxena for latest updates.
> >
> > Thanks
> > Govind
> >
> >> -Original Message-
> >> From: vpp-dev@lists.fd.io  On Behalf Of Shmuel H.
> via
> >> lists.fd.io
> >> Sent: Thursday, May 14, 2020 9:53 AM
> >> To: vpp-dev@lists.fd.io
> >> Subject: [vpp-dev] Performance on the Macchiatobin
> >>
> >> Hi,
> >>
> >> I have noticed that there has been some work on the Macchiatobin on vpp
> >> (as a part of the aarach64 project).
> >>
> >> However, I have not managed to find any performance results for the
> >> Macchiatobin as it is not a part of CSIT.
> >>
> >> I have found this related(?) [1] JIRA task from 2019/2018, but with no
> actual
> >> results.
> >>
> >> If someone has some information about what performance I should expect
> >> from the Macchiatobin, I would appreciate it a lot.
> >>
> >> [1]: https://jira.fd.io/browse/VPP-1267
> >>
> >> Regards,
> >> --
> >> - Shmuel Hazan
> >>
> >> mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il
> >
> >
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16394): https://lists.fd.io/g/vpp-dev/message/16394
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Shmuel H.]

Hi Govindarajan,

Thanks for your response.

I have just tried a similar test on my board (restricted to 1.6GHz),
with VPP 19.01 and the marvell mvpp2 plugin and got 5.4Mpps (2.7Mpps RX
from each port) with two cores (one for each port, one RXQ).

I am getting the same results from the DPDK driver, I am not sure what
could be the cause of those poor results. Maybe something on the kernel
/ a regression from the new Marvell SDK.

I will try to test it again with one core and report back. 

On 5/14/20 9:07 PM, Govindarajan Mohandoss wrote:
> Hi Shmuel,
>   We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
> Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used Marvell 
> I/O Plugin (PMD) and not DPDK. We don’t have the latest performance numbers.  
> Currently, our Macchiatobin boards are not functional.  Please contact Nitin 
> Saxena for latest updates.
>
> Thanks
> Govind
>
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Shmuel H. via
>> lists.fd.io
>> Sent: Thursday, May 14, 2020 9:53 AM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Performance on the Macchiatobin
>>
>> Hi,
>>
>> I have noticed that there has been some work on the Macchiatobin on vpp
>> (as a part of the aarach64 project).
>>
>> However, I have not managed to find any performance results for the
>> Macchiatobin as it is not a part of CSIT.
>>
>> I have found this related(?) [1] JIRA task from 2019/2018, but with no actual
>> results.
>>
>> If someone has some information about what performance I should expect
>> from the Macchiatobin, I would appreciate it a lot.
>>
>> [1]: https://jira.fd.io/browse/VPP-1267
>>
>> Regards,
>> --
>> - Shmuel Hazan
>>
>> mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il

-- 
- Shmuel Hazan

mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16393): https://lists.fd.io/g/vpp-dev/message/16393
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Christian Hopps]

We've also seen good numbers from these boxes; however, we have had issues with 
connecting them back-to-back (they don't work that way the interfaces flap), 
they seem to work when connected through a *non-marvel* based switch. IOW 
there's something weird going on with the marvel chip talking to itself (or the 
parts they use).

Then we experienced problems with the HW encryption offload, getting into 
non-recoverable failures. This could be sw bugs either in DPDK or in the 
marvell mvsam support code, or perhaps in the HW, who knows.

Not seeing any recent development and being unable to get support from anyone 
for the boxes caused us to stop using them. It's a shame the performance was 
fantastic with the encryption HW so it was a hard choice to make.

If you have time to debug these issues then they might be good HW, otherwise ...

Thanks,
Chris.



> On May 14, 2020, at 2:07 PM, Govindarajan Mohandoss 
>  wrote:
> 
> Hi Shmuel,
>  We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
> Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used Marvell 
> I/O Plugin (PMD) and not DPDK. We don’t have the latest performance numbers.  
> Currently, our Macchiatobin boards are not functional.  Please contact Nitin 
> Saxena for latest updates.
> 
> Thanks
> Govind
> 
>> -Original Message-
>> From: vpp-dev@lists.fd.io  On Behalf Of Shmuel H. via
>> lists.fd.io
>> Sent: Thursday, May 14, 2020 9:53 AM
>> To: vpp-dev@lists.fd.io
>> Subject: [vpp-dev] Performance on the Macchiatobin
>> 
>> Hi,
>> 
>> I have noticed that there has been some work on the Macchiatobin on vpp
>> (as a part of the aarach64 project).
>> 
>> However, I have not managed to find any performance results for the
>> Macchiatobin as it is not a part of CSIT.
>> 
>> I have found this related(?) [1] JIRA task from 2019/2018, but with no actual
>> results.
>> 
>> If someone has some information about what performance I should expect
>> from the Macchiatobin, I would appreciate it a lot.
>> 
>> [1]: https://jira.fd.io/browse/VPP-1267
>> 
>> Regards,
>> --
>> - Shmuel Hazan
>> 
>> mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16392): https://lists.fd.io/g/vpp-dev/message/16392
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Performance on the Macchiatobin

[Govindarajan Mohandoss]

Hi Shmuel,
  We measured IPv4 forwarding performance in the beginning of 2019 (with VPP 
Master branch) and got 5.77 MPPS with single core @ 1.6 GHz. We used Marvell 
I/O Plugin (PMD) and not DPDK. We don’t have the latest performance numbers.  
Currently, our Macchiatobin boards are not functional.  Please contact Nitin 
Saxena for latest updates.

Thanks
Govind

> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Shmuel H. via
> lists.fd.io
> Sent: Thursday, May 14, 2020 9:53 AM
> To: vpp-dev@lists.fd.io
> Subject: [vpp-dev] Performance on the Macchiatobin
> 
> Hi,
> 
> I have noticed that there has been some work on the Macchiatobin on vpp
> (as a part of the aarach64 project).
> 
> However, I have not managed to find any performance results for the
> Macchiatobin as it is not a part of CSIT.
> 
> I have found this related(?) [1] JIRA task from 2019/2018, but with no actual
> results.
> 
> If someone has some information about what performance I should expect
> from the Macchiatobin, I would appreciate it a lot.
> 
> [1]: https://jira.fd.io/browse/VPP-1267
> 
> Regards,
> --
> - Shmuel Hazan
> 
> mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16391): https://lists.fd.io/g/vpp-dev/message/16391
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev]: Buffer Memory Allocation on 2 NUMA socket platform

[Mrityunjay Kumar]

Hi Team

Please correct me if I understood wrongly.

I believe, socket-mem inside dpdk { .. } section has been deprecated. Now
in latest VPP release , there is no option to specifying socket-mem
manually. VPP has taken forcefully ownership on platform and allocating
memory from both NUMA socket.

-

What if I don’t want to share memory from both NUMA socket.  I have one use
case, where  we are running our docker container on different NUMA socket.
VPP is failing when we are trying to bring up on NUMA 1.

-

With few lines of fixes, we are able to run VPP successfully. If someone
can point me, how to bring up VPP on specific NUMA socket without any
changes in VPP 20.01, it will really appreciated.



//MJ



 *Regards*,
Mrityunjay Kumar.
Mobile: +91 - 9731528504
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16390): https://lists.fd.io/g/vpp-dev/message/16390
Mute This Topic: https://lists.fd.io/mt/74209249/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Question regarding ICMP NAT behavior

[Jon Loeliger via lists.fd.io]

Hi vpp-devers,

We have a report of an unexpected behavior when using a static NAT with
ICMP.
It appears that configuring an outside interface to allow ICMP also allows
forwarding
of all protocols as well.

If you start with, say, a blocked TCP on port 22 and an SNMP on port 161,
then
adding a NAT static map of ICMP on an inside-facing address of 192.16.0.53
for
an outside interface of TenGigabitEthernet6/0/0, then suddenly TCP/UDP are
accessible from the outside using SSH and SNMP.  (No, this isn't vppctl
syntax. :-))

(config)# nat static map icmp local 192.168.0.53 external outside
(config)# show nat static
Static Mappings

Proto Local IP Port External IP  Port Interface Twice NAT Out to
In Route Table
-     - -
- ---
  192.168.0.530 0.0.0.0 0   outside
  ipv4-VRF:0
  192.168.0.530 192.168.0.530
  ipv4-VRF:0

Did the ICMP mapping open more than was expected or intended here?

I chased this down in the code a bit, but I'm not sure what the _intent_ is
supposed to be.
When "address only" is true (ie, both ports are 0), then the protocol
appears not to be
used in any of the NAT-entry lookups.  Is that somehow allowing UDP and TCP
to slide
through?

Thanks,
jdl
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16389): https://lists.fd.io/g/vpp-dev/message/16389
Mute This Topic: https://lists.fd.io/mt/74208726/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on Wednesday 20th May

[Dave Wallace]

Excellent!

Thanks getting RC1 completed in a timely fashion -- well done.
-daw-

On 5/13/2020 7:04 PM, Andrew Yourtchenko wrote:

Hi all,

This is to announce that the VPP 20.05 RC1 milestone is complete!

The newly created stable/2005 branch is ready for your fixes in
preparation for the RC2 milestone.

They need to have a Jira ticket for the issue, and to avoid forgetting
adding them to master, where practical, *should* be first merged there
and then cherry-picked into the stable/2005 branch - but as soon as
the Jira ticket is mentioned in the commit message and the fix ends up
in both master and stable/2005 (and if it is important/urgent - maybe
earlier branches), then either order is fine.

The installation packages for the RC1 for Ubuntu 18.04 and Centos 7
from the new branch are available on
https://packagecloud.io/fdio/2005/

The master branch is open for all commits.

Our next milestone for VPP 20.05 is RC2, happening next Wednesday 20th May.

Thanks a lot to Vanessa Valderrama, Dave Wallace and Ed Warnicke for the help!

--a
/* Your friendly 2005 release manager */




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16388): https://lists.fd.io/g/vpp-dev/message/16388
Mute This Topic: https://lists.fd.io/mt/74194208/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Well would be a nice tool if one could use it while developing with changes to 
the API, but the changed API issue (an added function, which *should* be 
backward compatible) and go packaging nightmare with trying to regenerate the 
vpp API inside dependent packages (vpp-agent which then breaks cn-infra, etc..) 
appear to be insurmountable in a reasonable amount of time (a few hours)

Too bad, looked interesting. I'm going to try and back out my API changes and 
see if that works as a stop-gap, but otherwise will have to punt.

Thanks,
Chris.

> On May 14, 2020, at 7:24 AM, Christian Hopps  wrote:
> 
> Has anyone ever tried to build this with a development version (where APIs 
> have changed)?
> 
> After some go fmt.Printf debugging I see that I have 3 incompatible API calls 
> with vpp1908 (.01) binapi. It's really not obvious since i'm not a go 
> packaging expert how to update everything. I did find some info on generating 
> a vpp go api, but how to do that with the downloaded go module dependencies 
> (or how to update those to include my development version) is not obvious.
> 
> Thanks,
> Chris.
> 
>> On May 14, 2020, at 5:27 AM, Christian Hopps  wrote:
>> 
>> Yes, looking at that now. Should have googled first :)
>> 
>> I think the only thing missing here is the usage bar to present "idleness" 
>> per-worker/core. But this is a great start.
>> 
>> Thanks,
>> Chris.
>> 
>>> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>>>  wrote:
>>> 
>>> Hi Chris,
>>> 
 Has anyone contemplated the possibility of a "vpptop" like utility for
 VPP? The thought crossed my mind while I was using htop to help debug some
 performance problems I've been having that aren't directly related to vpp
 processing.
>>> 
>>> Something like this https://github.com/PantheonTechnologies/vpptop ?
>>> 
>>> ben
>>> 
>> 
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16385): https://lists.fd.io/g/vpp-dev/message/16385
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Performance on the Macchiatobin

[Shmuel H.]

Hi,

I have noticed that there has been some work on the Macchiatobin on vpp
(as a part of the aarach64 project).

However, I have not managed to find any performance results for the
Macchiatobin as it is not a part of CSIT.

I have found this related(?) [1] JIRA task from 2019/2018, but with no
actual results.

If someone has some information about what performance I should expect
from the Macchiatobin, I would appreciate it a lot.

[1]: https://jira.fd.io/browse/VPP-1267

Regards,
-- 
- Shmuel Hazan

mailto:s...@tkos.co.il | tel:+972-523-746-435 | http://tkos.co.il

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16386): https://lists.fd.io/g/vpp-dev/message/16386
Mute This Topic: https://lists.fd.io/mt/74206564/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on Wednesday 20th May

[Matthew Smith via lists.fd.io]

Sorry for polluting this thread. I just noticed (too late) that Chris
started a new thread with a different subject to address Govind's issue.

Thanks,
-Matt


On Thu, May 14, 2020 at 9:38 AM Matthew Smith via lists.fd.io  wrote:

>
> Hi Govind,
>
> I recently started seeing errors similar to the one you reported when
> running git review on CentOS 7. I updated to a newer version of git review
> and that fixed the issue.
>
> Older versions of git review use a branch format that is deprecated in
> gerrit. I think it changed at some point from refs/publish/master to
> refs/for/master. Gerrit still allowed the old format for a while, but at
> some point it stopped supporting it.
>
> -Matt
>
>
>
> On Wed, May 13, 2020 at 6:24 PM Govindarajan Mohandoss <
> govindarajan.mohand...@arm.com> wrote:
>
>> Hello Maintainers,
>>  I am doing the patch submission for the first time.
>>  I am following the page
>> https://wiki.fd.io/view/VPP/Pulling,_Building,_Running,_Hacking_and_Pushing_VPP_Code#Pulling
>> and getting the error below. Can you please help to fix this ?
>>
>> #:~/vpp_external/vpp$ git review
>> remote: error: branch refs/publish/master:
>> remote: You need 'Create' rights to create new references.
>> remote: User: mgovind
>> remote: Contact an administrator to fix the permissions
>> remote:
>> remote: Processing changes: refs: 1
>> remote: Processing changes: refs: 1, done
>> To ssh://gerrit.fd.io:29418/vpp
>>  ! [remote rejected] HEAD -> refs/publish/master (prohibited by
>> Gerrit: not permitted: create)
>> error: failed to push some refs to 'ssh://mgov...@gerrit.fd.io:29418/vpp'
>>
>> Thanks
>> Govind
>>
>> > -Original Message-
>> > From: vpp-dev@lists.fd.io  On Behalf Of Andrew
>> > Yourtchenko via lists.fd.io
>> > Sent: Wednesday, May 13, 2020 6:05 PM
>> > To: vpp-dev 
>> > Subject: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on
>> Wednesday
>> > 20th May
>> >
>> > Hi all,
>> >
>> > This is to announce that the VPP 20.05 RC1 milestone is complete!
>> >
>> > The newly created stable/2005 branch is ready for your fixes in
>> preparation
>> > for the RC2 milestone.
>> >
>> > They need to have a Jira ticket for the issue, and to avoid forgetting
>> adding
>> > them to master, where practical, *should* be first merged there and then
>> > cherry-picked into the stable/2005 branch - but as soon as the Jira
>> ticket is
>> > mentioned in the commit message and the fix ends up in both master and
>> > stable/2005 (and if it is important/urgent - maybe earlier branches),
>> then
>> > either order is fine.
>> >
>> > The installation packages for the RC1 for Ubuntu 18.04 and Centos 7
>> from the
>> > new branch are available on https://packagecloud.io/fdio/2005/
>> >
>> > The master branch is open for all commits.
>> >
>> > Our next milestone for VPP 20.05 is RC2, happening next Wednesday 20th
>> > May.
>> >
>> > Thanks a lot to Vanessa Valderrama, Dave Wallace and Ed Warnicke for the
>> > help!
>> >
>> > --a
>> > /* Your friendly 2005 release manager */
>>
>> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16384): https://lists.fd.io/g/vpp-dev/message/16384
Mute This Topic: https://lists.fd.io/mt/74194208/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on Wednesday 20th May

[Matthew Smith via lists.fd.io]

Hi Govind,

I recently started seeing errors similar to the one you reported when
running git review on CentOS 7. I updated to a newer version of git review
and that fixed the issue.

Older versions of git review use a branch format that is deprecated in
gerrit. I think it changed at some point from refs/publish/master to
refs/for/master. Gerrit still allowed the old format for a while, but at
some point it stopped supporting it.

-Matt



On Wed, May 13, 2020 at 6:24 PM Govindarajan Mohandoss <
govindarajan.mohand...@arm.com> wrote:

> Hello Maintainers,
>  I am doing the patch submission for the first time.
>  I am following the page
> https://wiki.fd.io/view/VPP/Pulling,_Building,_Running,_Hacking_and_Pushing_VPP_Code#Pulling
> and getting the error below. Can you please help to fix this ?
>
> #:~/vpp_external/vpp$ git review
> remote: error: branch refs/publish/master:
> remote: You need 'Create' rights to create new references.
> remote: User: mgovind
> remote: Contact an administrator to fix the permissions
> remote:
> remote: Processing changes: refs: 1
> remote: Processing changes: refs: 1, done
> To ssh://gerrit.fd.io:29418/vpp
>  ! [remote rejected] HEAD -> refs/publish/master (prohibited by
> Gerrit: not permitted: create)
> error: failed to push some refs to 'ssh://mgov...@gerrit.fd.io:29418/vpp'
>
> Thanks
> Govind
>
> > -Original Message-
> > From: vpp-dev@lists.fd.io  On Behalf Of Andrew
> > Yourtchenko via lists.fd.io
> > Sent: Wednesday, May 13, 2020 6:05 PM
> > To: vpp-dev 
> > Subject: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on
> Wednesday
> > 20th May
> >
> > Hi all,
> >
> > This is to announce that the VPP 20.05 RC1 milestone is complete!
> >
> > The newly created stable/2005 branch is ready for your fixes in
> preparation
> > for the RC2 milestone.
> >
> > They need to have a Jira ticket for the issue, and to avoid forgetting
> adding
> > them to master, where practical, *should* be first merged there and then
> > cherry-picked into the stable/2005 branch - but as soon as the Jira
> ticket is
> > mentioned in the commit message and the fix ends up in both master and
> > stable/2005 (and if it is important/urgent - maybe earlier branches),
> then
> > either order is fine.
> >
> > The installation packages for the RC1 for Ubuntu 18.04 and Centos 7 from
> the
> > new branch are available on https://packagecloud.io/fdio/2005/
> >
> > The master branch is open for all commits.
> >
> > Our next milestone for VPP 20.05 is RC2, happening next Wednesday 20th
> > May.
> >
> > Thanks a lot to Vanessa Valderrama, Dave Wallace and Ed Warnicke for the
> > help!
> >
> > --a
> > /* Your friendly 2005 release manager */
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16383): https://lists.fd.io/g/vpp-dev/message/16383
Mute This Topic: https://lists.fd.io/mt/74194208/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [SUSPECTED SPAM] [vpp-dev] Troubleshooting IPsec peer behind NAT (AWS instance)

[Muthu Raj]

Hi Neale,

So I've since tried out setting SPD on the interface with the IPv6 address,
and even though I am not able to ping the interface, I see that it does
receive and process packets (which I had erroneously assumed it did not
when it became unpingable).


I added a new SPD and added a policy like so
 ipsec policy add spd 1 priority 10 inbound  action protect sa 20
local-ip-range  -   remote-ip-range  -


This is what the trace looks like:

Packet 10

01:02:05:902414: dpdk-input
  lan0 rx queue 0
  buffer 0x13daa8: current data 0, length 96, buffer-pool 1, ref-count 1,
totlen-nifb 0, trace handle 0x9
   ext-hdr-valid
   l4-cksum-computed l4-cksum-correct
  PKT MBUF: port 0, nb_segs 1, pkt_len 96
buf_len 2176, data_len 96, ol_flags 0x181, data_off 128, phys_addr
0xe2f6aa80
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
Packet Offload Flags
01:02:05:868297: ethernet-input
  frame: flags 0x3, hw-if-index 2, sw-if-index 2
  IP6: 5c:5e:ab:d0:29:f0 -> b4:96:91:18:eb:be 802.1q vlan 67
01:02:05:868299: ip6-input
  IPSEC_ESP: 2001::1 -> 2001::2
tos 0x28, flow label 0x0, hop limit 238, payload length 132
01:02:05:868299: ipsec6-input-feature
  IPSEC_ESP: sa_id 0 spd 2 policy 0 spi 1000 (0x03e8) seq 13693
01:02:05:868300: ip6-lookup
  fib 0 dpo-idx 8 flow hash: 0x
  IPSEC_ESP: 2001::1 -> 2001::2
tos 0x28, flow label 0x0, hop limit 238, payload length 132
01:02:05:868301: ip6-local
IPSEC_ESP: 2001::1 -> 2001::2
  tos 0x28, flow label 0x0, hop limit 238, payload length 132
01:02:05:868301: ip6-punt
IPSEC_ESP: 2001::1 -> 2001::2
  tos 0x28, flow label 0x0, hop limit 238, payload length 132
01:02:05:868302: error-punt
  rx:wan0.67
01:02:05:868302: punt
  ip6-input: valid ip6 packets


  IPSEC_ESP: sa_id 0 spd 2 policy 0 spi 1000 (0x03e8) seq 13693

Here, the spd 2 actually does not have any policy in the 0 index.
here is what show ipsec spd 2 looks like:

 ip6-inbound-protect:
   [4] priority 100 action protect type ip6-inbound-protect protocol any sa
20
 local addr range 2001::2 - 2001::2 port range 0 - 65535
 remote addr range 2001::1 - 2001::1 port range 0 - 65535
 packets 0 bytes 0

Is there a mistake in the way SPD has been added?
Or is something else the issue?

Here is trace as seen by the sender:

Packet 1

20:48:33:279228: dpdk-input
  eth0 rx queue 0
  buffer 0x8ee28: current data 0, length 98, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace handle 0x0
  ext-hdr-valid
  l4-cksum-computed l4-cksum-correct
  PKT MBUF: port 1, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x0, data_off 128, phys_addr
0xb3fb8a80
packet_type 0x10 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0xed0f5862 fdir.hi 0x0 fdir.lo 0xed0f5862
Packet Types
  RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
  IP4: 12:6a:27:79:97:3c -> 12:71:86:3a:04:7b
  ICMP: 10.6.33.33 -> 172.30.0.2
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN
fragment id 0x6e42, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x15b1
20:48:33:279234: ethernet-input
  frame: flags 0x3, hw-if-index 2, sw-if-index 2
  IP4: 12:6a:27:79:97:3c -> 12:71:86:3a:04:7b
20:48:33:279239: ip4-input-no-checksum
  ICMP: 10.6.33.33 -> 172.30.0.2
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN
fragment id 0x6e42, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x15b1
20:48:33:279240: ip4-lookup
  fib 0 dpo-idx 6 flow hash: 0x
  ICMP: 10.6.33.33 -> 172.30.0.2
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN
fragment id 0x6e42, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x15b1
20:48:33:279242: ip4-rewrite
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN
fragment id 0x6e42, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x15b1
20:48:33:279242: ip4-rewrite
  tx_sw_if_index 3 dpo-idx 6 : ipv4 via 11.11.11.10 loop0: mtu:9000 next:4
246e969ce5
dfdead0800 flow hash: 0x
  : 246e969ce5dfdead080045546e4240003f01f61f0a062121ac1e
  0020: 0002080015b178da0011bd5ec60f05001011
20:48:33:279244: ipsec4-output-feature
  spd 1 policy 2
20:48:33:279246: esp4-encrypt
  esp: sa-index 0 spi 1000 (0x03e8) seq 13028 sa-seq-hi 0 crypto
aes-cbc-128 inte
grity sha1-96
20:48:33:279240: ip4-lookup
  fib 0 dpo-idx 6 flow hash: 0x
  ICMP: 10.6.33.33 -> 172.30.0.2
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN
fragment id 0x6e42, flags DONT_FRAGMENT
  ICMP echo_request checksum 0x15b1
20:48:33:279242: ip4-rewrite
  tx_sw_if_index 3 dpo-idx 6 : ipv4 via 11.11.11.10 loop0: mtu:9000 next:4
246e969ce5
20:48:33:279240: ip4-lookup
  fib 0 dpo-idx 6 flow hash: 0x
  ICMP: 10.6.33.33 -> 172.30.0.2
tos 0x00, ttl 64, length 84, checksum 0xf51f dscp CS0 ecn NON_ECN

Re: [vpp-dev] Proposal for VPP binary API stability

[Christian Hopps]

API stability is borderline critical for a project like VPP I think. Yes, it 
can be used stand-alone, but real value is added by building products on top of 
it.

Also important is having an API framework that allows for backward-compatible 
changes to the API for making improvements.

I'm not sure if VPP has the latter, but if there's too much pain with your 
proposed rules I think the latter should addressed rather than sacrificing the 
former. So I support your rules.

Thanks,
Chris.

> On May 14, 2020, at 8:28 AM, Ole Troan  wrote:
> 
> Andrew and I have discussed a process around API changes with the goal to 
> limit the impact on VPP consumers.
> One big painpoint in tracking VPP releases for users of the VPP binary has 
> been API changes.
> 
> We want to ensure:
> - A production API never changes.
> 
> - A production API can be deprecated with one release notice.
> E.g. show_version_2 is introduced in 20.01 and show_version_1 is marked as 
> deprecated.
> show_version_1 is marked with "option delete_by = "v20.05";
> 
> - APIs marked as experimental / not released can change with no notice.
> Added support for option status = "in_progress";
> 
> This patch:
> https://gerrit.fd.io/r/c/vpp/+/26881
> 
> has a tool "crcchecker" that we intend to run as part of the verify job.
> It will block modifications to existing production APIs.
> make checkstyle-api
> It also supports dumping API manifests and make API comparisions across 
> releases/revisions/commits.
> 
> A production API is defined by having .api semantic major version > 0.
> New messages can override that by setting status="in_progress".
> 
> The consequence of this is that developers must maintain two (or more) 
> versions of an API for some time.
> This is obviously painful, but at least it aligns cost and benefit better 
> than when we forced the API users to do it.
> 
> Looking back from 17.01 onwards, it looks like we on average have done about 
> 50 backwards incompatible changes per release.
> That's ignoring the work on API typing, which have resulted in significantly 
> more changes.
> 
> The hope is that we can start v20.09 development with this process in place.
> 
> What do people think?
> 
> Best regards,
> Ole

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16381): https://lists.fd.io/g/vpp-dev/message/16381
Mute This Topic: https://lists.fd.io/mt/74203477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Proposal for VPP binary API stability

[Ole Troan]

Andrew and I have discussed a process around API changes with the goal to limit 
the impact on VPP consumers.
One big painpoint in tracking VPP releases for users of the VPP binary has been 
API changes.

We want to ensure:
- A production API never changes.

- A production API can be deprecated with one release notice.
E.g. show_version_2 is introduced in 20.01 and show_version_1 is marked as 
deprecated.
show_version_1 is marked with "option delete_by = "v20.05";

- APIs marked as experimental / not released can change with no notice.
 Added support for option status = "in_progress";

This patch:
https://gerrit.fd.io/r/c/vpp/+/26881

has a tool "crcchecker" that we intend to run as part of the verify job.
It will block modifications to existing production APIs.
make checkstyle-api
It also supports dumping API manifests and make API comparisions across 
releases/revisions/commits.

A production API is defined by having .api semantic major version > 0.
New messages can override that by setting status="in_progress".

The consequence of this is that developers must maintain two (or more) versions 
of an API for some time.
This is obviously painful, but at least it aligns cost and benefit better than 
when we forced the API users to do it.

Looking back from 17.01 onwards, it looks like we on average have done about 50 
backwards incompatible changes per release.
That's ignoring the work on API typing, which have resulted in significantly 
more changes.

The hope is that we can start v20.09 development with this process in place.

What do people think?

Best regards,
Ole-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16380): https://lists.fd.io/g/vpp-dev/message/16380
Mute This Topic: https://lists.fd.io/mt/74203477/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Has anyone ever tried to build this with a development version (where APIs have 
changed)?

After some go fmt.Printf debugging I see that I have 3 incompatible API calls 
with vpp1908 (.01) binapi. It's really not obvious since i'm not a go packaging 
expert how to update everything. I did find some info on generating a vpp go 
api, but how to do that with the downloaded go module dependencies (or how to 
update those to include my development version) is not obvious.

Thanks,
Chris.

> On May 14, 2020, at 5:27 AM, Christian Hopps  wrote:
> 
> Yes, looking at that now. Should have googled first :)
> 
> I think the only thing missing here is the usage bar to present "idleness" 
> per-worker/core. But this is a great start.
> 
> Thanks,
> Chris.
> 
>> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>>  wrote:
>> 
>> Hi Chris,
>> 
>>> Has anyone contemplated the possibility of a "vpptop" like utility for
>>> VPP? The thought crossed my mind while I was using htop to help debug some
>>> performance problems I've been having that aren't directly related to vpp
>>> processing.
>> 
>> Something like this https://github.com/PantheonTechnologies/vpptop ?
>> 
>> ben
>> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16379): https://lists.fd.io/g/vpp-dev/message/16379
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Help with creating patch

[Andrew Yourtchenko]

FWIW, I had documented (mostly for myself) the process I used from a while ago: 
http://stdio.be/blog/2017-06-06-Creating-drafts-and-changes-for-VPP/


It’s a bit more typing, but works well over HTTPS (IIRC git-review was very 
insistent on SSH to gerrit port.. it does ask for the password - which is the 
same as the GUI - but this ensures I remember it :-) 

so I had never gotten used to using git-review.

 I wonder if the “publish” namespace is some sort of older feature left for 
compatibility... (“refs/for/drafts” stopped working about half a year ago, IIRC)

I heard on the grapevine upgrading the git-review to 2.0 seemed to do the 
trick, maybe the folks that did it might chime in with some first-hand 
experiences...

--a

> On 14 May 2020, at 12:02, Juraj Linkeš  wrote:
> 
> Hi folks,
> 
> I've ran into the same issue when trying to submit a CSIT patch after 
> upgrading my Debian distro (which also updated git review). It turns out that 
> my git review (version 1.25.0) is using improper default gerrit namespace. 
> From git review 1.27.0 release notes [0]:
>Update default gerrit namespace for newer gerrit. According to Gerrit 
> documentation for 2.15.3, refs/for/’branch’ should be used when pushing 
> changes to Gerrit instead of refs/publish/’branch’.
> 
> I didn't figure out how to configure the default namespace for git review 
> (maybe it could be done in .gitreview, but the docs don't mention that) so I 
> just submitted the change manually:
> git push ssh://juraj.lin...@gerrit.fd.io:29418/csit.git 
> HEAD:refs/for/master%wip -o topic=1n-tx2_doc_update
> 
> Regards,
> Juraj
> 
> [0] 
> https://docs.openstack.org/infra/git-review/releasenotes.html#relnotes-1-27-0
> 
>> -Original Message-
>> From: Govindarajan Mohandoss 
>> Sent: Thursday, May 14, 2020 4:31 AM
>> To: Luke, Chris ; vpp-dev 
>> Cc: nd ; nd 
>> Subject: Re: [vpp-dev] Help with creating patch
>> 
>> Hi Chris,
>>  I didn't create a local branch.  Thanks !!
>>  I didn’t change the subject thinking that it could be related to code 
>> freeze.
>> Sorry for that.
>> 
>> Thanks
>> Govind
>> 
>>> -Original Message-
>>> From: Luke, Chris 
>>> Sent: Wednesday, May 13, 2020 9:12 PM
>>> To: Govindarajan Mohandoss ; vpp-
>> dev
>>> 
>>> Cc: nd 
>>> Subject: RE: [vpp-dev] Help with creating patch
>>> 
>>> Govind,
>>> 
>>> Did you create a branch locally before making a commit? It looks like
>>> you tried to push to master which won't work. A typical workflow
>>> involves creating a local branch, making some changes and commits and
>>> then pushing to Gerrit.
>>> 
>>> Also, I changed the email subject; you should really have started a
>>> new thread instead of replying to an existing thread with something 
>>> unrelated.
>>> 
>>> Chris.
>>> 
>>> -Original Message-
>>> From: vpp-dev@lists.fd.io  On Behalf Of
>>> Govindarajan Mohandoss
>>> Sent: Wednesday, May 13, 2020 19:24
>>> To: ayour...@gmail.com; vpp-dev 
>>> Cc: nd 
>>> Subject: [EXTERNAL] Re: [vpp-dev] VPP 20.05 RC1 milestone is complete!
>>> RC2
>>> - on Wednesday 20th May
>>> 
>>> Hello Maintainers,
>>> I am doing the patch submission for the first time.
>>> I am following the page
>>> https://urldefense.com/v3/__https://wiki.fd.io/view/VPP/Pulling,_Build
>>> ing,_
>>> Running,_Hacking_and_Pushing_VPP_Code*Pulling__;Iw!!CQl3mcHX2A!X_Yl
>>> Df6H02w8Ew6AQDrBpiMP7UZ5XJeDWGNgAaY0wqMSqos0VyWPgbGH8cP27P
>>> ol6w$  and getting the error below. Can you please help to fix this ?
>>> 
>>> #:~/vpp_external/vpp$ git review
>>> remote: error: branch refs/publish/master:
>>> remote: You need 'Create' rights to create new references.
>>> remote: User: mgovind
>>> remote: Contact an administrator to fix the permissions
>>> remote:
>>> remote: Processing changes: refs: 1
>>> remote: Processing changes: refs: 1, done To ssh://gerrit.fd.io:29418/vpp
>>> ! [remote rejected] HEAD -> refs/publish/master (prohibited by Gerrit: 
>>> not
>>> permitted: create)
>>> error: failed to push some refs to 'ssh://mgov...@gerrit.fd.io:29418/vpp'
>>> 
>>> Thanks
>>> Govind
>>> 
 -Original Message-
 From: vpp-dev@lists.fd.io  On Behalf Of Andrew
 Yourtchenko via lists.fd.io
 Sent: Wednesday, May 13, 2020 6:05 PM
 To: vpp-dev 
 Subject: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on
 Wednesday 20th May
 
 Hi all,
 
 This is to announce that the VPP 20.05 RC1 milestone is complete!
 
 The newly created stable/2005 branch is ready for your fixes in
 preparation for the RC2 milestone.
 
 They need to have a Jira ticket for the issue, and to avoid
 forgetting adding them to master, where practical, *should* be first
 merged there and then cherry-picked into the stable/2005 branch -
 but as soon as the Jira ticket is mentioned in the commit message
 and the fix ends up in both master and
 stable/2005 (and if it is important/urgent - maybe earlier
 branches), then either order is fine.
 

Re: [vpp-dev] Help with creating patch

[Juraj Linkeš]

Hi folks,

I've ran into the same issue when trying to submit a CSIT patch after upgrading 
my Debian distro (which also updated git review). It turns out that my git 
review (version 1.25.0) is using improper default gerrit namespace. From git 
review 1.27.0 release notes [0]:
Update default gerrit namespace for newer gerrit. According to Gerrit 
documentation for 2.15.3, refs/for/’branch’ should be used when pushing changes 
to Gerrit instead of refs/publish/’branch’.

I didn't figure out how to configure the default namespace for git review 
(maybe it could be done in .gitreview, but the docs don't mention that) so I 
just submitted the change manually:
git push ssh://juraj.lin...@gerrit.fd.io:29418/csit.git 
HEAD:refs/for/master%wip -o topic=1n-tx2_doc_update

Regards,
Juraj

[0] 
https://docs.openstack.org/infra/git-review/releasenotes.html#relnotes-1-27-0

> -Original Message-
> From: Govindarajan Mohandoss 
> Sent: Thursday, May 14, 2020 4:31 AM
> To: Luke, Chris ; vpp-dev 
> Cc: nd ; nd 
> Subject: Re: [vpp-dev] Help with creating patch
> 
> Hi Chris,
>   I didn't create a local branch.  Thanks !!
>   I didn’t change the subject thinking that it could be related to code 
> freeze.
> Sorry for that.
> 
> Thanks
> Govind
> 
> > -Original Message-
> > From: Luke, Chris 
> > Sent: Wednesday, May 13, 2020 9:12 PM
> > To: Govindarajan Mohandoss ; vpp-
> dev
> > 
> > Cc: nd 
> > Subject: RE: [vpp-dev] Help with creating patch
> >
> > Govind,
> >
> > Did you create a branch locally before making a commit? It looks like
> > you tried to push to master which won't work. A typical workflow
> > involves creating a local branch, making some changes and commits and
> > then pushing to Gerrit.
> >
> > Also, I changed the email subject; you should really have started a
> > new thread instead of replying to an existing thread with something 
> > unrelated.
> >
> > Chris.
> >
> > -Original Message-
> > From: vpp-dev@lists.fd.io  On Behalf Of
> > Govindarajan Mohandoss
> > Sent: Wednesday, May 13, 2020 19:24
> > To: ayour...@gmail.com; vpp-dev 
> > Cc: nd 
> > Subject: [EXTERNAL] Re: [vpp-dev] VPP 20.05 RC1 milestone is complete!
> > RC2
> > - on Wednesday 20th May
> >
> > Hello Maintainers,
> >  I am doing the patch submission for the first time.
> >  I am following the page
> > https://urldefense.com/v3/__https://wiki.fd.io/view/VPP/Pulling,_Build
> > ing,_
> > Running,_Hacking_and_Pushing_VPP_Code*Pulling__;Iw!!CQl3mcHX2A!X_Yl
> > Df6H02w8Ew6AQDrBpiMP7UZ5XJeDWGNgAaY0wqMSqos0VyWPgbGH8cP27P
> > ol6w$  and getting the error below. Can you please help to fix this ?
> >
> > #:~/vpp_external/vpp$ git review
> > remote: error: branch refs/publish/master:
> > remote: You need 'Create' rights to create new references.
> > remote: User: mgovind
> > remote: Contact an administrator to fix the permissions
> > remote:
> > remote: Processing changes: refs: 1
> > remote: Processing changes: refs: 1, done To ssh://gerrit.fd.io:29418/vpp
> >  ! [remote rejected] HEAD -> refs/publish/master (prohibited by Gerrit: 
> > not
> > permitted: create)
> > error: failed to push some refs to 'ssh://mgov...@gerrit.fd.io:29418/vpp'
> >
> > Thanks
> > Govind
> >
> > > -Original Message-
> > > From: vpp-dev@lists.fd.io  On Behalf Of Andrew
> > > Yourtchenko via lists.fd.io
> > > Sent: Wednesday, May 13, 2020 6:05 PM
> > > To: vpp-dev 
> > > Subject: [vpp-dev] VPP 20.05 RC1 milestone is complete! RC2 - on
> > > Wednesday 20th May
> > >
> > > Hi all,
> > >
> > > This is to announce that the VPP 20.05 RC1 milestone is complete!
> > >
> > > The newly created stable/2005 branch is ready for your fixes in
> > > preparation for the RC2 milestone.
> > >
> > > They need to have a Jira ticket for the issue, and to avoid
> > > forgetting adding them to master, where practical, *should* be first
> > > merged there and then cherry-picked into the stable/2005 branch -
> > > but as soon as the Jira ticket is mentioned in the commit message
> > > and the fix ends up in both master and
> > > stable/2005 (and if it is important/urgent - maybe earlier
> > > branches), then either order is fine.
> > >
> > > The installation packages for the RC1 for Ubuntu 18.04 and Centos 7
> > > from the new branch are available on
> > > https://urldefense.com/v3/__https://packagecloud.io/fdio/2005/__;!!C
> > > Ql
> > >
> > 3mcHX2A!X_YlDf6H02w8Ew6AQDrBpiMP7UZ5XJeDWGNgAaY0wqMSqos0Vy
> > WPgbGH8cM1me
> > > QxkA$
> > >
> > > The master branch is open for all commits.
> > >
> > > Our next milestone for VPP 20.05 is RC2, happening next Wednesday
> > > 20th May.
> > >
> > > Thanks a lot to Vanessa Valderrama, Dave Wallace and Ed Warnicke for
> > > the help!
> > >
> > > --a
> > > /* Your friendly 2005 release manager */
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16377): https://lists.fd.io/g/vpp-dev/message/16377
Mute This Topic: https://lists.fd.io/mt/74197290/21656
Group Owner: 

Re: [vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Yes, looking at that now. Should have googled first :)

I think the only thing missing here is the usage bar to present "idleness" 
per-worker/core. But this is a great start.

Thanks,
Chris.

> On May 14, 2020, at 5:25 AM, Benoit Ganne (bganne) via lists.fd.io 
>  wrote:
> 
> Hi Chris,
> 
>> Has anyone contemplated the possibility of a "vpptop" like utility for
>> VPP? The thought crossed my mind while I was using htop to help debug some
>> performance problems I've been having that aren't directly related to vpp
>> processing.
> 
> Something like this https://github.com/PantheonTechnologies/vpptop ?
> 
> ben
> 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16376): https://lists.fd.io/g/vpp-dev/message/16376
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Benoit Ganne (bganne) via lists.fd.io]

Hi Chris,

> Has anyone contemplated the possibility of a "vpptop" like utility for
> VPP? The thought crossed my mind while I was using htop to help debug some
> performance problems I've been having that aren't directly related to vpp
> processing.

Something like this https://github.com/PantheonTechnologies/vpptop ?

ben
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16374): https://lists.fd.io/g/vpp-dev/message/16374
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] blinkenlights .. vpptop?

[Jerome Tollet via lists.fd.io]

https://github.com/PantheonTechnologies/vpptop may be of interest to you


De :  au nom de Christian Hopps 
Date : jeudi 14 mai 2020 à 11:19
À : vpp-dev 
Cc : Christian Hopps 
Objet : [vpp-dev] blinkenlights .. vpptop?

Has anyone contemplated the possibility of a "vpptop" like utility for VPP? The 
thought crossed my mind while I was using htop to help debug some performance 
problems I've been having that aren't directly related to vpp processing.

I'm thinking that vpptop could present a dashboard like top does showing the 
(top?) node run times, etc -- with the ability to narrow to per-core. A good 
starting point for the column data would be show runtime output.

More importantly maybe, it would also present per-core usage bars giving a 
"graphical" representation of "idleness" per core so one could quickly know how 
efficiently things were running on each core, and thus if workload was 
distributed correctly.

For memory it could show a graphical bar depicting buffer usage as well.

Thanks,
Chris.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16375): https://lists.fd.io/g/vpp-dev/message/16375
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] blinkenlights .. vpptop?

[Christian Hopps]

Has anyone contemplated the possibility of a "vpptop" like utility for VPP? The 
thought crossed my mind while I was using htop to help debug some performance 
problems I've been having that aren't directly related to vpp processing.

I'm thinking that vpptop could present a dashboard like top does showing the 
(top?) node run times, etc -- with the ability to narrow to per-core. A good 
starting point for the column data would be show runtime output.

More importantly maybe, it would also present per-core usage bars giving a 
"graphical" representation of "idleness" per core so one could quickly know how 
efficiently things were running on each core, and thus if workload was 
distributed correctly.

For memory it could show a graphical bar depicting buffer usage as well.

Thanks,
Chris.-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16373): https://lists.fd.io/g/vpp-dev/message/16373
Mute This Topic: https://lists.fd.io/mt/74201203/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-