[vpp-dev] NOTIFICATION: VPP 1801 verify and merge jobs

[Vanessa Valderrama]

The 1801 VPP verify and merge jobs in production have been configured to
run on the CPOC nodes.  Please run the necessary steps to produce new
artifacts.  Once the artifacts have been published, please let me know
so I can change the jobs in production to build on the LF nodes.

Thank you,
Vanessa



signature.asc
Description: OpenPGP digital signature
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[Jon Loeliger]

John,

On Mon, Feb 5, 2018 at 2:14 PM, John Lo (loj)  wrote:

> Thanks Jon.  Seems we have an agreed upon an approach now.  Just to be
> sure, let me list the specifics:
>
>
>
> 1.  We will get rid of custom_instance_by_real_instance vector in
> vxlan_main_t
>
See #3 below.

> 2.  We will add something like user_instance_by_real_instance hash in
> vxlan_main_t to track instance usage.
>
Yes.

> 3.  We will add u32 instance in vxlan_tunnel_t to keep user specified
> instance, which will be the same as real instance or index of vxlan tunnel
> in the tunnels vector in vxlan_main.
>
> I am happy to do this, yes.  However, there is some reason that this
number can not be used directly within the format() of the vxlan_tunnel%d
name.
At the time that the reuse of the HW IF happens, the necessary
name_renumber()
call fails to get the right instance number as the hw_if_index isn't set in
the hi struct
properly yet.  This is why the custom_instance_by_real_instance mechanism
works:
it is always available.

I will endeavor to use just the hash and the instance on the tunnel struct.
I will attempt to resolve the issue described in the previous paragraph in
more detail.
I know that was a bit vague, so I'll attempt to nail-down the real issue I
was seeing.

4.  We will replace the 64 byte if_name in the vxlan_tunnel_details API
> with u32 instance.
>
Yes.

> 5.  We will generate instance-in-use API error if vxlan tunnel
> creation cannot use the desired instance, either natural or specified.
>
Yes.

We also have to guarantee that: in the case of default allocation (ie, not
specified by the user/API), that there still always a successful allocation.
(Ie, a bit of searching for a valid instance might take place.)  I don't
think
this will be a problem in general.  it will likely only be noticed when both
default and specified allocations are intermixed.


> Will review your new patch set when it shows up.  Once you patch is
> merged, I will copy the same mechanism into the GRE tunnel update I am
> currently working. I will submit my GRE patch for review sometime later and
> add you as one of the reviewers.
>

Awesome!  Sounds great!


> Thanks,
>
> John
>

Thanks,
jdl
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[John Lo (loj)]

Thanks Jon.  Seems we have an agreed upon an approach now.  Just to be sure, 
let me list the specifics:


1.  We will get rid of custom_instance_by_real_instance vector in 
vxlan_main_t.

2.  We will add something like user_instance_by_real_instance hash in 
vxlan_main_t to track instance usage.

3.  We will add u32 instance in vxlan_tunnel_t to keep user specified 
instance, which will be the same as real instance or index of vxlan tunnel in 
the tunnels vector in vxlan_main.

4.  We will replace the 64 byte if_name in the vxlan_tunnel_details API 
with u32 instance.

5.  We will generate instance-in-use API error if vxlan tunnel creation 
cannot use the desired instance, either natural or specified.

Will review your new patch set when it shows up.  Once you patch is merged, I 
will copy the same mechanism into the GRE tunnel update I am currently working. 
I will submit my GRE patch for review sometime later and add you as one of the 
reviewers.

Thanks,
John

From: Jon Loeliger [mailto:j...@netgate.com]
Sent: Monday, February 05, 2018 2:26 PM
To: John Lo (loj) 
Cc: vpp-dev 
Subject: Re: [vpp-dev] VXLAN Tunnel IF Names

On Mon, Feb 5, 2018 at 11:48 AM, John Lo (loj) 
> wrote:
One more thang, I would prefer to have a u32 instance number instead of the 64 
bytes if_name in the vxlan_tunnel_details API.   -John

Hi John,

I'll work up a patch re-spin using a hash-table, and I will
include the if_name -> sw_if_index change mentioned above!

Thanks,
jdl


___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[Jon Loeliger]

On Mon, Feb 5, 2018 at 11:48 AM, John Lo (loj)  wrote:

> One more thang, I would prefer to have a u32 instance number instead of
> the 64 bytes if_name in the vxlan_tunnel_details API.   -John
>

Hi John,

I'll work up a patch re-spin using a hash-table, and I will
include the if_name -> sw_if_index change mentioned above!

Thanks,
jdl
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[John Lo (loj)]

One more thang, I would prefer to have a u32 instance number instead of the 64 
bytes if_name in the vxlan_tunnel_details API.   -John

From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of John Lo (loj)
Sent: Monday, February 05, 2018 12:43 PM
To: Jon Loeliger 
Cc: vpp-dev 
Subject: Re: [vpp-dev] VXLAN Tunnel IF Names

Hi Jon,

I think a good approach would be to store the insurance number in the tunnel 
struct and use a hash table to track instance number usage. This is a more 
effective way to not waste memory even for user who like to use big numbers for 
instance while not using that many tunnels.  The instance number in the tunnel 
struct would be the natural one if not specified by user. We will need an 
“instance number in-use” API error.  For users who always use default or always 
specify instance number, there should not be any problem and the current 
default instance behavior is preserved. For users that use mixed mode to create 
tunnels, if a natural one is not available, the API error will force the user 
to specify a usable instance number.  What do you think?

About the renumber param in API, I suppose it is better in the sense that 
Python binding always default non-specified parameters to 0 and thus more user 
friendly. An alternative would be to use instance 0 as “not specified” so any 
user specified instance number must be > 0.

As for GRE tunnel, I am in the process of making a major change to it to 
support ERSPAN encap that is built on top of TEB support.  There were two 
interface names used, greN or teb-greN, depending on payload type (and also 
tunnel interface type) while sharing the same instance space for N. This is 
kind of awkward to me and I have changed this to use greN interface name only. 
I removed the interface reuse logic on tunnel creation to prevent the problem 
caused by having to reuse the right interface type. Thus tunnel create/delete 
always create and delete associated interfaces. Once we agreed on instance 
mechanism for VXLAN, I will also include it in my change to GRE since I am 
changing the API anyway.

Regards,
John

From: Jon Loeliger [mailto:j...@netgate.com]
Sent: Monday, February 05, 2018 11:49 AM
To: John Lo (loj) >
Cc: Dave Barach (dbarach) >; 
vpp-dev >
Subject: Re: [vpp-dev] VXLAN Tunnel IF Names

On Mon, Feb 5, 2018 at 9:55 AM, Jon Loeliger 
> wrote:


I think we must check and reject instance misuse for this to be acceptable. I 
doubt the renumber is really used much for vhost and tap interface (others 
please correct me if not true).

We do use them on every call we make to these functions.
As well as on loopbacks.  Knowing what object (by name) will b
created is essential for us.


In your previous patch set #1, the usage of a bitmap to track instance usage 
seem a reasonable solution. It does imply we will never use “natural” instance 
and would always use the instance allocated from the bitmap and stored in the 
tunnel struct.

That said, we can combine the two approaches.  That is, use the
show_instance_by_real_instance vector for knowing what name to use,
but also use a bitmap to record what  was used and allocated.

One question on the bitmap approach.  It had a hard-coded upper limit of 16K
bits in my previous patch.  Are people OK with that?  It is arbitrary, and could
be larger.  It simply depends on how much memory folks are willing to spend
recording in-use bits.  That or a search of the show_by_real vector.  Or we
could convert to a hash-map to check "in-use" status.

Preference?

John,

Also as a point of information, I'd like to follow this approach for GRE as well
as soon as we agree on an approach here!

Thanks!
jdl

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[John Lo (loj)]

Hi Jon,

I think a good approach would be to store the insurance number in the tunnel 
struct and use a hash table to track instance number usage. This is a more 
effective way to not waste memory even for user who like to use big numbers for 
instance while not using that many tunnels.  The instance number in the tunnel 
struct would be the natural one if not specified by user. We will need an 
“instance number in-use” API error.  For users who always use default or always 
specify instance number, there should not be any problem and the current 
default instance behavior is preserved. For users that use mixed mode to create 
tunnels, if a natural one is not available, the API error will force the user 
to specify a usable instance number.  What do you think?

About the renumber param in API, I suppose it is better in the sense that 
Python binding always default non-specified parameters to 0 and thus more user 
friendly. An alternative would be to use instance 0 as “not specified” so any 
user specified instance number must be > 0.

As for GRE tunnel, I am in the process of making a major change to it to 
support ERSPAN encap that is built on top of TEB support.  There were two 
interface names used, greN or teb-greN, depending on payload type (and also 
tunnel interface type) while sharing the same instance space for N. This is 
kind of awkward to me and I have changed this to use greN interface name only. 
I removed the interface reuse logic on tunnel creation to prevent the problem 
caused by having to reuse the right interface type. Thus tunnel create/delete 
always create and delete associated interfaces. Once we agreed on instance 
mechanism for VXLAN, I will also include it in my change to GRE since I am 
changing the API anyway.

Regards,
John

From: Jon Loeliger [mailto:j...@netgate.com]
Sent: Monday, February 05, 2018 11:49 AM
To: John Lo (loj) 
Cc: Dave Barach (dbarach) ; vpp-dev 
Subject: Re: [vpp-dev] VXLAN Tunnel IF Names

On Mon, Feb 5, 2018 at 9:55 AM, Jon Loeliger 
> wrote:


I think we must check and reject instance misuse for this to be acceptable. I 
doubt the renumber is really used much for vhost and tap interface (others 
please correct me if not true).

We do use them on every call we make to these functions.
As well as on loopbacks.  Knowing what object (by name) will b
created is essential for us.


In your previous patch set #1, the usage of a bitmap to track instance usage 
seem a reasonable solution. It does imply we will never use “natural” instance 
and would always use the instance allocated from the bitmap and stored in the 
tunnel struct.

That said, we can combine the two approaches.  That is, use the
show_instance_by_real_instance vector for knowing what name to use,
but also use a bitmap to record what  was used and allocated.

One question on the bitmap approach.  It had a hard-coded upper limit of 16K
bits in my previous patch.  Are people OK with that?  It is arbitrary, and could
be larger.  It simply depends on how much memory folks are willing to spend
recording in-use bits.  That or a search of the show_by_real vector.  Or we
could convert to a hash-map to check "in-use" status.

Preference?

John,

Also as a point of information, I'd like to follow this approach for GRE as well
as soon as we agree on an approach here!

Thanks!
jdl

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[Jon Loeliger]

On Mon, Feb 5, 2018 at 9:55 AM, Jon Loeliger  wrote:

>
>
>
>> I think we must check and reject instance misuse for this to be
>> acceptable. I doubt the renumber is really used much for vhost and tap
>> interface (others please correct me if not true).
>>
>
> We do use them on every call we make to these functions.
> As well as on loopbacks.  Knowing what object (by name) will b
> created is essential for us.
>
>
>>
>>
> In your previous patch set #1, the usage of a bitmap to track instance
>> usage seem a reasonable solution. It does imply we will never use “natural”
>> instance and would always use the instance allocated from the bitmap and
>> stored in the tunnel struct.
>>
>
> That said, we can combine the two approaches.  That is, use the
> show_instance_by_real_instance vector for knowing what name to use,
> but also use a bitmap to record what  was used and allocated.
>
> One question on the bitmap approach.  It had a hard-coded upper limit of
> 16K
> bits in my previous patch.  Are people OK with that?  It is arbitrary, and
> could
> be larger.  It simply depends on how much memory folks are willing to spend
> recording in-use bits.  That or a search of the show_by_real vector.  Or we
> could convert to a hash-map to check "in-use" status.
>
> Preference?
>

John,

Also as a point of information, I'd like to follow this approach for GRE as
well
as soon as we agree on an approach here!

Thanks!
jdl
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

[vpp-dev] 18.04 tentative release plan

[Luke, Chris]

All,

For any who didn't hear, somehow I've ended up the release manager for 18.04, 
following in the footsteps of others whose notes Dave Wallace has done a 
sterling job coalescing into something I think even I can grok.

I've posted the tentative release plan for 18.04 in the usual place 
(https://wiki.fd.io/view/Projects/vpp/Release_Plans/Release_Plan_18.04) - 
please review the proposed dates and feel free to add release deliverables you 
think you will be producing for 18.04.

Unfortunately I won't be able to make the dev call tomorrow, but please do 
raise any concerns with the release plan there or here on the dev list.

Cheers,
Chris.

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[Jon Loeliger]

On Mon, Feb 5, 2018 at 9:29 AM, John Lo (loj)  wrote:

> Hi Jon,
>

Hi John,


> I don’t think the renumber scheme, as used by vhost and tap interfaces, is
> foolproof and can cause problems if used extensively for vxlan.  On
> creating an interface with renumber, it does not check if the instance
> being requested is already used by another device. On creating an interface
> without renumber, it also does not check if the “natural instance’ may have
> already been used by another device’s renumber.  Thus, on instance
> collision, last interface will take over the interface name and leave the
> previous interface  not accessible by the duplicated interface name.
>

I was solving that the UI level, but it could be added into the
vxlan_add_del_interface() layer as well.


> I think we must check and reject instance misuse for this to be
> acceptable. I doubt the renumber is really used much for vhost and tap
> interface (others please correct me if not true).
>

We do use them on every call we make to these functions.
As well as on loopbacks.  Knowing what object (by name) will b
created is essential for us.


>   I see a “FIXME” warning in the existing vhost renumber function:
>
>
>
> static int
>
> vhost_user_name_renumber (vnet_hw_interface_t * hi, u32 new_dev_instance)
>
> {
>
>   // FIXME: check if the new dev instance is already used
>
>   vhost_user_main_t *vum = _user_main;
>
>   vec_validate_init_empty (vum->show_dev_instance_by_real_dev_instance,
>
>  hi->dev_instance, ~0);
>

Hmmm, yes, I see ...


> In your previous patch set #1, the usage of a bitmap to track instance
> usage seem a reasonable solution. It does imply we will never use “natural”
> instance and would always use the instance allocated from the bitmap and
> stored in the tunnel struct.
>

This procedure fails for some reason on the 5th test in VXLAN tests.
I am not certain, but I think it is because the renumbering effort was
trying to get the desired instance number out of the tunnel struct,
which lead to finding the corresponding vnet_hw_interface, and when
it needed to renumber, the hi->hw_if_index or hi->dev_instance wasn't
yet set, and it ended up creating IFs with wrong names (always 0?).

That said, we can combine the two approaches.  That is, use the
show_instance_by_real_instance vector for knowing what name to use,
but also use a bitmap to record what  was used and allocated.

One question on the bitmap approach.  It had a hard-coded upper limit of 16K
bits in my previous patch.  Are people OK with that?  It is arbitrary, and
could
be larger.  It simply depends on how much memory folks are willing to spend
recording in-use bits.  That or a search of the show_by_real vector.  Or we
could convert to a hash-map to check "in-use" status.

Preference?

As for the API/CLI, I though another parameter “instance” would suffice as
> ~0 can be used to state “not specified” rather than having an additional
> parameter “renumber”, although I can accept either way.
>

Two things here.  I'm fine with either as well.  I was following the
existing
pattern for the "renumber" approach.  Using ~0 effectively means that the
message constructors must all explicitly initialize this field to ~0.  I'm
good
with that (previous patch!) if you guys are as well.

Regards,
>
> John
>

Thanks for your feedback!

jdl
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] Calling a C function in one plugin from another plugin?

[Michael Lilja]

Perfect, that seems like a much easier solution :)

Thanks

/Michael

From: "Dave Barach (dbarach)" 
Sent: Monday, 5 February 2018, 4.17 PM
To: Michael Lilja; vpp-dev@lists.fd.io 
Subject: RE: Calling a C function in one plugin from another plugin?

You can ask vlib_get_plugin_symbol ("plugin_name", "function_name") for the 
address of a function...

Returns NULL if e.g. the plugin in question isn't loaded or the symbol is 
missing.

HTH... D.

-Original Message-
From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of Michael Lilja
Sent: Monday, February 5, 2018 9:54 AM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] Calling a C function in one plugin from another plugin?

Hi,

I'm looking at using DPDK rte_flow (generic flow API) for ACL offloading. From 
what I can see the only option I have is to implement a v1_msg_* receiver in 
the DPDK plugin to accept commands from ACL via the SHMEM rings. The concern I 
have is that this might be in conflict with the design of VPP, I'm not sure if 
VPP is designed to have inter-plugin-communication?

Does anyone have another approach to call DPDK functions from within another 
plugins instead of the v1_msg_* layer?

Thanks,
Michael
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] VXLAN Tunnel IF Names

[John Lo (loj)]

Hi Jon,

I don’t think the renumber scheme, as used by vhost and tap interfaces, is 
foolproof and can cause problems if used extensively for vxlan.  On creating an 
interface with renumber, it does not check if the instance being requested is 
already used by another device. On creating an interface without renumber, it 
also does not check if the “natural instance’ may have already been used by 
another device’s renumber.  Thus, on instance collision, last interface will 
take over the interface name and leave the previous interface  not accessible 
by the duplicated interface name.

I think we must check and reject instance misuse for this to be acceptable. I 
doubt the renumber is really used much for vhost and tap interface (others 
please correct me if not true).  I see a “FIXME” warning in the existing vhost 
renumber function:

static int
vhost_user_name_renumber (vnet_hw_interface_t * hi, u32 new_dev_instance)
{
  // FIXME: check if the new dev instance is already used
  vhost_user_main_t *vum = _user_main;
  vec_validate_init_empty (vum->show_dev_instance_by_real_dev_instance,
 hi->dev_instance, ~0);

In your previous patch set #1, the usage of a bitmap to track instance usage 
seem a reasonable solution. It does imply we will never use “natural” instance 
and would always use the instance allocated from the bitmap and stored in the 
tunnel struct.

As for the API/CLI, I though another parameter “instance” would suffice as ~0 
can be used to state “not specified” rather than having an additional parameter 
“renumber”, although I can accept either way.

Regards,
John

From: Jon Loeliger [mailto:j...@netgate.com]
Sent: Sunday, February 04, 2018 3:38 PM
To: John Lo (loj) ; Dave Barach (dbarach) 
Cc: vpp-dev 
Subject: Re: [vpp-dev] VXLAN Tunnel IF Names

On Wed, Jan 31, 2018 at 6:41 PM, John Lo (loj) 
> wrote:
Hi Jon,

All VPP tunnel creation uses the mechanism of returning a sw_if_index of the 
created tunnel. The name of the tunnel is then followed by a number being the 
instance or index to the tunnel struct vector. Thus, the first VXLAN tunnel 
created is called vxlan_tunnel0 followed by vxlan_tunnel1, etc. The number is 
incrementing unless tunnels are deleted and created again where a newly created 
tunnel will reuse the last deleted tunnel, hence its name. If all previously 
deleted tunnels are used up, then the tunnel name number will start 
incrementing again.  I am not sure if it is feasible to follow this behavior to 
“predict” tunnel name.

John and Dave,

I have uploaded a new patch set.  It is being verified at the time of this 
writing.

This patch has a different implementation than the previous version.
The API changes, including VAT-n-friends, have been updated a bit,
but the basic mechanism of obtaining predictable SW IF names is
different from my original patch.

In the current form, I followed an existing "custom device instance" pattern
that was already present on several other objects already.  Much simpler,
and I expect it to verify now. :-)

So, I guess I realize my mistake in my previous discussions of this general 
problem
I apparently wasn't using the right words.  I was saying something like "The 
User
needs to be able to know or predict the SW IF name."  The phrase that seems to
be used specifically for this purpose in many other places is "Let the user 
assign
a custom device instance id."  And that is achieved via the use of the so-called
*_name_renumber() class function.

So my earlier plea for a "can we somehow solve this at a higher level" might
be more properly stated as:  Where objects are created as a side-effect of
some other creation request, a "renumber flag and custom device instance"
would be really nice on the Create API call.

HTH,
jdl

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] Calling a C function in one plugin from another plugin?

[Dave Barach (dbarach)]

You can ask vlib_get_plugin_symbol ("plugin_name", "function_name") for the 
address of a function... 

Returns NULL if e.g. the plugin in question isn't loaded or the symbol is 
missing.

HTH... D.

-Original Message-
From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of Michael Lilja
Sent: Monday, February 5, 2018 9:54 AM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] Calling a C function in one plugin from another plugin?

Hi,

I'm looking at using DPDK rte_flow (generic flow API) for ACL offloading. From 
what I can see the only option I have is to implement a v1_msg_* receiver in 
the DPDK plugin to accept commands from ACL via the SHMEM rings. The concern I 
have is that this might be in conflict with the design of VPP, I'm not sure if 
VPP is designed to have inter-plugin-communication?

Does anyone have another approach to call DPDK functions from within another 
plugins instead of the v1_msg_* layer?

Thanks,
Michael
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

[vpp-dev] Calling a C function in one plugin from another plugin?

[Michael Lilja]

Hi,

I'm looking at using DPDK rte_flow (generic flow API) for ACL offloading. From 
what I can see the only option I have is to implement a v1_msg_* receiver in 
the DPDK plugin to accept commands from ACL via the SHMEM rings. The concern I 
have is that this might be in conflict with the design of VPP, I'm not sure if 
VPP is designed to have inter-plugin-communication?

Does anyone have another approach to call DPDK functions from within another 
plugins instead of the v1_msg_* layer?

Thanks,
Michael
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev