Hello all,
I have tested GTP Encapsulation & Decapsulation, as well the GTP recirculation
cases.
I would kindly ask you to share your thoughts on the following change:
out_port = odp_port;
if (ovs_native_tunneling_is_on(ctx->xbridge->ofproto)) {
xlate_report(ctx, OFT_DETAIL, "output to native tunnel");
if(ctx->pending_encap && xport->pt_mode == NETDEV_PT_LEGACY_L3){
VLOG_INFO("For L3 Output Tunnel (%s) with type (%d), unsetting
the pending encapsulation flag.\n", netdev_get_name(xport->netdev),
xport->pt_mode);
ctx->pending_encap = false;
}
is_native_tunnel = true;
} else {
Once we know, that the output port is a tunnel, we could only check if the
pending encapsulation flag is pending. This would be only the case for
recirculation (considering normal GTP traffic // I would handle GTP-Echo
messages as packet in). If the output port is a Layer 3 port/tunnel, we could
simply deactivate the flag.
With this all GTP traffic successfully passed through.
Kind Regards,
Dincer
Dincer Beken
Software Developer
t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
e: [email protected]<mailto:[email protected]>
w: https://blackned.de
[cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
[cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
geschäftsführer: timo haas · hrb memmingen 17319
Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und löschen Sie diese eMail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
gestattet.
This eMail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this eMail in error)
please notify the sender immediately and delete this eMail.
Any unauthorized copying, disclosure or distribution of the contents in this
eMail is strictly forbidden.
________________________________
Von: dev <[email protected]> im Auftrag von Dincer Beken
<[email protected]>
Gesendet: Dienstag, 29. März 2022 13:11
An: Ilya Maximets <[email protected]>; [email protected]
<[email protected]>; Jan Scheurich <[email protected]>;
[email protected] <[email protected]>
Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS Crash
Hello all,
>From "https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365944.html";
I see. So in this case, I image OVS works like this
0) create gtpu0 (teid=10) and gtpu1 (teid=11) tunnel ports
and GTP-U packets arrives at an port
1) OVS decaps the rx packet, forwards to gtpu0 dev
2) controller adds an openflow rule
in_port=gtpu0 actions=output:gtpu1
3) gtpu1 gets inner packet, and encap it with outer header
with new teid, and tx
However, the sequence number will not remain the same.
William
William describe the exact case that I am also describing, which does not work.
In my case, I have the additional Tunnel Id set commands in Openflow
cookie=0x1234, duration=274.699s, table=1, n_packets=1, n_bytes=34,
priority=1100,ip,in_port=gtpu8,nw_dst=192.168.10.3
actions=load:0x55667701->NXM_NX_TUN_ID[],load:0xc0a8b164->NXM_NX_TUN_IPV4_DST[],output:gtpu9
And in Tunnel.c, I did the following change, such that the flow information is
not overwritten by the GTP port tunnel identifiers:
if (!cfg->ip_dst_flow) {
/* Dincer Beken:
* For better configurability -> Always setting the destination based
on the received flow (instead of configuration).
*/
if ((strcmp(netdev_get_type(tnl_port->netdev), "gtpu") != 0) ||
!flow->tunnel.ip_dst) {
flow->tunnel.ip_dst =
in6_addr_get_mapped_ipv4(&tnl_port->match.ipv6_dst);
if (!flow->tunnel.ip_dst) {
flow->tunnel.ipv6_dst = tnl_port->match.ipv6_dst;
} else {
flow->tunnel.ipv6_dst = in6addr_any;
}
}
}
flow->tunnel.tp_dst = cfg->dst_port;
if (strcmp(netdev_get_type(tnl_port->netdev), "gtpu") != 0) {
if (!cfg->out_key_flow) {
flow->tunnel.tun_id = cfg->out_key;
}
}
I heard newly about setting remote_ip & key to flow.
My questions:
* What is the correct way to do the GTP recirculation now?
* Is a single GTP port with remote_ip & key set to flow now enough to handle
multiple GTP connections with OpenFlow?
* Is the removal of the check of the GTPU_PT_Type intentional? If so, how do
we recirculate the GTP packet to another GTP tunnel after it has been
decapsulated.
It is not possible from the patches and the mailing list to really understand
how to implement this GTP UserSpace tunneling.
Looking forward for your response,
Dincer
Mit freundlichen Grüßen
Dincer Beken
E-Mail: [email protected]<mailto:[email protected]>
Tel.: +49 8331 9959-852
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht
gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the
sender immediately and delete this e-mail. Any unauthorized copying, disclosure
or
distribution of the contents in this e-mail is strictly forbidden.
Dincer Beken
Software Developer
t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
e: [email protected]<mailto:[email protected]>
w: https://blackned.de
[cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
[cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
geschäftsführer: timo haas · hrb memmingen 17319
Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und löschen Sie diese eMail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
gestattet.
This eMail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this eMail in error)
please notify the sender immediately and delete this eMail.
Any unauthorized copying, disclosure or distribution of the contents in this
eMail is strictly forbidden.
________________________________
Von: dev <[email protected]> im Auftrag von Dincer Beken
<[email protected]>
Gesendet: Dienstag, 29. März 2022 12:43
An: Ilya Maximets <[email protected]>; [email protected]
<[email protected]>; Jan Scheurich <[email protected]>;
[email protected] <[email protected]>
Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS Crash
Hello
Looking at the following commit:
https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/365389.html
I see that @William thought of this issue and excluded GTP from adding the
additional encapsulation issue in 2019.
I am currently wondering, why this is not in the OVS 2.14 that I am using.
Mit freundlichen Grüßen
Dincer Beken
E-Mail: [email protected]<mailto:[email protected]>
Tel.: +49 8331 9959-852
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht
gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the
sender immediately and delete this e-mail. Any unauthorized copying, disclosure
or
distribution of the contents in this e-mail is strictly forbidden.
Dincer Beken
Software Developer
t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
e: [email protected]<mailto:[email protected]>
w: https://blackned.de
[cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
[cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
geschäftsführer: timo haas · hrb memmingen 17319
Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und löschen Sie diese eMail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
gestattet.
This eMail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this eMail in error)
please notify the sender immediately and delete this eMail.
Any unauthorized copying, disclosure or distribution of the contents in this
eMail is strictly forbidden.
________________________________
Von: dev <[email protected]> im Auftrag von Dincer Beken
<[email protected]>
Gesendet: Dienstag, 29. März 2022 12:11
An: Ilya Maximets <[email protected]>; [email protected]
<[email protected]>; Jan Scheurich <[email protected]>
Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS Crash
Hello Ilya,
Thank you for your response and informing other developers.
Using "Clear Actions" instruction to reset the stack of actions, or trying to
somehow reset the packet did not solve the issue.
I think, it would be cumbersome and not very performant, to route the packet
out and then back in into the OvS to do append the GTP header and send it to
the destination. We could as an alternative send the packet to another datapath
after the encapsulation, to somehow force a clean packet, but even then, we
need to append the GTP Decapsulation with ETH header pushing. Also, we need
some virtual interfaces, and in case of DPDK, they also need to be evaluated
for performance. At the end, we use the Userspace for performance.
I would really appreciate, if I could get a relatively fast response here,
since I really need to fix this very urgently.
What would be the safest solution here to quickly implement? If you guys could
tell, for example where the best place is, for the exclusion for the additional
ETH encapsulation in case of the GTP, that would save my life, until there is a
proper solution.
Regards,
Dincer
Mit freundlichen Grüßen
Dincer Beken
E-Mail: [email protected]<mailto:[email protected]>
Tel.: +49 8331 9959-852
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht
gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the
sender immediately and delete this e-mail. Any unauthorized copying, disclosure
or
distribution of the contents in this e-mail is strictly forbidden.
Dincer Beken
Software Developer
t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
e: [email protected]<mailto:[email protected]>
w: https://blackned.de
[cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
[cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
geschäftsführer: timo haas · hrb memmingen 17319
Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und löschen Sie diese eMail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
gestattet.
This eMail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this eMail in error)
please notify the sender immediately and delete this eMail.
Any unauthorized copying, disclosure or distribution of the contents in this
eMail is strictly forbidden.
________________________________
Von: Ilya Maximets <[email protected]>
Gesendet: Dienstag, 29. März 2022 11:59
An: Dincer Beken <[email protected]>; [email protected]
<[email protected]>; Jan Scheurich <[email protected]>
Cc: [email protected] <[email protected]>; William Tu <[email protected]>
Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS Crash
On 3/29/22 11:39, Dincer Beken wrote:
> Hello again,
>
>
>
> If the packet is recirculated back, and if it is a stripped-off L3 packet
> (GTP decapsulation), OvS does not recognize it as a former GTP packet and
> appends add the pending encapsulation flag on top of it. But the must be
> encapsulated with GTP first, and the ETH header will be appended on top.
>
> Setting the following in "xlate_actions"
> // ctx.pending_encap = true;
>
> fixes the issue (of course just for this case, this cannot be a general
> solution).
>
> But can it be, that the case of GTP recirculation is just not considered, and
> this condition of adding the L2 encapsulation has to be extended to check for
> GTP like tunnels?
>
> The crash occurs during commit_encap_decap_action -> when the packet type is
> not recognized (after the GTPU decap)
>
> If i did something wrong, I would kindly also ask for your advice.
The crash is definitely is not an expected outcome of any scenario,
so that should not happen and has to be fixed.
I'm not very familiar with the L3 support in OVS, so I can't say
what is going on from the top of my head, unfortunately.
Jan, you worked on implementation of L3 tunnels, do you have some thoughs?
On a quick glance, I'd say that OVS may not handle pop and subsequent
push in the same list of actions correctly for L3 tunnels. And we
defenitly don't have a relevant unit test for that, so I'd not exclude
the possibility that such scenario might be just broken, since it was
never tested before...
P.S. The likely reason why you don't have a coredump is that the
process terminates with ABRT in OVS_NOT_REACHED() inside the
commit_encap_decap_action(). It doesn't really crash.
Just a guess though.
Best regards, Ilya Maximets.
>
> Really interested in your feedback.
>
> Best Regard,
> Dincer
>
> Dincer Beken
> Software Developer
>
> t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
>
> e: [email protected]<mailto:[email protected]>
> w: https://blackned.de
>
>
>
> [cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
>
> [cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
>
>
> blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
> geschäftsführer: timo haas · hrb memmingen 17319
>
> Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie
> diese eMail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
> gestattet.
>
> This eMail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this eMail in error)
> please notify the sender immediately and delete this eMail.
> Any unauthorized copying, disclosure or distribution of the contents in this
> eMail is strictly forbidden.
>
> ________________________________
> Von: dev <[email protected]> im Auftrag von Dincer Beken
> <[email protected]>
> Gesendet: Dienstag, 29. März 2022 04:09
> An: [email protected] <[email protected]>
> Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS
> Crash
>
> Checking further into the code, I see that even if I put the ETH header in
> front, the packet is not matched against a gtp-vport, since the packet will
> be a layer2 packet. Even if I hardcodedly fetch a gtp vport, in tnl_find or
> tnl_find_exact, the rule of pushing back the GTP header will not be applied.
>
> Could you please confirm this?
>
> Regards,
> Dincer
>
> Dincer Beken
> Software Developer
>
> t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
>
> e: [email protected]<mailto:[email protected]>
> w: https://blackned.de
>
>
>
> [cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
>
> [cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
>
>
> blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
> geschäftsführer: timo haas · hrb memmingen 17319
>
> Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie
> diese eMail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
> gestattet.
>
> This eMail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this eMail in error)
> please notify the sender immediately and delete this eMail.
> Any unauthorized copying, disclosure or distribution of the contents in this
> eMail is strictly forbidden.
>
> ________________________________
> Von: dev <[email protected]> im Auftrag von Dincer Beken
> <[email protected]>
> Gesendet: Dienstag, 29. März 2022 03:12
> An: [email protected] <[email protected]>
> Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS
> Crash
>
> https://mail.openvswitch.org/pipermail/ovs-dev/2017-May/332897.html
>
> https://www.youtube.com/watch?v=rOp1tUGIbYs
>
> So essentially, this is what I am doing during the decapsulation, such that
> the OvS does not crash.
>
> An additional question would be, if I need to specify an Input VPORt, as well?
>
>
>
> Dincer Beken
> Software Developer
>
> t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
>
> e: [email protected]<mailto:[email protected]>
> w: https://blackned.de
>
>
>
> [cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
>
> [cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
>
>
> blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
> geschäftsführer: timo haas · hrb memmingen 17319
>
> Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie
> diese eMail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
> gestattet.
>
> This eMail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this eMail in error)
> please notify the sender immediately and delete this eMail.
> Any unauthorized copying, disclosure or distribution of the contents in this
> eMail is strictly forbidden.
>
> ________________________________
> Von: dev <[email protected]> im Auftrag von Dincer Beken
> <[email protected]>
> Gesendet: Dienstag, 29. März 2022 02:45
> An: [email protected] <[email protected]>
> Betreff: Re: [ovs-dev] [EXTERNAL] Proper way to recirculate packet // OvS
> Crash
>
> Hello,
>
> After looking further into the issue, after the decapsulation, the packet is
> a L3 packet without a L2 header anymore. If I push the ETH header manually,
> the received port cannot be matched anymore.
>
> Do we have to push the ETH header (some random dummy), onto the decapsulated
> packet, before we encapsulate it again?
>
> Thank you,
> Dincer
>
> Dincer Beken
> Software Developer
>
> t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
>
> e: [email protected]<mailto:[email protected]>
> w: https://blackned.de
>
>
>
> [cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
>
> [cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
>
>
> blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
> geschäftsführer: timo haas · hrb memmingen 17319
>
> Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie
> diese eMail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
> gestattet.
>
> This eMail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this eMail in error)
> please notify the sender immediately and delete this eMail.
> Any unauthorized copying, disclosure or distribution of the contents in this
> eMail is strictly forbidden.
>
> ________________________________
> Von: dev <[email protected]> im Auftrag von Dincer Beken
> <[email protected]>
> Gesendet: Dienstag, 29. März 2022 00:57
> An: [email protected] <[email protected]>
> Betreff: [EXTERNAL] [ovs-dev] Proper way to recirculate packet // OvS Crash
>
> Hello,
>
> I am using the userspace OvS (2.14.) and the userspace GTP tunnel.
> I am decapsulating a GTP packet in table0 -> sending the decapsulated packet
> over to Table1, where I again want to encapsulate it again (with a different
> tunnel identifier) and submit to (another?) gtp port.
>
> root@RiduxV4:~/Documents# ovs-vsctl show
> 3271f3b8-4aa6-40a2-af9b-37353238b104
> Bridge dp2
> Controller "tcp:127.0.0.1"
> is_connected: true
> datapath_type: netdev
> Port gtpu8
> Interface gtpu8
> type: gtpu
> options: {dst_port="2152", key=flow, remote_ip=flow}
> Port dp2
> Interface dp2
> type: internal
> Port gtpu9
> Interface gtpu9
> type: gtpu
> options: {dst_port="2152", key="0x00",
> remote_ip="192.168.177.101"}
> Port ethvlan1000
> Interface ethvlan1000
> type: internal
>
> root@RiduxV4:~/Documents# ovs-ofctl dump-flows dp2
> cookie=0x1234, duration=277.786s, table=0, n_packets=30, n_bytes=2314,
> priority=1250,ip,in_port=eth1 actions=output:ethvlan1000
>
> cookie=0x1234, duration=277.736s, table=0, n_packets=1, n_bytes=34,
> priority=1400,ip,in_port=gtpu8
> actions=mod_dl_dst:99:99:99:99:99:99,resubmit(,1)
>
> cookie=0x1234, duration=274.699s, table=1, n_packets=1, n_bytes=34,
> priority=1100,ip,in_port=gtpu8,nw_dst=192.168.10.3
> actions=load:0x55667701->NXM_NX_TUN_ID[],load:0xc0a8b164->NXM_NX_TUN_IPV4_DST[],output:gtpu8
>
>
> So to repeat, the packet got decapsulated in table 0 and then moved over to
> table1 where it should be encapsulated again.
>
> Encapsulation and Decapsulation work, only encapsulating a decapsulated
> packet, without leaving OvS does not work in Userspace.
>
> If I select the output interface as the intput interface, OvS recognize this
> and does not allow it, so far so good.
>
> But if I select the IN_PORt or another GTP tunnel (gtpu9), OvS crashes. I
> could not generate a core dump, yet.
>
> compose_output_action__ -> xlate_commit_actions(ctx);
>
> Is there a possiblity to implement this in the userspace? Is there a way, to
> clear the actions stack, and send a fresh decapsulated packet in the
> recirculation, such that OvS does not recognize that it was encapsulated
> before?
>
> Looking forward for your support,
>
> Dincer
>
> Dincer Beken
> Software Developer
>
> t: +49 8331 99 59-852<tel:+49%208331%2099%2059-852>
>
> e: [email protected]<mailto:[email protected]>
> w: https://blackned.de
>
>
>
> [cid:blackned_d1650eb0-ca34-4593-90da-c73d0e84f2cd.png]
>
> [cid:linkedin_d8fe2332-efbd-461a-923a-3153732a489e.png]<https://www.linkedin.com/company/8960382>
>
>
> blackned GmbH · zugspitzstraße 1 · 87751 heimertingen
> geschäftsführer: timo haas · hrb memmingen 17319
>
> Diese eMail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtümlich
> erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie
> diese eMail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser eMail ist nicht
> gestattet.
>
> This eMail may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this eMail in error)
> please notify the sender immediately and delete this eMail.
> Any unauthorized copying, disclosure or distribution of the contents in this
> eMail is strictly forbidden.
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
