Re: [ovs-dev] [PATCHv6] netdev-linux: Detect numa node id.

2019-11-19 Thread Eelco Chaudron
On 19 Nov 2019, at 22:39, William Tu wrote: Hi, Any feedback on this patch? LGTM! Acked-by: Eelco Chaudron On Thu, Oct 31, 2019 at 12:21:21PM -0700, William Tu wrote: The patch detects the numa node id from the name of the netdev, by reading the '/sys/class/net//device/numa_node'. If not

Re: [ovs-dev] [PATCH RFC] tests: Add netdev-afxdp specific tests.

2019-11-19 Thread Eelco Chaudron
On 20 Nov 2019, at 1:09, William Tu wrote: netdev-afxdp supports different configuration modes, with different limitations. The patch adds configuration and end-to-end tests using veth/tap devices. These tests look good, do we want to put in some negative test cases? Signed-off-by:

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Eelco Chaudron
On 19 Nov 2019, at 17:52, Ilya Maximets wrote: On 19.11.2019 17:16, Eelco Chaudron wrote: On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth'

Re: [ovs-dev] [PATCH ovn v2 00/13] OVN Interconnection

2019-11-19 Thread taoyunupt
Get it. Thanks Han. At 2019-11-20 11:51:17, "Han Zhou" wrote: Hi Yun, This feature in OVN doesn't require or put limit on the CMS systems such as Neutron, or kubernetes. To utilize this feature from CMS system, the CMS integration may be needed, e.g. add support in networking-ovn

[ovs-dev] Payment Notification from *ALBERT BYRON 915_8W5BNVMXD

2019-11-19 Thread paymentem...@fnb.co.za
Dear PAYMENT NOTIFICATION Please find attached a copy of your payment notification. How to open your payment notification? In order to open your payment notification you will need Winrar installed on your computer. If you don't have imageburn installed on your computer, please refer to the

Re: [ovs-dev] [PATCH ovn v2 00/13] OVN Interconnection

2019-11-19 Thread taoyunupt
Hi,Han,Numan, If I am not mistake, each AZ(OVN deployment) must have a corresponding networking-ovn(neurton). But if we want to have a global neutron above AZ, that is, multi AZ(OVN deployment) is controlled by one neutron, this would be not work? Am I right ? Did you consider this

Re: [ovs-dev] [PATCH ovn v2 00/13] OVN Interconnection

2019-11-19 Thread Han Zhou
Hi Yun, This feature in OVN doesn't require or put limit on the CMS systems such as Neutron, or kubernetes. To utilize this feature from CMS system, the CMS integration may be needed, e.g. add support in networking-ovn Neutron plugin or networking-ovn k8s plugin. However, the scenario you

[ovs-dev] 答复: 答复: [openvswitch.org代发] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread 杨燚
Ilya, got it, thanks a lot. -邮件原件- 发件人: Ilya Maximets [mailto:i.maxim...@ovn.org] 发送时间: 2019年11月19日 19:55 收件人: Yi Yang (杨燚)-云服务集团 ; i.maxim...@ovn.org; ovs-dev@openvswitch.org 主题: Re: 答复: [openvswitch.org代发][ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode. On

[ovs-dev] [PATCH RFC] tests: Add netdev-afxdp specific tests.

2019-11-19 Thread William Tu
netdev-afxdp supports different configuration modes, with different limitations. The patch adds configuration and end-to-end tests using veth/tap devices. Signed-off-by: William Tu --- This patch depends on [PATCHv2] netdev-afxdp: Best-effort configuration of XDP mode. --- tests/automake.mk

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Russell Bryant
On Tue, Nov 19, 2019 at 5:32 PM Han Zhou wrote: > > > On Tue, Nov 19, 2019 at 2:19 PM Russell Bryant wrote: > >> On Tue, Nov 19, 2019 at 4:44 PM Han Zhou wrote: >> > >> > >> > >> > On Tue, Nov 19, 2019 at 1:38 PM Han Zhou wrote: >> >> >> >> >> >> >> >> On Tue, Nov 19, 2019 at 3:33 AM Numan

Re: [ovs-dev] [PATCH RFC] dpif-netdev: Add ovs-appctl dpif-netdev/subtable-show.

2019-11-19 Thread William Tu
On Fri, Nov 08, 2019 at 03:23:44PM +, Emma Finn wrote: > Add an ovs-appctl command to iterate through the dpcls > and for each subtable output the miniflow bits for any > existing table. > > $ ovs-appctl dpif-netdev/subatable-show > pmd thread numa_id 0 > dpcls port 2: > subtable: >

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Han Zhou
On Tue, Nov 19, 2019 at 2:19 PM Russell Bryant wrote: > On Tue, Nov 19, 2019 at 4:44 PM Han Zhou wrote: > > > > > > > > On Tue, Nov 19, 2019 at 1:38 PM Han Zhou wrote: > >> > >> > >> > >> On Tue, Nov 19, 2019 at 3:33 AM Numan Siddique wrote: > >> > > >> > On Tue, Nov 19, 2019 at 7:04 AM

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Russell Bryant
On Tue, Nov 19, 2019 at 4:44 PM Han Zhou wrote: > > > > On Tue, Nov 19, 2019 at 1:38 PM Han Zhou wrote: >> >> >> >> On Tue, Nov 19, 2019 at 3:33 AM Numan Siddique wrote: >> > >> > On Tue, Nov 19, 2019 at 7:04 AM Russell Bryant wrote: >> > > >> > > While debugging some problems in a cluster

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Han Zhou
On Tue, Nov 19, 2019 at 1:38 PM Han Zhou wrote: > > > On Tue, Nov 19, 2019 at 3:33 AM Numan Siddique wrote: > > > > On Tue, Nov 19, 2019 at 7:04 AM Russell Bryant wrote: > > > > > > While debugging some problems in a cluster using ovn-kubernetes, I > > > noticed that we're creating two

Re: [ovs-dev] [PATCH v3] dpdk: Migrate to the new pdump API.

2019-11-19 Thread Stokes, Ian
On 11/13/2019 3:26 PM, Ilya Maximets wrote: On 13.11.2019 15:28, Stokes, Ian wrote: On 11/12/2019 7:40 PM, Ilya Maximets wrote: On 12.11.2019 19:51, Stokes, Ian wrote: On 11/12/2019 5:15 PM, David Marchand wrote: On Tue, Nov 12, 2019 at 6:07 PM Stokes, Ian wrote: On 11/11/2019 3:01

Re: [ovs-dev] [PATCHv6] netdev-linux: Detect numa node id.

2019-11-19 Thread William Tu
Hi, Any feedback on this patch? Thanks William On Thu, Oct 31, 2019 at 12:21:21PM -0700, William Tu wrote: > The patch detects the numa node id from the name of the netdev, > by reading the '/sys/class/net//device/numa_node'. > If not available, ex: virtual device, or any error happens, > return

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Han Zhou
On Tue, Nov 19, 2019 at 3:33 AM Numan Siddique wrote: > > On Tue, Nov 19, 2019 at 7:04 AM Russell Bryant wrote: > > > > While debugging some problems in a cluster using ovn-kubernetes, I > > noticed that we're creating two conflicting logical flows. These two > > flows only matched on the

Re: [ovs-dev] [PATCH] dpdk: Deprecate pdump support.

2019-11-19 Thread Stokes, Ian
On 11/19/2019 12:45 PM, David Marchand wrote: On Mon, Nov 11, 2019 at 7:53 PM Ilya Maximets wrote: The conventional way for packet dumping in OVS is to use ovs-tcpdump that works via traffic mirroring. DPDK pdump could probably be used for some lower level debugging, but it is not

[ovs-dev] [PATCH ovn v2] ovn-northd: ls_*_acl behavior not consistent for untracked flows

2019-11-19 Thread venugopali
From: venu iyer If one creates a port group and a MAC address set, and an ACL that prevents packets being output to a port in that Port Group from any MAC address in that address set, the outcome is not consistent. The outcome depends on whether there is a stateful rule on the switch or not.

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread William Tu
On Tue, Nov 19, 2019 at 09:26:40PM +0100, Ilya Maximets wrote: > On 19.11.2019 21:09, William Tu wrote: > > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > >> On 19.11.2019 20:45, William Tu wrote: > >>> On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > On

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread William Tu
On Tue, Nov 19, 2019 at 12:09:02PM -0800, William Tu wrote: > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > > On 19.11.2019 20:45, William Tu wrote: > > > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > > >> On 19.11.2019 17:16, Eelco Chaudron wrote: > > >>> > >

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Ilya Maximets
On 19.11.2019 21:09, William Tu wrote: > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: >> On 19.11.2019 20:45, William Tu wrote: >>> On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: On 19.11.2019 17:16, Eelco Chaudron wrote: > > > On 7 Nov 2019, at

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread William Tu
On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > On 19.11.2019 20:45, William Tu wrote: > > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > >> On 19.11.2019 17:16, Eelco Chaudron wrote: > >>> > >>> > >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >>> >

Re: [ovs-dev] [PATCH ovn] ovn-northd: ls_*_acl behavior not consistent for untracked flows

2019-11-19 Thread 0-day Robot
Bleep bloop. Greetings , I am a robot and I have tried out your patch. Thanks for your contribution. I encountered some error that I wasn't expecting. See the details below. checkpatch: WARNING: Line is 86 characters long (recommended limit is 79) #92 FILE: northd/ovn-northd.c:4916:

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Ilya Maximets
On 19.11.2019 20:45, William Tu wrote: > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: >> On 19.11.2019 17:16, Eelco Chaudron wrote: >>> >>> >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: >>> Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e.

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread William Tu
On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > On 19.11.2019 17:16, Eelco Chaudron wrote: > > > > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > > > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. > >> i.e. 'generic XDP' or 'native XDP with

Re: [ovs-dev] [PATCH RFC] dpif-netdev: Add ovs-appctl dpif-netdev/subtable-show.

2019-11-19 Thread Ilya Maximets
On 08.11.2019 16:23, Emma Finn wrote: > Add an ovs-appctl command to iterate through the dpcls > and for each subtable output the miniflow bits for any > existing table. Hi Emma, First of all, thanks for working on this. This might be useful for developers to determine the optimization points.

Re: [ovs-dev] [PATCH v2] netdev-afxdp: add afxdp specific maximum MTU check

2019-11-19 Thread William Tu
On Tue, Nov 12, 2019 at 04:46:09AM -0500, Eelco Chaudron wrote: > Drivers natively supporting AF_XDP will check that a configured MTU size > will not exceed the allowed size for AF_XDP. However, when the skb > compatibility mode is used there is no check and any value is accepted. > This, for

Re: [ovs-dev] [PATCH ovn] Skip IPv6 NS packets in router egress SNAT pipeline

2019-11-19 Thread Numan Siddique
On Tue, Nov 19, 2019 at 11:41 PM Mark Michelson wrote: > > Acked-by: Mark Michelson Thanks. I applied this to master. Numan > > On 11/19/19 7:22 AM, num...@ovn.org wrote: > > From: Numan Siddique > > > > When ovn-controller injects IPv6 NS packet to learn the mac, in some > > setups it is

[ovs-dev] [PATCH ovn] ovn-northd: ls_*_acl behavior not consistent for untracked flows

2019-11-19 Thread venugopali
From: venu iyer If one creates a port group and a MAC address set, and an ACL that prevents packets being output to a port in that Port Group from any MAC address in that address set, the outcome is not consistent. The outcome depends on whether there is a stateful rule on the switch or not.

Re: [ovs-dev] [PATCHv4] netdev-afxdp: Enable loading XDP program.

2019-11-19 Thread William Tu
Hi Eelco, Thanks for your testing. On Tue, Nov 12, 2019 at 11:25:55AM +0100, Eelco Chaudron wrote: > See one remark below, however when I did a quick test with a program that > would not load it goes into some re-try loop: > > 2019-11-12T10:13:21.658Z|01609|netdev_afxdp|INFO|eno1: Removing xdp

Re: [ovs-dev] include/sparse/rte_flow.h

2019-11-19 Thread Ilya Maximets
On 19.11.2019 19:01, Eli Britstein wrote: > > On 11/19/2019 7:46 PM, Ilya Maximets wrote: >> On 19.11.2019 18:29, Eli Britstein wrote: >>> On 11/19/2019 7:27 PM, Eli Britstein wrote: Hi I see this file has many inconsistencies against the one from DPDK (18.11.2). For

[ovs-dev] Cero rotación en su organización

2019-11-19 Thread Ultimas Conexiones
Buen día El cupo del curso está por llenarse y quise aprovechar la oportunidad de hacerte una última invitación: •Nombre: Rotación Cero •¿Cuándo?: Miércoles 27 de Noviembre •Formato: En línea con interacción en vivo. •Instructor: Lyz Escalante •

Re: [ovs-dev] [PATCH] netdev: Dynamic per-port Flow API.

2019-11-19 Thread Ilya Maximets
On 19.11.2019 18:20, Ophir Munk wrote: > Hi Ilya, > Thanks for the patch set which adds the dpif hint inside the netdev struct. > It is really helpful. > Our goal is to have flow_put() calls on vxlan devices, using the existing > dpdk flow API. > We added logic inside

Re: [ovs-dev] [PATCH v5 ovn 2/4] ovn-controller: Add per node states to I-P engine.

2019-11-19 Thread Han Zhou
On Tue, Nov 19, 2019 at 1:58 AM Dumitru Ceara wrote: > > On Tue, Nov 19, 2019 at 1:41 AM Han Zhou wrote: > > > > Thanks Dumitru. Please see my comments inline below. > > > > On Mon, Nov 18, 2019 at 6:07 AM Dumitru Ceara wrote: > > > > > > This commit transforms the 'changed' field in struct

Re: [ovs-dev] include/sparse/rte_flow.h

2019-11-19 Thread Eli Britstein
On 11/19/2019 7:46 PM, Ilya Maximets wrote: > On 19.11.2019 18:29, Eli Britstein wrote: >> On 11/19/2019 7:27 PM, Eli Britstein wrote: >>> Hi >>> >>> I see this file has many inconsistencies against the one from DPDK >>> (18.11.2). >>> >>> For example, this API: >>> >>> rte_flow_query(uint16_t

Re: [ovs-dev] [PATCH ovn] Skip IPv6 NS packets in router egress SNAT pipeline

2019-11-19 Thread Mark Michelson
Acked-by: Mark Michelson On 11/19/19 7:22 AM, num...@ovn.org wrote: From: Numan Siddique When ovn-controller injects IPv6 NS packet to learn the mac, in some setups it is observed that the packet is dropped by ovs-vswitchd. We see below logs

Re: [ovs-dev] include/sparse/rte_flow.h

2019-11-19 Thread Ilya Maximets
On 19.11.2019 18:29, Eli Britstein wrote: > > On 11/19/2019 7:27 PM, Eli Britstein wrote: >> Hi >> >> I see this file has many inconsistencies against the one from DPDK >> (18.11.2). >> >> For example, this API: >> >> rte_flow_query(uint16_t port_id, >>        struct rte_flow *flow, >>       

Re: [ovs-dev] [PATCH] netdev: Dynamic per-port Flow API.

2019-11-19 Thread Ophir Munk
Hi Ilya, Thanks for the patch set which adds the dpif hint inside the netdev struct. It is really helpful. Our goal is to have flow_put() calls on vxlan devices, using the existing dpdk flow API. We added logic inside netdev_dpdk_flow_api_supported() to accept vxlan devices and indeed we see

Re: [ovs-dev] [PATCH 00/10] Add support for offloading CT datapath rules to TC

2019-11-19 Thread Paul Blakey
On 10/30/2019 3:37 PM, Roi Dayan wrote: > The following patchset introduces hardware offload of OVS connection > tracking datapath rules. > > OVS uses ct() and recirc() (recirculation) actions and recirc_id()/ct_state() > matches to support connection tracking. > > The datapath rules are in the

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Ilya Maximets
On 19.11.2019 17:16, Eelco Chaudron wrote: > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. >> >> Devices like 'veth' interfaces in Linux supports native

[ovs-dev] Dearest In Christ God Bless As You Get Back To Me

2019-11-19 Thread pastor David mike
-- Dearest In Christ, My name is PASTOR DAVID MIKE . I am 57 years old please check attachment to see my picture. I am writing to you from my hospital bed, because recently my doctor confirmed that my cancer disease which I have been treating for about 2 years now risks taking my life. To

Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Eelco Chaudron
On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Russell Bryant
On Tue, Nov 19, 2019 at 6:33 AM Numan Siddique wrote: > > On Tue, Nov 19, 2019 at 7:04 AM Russell Bryant wrote: > > > > While debugging some problems in a cluster using ovn-kubernetes, I > > noticed that we're creating two conflicting logical flows. These two > > flows only matched on the

Re: [ovs-dev] [PATCH 00/10] Add support for offloading CT datapath rules to TC

2019-11-19 Thread Marcelo Ricardo Leitner
On Tue, Nov 19, 2019 at 02:56:43PM +, Paul Blakey wrote: > On 10/30/2019 3:37 PM, Roi Dayan wrote: > > The following patchset introduces hardware offload of OVS connection > > tracking datapath rules. > > Hey guys, > > Any comments on the series? :) Ahm.. no. I'm backlogged with current

Re: [ovs-dev] [PATCH] dpdk: Deprecate pdump support.

2019-11-19 Thread David Marchand
On Mon, Nov 11, 2019 at 7:53 PM Ilya Maximets wrote: > > The conventional way for packet dumping in OVS is to use ovs-tcpdump > that works via traffic mirroring. DPDK pdump could probably be used > for some lower level debugging, but it is not commonly used for > various reasons. > > There are

Re: [ovs-dev] [PATCH v3] dpif-netdev: log rxq assignment in isolated pmd

2019-11-19 Thread Ilya Maximets
Thanks for a new version. Applied to master. BTW, few nits about patch submission for a next time: * Commonly in OVS patch subject should start with a capital letter and end with a period, i.e. 'dpif-netdev: Log ... pmd.' * Please, don't send patches in reply to the previous version. This

[ovs-dev] [PATCH ovn] Skip IPv6 NS packets in router egress SNAT pipeline

2019-11-19 Thread numans
From: Numan Siddique When ovn-controller injects IPv6 NS packet to learn the mac, in some setups it is observed that the packet is dropped by ovs-vswitchd. We see below logs 2019-11-18T10:42:17.973Z|1|ofproto_dpif_upcall(handler6)|INFO|received packet on unassociated datapath port

Re: [ovs-dev] 答复: [openvswitch.org代发] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread Ilya Maximets
On 19.11.2019 10:00, Yi Yang (杨燚)-云服务集团 wrote: > Hi, Ilya > > Can you explain what kernel limitations are for TCP for veth? I can't > understand why veth has such limitations only for TCP. I saw a veth bug > (https://tech.vijayp.ca/linux-kernel-bug-delivers-corrupt-tcp-ip-data-to-mes >

Re: [ovs-dev] [PATCH ovn] northd: Match IPv4 or IPv6 for MAC resolution

2019-11-19 Thread Numan Siddique
On Tue, Nov 19, 2019 at 7:04 AM Russell Bryant wrote: > > While debugging some problems in a cluster using ovn-kubernetes, I > noticed that we're creating two conflicting logical flows. These two > flows only matched on the destination MAC address. It was not > deterministic whether you'd hit

Re: [ovs-dev] [PATCH v5 ovn 2/4] ovn-controller: Add per node states to I-P engine.

2019-11-19 Thread Dumitru Ceara
On Tue, Nov 19, 2019 at 1:41 AM Han Zhou wrote: > > Thanks Dumitru. Please see my comments inline below. > > On Mon, Nov 18, 2019 at 6:07 AM Dumitru Ceara wrote: > > > > This commit transforms the 'changed' field in struct engine_node in a > > 'state' field. Possible node states are: > > -

[ovs-dev] 答复: [openvswitch.org代发] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.

2019-11-19 Thread 杨�D
Hi, Ilya Can you explain what kernel limitations are for TCP for veth? I can't understand why veth has such limitations only for TCP. I saw a veth bug (https://tech.vijayp.ca/linux-kernel-bug-delivers-corrupt-tcp-ip-data-to-mes os-kubernetes-docker-containers-4986f88f7a19) but it has been fixed

Re: [ovs-dev] [PATCH v5 ovn 1/4] ovn-controller: Refactor I-P engine_run() tracking.

2019-11-19 Thread Dumitru Ceara
Hi Han, Thanks for reviewing this series. On Mon, Nov 18, 2019 at 8:49 PM Han Zhou wrote: > > > > On Mon, Nov 18, 2019 at 6:07 AM Dumitru Ceara wrote: > > > > This commit simplifies the logic of calling engine_run and engine_need_run > > in > > order to reduce the number of external variables