Hi Ananda. I am not aware of any workaround. I think it's a bug and it should be fixed in the driver or firmware.
вт, 14 авг. 2018 г. в 21:20, Ananda Sathyanarayana <[email protected]>: > > Hi Alex, Beilei, > > I am also facing the same issue. Do we have a possible solution/work-around > for this issue ? > > Really appreciate your response. > > Thanks, > Ananda > > > -----Original Message----- > From: users <[email protected]> On Behalf Of Alex Kiselev > Sent: Sunday, May 27, 2018 9:09 AM > To: Xing, Beilei <[email protected]>; [email protected]; Zhang, Qi Z > <[email protected]> > Subject: Re: [dpdk-users] problem with an additional MAC address on i40e > > Everything is fine with testmpd promisc mode, it was just my inattention. > The rest of my previous mail is still valid. > > > Hi Beilei. > > > I've found a way to reproduce the bug with the testpmd. > > > I beleive that the bug is that once an additional MAC is added to a > > i40e port the port stop receiving packets with source MAC addresses > > equal to the added MAC. > > > First, I modified testpmd and turned off promisc mode by commenting > > lines 2537, 2538 in testpmd.c > > > RTE_ETH_FOREACH_DEV(port_id) > > rte_eth_promiscuous_enable(port_id); > > > It's very strange, but without this modification even when promisc > > mode was turned off by the command "set promisc 0 off" > > a port was still receiving packets with any MAC addresses, for example > > 60:A4:4C:41:0A:F1 in the log below > > > set promisc 0 off > > set allmulti 0 on > > set fwd rxonly > > set verbose 1 > > start > > > h5 test-pmd # ./testpmd --proc-type=primary -c 0xF > > --lcores='0@0,1@1,2@2,3@3' --syslog='daemon' -n2 -w 0000:01:00.0 -w > > 0000:01:00.1 -- -i > > Interactive-mode selected > > Configuring Port 0 (socket 0) > > Port 0: 68:05:CA:30:61:80 > > Configuring Port 1 (socket 0) > > Port 1: 68:05:CA:30:61:81 > > Checking link statuses... > > Done > > testpmd> > > Port 0: LSC event > > > > Port 1: LSC event > > > > testpmd> set promisc 0 off > > testpmd> set allmulti 0 on > > testpmd> set fwd rxonly > > Set rxonly packet forwarding mode > > testpmd> set verbose 1 > > Change verbose level from 0 to 1 > > testpmd> start > > rxonly packet forwarding - ports=2 - cores=1 - > > streams=2 - NUMA support enabled, MP over anonymous pages disabled > > Logical Core 1 (socket 0) forwards packets on 2 streams: > > RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) > > peer=02:00:00:00:00:01 > > RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) > > peer=02:00:00:00:00:00 > > > > rxonly packet forwarding packets/burst=32 > > nb forwarding cores=1 - nb forwarding ports=2 > > port 0: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > port 1: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > testpmd> port 1/queue 0: received 1 packets > > src=60:A4:4C:41:0A:24 - dst=60:A4:4C:41:0A:F1 - > > type=0x0800 - length=98 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_ICMP - sw ptype: L2_ETHER L3_IPV4 > > - l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > .. > > .. > > .. > > > > Telling cores to stop... > > Waiting for lcores to finish... > > > > ---------------------- Forward statistics for port 0 > > ---------------------- > > RX-packets: 29 RX-dropped: 0 > > RX-total: 29 > > RX-error: 1 > > RX-nombufs: 0 > > TX-packets: 0 TX-dropped: 0 > > TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > ---------------------- Forward statistics for port 1 > > ---------------------- > > RX-packets: 29 RX-dropped: 0 > > RX-total: 29 > > TX-packets: 0 TX-dropped: 0 > > TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > +++++++++++++++ Accumulated forward statistics for all > > ports+++++++++++++++ > > RX-packets: 58 RX-dropped: 0 > > RX-total: 58 > > TX-packets: 0 TX-dropped: 0 > > TX-total: 0 > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Done. > > testpmd> show port info 0 > > > > ********************* Infos for port 0 > > ********************* > > MAC address: 68:05:CA:30:61:80 > > Driver name: net_i40e > > Connect to socket: 0 > > memory allocation on the socket: 0 > > Link status: up > > Link speed: 10000 Mbps > > Link duplex: full-duplex > > MTU: 1500 > > Promiscuous mode: disabled > > Allmulticast mode: enabled > > Maximum number of MAC addresses: 64 > > Maximum number of MAC addresses of hash filtering: 0 > > VLAN offload: > > strip on > > filter on > > qinq(extend) off > > Hash key size in bytes: 52 > > Redirection table size: 512 > > Supported flow types: > > ipv4-frag > > ipv4-tcp > > ipv4-udp > > ipv4-sctp > > ipv4-other > > ipv6-frag > > ipv6-tcp > > ipv6-udp > > ipv6-sctp > > ipv6-other > > l2_payload > > Max possible RX queues: 320 > > Max possible number of RXDs per queue: 4096 > > Min possible number of RXDs per queue: 64 > > RXDs number alignment: 32 > > Max possible TX queues: 320 > > Max possible number of TXDs per queue: 4096 > > Min possible number of TXDs per queue: 64 > > TXDs number alignment: 32 > > > Then I started main tests with an additional mac address. > > I added MAC 00:00:5E:00:01:0A on port 0 and started two VRRP group on > > a switch that begin transmitting packets destined to multicast MAC > > 01:00:5E:00:00:12. > > > First VRRP group were transmitting packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 > > > > and second VRRP group were transmitting packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 > > > testpmd should've received both packets since allmuticast mode was > > turned on, but it did receive only packets with source MAC > > 00:00:5E:00:01:0B. > > > The cause of that behavior is that MAC address 00:00:5E:00:01:0A was added > > to the port 0. > > Once I deleted MAC 00:00:5E:00:01:0A from the port, testpmd started > > receiving both packets. > > > > Log 1, additional MAC 00:00:5E:00:01:0A is added on the port 0. > > Testpmd is ONLY receiving packets with src 00:00:5E:00:01:0B and > > skipping packets with src 00:00:5E:00:01:0A. > > > set promisc 0 off > > set allmulti 0 on > > set fwd rxonly > > set verbose 1 > > mac_addr add 0 00:00:5E:00:01:0A > > start > > > h5 test-pmd # ./testpmd --proc-type=primary -c 0xF > > --lcores='0@0,1@1,2@2,3@3' --syslog='daemon' -n2 -w 0000:01:00.0 -w > > 0000:01:00.1 -- -i > > Interactive-mode selected > > Configuring Port 0 (socket 0) > > Port 0: 68:05:CA:30:61:80 > > Configuring Port 1 (socket 0) > > Port 1: 68:05:CA:30:61:81 > > Checking link statuses... > > Done > > testpmd> > > testpmd> > > Port 0: LSC event > > > > Port 1: LSC event > > > > testpmd> set promisc 0 off > > testpmd> set allmulti 0 on > > testpmd> set fwd rxonly > > Set rxonly packet forwarding mode > > testpmd> set verbose 1 > > Change verbose level from 0 to 1 > > testpmd> mac_addr add 0 00:00:5E:00:01:0A > > testpmd> start > > rxonly packet forwarding - ports=2 - cores=1 - streams=2 - > > NUMA support enabled, MP over anonymous pages disabled > > Logical Core 1 (socket 0) forwards packets on 2 streams: > > RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) > > peer=02:00:00:00:00:01 > > RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) > > peer=02:00:00:00:00:00 > > > > rxonly packet forwarding packets/burst=32 > > nb forwarding cores=1 - nb forwarding ports=2 > > port 0: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > port 1: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > testpmd> > > testpmd> port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=342 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=342 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=342 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=342 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=10:7B:EF:58:59:80 - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=316 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=10:7B:EF:58:59:80 - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=316 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=10:7B:EF:58:59:80 - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=316 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=10:7B:EF:58:59:80 - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=316 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - hw ptype: L2_ETHER_ARP - sw > > ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=D4:CA:6D:7C:D0:DC - dst=FF:FF:FF:FF:FF:FF - > > type=0x0806 - length=60 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER_ARP - sw ptype: L2_ETHER - l2_len=14 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > stport 0/queue 0: received 1 packets > > src=00:1A:E3:67:83:47 - dst=01:00:5E:00:00:0D - > > type=0x0800 - length=68 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > op > > Telling cores to stop... > > Waiting for lcores to finish... > > > > ---------------------- Forward statistics for port 0 > > ---------------------- > > RX-packets: 37 RX-dropped: 0 RX-total: 37 > > RX-error: 1 > > RX-nombufs: 0 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > ---------------------- Forward statistics for port 1 > > ---------------------- > > RX-packets: 38 RX-dropped: 0 RX-total: 38 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > +++++++++++++++ Accumulated forward statistics for all > > ports+++++++++++++++ > > RX-packets: 75 RX-dropped: 0 RX-total: 75 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Done. > > testpmd> > > > > Log 2, additional MAC 00:00:5E:00:01:0A is removed on the port 0. > > Testpmd is receiving both packets with src 00:00:5E:00:01:0A and > > 00:00:5E:00:01:0B > > > > testpmd> mac_addr remove 0 00:00:5E:00:01:0A > > testpmd> start > > rxonly packet forwarding - ports=2 - cores=1 - streams=2 - > > NUMA support enabled, MP over anonymous pages disabled > > Logical Core 1 (socket 0) forwards packets on 2 streams: > > RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) > > peer=02:00:00:00:00:01 > > RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) > > peer=02:00:00:00:00:00 > > > > rxonly packet forwarding packets/burst=32 > > nb forwarding cores=1 - nb forwarding ports=2 > > port 0: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > port 1: > > CRC stripping enabled > > RX queues=1 - RX desc=128 - RX free threshold=32 > > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > > TX queues=1 - TX desc=512 - TX free threshold=32 > > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > > TX RS bit threshold=32 - TXQ flags=0xf01 > > testpmd> port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 1/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - VLAN tci=0x3 - hw ptype: > > L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 > > L4_UDP - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_VLAN PKT_RX_L4_CKSUM_GOOD > > PKT_RX_IP_CKSUM_GOOD PKT_RX_VLAN_STRIPPED > > port 0/queue 0: received 1 packets > > src=F8:32:E4:72:61:1B - dst=FF:FF:FF:FF:FF:FF - > > type=0x0800 - length=174 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_UDP - sw ptype: L2_ETHER L3_IPV4 L4_UDP - > > l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > port 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > sport 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0B - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > toport 0/queue 0: received 1 packets > > src=00:00:5E:00:01:0A - dst=01:00:5E:00:00:12 - > > type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER > > L3_IPV4_EXT_UNKNOWN L4_NONFRAG - sw ptype: L2_ETHER L3_IPV4 - > > l2_len=14 - l3_len=20 - Receive queue=0x0 > > ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD > > p > > Telling cores to stop... > > Waiting for lcores to finish... > > > > ---------------------- Forward statistics for port 0 > > ---------------------- > > RX-packets: 18 RX-dropped: 0 RX-total: 18 > > RX-error: 1 > > RX-nombufs: 0 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > ---------------------- Forward statistics for port 1 > > ---------------------- > > RX-packets: 18 RX-dropped: 0 RX-total: 18 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ---------------------------------------------------------------------- > > ------ > > > > +++++++++++++++ Accumulated forward statistics for all > > ports+++++++++++++++ > > RX-packets: 36 RX-dropped: 0 RX-total: 36 > > TX-packets: 0 TX-dropped: 0 TX-total: 0 > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Done. > > testpmd> > > > > > h5 test-pmd # echo $RTE_SDK > > /usr/src/dpdk-stable-17.11.1 > > > h5 src # ethtool -i enp1s0f0 > > driver: i40e > > version: 2.4.6 > > firmware-version: 6.01 0x80003483 1.1747.0 > > bus-info: 0000:01:00.0 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > h5 src # > > h5 src # ethtool -i enp1s0f1 > > driver: i40e > > version: 2.4.6 > > firmware-version: 6.01 0x80003483 1.1747.0 > > bus-info: 0000:01:00.1 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > > -- > > Alex > > > >> What's the firmware version you used? Did you try testpmd? > > >> I tried testpmd with X710 on my side, the steps are: > testpmd>>> set fwd rxonly > testpmd>>> set verbose 1 > testpmd>>> start > testpmd>>> mac_addr add 0 00:00:5E:00:01:0A stop start > > >> After that, packets with destination 01:00:5E:00:00:12 still can be > >> received. > > >> Best Regards > >> Beilei Xing > > >>> -----Original Message----- > >>> From: Alex Kiselev [mailto:[email protected]] > >>> Sent: Tuesday, May 22, 2018 6:42 PM > >>> To: [email protected]; Xing, Beilei <[email protected]>; Zhang, Qi > >>> Z <[email protected]> > >>> Subject: problem with an additional MAC address on i40e > > >>> Hi. > > >>> I am facing an issue trying to receive packets destined to the MAC > >>> address > >>> ipv4_mcast_12 01:00:5E:00:00:12. > > >>> The problem is that once I add to a port additional MAC address > >>> 00:00:5E:00:01:0A using rte_eth_dev_mac_addr_add(), the port stop > >>> receiving packets destinated to the address 01:00:5E:00:00:12. When > >>> there is no additinal mac on a port everything is ok. Also, there is > >>> no such issue when I am using intel X520 nic (ixgbe), I am facing > >>> this behavior only with > >>> X710 (i40e) intel NIC. > > >>> DPDK ver dpdk-stable-17.11.1 > > > >>> -- > >>> Alex Kiselev. > > > > > > > > -- > С уважением, > Alex mailto:[email protected] > -- -- Kiselev Alexander
