Re: [lwip-users] Poor RX performance, misconfigured lwipopts?
Sergio, Thank you for taking a look and thank you for the suggestions. After considering your idea about testing with just UDP I decided to try with ICMP packet loss measurements and found out that indeed it was my driver that was dropping frames on occasion. Ugh. Fixing my driver-side code has resolved all of the issues previously mentioned. - Joseph -- Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] Poor RX performance, misconfigured lwipopts?
I forgot to mention that I'm running 2.1.2, but I doubt that will make a difference. Thanks. -- Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
[lwip-users] Poor RX performance, misconfigured lwipopts?
Hello everyone, I've been troubleshooting an RX performance issue for a couple of weeks now and while I've made incremental gains and improved my own driver-side code, I'm still missing something big here and I suspect that I'm likely just misusing lwIP or have a bad config. Any help would be greatly appreciated. Setup: - Port: Unix - Hardware: Modern desktop-class hardware for both RX and TX - API: Socket (I know the raw API is more performant). - Protocol of concern: TCP Test setup: For each of the following tests I'm sending and receiving blobs of data of exponentially increasing size (from 4mb to 256mb). All sizes exhibit roughly the same problem over the course of their TCP stream. - (1) OS native sockets TX -> OS native sockets RX (for baseline comparison) - (2) lwIP socket API TX -> OS native RX (no issue here) - (3) OS native TX -> lwIP socket RX (poor performance) - (4) lwIP socket TX -> lwIP socket RX (exceedingly poor performance) - In the capture file I have attached (11.7.7.15) is the native OS stack sending to lwIP at (11.7.7.130) I have tested against macOS and linux stacks, both react similarly to the odd lwIP behavior. Observations: In all test configurations (1/2/3/4) at home on my LAN I get roughly comparable throughput and everything behaves just fine. However when I test from a workstation at my office to home things change. For test (2) I see about 90% of baseline, but for test (3) I usually see 25% of baseline (very rarely it performs around 90% like test (2)). And test (4) is absolutely horrendous with about 5% of baseline. When looking at a packet capture (attached), I see a smattering of DUP ACKS, Retransmissions, and in some cases ACKs that seem out of order or extremely old but that are not duplicates! It appears that eventually the sending side decides to reduce the segment size. I've turned on DEBUG types and observed delayed ACK from lwIP I've also turned on STATS and am not noticing any errors whatsoever. Conclusions: - Given that I only see this from my office I am suspecting that either packet loss or the increased latency is creating a situation that my current lwIP config isn't handling well. - Since I can observe >95% of baseline when on my LAN I doubt there is a bottleneck in my code or lwIP itself. - Since the performance seems to get worse with the addition of (lwIP RX) on either side I'm suspecting that if I fix my issue in test (3) it should also fix test (4). Theories: - After much reading a common theme of dropped frames comes up, I've inspected and simplified my ethernet driver to the point where I don't believe this is a possibility, especially given how well it performs on my LAN. - I've read that TCP_TMR_INTERVAL is tick based and not based on an actual timer. I've toyed around with this value lowering it to 1 and raising it to 4000 but I feel this is a shortsighted approach asking for trouble. I've looked at the unix port and it does look like it's using a clock so I'm somewhat confused on this point. - Initially I thought this could be a window scaling issue so I've bumped it up to a pretty high value for testing. Questions: - Am I out in the weeds? - What else can I do to narrow down the issue? - What are some reasonable values to explore in lwipopts.h given my setup? Thanks in advance for any assistance! - Joseph lwipopts.h: #define LWIP_MTU 2800 #define LWIP_CHKSUM_ALGORITHM 2 // memory #define MEMP_NUM_NETCONN 1024 #define MEMP_NUM_NETBUF 2 #define MEMP_NUM_TCPIP_MSG_API 64 #define MEMP_NUM_TCPIP_MSG_INPKT64 #define PBUF_POOL_SIZE 128 #define TCP_DEFAULT_LISTEN_BACKLOG 0xff // arp #define ARP_TABLE_SIZE 64 #define ARP_MAXAGE 300 #define ARP_QUEUEING1 #define ARP_QUEUE_LEN 3 // ip #define IP_REASS_MAXAGE 15 #define IP_REASS_MAX_PBUFS 32 // tcp #define TCP_TMR_INTERVAL100 #define TCP_WND 0x7fff8 #define TCP_MAXRTX 12 #define TCP_SYNMAXRTX 12 #define LWIP_TCP_SACK_OUT 1 #define LWIP_TCP_MAX_SACK_NUM 4 #define TCP_MSS (LWIP_MTU - 40) #define TCP_SND_BUF (64 * TCP_MSS) #define TCP_SND_QUEUELEN(64 * (2 * (TCP_SND_BUF/TCP_MSS))) #define TCP_SNDLOWAT(0x - (4*TCP_MSS) - 1) #define TCP_SNDQUEUELOWAT LWIP_MAX(((TCP_SND_QUEUELEN)/2), 5) #define TCP_WND_UPDATE_THRESHOLDLWIP_MIN((TCP_WND / 4), (TCP_MSS * 4)) #define LWIP_WND_SCALE 1 #define TCP_RCV_SCALE 3 // tcpip #define TCPIP_MBOX_SIZE 0 #define LWIP_TCPIP_CORE_LOCKING 1 #define LWIP_TCPIP_CORE_LOCKING_INPUT 1 // netconn #define LWIP_NETCONN_FULLDUPLEX 0 // netif #define LWIP_SINGLE_NETIF 0 #define
Re: [lwip-users] Socket API: Only six connections getting Accepted
Looks like you need to set your MEMP_NUM_TCP_PCB_LISTEN to something higher. -- Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] netif setup for IPv6 traffic (static /128 address assignment)
Upon closer inspection it might have been a copy/paste error on my part. I am using it in TCP/IP thread mode in this case and after replacing ethernet_input with tcpip_input in the following line: netif_add(, NULL, NULL, NULL, NULL, netif_init_6, tcpip_input); It works as intended now. I do appreciate your time, Simon. Thanks. -- Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] netif setup for IPv6 traffic (static /128 address assignment)
Thanks for responding. I've tried using netif_ip6_addr_set() per your
suggestion but while it sets the address correctly for the netif, it doesn't
seem to take me any closer to an operational state. I'm just not sure where
to find a good example to follow. Can you clarify "mixing responsibilities"?
Here's a snippet which might be closer to standard usage (but still exhibits
the same problem).
Am I doing something out of order? or does lwIP not support assigning a
static ipv6 address of this form?
Thanks.
static err_t netif_init(struct netif *netif)
{
netif->hwaddr_len = 6;
netif->name[0]= 'l';
netif->name[1]= '0'+lwipInterfacesCount;
netif->linkoutput = lwip_eth_tx;
netif->output = etharp_output;
netif->output_ip6 = ethip6_output;
netif->mtu= MY_MTU;
netif->flags = NETIF_FLAG_BROADCAST
| NETIF_FLAG_ETHARP
| NETIF_FLAG_ETHERNET
| NETIF_FLAG_IGMP
| NETIF_FLAG_MLD6
| NETIF_FLAG_LINK_UP
| NETIF_FLAG_UP;
_mac.copyTo(netif->hwaddr, netif->hwaddr_len);
netif->hwaddr_len = sizeof(netif->hwaddr);
return ERR_OK;
}
void setup(...) {
static ip6_addr_t ipaddr;
memcpy(&(ipaddr.addr), ip.rawIpData(), sizeof(ipaddr.addr));
netif_add(, NULL, NULL, NULL, NULL, netif_init, ethernet_input);
netif.state = this;
s8_t idx = 1;
netif_ip6_addr_set(, 1, );
netif_set_status_callback(, netif_status_callback);
netif_set_default();
netif_set_up();
netif_set_link_up();
}
--
Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html
___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
[lwip-users] netif setup for IPv6 traffic (static /128 address assignment)
Hello, I'm trying to assign a static /128 address to a netif but I think I'm doing something wrong. I'm using 2.0.3 (also tested latest mater). I can see packets are being correctly received but when ethip6_output() is called it can't seem to find the proper interface to send out on. I've stepped through the code for a bit and it seems like it can't get what it needs from the ND6 module, almost as if there are no destination cache/hints. Does my initialization code look sane? Do I need to set additional flags or enable a module of some sort? Thanks in advance guys. // For instance, I want to assign: fde4:da74:55b2:b9ee:6a99:930f:26b9:2df0 static ip6_addr_t ipaddr; memcpy(&(ipaddr.addr), ip.rawIpData(), sizeof(ipaddr.addr)); lwipdev->mtu = MY_MTU; lwipdev->name[0] = 'l'; lwipdev->name[1] = '0'+lwipInterfacesCount; lwipdev->hwaddr_len = 6; lwipdev->linkoutput = lwip_eth_tx; lwipdev->ip6_autoconfig_enabled = 1; mac.copyTo(lwipdev->hwaddr, lwipdev->hwaddr_len); netif_add(lwipdev, NULL, NULL, NULL, NULL, tapif_init, ethernet_input); lwipdev->state = this; netif_create_ip6_linklocal_address(lwipdev, 1); s8_t idx = 1; netif_add_ip6_address(lwipdev, , ); netif_set_default(lwipdev); netif_set_up(lwipdev); netif_set_link_up(lwipdev); netif_ip6_addr_set_state(lwipdev, 1, IP6_ADDR_TENTATIVE); lwipdev->output_ip6 = ethip6_output; -- Sent from: http://lwip.100.n7.nabble.com/lwip-users-f3.html ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] lwIP 2.0.0 IPV6 Thread Awareness
Thanks for the advice, Dirk. I've taken a look and it appears that this is designed to be used with NO_SYS==0, currently for my application I have NO_SYS==1. It appears that I need to do some significant re-engineering of the core's locking mechanism to make this work for my context, but I suspect that's a bad assumption. Any ideas? Whatever the solution I arrive at I'll detail it here for future reference. Thanks. -- View this message in context: http://lwip.100.n7.nabble.com/lwIP-2-0-0-IPV6-Thread-Awareness-tp27528p27569.html Sent from the lwip-users mailing list archive at Nabble.com. ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
[lwip-users] lwIP 2.0.0 IPV6 Thread Awareness
Hello again everyone,
I've got a question about thread-safety using 2.0.0 in IPV6 mode. I've tried
searching for a solution on here but I'm somewhat stumped and could use some
help.
Prior to this I've been using the 1.4.1 core in IPV4 mode, each call to the
core was surrounded with a mutex. Like such:
inline void __tcp_tmr(void) throw() { Mutex::Lock _l(_lock); return
_tcp_tmr(); }
This functioned very well (as a prototype) using one global mutex for all
calls. However after upgrading to 2.0.0 and using it in IPV6 mode, I get a
number of deadlock scenarios. I've resolved one of them by introducing a
separate mutex for pbuf_alloc (say, _lock_mem) and one for everything else
(_lock). This works 99% of the time, but I keep getting into the following
quagmire:
On line 1093 of tcp_out.c I've observed that useg is NULL. I've read that
this is due to a thread conflict in almost all cases. So I've tried
introducing finer granularity to my locks but I can't seem to pin down a
solution.
Is there any sort of guideline to how calls should be locked? Short of
diagramming out the entire core and finding every possible conflict and
determining some set of locks I can use, I'm not sure how to go about this.
Any help would be greatly appreciated!
- Joseph
--
View this message in context:
http://lwip.100.n7.nabble.com/lwIP-2-0-0-IPV6-Thread-Awareness-tp27528.html
Sent from the lwip-users mailing list archive at Nabble.com.
___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] lwIP 2.0.0 IPv6 Support (specifically interface initialization)
Dirk, thank you. This was a big help and worked perfectly. Just for future reference in case anyone else comes across the thread, my netif initialization code ended up taking the form: IP6_ADDR2(, a, b, c, d, e, f, g, h); interface6.mtu = _mtu; interface6.name[0] = 't'; interface6.name[1] = 'p'; interface6.hwaddr_len = 6; interface6.linkoutput = low_level_output; interface6.ip6_autoconfig_enabled = 1; netif_create_ip6_linklocal_address(, 1); netif_add(, NULL, tapif_init, ethernet_input); netif_set_default(); netif_set_up(); netif_ip6_addr_set_state(, 1, IP6_ADDR_TENTATIVE); _mac.copyTo(interface6.hwaddr, interface6.hwaddr_len); ip6_addr_copy(ip_2_ip6(interface6.ip6_addr[1]), addr6); interface6.output_ip6 = ethip6_output; interface6.state = this; interface6.flags = NETIF_FLAG_LINK_UP | NETIF_FLAG_UP; -- View this message in context: http://lwip.100.n7.nabble.com/lwIP-2-0-0-IPv6-Support-specifically-interface-initialization-tp27397p27444.html Sent from the lwip-users mailing list archive at Nabble.com. ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
[lwip-users] lwIP 2.0.0 IPv6 Support (specifically interface initialization)
Hello everyone! I've got a few questions about IPv6 support. First, I've decided to try to use 2.0.0 RC2 for IPv6 support (is this wise?). Second, I've got a few questions about how to properly initialize an interface for IPv6. I have experience working with lwip 1.4.1 in IPv4 mode with no problem. However, I've spent a while trying to piece together clues for IPv6 but I can't seem to find much in the way of documentation or examples. Maybe you guys can send me in the right direction. Here's my initialization code (modeled loosely after how I'd do things in IPv4 mode). static ip6_addr_t addr6; IP6_ADDR2(, 0xfd56, 0x5799, 0xd8f6, 0x1238, 0x8c99, 0x93b4, 0x9d8e, 0x24f6); ip_2_ip6(_addr[0])->addr[0] = addr6.addr[0]; ip_2_ip6(_addr[0])->addr[1] = addr6.addr[1]; ip_2_ip6(_addr[0])->addr[2] = addr6.addr[2]; ip_2_ip6(_addr[0])->addr[3] = addr6.addr[3]; netif_ip6_addr_set_state(, 0, IP6_ADDR_TENTATIVE); netif_add(, NULL, tapif_init, ethernet_input); interface6.state = this; interface6.output_ip6 = _etharp_output; interface6.mtu = _mtu; interface6.name[0] = 't'; interface6.name[1] = 'p'; interface6.hwaddr_len = 6; _mac.copyTo(interface6.hwaddr, interface.hwaddr_len); netif_set_default(); netif_set_up(); where IP6_ADDR2 is a macro I took and renamed from lwIP 1.4.1, which for some reason is removed from lwIP 2.0.0? So, in short, this initialization code is faulty/incorrect in some way but I can't figure out how. Any suggestions? Thanks in advance! -- View this message in context: http://lwip.100.n7.nabble.com/lwIP-2-0-0-IPv6-Support-specifically-interface-initialization-tp27397.html Sent from the lwip-users mailing list archive at Nabble.com. ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
