Re: [lwip-users] long time tcp receive lwip packets lost
> I wrote the bridging part and for testing purposes I didn't implement > buffer to TCP to serial. So Some while receiving data some data piles > up in somewhere in tcp thread (I think). No, don't "think", debug, do know exactly where you have your problem, because it is not lwIP, it is you, and you are the only one who can solve it. > I wrote a python script that sends 2096 characters every second (some > lorem ipsum text). serial speed is 115200. So there is enough time to > send data and free memory. 2096 x 10 = 20960 bits/115200 bps = 182ms Yes, you should have a pulsing output assuming your TCP data has no problems in arrival and gets there on time. Have you (for example) moved a pin to know when your app runs and what it does ? Either you are not freeing memory correctly, or you are not assigning priorities and run time correctly in your OS, or your port is broken or your driver is broken (leaky). Do yourself some favors and - Study X.10 (the guys at ITU-T solved the serial packetization issue for you some decades ago) - Check your port and driver by running a known to work application: the examples. - Post the section of your code that interfaces with lwIP. Not my case, but you sure will find someone versed in socket API if you label your message with the correct subject. - Give details on your OS configuration for your particular app. Again, not my case, but there are some RTOS users lurking here, do use the "subject" field wisely. ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
Re: [lwip-users] long time tcp receive lwip packets lost
Hi, I am not sure I understand you. You need buffering basically when you send data from UART to TCP. The reasons are: Not to send small chunks of data in every TCP packet and secondly to read data from UART. Your UART does not have a FIFO and does not have any means to block or control the rate. If you block you loose data. So for that reason you must have a way to collect data while some of it is sent via TCP. I can suggest using the OS queue. On the UART side you write to end of queue in the ISR. >From the TCP side you read data and send it to the remote side. Regarding TCP to UART you do not need a buffer unless you are not able to send out all you get for some reason. TCP menages the rate and as soon as it frees the RX buffer it will inform the remote side about it. This way you control the rate. My bridging software was written for RAW API and handled 6 UART's. Every UART had two queues. One for RX one for TX. The only module I wrote using Socket API was a Websocket client. I do not have lots of experience with Socket API. Hope that helped. BR, Noam. From: lwip-users <lwip-users-bounces+noam=silrd@nongnu.org> on behalf of Erkan Ersoy <ice_ffiir...@yahoo.com> Sent: Sunday, September 18, 2016 7:45 PM To: Mailing list for lwIP users Subject: Re: [lwip-users] long time tcp receive lwip packets lost yes second part of mail was about uart-> tcp. Still I am not using buffer but lwip packs and sends it for me my mss setting is #define TCP_MSS 700 I am using socket API (I needed simultaneous sending and receiving but there is not enough documentation about non-blocking netconn) My current confusion is about tcp->uart: the bufferless configuration works fine at first but after a couple of hours It doesn't. It receives data as 80 byte small chunks (whole data is 2096 bytes small chunks are random parts in them) and the data piles in somewhere in lwip. after I stop sending data, I see it continues send data through uart then it stops. So i think it emptied its buffer but if I begin to send again behaviour is the same; until I restart device. And stats show no memory is overloaded. if I add buffer the problem may disappear for couple of hours up time but I think it will pile up data slower and become a problem in several days or something. On Sunday, September 18, 2016 7:27 PM, Noam Weissman <n...@silrd.com> wrote: Hi, 1. You must add buffering 2. RS232 is character based but TCP is packet based. As a result of the different data handling if you do not buffer data it means that for every byte you send from RS232 you send a TCP packet that has one data byte. That means you have a huge overhead on every byte. This is very un-efficient and causes the system to get very slow. I have implemented a Bridge, used a 255 byte buffer and set two parameters: size and time. Size: If you reach a predefined number of bytes you send them in one TCP packet. Time: If you do not get any new data into the TCP buffer after a predefined time you send what you have. This way you collect data from UART and send it in chunks/packets. For example you define 100 bytes as threshold or 30ms as time threshold. Every byte added from UART To the send buffer is counted. Once the counter reaches 100 or if no new byte is added after 30ms the TCP buffer is sent and parameters are cleared. I also suggest using a small MSS size (I use 536 bytes). BR, Noam. From: lwip-users [mailto:lwip-users-bounces+noam=silrd@nongnu.org] On Behalf Of Erkan Ersoy Sent: Saturday, September 17, 2016 1:42 PM To: Mailing List for LwIP Users Subject: [lwip-users] long time tcp receive lwip packets lost Hello I am making a serial tcp bridge device. it works in server mode I have freeRTOS 9, lwip 1.4.1 and STM32F107 device I wrote the bridging part and for testing purposes I didn't implement buffer to TCP to serial. So Some while receiving data some data piles up in somewhere in tcp thread (I think). I wrote a python script that sends 2096 characters every second (some lorem ipsum text). serial speed is 115200. So there is enough time to send data and free memory. But after couple of hours (it is random. sometimes 2 hours sometimes 4 and yesterday it was 12 hours) device cant receive whole data somewhat it is 80 byte parts.and it piles up in tcp part. And after I stop my script I continue to get data from my serial window a few seconds. After memory is empty if I start my script again still same problem. I enabled stats to see if I am getting out of memory but it looks fine. I couldn't find where my problem is. There is a web server and an udp server running in device (I didnt send request both server while testing) I am using socket api my stats before running test (it sends stat to seperate uart every 10 seconds) accept_function: newpcb->state: ESTABLISHED socket Connected ETHAR
Re: [lwip-users] long time tcp receive lwip packets lost
So I am already not using buffer for tcp to uart (my reasoning is tcp is faster
then UART so I need buffer I once used microchip tcp library; there I needed
it).
So I am not using buffer for TCP to UART and after couple of hours I have that
problem that I explained. I think I have some kind of memory leak.
On Sunday, September 18, 2016 10:51 PM, Noam Weissman <n...@silrd.com>
wrote:
#yiv6886118056 #yiv6886118056 -- P
{margin-top:0;margin-bottom:0;}#yiv6886118056 Hi,
I am not sure I understand you.
You need buffering basically when you send data from UART to TCP. The reasons
are:Not to send small chunks of data in every TCP packet and secondly to read
data fromUART. Your UART does not have a FIFO and does not have any means to
block or control
the rate. If you block you loose data. So for that reason you must have a way
to collect datawhile some of it is sent via TCP.
I can suggest using the OS queue. On the UART side you write to end of queue in
the ISR.From the TCP side you read data and send it to the remote side.
Regarding TCP to UART you do not need a buffer unless you are not able to send
outall you get for some reason. TCP menages the rate and as soon as it frees
the RX bufferit will inform the remote side about it. This way you control the
rate.
My bridging software was written for RAW API and handled 6 UART's. Every UART
had twoqueues. One for RX one for TX.
The only module I wrote using Socket API was a Websocket client. I do not have
lotsof experience with Socket API.
Hope that helped.
BR,Noam.
From: lwip-users <lwip-users-bounces+noam=silrd@nongnu.org> on behalf of
Erkan Ersoy <ice_ffiir...@yahoo.com>
Sent: Sunday, September 18, 2016 7:45 PM
To: Mailing list for lwIP users
Subject: Re: [lwip-users] long time tcp receive lwip packets lost yes second
part of mail was about uart-> tcp. Still I am not using buffer but lwip packs
and sends it for memy mss setting is
#define TCP_MSS 700
I am using socket API (I needed simultaneous sending and receiving but there
is not enough documentation about non-blocking netconn)
My current confusion is about tcp->uart:the bufferless configuration works fine
at first but after a couple of hours It doesn't. It receives data as 80 byte
small chunks (whole data is 2096 bytes small chunks are random parts in them)
and the data piles in somewhere in lwip. after I stop sending data, I see it
continues send data through uart then it stops. So i think it emptied its
buffer but if I begin to send again behaviour is the same; until I restart
device. And stats show no memory is overloaded.
if I add buffer the problem may disappear for couple of hours up time but I
think it will pile up data slower and become a problem in several days or
something.
On Sunday, September 18, 2016 7:27 PM, Noam Weissman <n...@silrd.com> wrote:
Hi, 1. You must add buffering2. RS232 is character based but TCP is
packet based. As a result of the different data handling if you do not buffer
data it means that for every byte you send fromRS232 you send a TCP packet that
has one data byte. That means you have a huge overhead on every byte.This is
very un-efficient and causes the system to get very slow. I have implemented a
Bridge, used a 255 byte buffer and set two parameters: size and time. Size: If
you reach a predefined number of bytes you send them in one TCP packet.Time: If
you do not get any new data into the TCP buffer after a predefined time you
send what you have. This way you collect data from UART and send it in
chunks/packets. For example you define 100 bytes as threshold or 30ms as time
threshold. Every byte added from UARTTo the send buffer is counted. Once the
counter reaches 100 or if no new byte is added after 30ms theTCP buffer is sent
and parameters are cleared. I also suggest using a small MSS size (I use 536
bytes). BR,Noam. From: lwip-users
[mailto:lwip-users-bounces+noam=silrd@nongnu.org]On Behalf Of Erkan Ersoy
Sent: Saturday, September 17, 2016 1:42 PM
To: Mailing List for LwIP Users
Subject: [lwip-users] long time tcp receive lwip packets lost Hello I am making
a serial tcp bridge device. it works in server mode I have freeRTOS 9, lwip
1.4.1 and STM32F107 device I wrote the bridging part and for testing purposes I
didn't implement buffer to TCP to serial. So Some while receiving data some
data piles up in somewhere in tcp thread (I think).I wrote a python script that
sends 2096 characters every second (some lorem ipsum text). serial speed is
115200. So there is enough time to send data and free memory. But after couple
of hours (it is random. sometimes 2 hours sometimes 4 and yesterday it was 12
hours) device cant receive whole data somewhat it is 80 byte parts.and it piles
up in tcp part. And after I stop my script I continue to get data from my
serial window a few seconds. After memory is empty if I start my script again
still
Re: [lwip-users] long time tcp receive lwip packets lost
Hi, 1. You must add buffering 2. RS232 is character based but TCP is packet based. As a result of the different data handling if you do not buffer data it means that for every byte you send from RS232 you send a TCP packet that has one data byte. That means you have a huge overhead on every byte. This is very un-efficient and causes the system to get very slow. I have implemented a Bridge, used a 255 byte buffer and set two parameters: size and time. Size: If you reach a predefined number of bytes you send them in one TCP packet. Time: If you do not get any new data into the TCP buffer after a predefined time you send what you have. This way you collect data from UART and send it in chunks/packets. For example you define 100 bytes as threshold or 30ms as time threshold. Every byte added from UART To the send buffer is counted. Once the counter reaches 100 or if no new byte is added after 30ms the TCP buffer is sent and parameters are cleared. I also suggest using a small MSS size (I use 536 bytes). BR, Noam. From: lwip-users [mailto:lwip-users-bounces+noam=silrd@nongnu.org] On Behalf Of Erkan Ersoy Sent: Saturday, September 17, 2016 1:42 PM To: Mailing List for LwIP Users Subject: [lwip-users] long time tcp receive lwip packets lost Hello I am making a serial tcp bridge device. it works in server mode I have freeRTOS 9, lwip 1.4.1 and STM32F107 device I wrote the bridging part and for testing purposes I didn't implement buffer to TCP to serial. So Some while receiving data some data piles up in somewhere in tcp thread (I think). I wrote a python script that sends 2096 characters every second (some lorem ipsum text). serial speed is 115200. So there is enough time to send data and free memory. But after couple of hours (it is random. sometimes 2 hours sometimes 4 and yesterday it was 12 hours) device cant receive whole data somewhat it is 80 byte parts.and it piles up in tcp part. And after I stop my script I continue to get data from my serial window a few seconds. After memory is empty if I start my script again still same problem. I enabled stats to see if I am getting out of memory but it looks fine. I couldn't find where my problem is. There is a web server and an udp server running in device (I didnt send request both server while testing) I am using socket api my stats before running test (it sends stat to seperate uart every 10 seconds) accept_function: newpcb->state: ESTABLISHED socket Connected ETHARP xmit: 5 recv: 43 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 32 IP xmit: 36 recv: 53 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 TCP xmit: 2 recv: 35 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 MEM HEAP avail: 10240 used: 68 max: 456 err: 0 MEM UDP_PCB avail: 6 used: 2 max: 3 err: 0 MEM TCP_PCB avail: 20 used: 1 max: 1 err: 0 MEM TCP_PCB_LISTEN avail: 5 used: 2 max: 2 err: 0 MEM TCP_SEG avail: 12 used: 0 max: 1 err: 0 MEM NETBUF avail: 8 used: 0 max: 0 err: 0 MEM NETCONN avail: 16 used: 4 max: 4 err: 0 MEM TCPIP_MSG_API avail: 8 used: 0 max: 0 err: 0 MEM TCPIP_MSG_INPKT avail: 8 used: 0 max: 3 err: 0 MEM SYS_TIMEOUT avail: 20 used: 6 max: 6 err: 0 MEM NETDB avail: 1 used: 0 max: 0 err: 0 MEM PBUF_REF/ROM avail: 25 used: 0 max: 0 err: 0 MEM PBUF_POOL avail: 12 used: 0 max: 5 err: 0 SYS sem.used: 4 sem.max: 5 sem.err: 0 mutex.used: 1 mutex.max: 1 mutex.err: 0 mbox.used: 5 mbox.max: 5 mbox.err: 0 And after the problem (Nothing seems suspicious except cachehit I could't figure out what it is) ETHARP xmit: 21 recv: 2811 fw: 0 drop: 9 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 9 opterr: 0 err: 0 cachehit: 41446 IP xmit: 41449 recv: 37649 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 TCP xmit: 1 recv: 36784 fw: 0 drop: 0 chkerr
[lwip-users] long time tcp receive lwip packets lost
Hello I am making a serial tcp bridge device. it works in server mode I have freeRTOS 9, lwip 1.4.1 and STM32F107 device I wrote the bridging part and for testing purposes I didn't implement buffer to TCP to serial. So Some while receiving data some data piles up in somewhere in tcp thread (I think).I wrote a python script that sends 2096 characters every second (some lorem ipsum text). serial speed is 115200. So there is enough time to send data and free memory. But after couple of hours (it is random. sometimes 2 hours sometimes 4 and yesterday it was 12 hours) device cant receive whole data somewhat it is 80 byte parts.and it piles up in tcp part. And after I stop my script I continue to get data from my serial window a few seconds. After memory is empty if I start my script again still same problem. I enabled stats to see if I am getting out of memory but it looks fine. I couldn't find where my problem is. There is a web server and an udp server running in device (I didnt send request both server while testing)I am using socket api my stats before running test (it sends stat to seperate uart every 10 seconds) accept_function: newpcb->state: ESTABLISHEDsocket Connected ETHARP xmit: 5 recv: 43 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 32 IP xmit: 36 recv: 53 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 TCP xmit: 2 recv: 35 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 MEM HEAP avail: 10240 used: 68 max: 456 err: 0 MEM UDP_PCB avail: 6 used: 2 max: 3 err: 0 MEM TCP_PCB avail: 20 used: 1 max: 1 err: 0 MEM TCP_PCB_LISTEN avail: 5 used: 2 max: 2 err: 0 MEM TCP_SEG avail: 12 used: 0 max: 1 err: 0 MEM NETBUF avail: 8 used: 0 max: 0 err: 0 MEM NETCONN avail: 16 used: 4 max: 4 err: 0 MEM TCPIP_MSG_API avail: 8 used: 0 max: 0 err: 0 MEM TCPIP_MSG_INPKT avail: 8 used: 0 max: 3 err: 0 MEM SYS_TIMEOUT avail: 20 used: 6 max: 6 err: 0 MEM NETDB avail: 1 used: 0 max: 0 err: 0 MEM PBUF_REF/ROM avail: 25 used: 0 max: 0 err: 0 MEM PBUF_POOL avail: 12 used: 0 max: 5 err: 0 SYS sem.used: 4 sem.max: 5 sem.err: 0 mutex.used: 1 mutex.max: 1 mutex.err: 0 mbox.used: 5 mbox.max: 5 mbox.err: 0 And after the problem (Nothing seems suspicious except cachehit I could't figure out what it is) ETHARP xmit: 21 recv: 2811 fw: 0 drop: 9 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 9 opterr: 0 err: 0 cachehit: 41446 IP xmit: 41449 recv: 37649 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 TCP xmit: 1 recv: 36784 fw: 0 drop: 0 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 MEM HEAP avail: 10240 used: 68 max: 456 err: 0 MEM UDP_PCB avail: 6 used: 2 max: 3 err: 0 MEM TCP_PCB avail: 20 used: 1 max: 1 err: 0 MEM TCP_PCB_LISTEN avail: 5 used: 2 max: 2 err: 0 MEM TCP_SEG avail: 12 used: 0 max: 1 err: 0 MEM NETBUF avail: 8 used: 0 max: 0 err: 0 MEM NETCONN avail: 16 used: 4 max: 4 err: 0 MEM TCPIP_MSG_API avail: 8 used: 0 max: 0 err: 0 MEM TCPIP_MSG_INPKT avail: 8 used: 0 max: 3 err: 0 MEM SYS_TIMEOUT avail: 20 used: 6 max: 6 err: 0 MEM NETDB avail: 1 used: 0 max: 0 err: 0 MEM PBUF_REF/ROM avail: 25 used: 0 max: 0 err: 0 MEM PBUF_POOL avail: 12 used: 0 max: 6 err: 0 SYS sem.used: 4 sem.max: 5 sem.err: 0 mutex.used: 1 mutex.max: 1 mutex.err: 0 mbox.used: 5 mbox.max: 5 mbox.err: 0 ___ lwip-users mailing list lwip-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/lwip-users
