Re: Missing bytes on serial port
> Hmmm... the UART device will closed when the kernel thread is killed, right?
> When the device is closed, it is also flushed I believe. If those two
> things are true, then data loss would be expected, right?
Yes, the UART is closed just before we perform a thread_delete(0).
Data loss is accounted for in the protocol. We send a known packet to the ESP
and it then responds to the request. This happens several times (we try 40
times). The logic analyser is showing the traffic between the chips and I have
manually checked the packets and the data looks good.
> Note that for a UART console, stdin is inherited so that the UART is not
> closed when the kernel thread exits.
The UART is opened at the start of the thread not prior to the thread being
started.
We are doing the following:
int start_uart_thread()
{
// Configure the thread properties
handle = kthread_create(… monitor_thread …);
}
void monitor_thread(…)
{
int uart = open(….);
// Setup signalling.
while (!shutting_down) { read and process data }
close(uart);
kthread_delete(0);
}
In this case is does the inheritance go both ways? As in if I open the UART
in the kthread does the parent thread keep it open?
Regards,
Mark
—
Mark Stevens
mark.stev...@wildernesslabs.co
> On 10 May 2024, at 05:38, Gregory Nutt wrote:
>
>
> On 5/9/2024 10:23 PM, Mark Stevens wrote:
>> The issue arises if I try to kill the kernel thread reading from the UART
>> and then try to start a user thread reading from the same UART.
>
> Hmmm... the UART device will closed when the kernel thread is killed, right?
> When the device is closed, it is also flushed I believe. If those two
> things are true, then data loss would be expected, right?
>
> Note that for a UART console, stdin is inherited so that the UART is not
> closed when the kernel thread exits.
>
>> Also, the issue can be reproduced within the first 15 seconds after board
>> reset so I am confident that the issue will not be related to long term
>> drift.
>
> From what I have read, any kind of slow drift is not possible due to the
> start and stop bits. Those transitions are necessary to synchronize on each
> byte. I also read that in the event of bad start bit, several bytes could be
> lost while the UART recovers.
>
> But that does not seem to be these issue here.
Re: Missing bytes on serial port
On 5/9/2024 10:23 PM, Mark Stevens wrote: The issue arises if I try to kill the kernel thread reading from the UART and then try to start a user thread reading from the same UART. Hmmm... the UART device will closed when the kernel thread is killed, right? When the device is closed, it is also flushed I believe. If those two things are true, then data loss would be expected, right? Note that for a UART console, stdin is inherited so that the UART is not closed when the kernel thread exits. Also, the issue can be reproduced within the first 15 seconds after board reset so I am confident that the issue will not be related to long term drift. From what I have read, any kind of slow drift is not possible due to the start and stop bits. Those transitions are necessary to synchronize on each byte. I also read that in the event of bad start bit, several bytes could be lost while the UART recovers. But that does not seem to be these issue here.
Re: Missing bytes on serial port
Greg, Both pieces of code work fine if they are executed on their own so I am confident that UART connection is good. The issue arises if I try to kill the kernel thread reading from the UART and then try to start a user thread reading from the same UART. Also, the issue can be reproduced within the first 15 seconds after board reset so I am confident that the issue will not be related to long term drift. Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co > On 9 May 2024, at 22:51, Gregory Nutt wrote: > > This problem is reported for a lot a platforms and seems to be a hardware > issue, usually associated with FIFOs and buffers. > > If you rule everything else out, then you can also consider some of the more > unusual causes. On some hardware, small differences in BAUD can result in > the kind of data loss you describe. Other hardware is more resilient. > > Since you are using two different MCUs, there is a high probability that the > BAUD rates are not exactly identical and you may be relying on marginally > sized start and stop bits to keep in synchronization. The half bit times > typically used in the start and stop bits normally account for this. > > Here is a long discussion of some hardware behaviors: > https://www.eevblog.com/forum/beginners/uart-question/ Searching for "uart > data loss due to small baud differences" finds several others. > > On 5/9/2024 3:20 PM, Mark Stevens wrote: >> I’m not writing to the UART - I am reading. >> >> Regards, >> Mark >> --- >> Mark Stevens >> Blog: blog.mark-stevens.co.uk >> >> >>> On 9 May 2024, at 17:40, Mark Stevens >>> wrote: >>> >>> This is a direct connection between the two chips on a PCB. >>> >>> Regards, >>> Mark >>> — >>> Mark Stevens >>> mark.stev...@wildernesslabs.co >>> >>> >>> >>> >>> >>> On 9 May 2024, at 17:38, Bill Rees wrote: I've seen this problem before which revolved around flow control; essentially soft versus hard flow control (xmit off/ xmit on) Are you using a null modem cable? If not that may give you the accuracy you're looking for, else hardware flow control is the only other possibility if it is flow control. Bill On 5/9/2024 9:24 AM, Tomek CEDRO wrote: > On Thu, May 9, 2024 at 6:15 PM Mark Stevens > wrote: >> Yes, I am sure both side are configured correctly. >> If I run the kernel code only then all works as expected. >> If I run user space code alone all works as expected. >> The problems happen when I transition from kernel use of the UART to >> user space use of the UART. >> I have also connected a logic analyser to the system and all looks good. >> Also, my current problem is NuttX reading data not sending it. Sending >> may also be a problem but I have not got that far at the moment. > Which UART do you use? What happens when you use different UART? Are > you sure it does not interfere with console? > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >>
Re: Missing bytes on serial port
This problem is reported for a lot a platforms and seems to be a hardware issue, usually associated with FIFOs and buffers. If you rule everything else out, then you can also consider some of the more unusual causes. On some hardware, small differences in BAUD can result in the kind of data loss you describe. Other hardware is more resilient. Since you are using two different MCUs, there is a high probability that the BAUD rates are not exactly identical and you may be relying on marginally sized start and stop bits to keep in synchronization. The half bit times typically used in the start and stop bits normally account for this. Here is a long discussion of some hardware behaviors: https://www.eevblog.com/forum/beginners/uart-question/ Searching for "uart data loss due to small baud differences" finds several others. On 5/9/2024 3:20 PM, Mark Stevens wrote: I’m not writing to the UART - I am reading. Regards, Mark --- Mark Stevens Blog: blog.mark-stevens.co.uk On 9 May 2024, at 17:40, Mark Stevens wrote: This is a direct connection between the two chips on a PCB. Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co On 9 May 2024, at 17:38, Bill Rees wrote: I've seen this problem before which revolved around flow control; essentially soft versus hard flow control (xmit off/ xmit on) Are you using a null modem cable? If not that may give you the accuracy you're looking for, else hardware flow control is the only other possibility if it is flow control. Bill On 5/9/2024 9:24 AM, Tomek CEDRO wrote: On Thu, May 9, 2024 at 6:15 PM Mark Stevens wrote: Yes, I am sure both side are configured correctly. If I run the kernel code only then all works as expected. If I run user space code alone all works as expected. The problems happen when I transition from kernel use of the UART to user space use of the UART. I have also connected a logic analyser to the system and all looks good. Also, my current problem is NuttX reading data not sending it. Sending may also be a problem but I have not got that far at the moment. Which UART do you use? What happens when you use different UART? Are you sure it does not interfere with console? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
I’m not writing to the UART - I am reading. Regards, Mark --- Mark Stevens Blog: blog.mark-stevens.co.uk > On 9 May 2024, at 17:40, Mark Stevens > wrote: > > This is a direct connection between the two chips on a PCB. > > Regards, > Mark > — > Mark Stevens > mark.stev...@wildernesslabs.co > > > > > > >> On 9 May 2024, at 17:38, Bill Rees wrote: >> >> >> I've seen this problem before which revolved around flow control; >> essentially soft versus hard flow control (xmit off/ xmit on) >> >> Are you using a null modem cable? If not that may give you the accuracy >> you're looking for, else hardware flow control is the only other possibility >> if it is flow control. >> >> Bill >> >> On 5/9/2024 9:24 AM, Tomek CEDRO wrote: >>> On Thu, May 9, 2024 at 6:15 PM Mark Stevens >>> wrote: Yes, I am sure both side are configured correctly. If I run the kernel code only then all works as expected. If I run user space code alone all works as expected. The problems happen when I transition from kernel use of the UART to user space use of the UART. I have also connected a logic analyser to the system and all looks good. Also, my current problem is NuttX reading data not sending it. Sending may also be a problem but I have not got that far at the moment. >>> Which UART do you use? What happens when you use different UART? Are >>> you sure it does not interfere with console? >>> >>> -- >>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >> >
Re: Missing bytes on serial port
When writing to the port, write() should indicate number of bytes written. Are you checking the return value and is it less than expected? Note that write() enqueues the bytes but returns before write complete. If port is closed before hardware finishes shifting all the bits out, message will be truncated. Note that most MCUs have errata related to UART completion: usually the "busy" bit (or whatever a particular micro calls it) indicates transmission done before it actually shifts out the last bits. On Thu, May 9, 2024 at 12:41 PM Mark Stevens wrote: > This is a direct connection between the two chips on a PCB. > > Regards, > Mark > — > Mark Stevens > mark.stev...@wildernesslabs.co > > > > > > > > On 9 May 2024, at 17:38, Bill Rees > wrote: > > > > > > I've seen this problem before which revolved around flow control; > essentially soft versus hard flow control (xmit off/ xmit on) > > > > Are you using a null modem cable? If not that may give you the accuracy > you're looking for, else hardware flow control is the only other > possibility if it is flow control. > > > > Bill > > > > On 5/9/2024 9:24 AM, Tomek CEDRO wrote: > >> On Thu, May 9, 2024 at 6:15 PM Mark Stevens > >> wrote: > >>> Yes, I am sure both side are configured correctly. > >>> If I run the kernel code only then all works as expected. > >>> If I run user space code alone all works as expected. > >>> The problems happen when I transition from kernel use of the UART to > user space use of the UART. > >>> I have also connected a logic analyser to the system and all looks > good. > >>> Also, my current problem is NuttX reading data not sending it. > Sending may also be a problem but I have not got that far at the moment. > >> Which UART do you use? What happens when you use different UART? Are > >> you sure it does not interfere with console? > >> > >> -- > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > >
Re: Missing bytes on serial port
This is a direct connection between the two chips on a PCB. Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co > On 9 May 2024, at 17:38, Bill Rees wrote: > > > I've seen this problem before which revolved around flow control; essentially > soft versus hard flow control (xmit off/ xmit on) > > Are you using a null modem cable? If not that may give you the accuracy > you're looking for, else hardware flow control is the only other possibility > if it is flow control. > > Bill > > On 5/9/2024 9:24 AM, Tomek CEDRO wrote: >> On Thu, May 9, 2024 at 6:15 PM Mark Stevens >> wrote: >>> Yes, I am sure both side are configured correctly. >>> If I run the kernel code only then all works as expected. >>> If I run user space code alone all works as expected. >>> The problems happen when I transition from kernel use of the UART to user >>> space use of the UART. >>> I have also connected a logic analyser to the system and all looks good. >>> Also, my current problem is NuttX reading data not sending it. Sending may >>> also be a problem but I have not got that far at the moment. >> Which UART do you use? What happens when you use different UART? Are >> you sure it does not interfere with console? >> >> -- >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
Re: Missing bytes on serial port
I've seen this problem before which revolved around flow control; essentially soft versus hard flow control (xmit off/ xmit on) Are you using a null modem cable? If not that may give you the accuracy you're looking for, else hardware flow control is the only other possibility if it is flow control. Bill On 5/9/2024 9:24 AM, Tomek CEDRO wrote: On Thu, May 9, 2024 at 6:15 PM Mark Stevens wrote: Yes, I am sure both side are configured correctly. If I run the kernel code only then all works as expected. If I run user space code alone all works as expected. The problems happen when I transition from kernel use of the UART to user space use of the UART. I have also connected a logic analyser to the system and all looks good. Also, my current problem is NuttX reading data not sending it. Sending may also be a problem but I have not got that far at the moment. Which UART do you use? What happens when you use different UART? Are you sure it does not interfere with console? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
On Thu, May 9, 2024 at 6:26 PM Mark Stevens wrote: > I can’t use a different UART, this is a direct connection between the two > chips on the PCB. I mean different internal UARTn/USARTn peripheral, configured to use the same physical connection, verify RX/TX BUFSIZE, and/or verify/eliminate possible console interference, see: https://nuttx.apache.org/docs/latest/platforms/arm/stm32f1/boards/stm32vldiscovery/index.html#uart -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
I can’t use a different UART, this is a direct connection between the two chips on the PCB. Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co > On 9 May 2024, at 17:24, Tomek CEDRO wrote: > > On Thu, May 9, 2024 at 6:15 PM Mark Stevens > wrote: >> Yes, I am sure both side are configured correctly. >> If I run the kernel code only then all works as expected. >> If I run user space code alone all works as expected. >> The problems happen when I transition from kernel use of the UART to user >> space use of the UART. >> I have also connected a logic analyser to the system and all looks good. >> Also, my current problem is NuttX reading data not sending it. Sending may >> also be a problem but I have not got that far at the moment. > > Which UART do you use? What happens when you use different UART? Are > you sure it does not interfere with console? > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
On Thu, May 9, 2024 at 6:15 PM Mark Stevens wrote: > Yes, I am sure both side are configured correctly. > If I run the kernel code only then all works as expected. > If I run user space code alone all works as expected. > The problems happen when I transition from kernel use of the UART to user > space use of the UART. > I have also connected a logic analyser to the system and all looks good. > Also, my current problem is NuttX reading data not sending it. Sending may > also be a problem but I have not got that far at the moment. Which UART do you use? What happens when you use different UART? Are you sure it does not interfere with console? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
One extra piece of information that has occurred to me, I am using a `SIGINT` kill signal to tell the kernel thread that it should close the connection. Regards, Mark --- Mark Stevens Blog: blog.mark-stevens.co.uk > On 9 May 2024, at 17:11, Tomek CEDRO wrote: > > On Thu, May 9, 2024 at 5:40 PM Mark Stevens > wrote: >> For example, I am seeing on the logic analyser the ESP sending the >> following hex bytes: >> c0 01 08 04 00 07 12 20 55 00 00 00 00 c0 >> NuttX is seeing: >> c0 01 04 20 55 00 00 c0 > > Can you add small delay between each byte transmitted? > > Can you divide transmitted data into chunks with a small delay between them? > > By adjusting delay between bytes and adjusting chunk size and delay > between chunks you may identify timing and/or configuration problems. > > Are you sure port have exactly the same configuration on both sides > (i.e. 115200-8-N-1)? > > Do you use cable for transmission? If so wow long is the cable between > devices? > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
Yes, I am sure both side are configured correctly. If I run the kernel code only then all works as expected. If I run user space code alone all works as expected. The problems happen when I transition from kernel use of the UART to user space use of the UART. I have also connected a logic analyser to the system and all looks good. Also, my current problem is NuttX reading data not sending it. Sending may also be a problem but I have not got that far at the moment. Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co > On 9 May 2024, at 17:11, Tomek CEDRO wrote: > > On Thu, May 9, 2024 at 5:40 PM Mark Stevens > wrote: >> For example, I am seeing on the logic analyser the ESP sending the >> following hex bytes: >> c0 01 08 04 00 07 12 20 55 00 00 00 00 c0 >> NuttX is seeing: >> c0 01 04 20 55 00 00 c0 > > Can you add small delay between each byte transmitted? > > Can you divide transmitted data into chunks with a small delay between them? > > By adjusting delay between bytes and adjusting chunk size and delay > between chunks you may identify timing and/or configuration problems. > > Are you sure port have exactly the same configuration on both sides > (i.e. 115200-8-N-1)? > > Do you use cable for transmission? If so wow long is the cable between > devices? > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
On Thu, May 9, 2024 at 5:40 PM Mark Stevens wrote: > For example, I am seeing on the logic analyser the ESP sending the following > hex bytes: > c0 01 08 04 00 07 12 20 55 00 00 00 00 c0 > NuttX is seeing: > c0 01 04 20 55 00 00 c0 Can you add small delay between each byte transmitted? Can you divide transmitted data into chunks with a small delay between them? By adjusting delay between bytes and adjusting chunk size and delay between chunks you may identify timing and/or configuration problems. Are you sure port have exactly the same configuration on both sides (i.e. 115200-8-N-1)? Do you use cable for transmission? If so wow long is the cable between devices? -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: Missing bytes on serial port
It is random. So in a 50-ish byte packet I will see sequences of missing bytes which can be 1, 2 or event 3 bytes long missing. For example, I am seeing on the logic analyser the ESP sending the following hex bytes: c0 01 08 04 00 07 12 20 55 00 00 00 00 c0 NuttX is seeing: c0 01 04 20 55 00 00 c0 Regards, Mark — Mark Stevens mark.stev...@wildernesslabs.co > On 9 May 2024, at 14:56, Nathan Hartman wrote: > > On Thu, May 9, 2024 at 3:31 AM Mark Stevens > wrote: > >> So we have a two chip board: >> >> * STM32 running NuttX (v7.5 I believe) >> * ESP32 acting as a coprocessor running custom firmware >> >> The STM32 runs the show and the ESP32 provides services to the STM32 code. >> >> In normal run mode, NuttX has a kernel thread that reads data from the >> ESP32 over UART (/dev/ttyS2) and then processes the data. This is working >> fine as is. >> >> The UART is configured to use a 512 byte buffer. >> >> Every now and then we want to upload new firmware to the ESP32. This is >> done by a user mode thread and it goes through the following steps: >> >> * Signals to the kernel thread that it should close the UART and exit. >> * Opens the serial port >> * Starts the programming sequence >> >> If we try to do this then the user mode thread misses bytes in the byte >> stream. >> >> Kernel mode thread only: >> When the system starts then this thread works fine, no bytes are lost. >> >> User mode thread only: >> If we do not start the kernel mode thread then the programming works fine, >> no bytes are lost. >> >> Both threads: >> Starting the kernel works fine, we do not miss any bytes. The kernel >> thread can be stopped, the UART is closed correctly. >> >> The user thread can open the serial port correctly after the kernel thread >> has stopped but now it misses bytes. >> >> So we know the individual threads work as expected when used on their own >> but not together. >> >> How anyone seen this or have any advice on how we can resolve the issue? >> >> Regards, >> Mark >> __ >> mark.stev...@wildernesslabs.co >> >> >> >> > > Which bytes are missing? Are they the ones at the beginning of the message, > or the end? > > Nathan
Re: Missing bytes on serial port
On Thu, May 9, 2024 at 3:31 AM Mark Stevens wrote: > So we have a two chip board: > > * STM32 running NuttX (v7.5 I believe) > * ESP32 acting as a coprocessor running custom firmware > > The STM32 runs the show and the ESP32 provides services to the STM32 code. > > In normal run mode, NuttX has a kernel thread that reads data from the > ESP32 over UART (/dev/ttyS2) and then processes the data. This is working > fine as is. > > The UART is configured to use a 512 byte buffer. > > Every now and then we want to upload new firmware to the ESP32. This is > done by a user mode thread and it goes through the following steps: > > * Signals to the kernel thread that it should close the UART and exit. > * Opens the serial port > * Starts the programming sequence > > If we try to do this then the user mode thread misses bytes in the byte > stream. > > Kernel mode thread only: > When the system starts then this thread works fine, no bytes are lost. > > User mode thread only: > If we do not start the kernel mode thread then the programming works fine, > no bytes are lost. > > Both threads: > Starting the kernel works fine, we do not miss any bytes. The kernel > thread can be stopped, the UART is closed correctly. > > The user thread can open the serial port correctly after the kernel thread > has stopped but now it misses bytes. > > So we know the individual threads work as expected when used on their own > but not together. > > How anyone seen this or have any advice on how we can resolve the issue? > > Regards, > Mark > __ > mark.stev...@wildernesslabs.co > > > > Which bytes are missing? Are they the ones at the beginning of the message, or the end? Nathan
Missing bytes on serial port
So we have a two chip board: * STM32 running NuttX (v7.5 I believe) * ESP32 acting as a coprocessor running custom firmware The STM32 runs the show and the ESP32 provides services to the STM32 code. In normal run mode, NuttX has a kernel thread that reads data from the ESP32 over UART (/dev/ttyS2) and then processes the data. This is working fine as is. The UART is configured to use a 512 byte buffer. Every now and then we want to upload new firmware to the ESP32. This is done by a user mode thread and it goes through the following steps: * Signals to the kernel thread that it should close the UART and exit. * Opens the serial port * Starts the programming sequence If we try to do this then the user mode thread misses bytes in the byte stream. Kernel mode thread only: When the system starts then this thread works fine, no bytes are lost. User mode thread only: If we do not start the kernel mode thread then the programming works fine, no bytes are lost. Both threads: Starting the kernel works fine, we do not miss any bytes. The kernel thread can be stopped, the UART is closed correctly. The user thread can open the serial port correctly after the kernel thread has stopped but now it misses bytes. So we know the individual threads work as expected when used on their own but not together. How anyone seen this or have any advice on how we can resolve the issue? Regards, Mark __ mark.stev...@wildernesslabs.co
