Re: [fpc-pascal] Reading Serial Hex Data
El 28/12/2020 a las 2:02, James Richters via fpc-pascal escribió: I think I figured out why my writeln's are causing an issue.. they are introducing a delay between the SerWrite and SerRead... and the device I am reading is timing out and sending it's response a second time. Hello, None of the serial devices I have seen do something like that, in fact usually they do exactly the opposite, fire and forget, or in other words send a message and do not check for errors unless they wait for an answer from the host. Your problem seems to be or a documentation misinterpretation or the connection or the device is not using any kind of flow control. I was under the impression that there was a hardware buffer on the serial ports, my packets are very small, some only a few bytes, but even those are not going into any kind of a hardware buffer. Hardware buffers in serial device is 16 bytes (In fact 14 for receive) at most and it is managed by the Windows serial driver to garantee no byte is lost. The operation scheme is something like: 1- Serial byte received 2- IRQ signal issued 3- Windows serial driver read serial port and copy byte(s) to driver buffer. 4- Signal IRQ as handled. The 16 bytes hardware (FIFO) buffer is present to garantee that if steps between 2 and 4 takes too much time (this is time scaled in 1990's CPU power) no byte is lost. If an Intel 8088 can read 115200bps without any byte lost think what a today computer can read. In order to perform tests there are programs where you create an hex string to be sent and inspect the answer from the device. Read documentation about how to open the port, specially about the flow control sync. XOn/XOff, RTS/CTS. In example some need that you raise the RTS line and wait for CTS line to be high before write, if you do not wait the device can misinterpret your request and hang or answer something stupid, this is automatically done by the Windows serial driver if you select the RTS/CTS flow control when open the port. Have a nice 2021! -- ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Mon, 28 Dec 2020 14:12:02 +0100, Jean SUZINEAU via fpc-pascal
wrote:
>Le 28/12/2020 à 13:16, Bo Berglund via fpc-pascal a écrit :
>> Synchronize(CallEvent); //Supply received data in FBuffer to
>> caller
>
>You are using TThread.Synchronize.
>
>In a console app, it's likely that somewhere else in your main thread
>you'll need to call regularly Classes.CheckSynchronize to allow the
>actual execution of CallEvent.
>
>In my case, linux server dll called by a 4js 4GL console program (no GUI
>on linux server), I need to call regurlarly Classes.CheckSynchronize
>froml 4gl through function exported by the pascal code.
>If I don't do this, the CallEvent is nether called.
>
Where I use the fpserial class for communications over RS232 there is
an "eternal" loop in the main program, which for a console program is
what is really running:
{$IFDEF UNIX} //What to do for system messages
procedure handleSignal(signum: CInt); cdecl;
begin
case signum of
SIGHUP : bSHup := true;
SIGTERM : bSTerm := true;
SIGINT : bSInt := true
end;
end;
{$ENDIF}
program
//Create all objects and start processes
{$IFDEF UNIX} //Make sure there is a signal handler
fpSignal(SigTerm, SignalHandler(@handleSignal));
fpSignal(SigInt, SignalHandler(@handleSignal));
fpSignal(SigHup, SignalHandler(@handleSignal));
{$ENDIF}
//Now wait for signals:
While not (bSTerm or bSInt or bsHup) do //React to system signals
begin
//Here is where the server runs as defined elsewhere
//Eternal loop to wait for system messages
Sleep(1); //To not hog the CPU
CheckSynchronize; //Enable events to fire off handlers
end;
//cleanup and exit
...
end.
The program is intended to run as a Linux service and so it reacts to
the service manager's messages which are coming in via the
handleSignal procedure.
The main action is controlled by socket handlers started by the
preliminaries in the main program and which are triggered by events
from various sources like TCP/IP connections, timers etc.
--
Bo Berglund
Developer in Sweden
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
Le 28/12/2020 à 13:16, Bo Berglund via fpc-pascal a écrit : Synchronize(CallEvent); //Supply received data in FBuffer to caller You are using TThread.Synchronize. In a console app, it's likely that somewhere else in your main thread you'll need to call regularly Classes.CheckSynchronize to allow the actual execution of CallEvent. In my case, linux server dll called by a 4js 4GL console program (no GUI on linux server), I need to call regurlarly Classes.CheckSynchronize froml 4gl through function exported by the pascal code. If I don't do this, the CallEvent is nether called. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 18:20:32 -0500, James Richters via fpc-pascal
wrote:
>>On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal
>> wrote:
>>
>>>I'm trying to figure out how to read a packet of HEX Bytes of an
>>>unknown length that have no specific termination character over a
>>>serial port as efficiently as practical.
>>
>>What exactly do you mean by HEX Bytes?
>>Hex transfers using two characters for each byte? like 1F A4 etc?
>
>What I meant by 'Hex Data' is that I am reading data in the form of bytes but
>they do not represent ASCII characters.
>So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage
>return. There is no specific code for end of line,
>the last two bytes are a checksum, but I won't know they are the last ones
>until I'm done reading.
>I'm reading all the data available, then analyzing it.
>
This is NOT wahat is generally meant by hex data, hex data means bytes
are encoded using the ASCII characters 0..9, A..F thus expanding data
size by x2 while making it possible to send over channels that use
control characters to handle transfer details such as line feeds,
start of text and end of text etc.
What you have is straight binary data
>>
>>Is your program a console or GUI program?
>
>This is a console application.
I suspected as much. Many of the serial components available are
geared towards GUI apps and so for instance LazSerial links in stuff
like Forms and menus and the like, which causes exceptrions when
running in a straight GUI-less environment like a Linux server.
So I created a class named fpserial where I packaged the functions I
needed and used the built-in Serial unit as the basic execution
handler.
This class uses an event model to supply incoming data, which are
received inside a thread and supplied via a synchronized call to the
event function in the main thread of the program.
Being a console program it has a loop which includes CheckSynchronize
to handle this.
After I did that I have had no problems talking to external hardware
sending and receiving data in binary streams.
But there is a "protocol" for the transfers implemented to allow some
control. It looks something like this:
Where length is a 4-byte integer containing the length of the transfer
body and checksum is a 16 bit sum over all of the bytes inside the
body (excluding the checksum).
The messages are sent following a handshake between the two sides.
My thread code looks like this:
{ TComPortReadThread }
TComPortReadThread=class(TThread)
private
FBuffer: TBytes;
FReadPacketSize: integer;
FReadTimeout: integer;
public
MustDie: boolean;
Owner: TFpSerialPort;
property ReadPacketSize: integer read FReadPacketSize write
FReadPacketSize; //How many bytes to read in each operation
property ReadTimeout: integer read FReadTimeout write
FReadTimeout;//Max time to wait for data in thread
protected
procedure CallEvent;
procedure Execute; override;
published
property Terminated;
end;
implementation
{ TComPortReadThread }
procedure TComPortReadThread.CallEvent;
begin
if Assigned(Owner.FOnRxData) then
begin
Owner.FOnRxData(Owner, FBuffer);
end;
end;
procedure TComPortReadThread.Execute;
var
Cnt: integer;
begin
try
SetLength(FBuffer, BUFFERSIZE); //Set buffer size to 8192 bytes.
while not MustDie do
begin
cnt := SerReadTimeout(Owner.FHandle, FBuffer[0],
FReadPacketSize, FReadTimeout); //Read FReadPacketSize bytes with
timeout of FReadTimeout ms
if cnt > 0 then
begin
SetLength(FBuffer, cnt); //Reduce size to fit received data
Synchronize(CallEvent); //Supply received data in FBuffer to
caller
SetLength(FBuffer, BUFFERSIZE); //Reset buffer size to 8192
bytes.
end;
end;
finally
Terminate;
end;
end;
HTH
--
Bo Berglund
Developer in Sweden
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
Op 12/27/2020 om 4:47 PM schreef James Richters via fpc-pascal: I'm trying to figure out how to read a packet of HEX Bytes of an unknown length that have no specific termination character over a serial port as efficiently as practical. The packet ends when there is just no more data to read. I have something that is working that I wrote using SerRead from the Serial unit in FPC, and it works at 9600bps, but the method I am using will not work well at anything higher unless I can figure out how to a very accurate very short delay. At 9600, I need a delay just a little less than 1ms, so I use Sleep(1) for my delay, but since 1ms is the minimum for sleep, and sleep itself is not really that accurate, I am doing a lot more delay than I need to in order to make sure it works. You don't mention OS, but for Windows no sleep should not be necessary, and is unnecessarily delaying. Look at delphi tcomport which has a waitforevent based solution. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
I think I figured out why my writeln's are causing an issue.. they are introducing a delay between the SerWrite and SerRead... and the device I am reading is timing out and sending it's response a second time. This seems to be the case because it's re-sending bytes that have already been received. I was under the impression that there was a hardware buffer on the serial ports, my packets are very small, some only a few bytes, but even those are not going into any kind of a hardware buffer. I'm on Windows, and I remember seeing something about buffers in device manager.. and so I went into the settings On the serial port I am using... and it turns out, it doesn't have a buffer... because it's a USB to serial converter. I checked the settings on real serial port, and those indeed have a hardware buffer I can turn on, but the USB serial port Does not. I think the best solution is to make the separate thread that reads everything available all the time and create a software Buffer. James -Original Message- From: fpc-pascal On Behalf Of James Richters via fpc-pascal Sent: Sunday, December 27, 2020 6:21 PM To: bo.bergl...@gmail.com; 'FPC-Pascal users discussions' Cc: James Richters Subject: Re: [fpc-pascal] Reading Serial Hex Data What I meant by 'Hex Data' is that I am reading data in the form of bytes but they do not represent ASCII characters. So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage return. There is no specific code for end of line, the last two bytes are a checksum, but I won't know they are the last ones until I'm done reading. I'm reading all the data available, then analyzing it. This is a console application. I have managed to get a bit more accurate timing by setting a time_variable := NOW and then repeating until (Now-TimeVariable) >= Minimum_Time_I_Want. I'm not sure what the resolution of NOW is, but it's working quite a bit better than Sleep() I think I do need to put this in a thread.. I notice one strange behavior... if I just run my program, then it works exactly as I expect, and I'm not wasting a lot of time waiting for the end of Data. However if I do a writeln to the console before the function that reads the serial port, then I am missing the end of the data sometimes, I have to put an additional delay equal to about 10 Bytes of data to make it work... but without the writeln's I need only to wait for the amount of time of 2 Bytes, which is what I expected. I'm not using any threads yet, so this is strange behavior, doing a writeln before I even start the procedure that does the serial communication should not effect it at all, because the writeln should be completely done before it starts the serial communication, yet this is what I observe... If I don't do the writeln's everything is fine, if I do them, my data timing is off... it's still there, but I have to wait longer to make sure I get all the data that there is. What seems to be happening is that I'm I'm using SerRead, it does seem that I never miss anything, it's just that I'm not detecting the end of data accurately if I have writeln's right before I read the serial port. James -Original Message- From: fpc-pascal On Behalf Of Bo Berglund via fpc-pascal Sent: Sunday, December 27, 2020 5:08 PM To: fpc-pascal@lists.freepascal.org Cc: Bo Berglund Subject: Re: [fpc-pascal] Reading Serial Hex Data On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an >unknown length that have no specific termination character over a >serial port as efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the >Serial unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
What I meant by 'Hex Data' is that I am reading data in the form of bytes but they do not represent ASCII characters. So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage return. There is no specific code for end of line, the last two bytes are a checksum, but I won't know they are the last ones until I'm done reading. I'm reading all the data available, then analyzing it. This is a console application. I have managed to get a bit more accurate timing by setting a time_variable := NOW and then repeating until (Now-TimeVariable) >= Minimum_Time_I_Want. I'm not sure what the resolution of NOW is, but it's working quite a bit better than Sleep() I think I do need to put this in a thread.. I notice one strange behavior... if I just run my program, then it works exactly as I expect, and I'm not wasting a lot of time waiting for the end of Data. However if I do a writeln to the console before the function that reads the serial port, then I am missing the end of the data sometimes, I have to put an additional delay equal to about 10 Bytes of data to make it work... but without the writeln's I need only to wait for the amount of time of 2 Bytes, which is what I expected. I'm not using any threads yet, so this is strange behavior, doing a writeln before I even start the procedure that does the serial communication should not effect it at all, because the writeln should be completely done before it starts the serial communication, yet this is what I observe... If I don't do the writeln's everything is fine, if I do them, my data timing is off... it's still there, but I have to wait longer to make sure I get all the data that there is. What seems to be happening is that I'm I'm using SerRead, it does seem that I never miss anything, it's just that I'm not detecting the end of data accurately if I have writeln's right before I read the serial port. James -Original Message- From: fpc-pascal On Behalf Of Bo Berglund via fpc-pascal Sent: Sunday, December 27, 2020 5:08 PM To: fpc-pascal@lists.freepascal.org Cc: Bo Berglund Subject: Re: [fpc-pascal] Reading Serial Hex Data On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an >unknown length that have no specific termination character over a >serial port as efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the >Serial unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an unknown >length that have no specific termination character over a serial port as >efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the Serial >unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 21:15:03 +0100, Jean SUZINEAU via fpc-pascal wrote: >I think TBlockSerial can work in your console program context. May be >TLazSerial can be more tricky to use in your context, you'll need to use >events/callbacks and FCL. I have struggled with LazSerial in a console program and it just is not intended for that usage. Won't work. Throws exceptions if run in a device without a GUI like a Linux server. But instead I used the built-in serial unit and packaged the whole reading stuff into a thread that will read the serial line constantly using the blocking read and supply the data to the main application as it arrives. Not a big deal to do and get working. You need a read timeout in the execute loop so that the thread can detect if it has been commanded to terminate, that is all I think. Also, for the OP: What do you mean by receiving hex data? Are you saying that you receive byte data as 2-character ASCII hex code? I.e. every byte you want to receive is represented by *two* characters in the incoming stream? In that case if you are not using any delimiters you will probably soon be out of sync and mixing most and least significant hex characters in the transmission so that for example 1a 2b 3c could become ..1 a2 b3 c.. Very dangerous situation. You really must use some kind of delimiter in the data flow... -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
May be using TBlockSerial from unit synaser from Ararat Synapse ? ( doc: http://synapse.ararat.cz/doc/help/synaser.TBlockSerial.html , download: http://synapse.ararat.cz/doku.php/download ). Given a variable (SynSer: TBlockSerial;), you can test if data is available with SynSer.CanReadEx(0) and then use SynSer.RecvPacket(Timeout) to get your data. I think it should work, I haven't used directly TBlockSerial this way, but through TLazSerial component for reading data from an Arduino in a program running on Windows, Linux x86, or Raspberry. I think TBlockSerial can work in your console program context. May be TLazSerial can be more tricky to use in your context, you'll need to use events/callbacks and FCL. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Reading Serial Hex Data
I'm trying to figure out how to read a packet of HEX Bytes of an unknown length that have no specific termination character over a serial port as efficiently as practical. The packet ends when there is just no more data to read. I have something that is working that I wrote using SerRead from the Serial unit in FPC, and it works at 9600bps, but the method I am using will not work well at anything higher unless I can figure out how to a very accurate very short delay. At 9600, I need a delay just a little less than 1ms, so I use Sleep(1) for my delay, but since 1ms is the minimum for sleep, and sleep itself is not really that accurate, I am doing a lot more delay than I need to in order to make sure it works. The procedure is: Check the serial port for a byte in a loop until you either get a byte or you get a timeout Then once you have one byte, you delay enough time for few bytes to be in the buffer Next read one byte at time, Delay at the correct rate for the baudrate until you run out of bytes or hit a maximum number of bytes. What is needed is to stop at a timeout in case there is nothing to read, OR to stop when maximum bytes are read OR stop when 2 bytes are missed meaning you ran out of data to read. But once you read one byte, you abandon the timeout. SerReadTimeout has a timeout, but it seems to be a timeout for the whole operation.. what I need is the first timeout to be a long timeout to wait for the device to respond, but once you get one byte, you cancel the timeout and never use it again.. now you read as many bytes as you can until you miss two in a row... then you stop. Either that or have two timeouts, one timeout to delay for receiving the first byte and the second timeout a way to tell when you have no more bytes left to read. It Seems like lots of people would want to read serial ports in this way, because you never know how many bytes are coming, and you don't want to always wait the entire timeout, that is just there to stop it in case there is no data. Then again, I guess most applications can read until they get a linefeed or carriage return, which I will not be. I'm thinking this has already been done somewhere and maybe there is already a solution that is tested and works, but I don't know where it is or what it's called. I could not even find any documentation of SerReadTimout. Here is pseudocode of the procedure that would be needed... the only thing I am really lacking is the accurate time delay. So if there is not already a function somewhere that works this way, then a recommendation on very fast and accurate timing delays would be helpful. repeat Read the serial port and check for a byte Delay the amount of time one byte should take Inc(Timeout); Until (Timeout=Max_Allowed) or Byte_ is_received If Byte is received Begin Count:=-0; String:=String+ByteRead; Inc(Bytes_Read); End; Delay enough time for a few more bytes to get in the buffer Repeat Read serial port Delay the amount of time one byte should take If Byte is received Begin Count:=-0; String:=String+ByteRead; Inc(Bytes_Read); End Else Inc(Count); Until MaxBytesRead Or (Count=2) James ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
