Re: [fpc-pascal] Reading Serial Hex Data
El 28/12/2020 a las 2:02, James Richters via fpc-pascal escribió: I think I figured out why my writeln's are causing an issue.. they are introducing a delay between the SerWrite and SerRead... and the device I am reading is timing out and sending it's response a second time. Hello, None of the serial devices I have seen do something like that, in fact usually they do exactly the opposite, fire and forget, or in other words send a message and do not check for errors unless they wait for an answer from the host. Your problem seems to be or a documentation misinterpretation or the connection or the device is not using any kind of flow control. I was under the impression that there was a hardware buffer on the serial ports, my packets are very small, some only a few bytes, but even those are not going into any kind of a hardware buffer. Hardware buffers in serial device is 16 bytes (In fact 14 for receive) at most and it is managed by the Windows serial driver to garantee no byte is lost. The operation scheme is something like: 1- Serial byte received 2- IRQ signal issued 3- Windows serial driver read serial port and copy byte(s) to driver buffer. 4- Signal IRQ as handled. The 16 bytes hardware (FIFO) buffer is present to garantee that if steps between 2 and 4 takes too much time (this is time scaled in 1990's CPU power) no byte is lost. If an Intel 8088 can read 115200bps without any byte lost think what a today computer can read. In order to perform tests there are programs where you create an hex string to be sent and inspect the answer from the device. Read documentation about how to open the port, specially about the flow control sync. XOn/XOff, RTS/CTS. In example some need that you raise the RTS line and wait for CTS line to be high before write, if you do not wait the device can misinterpret your request and hang or answer something stupid, this is automatically done by the Windows serial driver if you select the RTS/CTS flow control when open the port. Have a nice 2021! -- ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Mon, 28 Dec 2020 14:12:02 +0100, Jean SUZINEAU via fpc-pascal wrote: >Le 28/12/2020 à 13:16, Bo Berglund via fpc-pascal a écrit : >> Synchronize(CallEvent); //Supply received data in FBuffer to >> caller > >You are using TThread.Synchronize. > >In a console app, it's likely that somewhere else in your main thread >you'll need to call regularly Classes.CheckSynchronize to allow the >actual execution of CallEvent. > >In my case, linux server dll called by a 4js 4GL console program (no GUI >on linux server), I need to call regurlarly Classes.CheckSynchronize >froml 4gl through function exported by the pascal code. >If I don't do this, the CallEvent is nether called. > Where I use the fpserial class for communications over RS232 there is an "eternal" loop in the main program, which for a console program is what is really running: {$IFDEF UNIX} //What to do for system messages procedure handleSignal(signum: CInt); cdecl; begin case signum of SIGHUP : bSHup := true; SIGTERM : bSTerm := true; SIGINT : bSInt := true end; end; {$ENDIF} program //Create all objects and start processes {$IFDEF UNIX} //Make sure there is a signal handler fpSignal(SigTerm, SignalHandler(@handleSignal)); fpSignal(SigInt, SignalHandler(@handleSignal)); fpSignal(SigHup, SignalHandler(@handleSignal)); {$ENDIF} //Now wait for signals: While not (bSTerm or bSInt or bsHup) do //React to system signals begin //Here is where the server runs as defined elsewhere //Eternal loop to wait for system messages Sleep(1); //To not hog the CPU CheckSynchronize; //Enable events to fire off handlers end; //cleanup and exit ... end. The program is intended to run as a Linux service and so it reacts to the service manager's messages which are coming in via the handleSignal procedure. The main action is controlled by socket handlers started by the preliminaries in the main program and which are triggered by events from various sources like TCP/IP connections, timers etc. -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
Le 28/12/2020 à 13:16, Bo Berglund via fpc-pascal a écrit : Synchronize(CallEvent); //Supply received data in FBuffer to caller You are using TThread.Synchronize. In a console app, it's likely that somewhere else in your main thread you'll need to call regularly Classes.CheckSynchronize to allow the actual execution of CallEvent. In my case, linux server dll called by a 4js 4GL console program (no GUI on linux server), I need to call regurlarly Classes.CheckSynchronize froml 4gl through function exported by the pascal code. If I don't do this, the CallEvent is nether called. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 18:20:32 -0500, James Richters via fpc-pascal wrote: >>On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal >> wrote: >> >>>I'm trying to figure out how to read a packet of HEX Bytes of an >>>unknown length that have no specific termination character over a >>>serial port as efficiently as practical. >> >>What exactly do you mean by HEX Bytes? >>Hex transfers using two characters for each byte? like 1F A4 etc? > >What I meant by 'Hex Data' is that I am reading data in the form of bytes but >they do not represent ASCII characters. >So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage >return. There is no specific code for end of line, >the last two bytes are a checksum, but I won't know they are the last ones >until I'm done reading. >I'm reading all the data available, then analyzing it. > This is NOT wahat is generally meant by hex data, hex data means bytes are encoded using the ASCII characters 0..9, A..F thus expanding data size by x2 while making it possible to send over channels that use control characters to handle transfer details such as line feeds, start of text and end of text etc. What you have is straight binary data >> >>Is your program a console or GUI program? > >This is a console application. I suspected as much. Many of the serial components available are geared towards GUI apps and so for instance LazSerial links in stuff like Forms and menus and the like, which causes exceptrions when running in a straight GUI-less environment like a Linux server. So I created a class named fpserial where I packaged the functions I needed and used the built-in Serial unit as the basic execution handler. This class uses an event model to supply incoming data, which are received inside a thread and supplied via a synchronized call to the event function in the main thread of the program. Being a console program it has a loop which includes CheckSynchronize to handle this. After I did that I have had no problems talking to external hardware sending and receiving data in binary streams. But there is a "protocol" for the transfers implemented to allow some control. It looks something like this: Where length is a 4-byte integer containing the length of the transfer body and checksum is a 16 bit sum over all of the bytes inside the body (excluding the checksum). The messages are sent following a handshake between the two sides. My thread code looks like this: { TComPortReadThread } TComPortReadThread=class(TThread) private FBuffer: TBytes; FReadPacketSize: integer; FReadTimeout: integer; public MustDie: boolean; Owner: TFpSerialPort; property ReadPacketSize: integer read FReadPacketSize write FReadPacketSize; //How many bytes to read in each operation property ReadTimeout: integer read FReadTimeout write FReadTimeout;//Max time to wait for data in thread protected procedure CallEvent; procedure Execute; override; published property Terminated; end; implementation { TComPortReadThread } procedure TComPortReadThread.CallEvent; begin if Assigned(Owner.FOnRxData) then begin Owner.FOnRxData(Owner, FBuffer); end; end; procedure TComPortReadThread.Execute; var Cnt: integer; begin try SetLength(FBuffer, BUFFERSIZE); //Set buffer size to 8192 bytes. while not MustDie do begin cnt := SerReadTimeout(Owner.FHandle, FBuffer[0], FReadPacketSize, FReadTimeout); //Read FReadPacketSize bytes with timeout of FReadTimeout ms if cnt > 0 then begin SetLength(FBuffer, cnt); //Reduce size to fit received data Synchronize(CallEvent); //Supply received data in FBuffer to caller SetLength(FBuffer, BUFFERSIZE); //Reset buffer size to 8192 bytes. end; end; finally Terminate; end; end; HTH -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
Op 12/27/2020 om 4:47 PM schreef James Richters via fpc-pascal: I'm trying to figure out how to read a packet of HEX Bytes of an unknown length that have no specific termination character over a serial port as efficiently as practical. The packet ends when there is just no more data to read. I have something that is working that I wrote using SerRead from the Serial unit in FPC, and it works at 9600bps, but the method I am using will not work well at anything higher unless I can figure out how to a very accurate very short delay. At 9600, I need a delay just a little less than 1ms, so I use Sleep(1) for my delay, but since 1ms is the minimum for sleep, and sleep itself is not really that accurate, I am doing a lot more delay than I need to in order to make sure it works. You don't mention OS, but for Windows no sleep should not be necessary, and is unnecessarily delaying. Look at delphi tcomport which has a waitforevent based solution. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
I think I figured out why my writeln's are causing an issue.. they are introducing a delay between the SerWrite and SerRead... and the device I am reading is timing out and sending it's response a second time. This seems to be the case because it's re-sending bytes that have already been received. I was under the impression that there was a hardware buffer on the serial ports, my packets are very small, some only a few bytes, but even those are not going into any kind of a hardware buffer. I'm on Windows, and I remember seeing something about buffers in device manager.. and so I went into the settings On the serial port I am using... and it turns out, it doesn't have a buffer... because it's a USB to serial converter. I checked the settings on real serial port, and those indeed have a hardware buffer I can turn on, but the USB serial port Does not. I think the best solution is to make the separate thread that reads everything available all the time and create a software Buffer. James -Original Message- From: fpc-pascal On Behalf Of James Richters via fpc-pascal Sent: Sunday, December 27, 2020 6:21 PM To: bo.bergl...@gmail.com; 'FPC-Pascal users discussions' Cc: James Richters Subject: Re: [fpc-pascal] Reading Serial Hex Data What I meant by 'Hex Data' is that I am reading data in the form of bytes but they do not represent ASCII characters. So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage return. There is no specific code for end of line, the last two bytes are a checksum, but I won't know they are the last ones until I'm done reading. I'm reading all the data available, then analyzing it. This is a console application. I have managed to get a bit more accurate timing by setting a time_variable := NOW and then repeating until (Now-TimeVariable) >= Minimum_Time_I_Want. I'm not sure what the resolution of NOW is, but it's working quite a bit better than Sleep() I think I do need to put this in a thread.. I notice one strange behavior... if I just run my program, then it works exactly as I expect, and I'm not wasting a lot of time waiting for the end of Data. However if I do a writeln to the console before the function that reads the serial port, then I am missing the end of the data sometimes, I have to put an additional delay equal to about 10 Bytes of data to make it work... but without the writeln's I need only to wait for the amount of time of 2 Bytes, which is what I expected. I'm not using any threads yet, so this is strange behavior, doing a writeln before I even start the procedure that does the serial communication should not effect it at all, because the writeln should be completely done before it starts the serial communication, yet this is what I observe... If I don't do the writeln's everything is fine, if I do them, my data timing is off... it's still there, but I have to wait longer to make sure I get all the data that there is. What seems to be happening is that I'm I'm using SerRead, it does seem that I never miss anything, it's just that I'm not detecting the end of data accurately if I have writeln's right before I read the serial port. James -Original Message- From: fpc-pascal On Behalf Of Bo Berglund via fpc-pascal Sent: Sunday, December 27, 2020 5:08 PM To: fpc-pascal@lists.freepascal.org Cc: Bo Berglund Subject: Re: [fpc-pascal] Reading Serial Hex Data On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an >unknown length that have no specific termination character over a >serial port as efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the >Serial unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
What I meant by 'Hex Data' is that I am reading data in the form of bytes but they do not represent ASCII characters. So if I read a $0D, it's just a byte $0D, it doesn't represent a carriage return. There is no specific code for end of line, the last two bytes are a checksum, but I won't know they are the last ones until I'm done reading. I'm reading all the data available, then analyzing it. This is a console application. I have managed to get a bit more accurate timing by setting a time_variable := NOW and then repeating until (Now-TimeVariable) >= Minimum_Time_I_Want. I'm not sure what the resolution of NOW is, but it's working quite a bit better than Sleep() I think I do need to put this in a thread.. I notice one strange behavior... if I just run my program, then it works exactly as I expect, and I'm not wasting a lot of time waiting for the end of Data. However if I do a writeln to the console before the function that reads the serial port, then I am missing the end of the data sometimes, I have to put an additional delay equal to about 10 Bytes of data to make it work... but without the writeln's I need only to wait for the amount of time of 2 Bytes, which is what I expected. I'm not using any threads yet, so this is strange behavior, doing a writeln before I even start the procedure that does the serial communication should not effect it at all, because the writeln should be completely done before it starts the serial communication, yet this is what I observe... If I don't do the writeln's everything is fine, if I do them, my data timing is off... it's still there, but I have to wait longer to make sure I get all the data that there is. What seems to be happening is that I'm I'm using SerRead, it does seem that I never miss anything, it's just that I'm not detecting the end of data accurately if I have writeln's right before I read the serial port. James -Original Message- From: fpc-pascal On Behalf Of Bo Berglund via fpc-pascal Sent: Sunday, December 27, 2020 5:08 PM To: fpc-pascal@lists.freepascal.org Cc: Bo Berglund Subject: Re: [fpc-pascal] Reading Serial Hex Data On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an >unknown length that have no specific termination character over a >serial port as efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the >Serial unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 10:47:59 -0500, James Richters via fpc-pascal wrote: >I'm trying to figure out how to read a packet of HEX Bytes of an unknown >length that have no specific termination character over a serial port as >efficiently as practical. What exactly do you mean by HEX Bytes? Hex transfers using two characters for each byte? like 1F A4 etc? >The packet ends when there is just no more data to read. > >I have something that is working that I wrote using SerRead from the Serial >unit in FPC, and it works at 9600bps, If you use the serial unit and just read the data coming in using blocking calls you will not miss anything, what you do not read will stay in the input buffer until you read again. If you need to do something inbetween then you can put the reading in a thread so that is handled there and then transfer the data to the main thread via a synchronize call to a receive function there. Is your program a console or GUI program? -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
On Sun, 27 Dec 2020 21:15:03 +0100, Jean SUZINEAU via fpc-pascal wrote: >I think TBlockSerial can work in your console program context. May be >TLazSerial can be more tricky to use in your context, you'll need to use >events/callbacks and FCL. I have struggled with LazSerial in a console program and it just is not intended for that usage. Won't work. Throws exceptions if run in a device without a GUI like a Linux server. But instead I used the built-in serial unit and packaged the whole reading stuff into a thread that will read the serial line constantly using the blocking read and supply the data to the main application as it arrives. Not a big deal to do and get working. You need a read timeout in the execute loop so that the thread can detect if it has been commanded to terminate, that is all I think. Also, for the OP: What do you mean by receiving hex data? Are you saying that you receive byte data as 2-character ASCII hex code? I.e. every byte you want to receive is represented by *two* characters in the incoming stream? In that case if you are not using any delimiters you will probably soon be out of sync and mixing most and least significant hex characters in the transmission so that for example 1a 2b 3c could become ..1 a2 b3 c.. Very dangerous situation. You really must use some kind of delimiter in the data flow... -- Bo Berglund Developer in Sweden ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Reading Serial Hex Data
May be using TBlockSerial from unit synaser from Ararat Synapse ? ( doc: http://synapse.ararat.cz/doc/help/synaser.TBlockSerial.html , download: http://synapse.ararat.cz/doku.php/download ). Given a variable (SynSer: TBlockSerial;), you can test if data is available with SynSer.CanReadEx(0) and then use SynSer.RecvPacket(Timeout) to get your data. I think it should work, I haven't used directly TBlockSerial this way, but through TLazSerial component for reading data from an Arduino in a program running on Windows, Linux x86, or Raspberry. I think TBlockSerial can work in your console program context. May be TLazSerial can be more tricky to use in your context, you'll need to use events/callbacks and FCL. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Reading Serial Hex Data
I'm trying to figure out how to read a packet of HEX Bytes of an unknown length that have no specific termination character over a serial port as efficiently as practical. The packet ends when there is just no more data to read. I have something that is working that I wrote using SerRead from the Serial unit in FPC, and it works at 9600bps, but the method I am using will not work well at anything higher unless I can figure out how to a very accurate very short delay. At 9600, I need a delay just a little less than 1ms, so I use Sleep(1) for my delay, but since 1ms is the minimum for sleep, and sleep itself is not really that accurate, I am doing a lot more delay than I need to in order to make sure it works. The procedure is: Check the serial port for a byte in a loop until you either get a byte or you get a timeout Then once you have one byte, you delay enough time for few bytes to be in the buffer Next read one byte at time, Delay at the correct rate for the baudrate until you run out of bytes or hit a maximum number of bytes. What is needed is to stop at a timeout in case there is nothing to read, OR to stop when maximum bytes are read OR stop when 2 bytes are missed meaning you ran out of data to read. But once you read one byte, you abandon the timeout. SerReadTimeout has a timeout, but it seems to be a timeout for the whole operation.. what I need is the first timeout to be a long timeout to wait for the device to respond, but once you get one byte, you cancel the timeout and never use it again.. now you read as many bytes as you can until you miss two in a row... then you stop. Either that or have two timeouts, one timeout to delay for receiving the first byte and the second timeout a way to tell when you have no more bytes left to read. It Seems like lots of people would want to read serial ports in this way, because you never know how many bytes are coming, and you don't want to always wait the entire timeout, that is just there to stop it in case there is no data. Then again, I guess most applications can read until they get a linefeed or carriage return, which I will not be. I'm thinking this has already been done somewhere and maybe there is already a solution that is tested and works, but I don't know where it is or what it's called. I could not even find any documentation of SerReadTimout. Here is pseudocode of the procedure that would be needed... the only thing I am really lacking is the accurate time delay. So if there is not already a function somewhere that works this way, then a recommendation on very fast and accurate timing delays would be helpful. repeat Read the serial port and check for a byte Delay the amount of time one byte should take Inc(Timeout); Until (Timeout=Max_Allowed) or Byte_ is_received If Byte is received Begin Count:=-0; String:=String+ByteRead; Inc(Bytes_Read); End; Delay enough time for a few more bytes to get in the buffer Repeat Read serial port Delay the amount of time one byte should take If Byte is received Begin Count:=-0; String:=String+ByteRead; Inc(Bytes_Read); End Else Inc(Count); Until MaxBytesRead Or (Count=2) James ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal