Re: [twsocket] udp multicast multiple packages -> TWSocket bug?
> What happens now is that the size of arriving packets gets bigger.
> So instead of getting packets of 1316 Byte every 2ms,
> I receive 2632 Byte, for example.
> To me it appears that TWSocket has buffered two arriving packets, right?
Wrong. Winsock has a buffer (8KB by default if memory serve me well). Since
your thread use a lot of CPU - atually it uses the most CPU it can - the
main thread do not get scheduled fast enough and winsock has time to receive
two packet before being able to deliver it to the main thread.
Note that this is just normal behaviour.
> rcvsize:=Socket.rcvdcnt;
> GetMem(rec,rcvsize);
> IF (rcvsize > 0) THEN
> BEGIN
>socket.receive(rec,rcvsize);
>MyFile.Write(rec^,rcvsize);
> END;
This is very bad code. First of all, you have a huge memory leak since you
never free the memory you allocate. Then you have a huge performance it
since you allocate a new buffer each time. And then you use RcvdCount which
is not reliable in all situations (that's what MS say in an MSDN article).
The correct way to do what you want is to use a fixed size buffer. A local
variable will be perfect. Something like this:
var
Buffer : array [0..1315]; // You are using UDP with 1316 bytes
datagram.
// You could also use a record
type with the correct structure
Count : Integer;
begin
Count := Socket.Receive(@Buffer, SizeOf(Buffer);
if Count <= 0 then
Exit;
MyFile.Write(Buffer, Count);
end;
Also please note that UDP is by design an UNRELIABLE protocol. You may loose
data, receive data multiple times, receive data in incorrect order and other
similar problems. UDP is unreliable but fast. You have to implement you own
error detection and correction. UDP is well suited for streaming
applications where it is not a problem to miss data or have it slightly
incorrect. Audio and video streaming are OK with UDP. File transfer is
defenitely not without implementing a detection and error correction. This
is not an ICS issue, it is a TCP/IP issue. TCP/IP has been designed like
that and it works like that on all platforms, using any component or
library.
If you need a RELIABLE data transport, use TCP. See TCP/UDP primer article
on the support page at my website.
--
Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html
--
[EMAIL PROTECTED]
http://www.overbyte.be
- Original Message -
From: "Christian Hinske" <[EMAIL PROTECTED]>
To: "ICS support mailing"
Sent: Saturday, January 13, 2007 12:37 PM
Subject: [twsocket] udp multicast multiple packages -> TWSocket bug?
> Hi,
> first of all thanks a lot for your help. Unfortunately, the thing is a bit
> different (maybe I haven't chosen the best of my code-fragments to
> present):
>
> It doesn't make a difference if I am using a ringbuffer or not or if the
> TWSocket-component is running in an extra thread or in the main unit.
>
> Let's assume the following scenario:
> I have the TWSocket-component which is declared in the main unit
> TForm1.Socket : TWSocket; TForm1.Socket:=TWSocket.Create(self);
> as well as a TFilestream-Component
> TForm1.MyFile : TFilestream;
> TForm1.MyFile:=TFileSTream.Create('C:\test.ts', fmCreate);
>
> The OnDataAvailable(Sender : TObject; Error : Word) procedure basically
> doesn't much apart from getting the data and writing it down to file, eg.
>
> procedure TForm1.OnDataAvailable(Sender : TObject; Error : Word);
> var rec : Pointer; rcvsize : Integer;
> begin
> rcvsize:=Socket.rcvdcnt;
> GetMem(rec,rcvsize);
> IF (rcvsize > 0) THEN
> BEGIN
>socket.receive(rec,rcvsize);
>MyFile.Write(rec^,rcvsize);
> END;
> end;
>
> So far, everything is running fine, until I introduce a thread, which
> completely doesn't have to do anything with the socket at all, e.g.
> procedure TMyThread.Execute;
> var x : Int64;
> begin
> x:=0;
> WHILE (not terminated) DO
>Inc(x,1);
> end;
>
> What happens now is that the size of arriving packets gets bigger. So
> instead of getting packets of 1316 Byte every 2ms, I receive 2632 Byte,
> for example. To me it appears that TWSocket has buffered two arriving
> packets, right? However, every packet > 1316Byte is damaged with the same
> structure: The first bytes are trash; Byte 188-1316 perfect; Byte 1317 -
> 2632 or more = empty!
>
> So that's why I thought that maybe the buffering doesn't work correctly,
> meaning that if OnDataAvailable is not executed because of the second
> thread and data has to be buffered by TWSocket, it doesn't append the new
> data at the end, but copies it into the first bytes, which would explain,
> why only the first bytes are damaged, bytes 188-1316
[twsocket] udp multicast multiple packages -> TWSocket bug?
Hi,
first of all thanks a lot for your help. Unfortunately, the thing is a bit
different (maybe I haven't chosen the best of my code-fragments to present):
It doesn't make a difference if I am using a ringbuffer or not or if the
TWSocket-component is running in an extra thread or in the main unit.
Let's assume the following scenario:
I have the TWSocket-component which is declared in the main unit TForm1.Socket
: TWSocket; TForm1.Socket:=TWSocket.Create(self);
as well as a TFilestream-Component
TForm1.MyFile : TFilestream; TForm1.MyFile:=TFileSTream.Create('C:\test.ts',
fmCreate);
The OnDataAvailable(Sender : TObject; Error : Word) procedure basically doesn't
much apart from getting the data and writing it down to file, eg.
procedure TForm1.OnDataAvailable(Sender : TObject; Error : Word);
var rec : Pointer; rcvsize : Integer;
begin
rcvsize:=Socket.rcvdcnt;
GetMem(rec,rcvsize);
IF (rcvsize > 0) THEN
BEGIN
socket.receive(rec,rcvsize);
MyFile.Write(rec^,rcvsize);
END;
end;
So far, everything is running fine, until I introduce a thread, which
completely doesn't have to do anything with the socket at all, e.g.
procedure TMyThread.Execute;
var x : Int64;
begin
x:=0;
WHILE (not terminated) DO
Inc(x,1);
end;
What happens now is that the size of arriving packets gets bigger. So instead
of getting packets of 1316 Byte every 2ms, I receive 2632 Byte, for example. To
me it appears that TWSocket has buffered two arriving packets, right? However,
every packet > 1316Byte is damaged with the same structure: The first bytes are
trash; Byte 188-1316 perfect; Byte 1317 - 2632 or more = empty!
So that's why I thought that maybe the buffering doesn't work correctly,
meaning that if OnDataAvailable is not executed because of the second thread
and data has to be buffered by TWSocket, it doesn't append the new data at the
end, but copies it into the first bytes, which would explain, why only the
first bytes are damaged, bytes 188-1316 are OK and 1317-x are empty. But maybe
I am completely wrong.
Thanks for any help,
Chris
Original-Nachricht
Datum: Fri, 12 Jan 2007 15:26:09 +0100
Von: Wilfried Mestdagh <[EMAIL PROTECTED]>
An: ICS support mailing
Betreff: Re: [twsocket] udp multicast multiple packages -> TWSocket bug?
> Hello,
>
> I did a quick look. It seems you start (or resume one from pool) a
> thread and give the received data to it.
>
> Probably you give data again to a thread that is still busy with the
> previous data. You have to check for this somewhere because you tell
> about corrupted data with many threads.
>
> ---
> Rgds, Wilfried [TeamICS]
> http://www.overbyte.be/eng/overbyte/teamics.html
> http://www.mestdagh.biz
>
> --
> To unsubscribe or change your settings for TWSocket mailing list
> please goto http://www.elists.org/mailman/listinfo/twsocket
> Visit our website at http://www.overbyte.be
--
Christian Hinske
Schleißheimerstraße 157
D-80797 München
Tel. :(++ 49 89) 36 0 37 445
Mobil :(++49)176 234 10 146
e-mail: [EMAIL PROTECTED]
Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://www.elists.org/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] udp multicast multiple packages -> TWSocket bug?
Hello, I did a quick look. It seems you start (or resume one from pool) a thread and give the received data to it. Probably you give data again to a thread that is still busy with the previous data. You have to check for this somewhere because you tell about corrupted data with many threads. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] udp multicast multiple packages -> TWSocket bug?
No time to look at your code. Sorry.
Could you tell me why you need a thread for TWSocket ? It is asynchronous
anyway.
If you use a thread, be sure to create TWSocket from the execute method and
to add a message pump in your thread. Also make sure to /not/ call the
message pump from any of the event generated by TWSocket, directly or
indirectly.
--
Contribute to the SSL Effort. Visit http://www.overbyte.be/eng/ssl.html
--
[EMAIL PROTECTED]
http://www.overbyte.be
- Original Message -
From: "Christian Hinske" <[EMAIL PROTECTED]>
To:
Sent: Wednesday, January 10, 2007 10:14 PM
Subject: [twsocket] udp multicast multiple packages -> TWSocket bug?
Hi everyone, I am quite new to this mailing list, but I hope anyone can
answer my question.
The problem I am experiencing is as follows:
I get MPEG2 TS-data from a realtime hardware encoder who is multicasting
into the local network. I am using TWSocket to receive the datapackes. As
some of you might know, MPEG2 Transport Stream (TS) consists of 188 Byte
including a 4Byte header, which I can check to see, if the received package
is OK.
Now, if I am working with only one thread, everything is fine. Once I am
adding a second thread (OnDataAvailable -> RingBuffer; RingBuffer->
Thread2 -> File), it seems that OnDataAvailable isn't executed for every
event and so buffers the data.
The hardware encoder packs 7 MPEG2 packages into one udp-packet, so the
standard-size package I get is 1316 Byte. Now, once I am applying the second
thread, packages get multiples of that (2632), but they all share the same
characteristics: The first header is damaged, 2nd - 7th are fine, 8th -
whatever size are empty or random data.
I was wondering if maybe TWSocket does not copy the second 1316Byte packet
into the buffer-area after the first 1316Byte package if OnDataAvailable
hasn't been called in between, but somewhere into the first bytes, which
would explain why the first header (with 188 Bytes following is broken).
This is how parts of my debugfile look like
1316 Byte udp package received
header 1 - OKAY, pid 256 -> payload video
header 2 - OKAY, pid 256 -> payload video
header 3 - OKAY, pid 256 -> payload video
header 4 - OKAY, pid 259 -> payload audio
header 5 - OKAY, pid 256 -> payload video
header 6 - OKAY, pid 256 -> payload video
header 7 - OKAY, pid 256 -> payload video
2632 Byte udp package received
header 1 - ERROR, pid 19440
header 2 - OKAY, pid 256
header 3 - OKAY, pid 256
header 4 - OKAY, pid 256
header 5 - OKAY, pid 256
header 6 - OKAY, pid 256
header 7 - OKAY, pid 256
header 8 - ERROR, pid 0
header 9 - ERROR, pid 0
header 10 - ERROR, pid 0
header 11 - ERROR, pid 0
header 12 - ERROR, pid 0
header 13 - ERROR, pid 0
header 14 - ERROR, pid 0
The code I am using is in analogy to the TWSocket multithreading example.
Code looks like this:
type
TPos = RECORD
Buffersize, Blocksize : Int64;
CurrentWPos, CurrentRPos, StartPos : Int64;
Diskrepanz : Int64;
Richtung : Byte;
END;
TStreamThread = class(TThread)
private
FAddrDatei : Pointer;
BlockMem : Pointer;
procedure CalculateDiscrepancy;
procedure WriteMemToFile;
procedure ResetPosition;
public
FDatenspeicher : Pointer;
FPos : ^TPos;
FAddrLink : Pointer;
FAusgabe : TLabel;
Ffinalizethread: Boolean;
FExchangeFile : Boolean;
procedure SetLabel;
procedure Execute; override;
end;
TChangeThread = class(TThread)
private
procedure ExchangeFile;
procedure PrepareFile;
public
Mode : Byte;
procedure Execute; override;
end;
TRingBuffer = class(TObject)
private
Speicher : Pointer;
Groesse: Int64;
Start : Pointer;
Writer : Int64;
Reader : Int64;
Diskrepanz : Int64;
Richtung : Integer;
public
fReadyToWrite : TEvent;
fReadyToRead : TEvent;
CS : TCriticalSection;
constructor Create;
procedure GetData(pData : Pointer; pSize : Int64);
procedure PutData(pData : Pointer; pSize : Int64);
end;
var
Datei : ARRAY[0..1] OF TFileStream;
AddrDatei : Pointer;
AddrLink : Pointer;
DoExchange: Byte;
RootName : String;
Socket:TWSocket;
StreamThread : TStreamThread;
ChangeThread : TChangeThread;
ExchangeThread: TChangeThread;
public
{ Public-Deklarationen }
DataRes: Pointer;
Pos: ^TPos;
MyVar : Integer;
CritSec: TCriticalSection;
Wholesize : Int64;
CurrentFileSize: Int64;
WrittenSize: Int64;
Testvariable : Int64;
ReceivedPacks : Int64;
MaxSize: Int64;
PrerunSize : Int64;
FileCount : Integer;
StreamID : Intege
[twsocket] udp multicast multiple packages -> TWSocket bug?
Hi everyone, I am quite new to this mailing list, but I hope anyone can answer
my question.
The problem I am experiencing is as follows:
I get MPEG2 TS-data from a realtime hardware encoder who is multicasting into
the local network. I am using TWSocket to receive the datapackes. As some of
you might know, MPEG2 Transport Stream (TS) consists of 188 Byte including a
4Byte header, which I can check to see, if the received package is OK.
Now, if I am working with only one thread, everything is fine. Once I am adding
a second thread (OnDataAvailable -> RingBuffer; RingBuffer-> Thread2 -> File),
it seems that OnDataAvailable isn't executed for every event and so buffers the
data.
The hardware encoder packs 7 MPEG2 packages into one udp-packet, so the
standard-size package I get is 1316 Byte. Now, once I am applying the second
thread, packages get multiples of that (2632), but they all share the same
characteristics: The first header is damaged, 2nd - 7th are fine, 8th -
whatever size are empty or random data.
I was wondering if maybe TWSocket does not copy the second 1316Byte packet into
the buffer-area after the first 1316Byte package if OnDataAvailable hasn't been
called in between, but somewhere into the first bytes, which would explain why
the first header (with 188 Bytes following is broken).
This is how parts of my debugfile look like
1316 Byte udp package received
header 1 - OKAY, pid 256 -> payload video
header 2 - OKAY, pid 256 -> payload video
header 3 - OKAY, pid 256 -> payload video
header 4 - OKAY, pid 259 -> payload audio
header 5 - OKAY, pid 256 -> payload video
header 6 - OKAY, pid 256 -> payload video
header 7 - OKAY, pid 256 -> payload video
2632 Byte udp package received
header 1 - ERROR, pid 19440
header 2 - OKAY, pid 256
header 3 - OKAY, pid 256
header 4 - OKAY, pid 256
header 5 - OKAY, pid 256
header 6 - OKAY, pid 256
header 7 - OKAY, pid 256
header 8 - ERROR, pid 0
header 9 - ERROR, pid 0
header 10 - ERROR, pid 0
header 11 - ERROR, pid 0
header 12 - ERROR, pid 0
header 13 - ERROR, pid 0
header 14 - ERROR, pid 0
The code I am using is in analogy to the TWSocket multithreading example. Code
looks like this:
type
TPos = RECORD
Buffersize, Blocksize : Int64;
CurrentWPos, CurrentRPos, StartPos : Int64;
Diskrepanz : Int64;
Richtung : Byte;
END;
TStreamThread = class(TThread)
private
FAddrDatei : Pointer;
BlockMem : Pointer;
procedure CalculateDiscrepancy;
procedure WriteMemToFile;
procedure ResetPosition;
public
FDatenspeicher : Pointer;
FPos : ^TPos;
FAddrLink : Pointer;
FAusgabe : TLabel;
Ffinalizethread: Boolean;
FExchangeFile : Boolean;
procedure SetLabel;
procedure Execute; override;
end;
TChangeThread = class(TThread)
private
procedure ExchangeFile;
procedure PrepareFile;
public
Mode : Byte;
procedure Execute; override;
end;
TRingBuffer = class(TObject)
private
Speicher : Pointer;
Groesse: Int64;
Start : Pointer;
Writer : Int64;
Reader : Int64;
Diskrepanz : Int64;
Richtung : Integer;
public
fReadyToWrite : TEvent;
fReadyToRead : TEvent;
CS : TCriticalSection;
constructor Create;
procedure GetData(pData : Pointer; pSize : Int64);
procedure PutData(pData : Pointer; pSize : Int64);
end;
var
Datei : ARRAY[0..1] OF TFileStream;
AddrDatei : Pointer;
AddrLink : Pointer;
DoExchange: Byte;
RootName : String;
Socket:TWSocket;
StreamThread : TStreamThread;
ChangeThread : TChangeThread;
ExchangeThread: TChangeThread;
public
{ Public-Deklarationen }
DataRes: Pointer;
Pos: ^TPos;
MyVar : Integer;
CritSec: TCriticalSection;
Wholesize : Int64;
CurrentFileSize: Int64;
WrittenSize: Int64;
Testvariable : Int64;
ReceivedPacks : Int64;
MaxSize: Int64;
PrerunSize : Int64;
FileCount : Integer;
StreamID : Integer;
RecordingID: Integer;
ErrorPacks : Int64;
procedure DataAvailable(Sender : TObject; Error : Word);
procedure WriteToMem;
end;
var
Form2: TForm2;
implementation
{$R *.dfm}
{procedure TForm2.SetLabel(lname: string);
begin
StatusLA.Caption:=lname;
end; }
constructor TForm2.TRingBuffer.Create;
begin
Inherited Create;
GetMem(Start,Groesse);
Writer:=0;
Reader:=0;
Richtung:=0;
CS:=TCriticalSection.Create;
end;
procedure TForm2.TRingbuffer.GetData(pData : Pointer; pSize : Int64);
var zwSize1,zwSize2 : Int64;
begin
IF fReadyToRead.WaitFor(INFINITE) = wrSignaled THEN
BEGIN
fReadyToWrite.ResetEvent;
try
CS.Enter;
Diskrepanz:=Writer+Groesse*Richtung - Reader;
IF (Di
