Re: [twsocket] OK to Receiving >2GB ?
1500 is the 'standard' MTU allocated. This is optimised for LAN However, most internet 'optimisers' tend to adjust this value based on your connection so this value is prone to change. (dial-up I believe is recommended to have an MTU of 576) There are programs out that that will read/set this value for you (http://62.172.198.79/broadband1/update/index.asp#20030131 shows one such program) You should never assume to know this value beforehand though, unless you are sure you are dealing with a fresh windows install. Dave Darin McGee wrote: >About 1500 bytes give or take a few over ethernet > >-Original Message- >From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] >On Behalf Of Kei >Sent: Saturday, October 29, 2005 3:26 PM >To: ICS support mailing >Subject: Re: [twsocket] OK to Receiving >2GB ? > >Hi! > >I think I'm going to accept the reality that... TCP packets are splitted >into arbitrary sizes.. but!!! >How do I know the maximum size possible? What is the maximum size that a >packet can be? I certainly don't want to malloc 100KB for a TCP >packet... > >Thanks! > >David > >Wilfried Mestdagh wrote: > > > >>Hello David, >> >> >> >> >> >>>Hi! I'm new to ICS! >>> >>> >>> >>> >>Welcome to the group :) >> >> >> >> >> >>>A->>B: msg hello >>>B->>A: msg yo! how's it going? >>> >>> >>> >>> >>Yes that's the way to go. Design a user made proto for what you intend >>to do. >> >> >> >> >> >>>If A is larger than the default buffer size (256 chars) then the A >>>(sender) will warn B in advance >>> >>> >>> >>> >>You can do that, but is not nececary. Also you will maybe have a very >>mutch allocation / deallocation of memory and you can eventually end up >> >> > > > >>with fragmented memory where you have not a nice large block in it at >>the moment you need it. But it can work, just think over carefully. A >>better idea is often to make a receive buffer that grows automatically >>if (and only if) needed, and then just reuse that buffer over and over >>again. Then you have some (re)allocation in begin but then stable. >> >> >> >> >> >>>TWSocket will automatically split it into packets, >>> >>> >>> >>> >>Winsock will split in packets as large as the MTU (around 1500 bytes). >>Eventually data can (and will if high speed) arrive as 1 large packet >>but not necacarely. >> >> >> >> >> >>>I really want the sender side to send the 1MB file all at once, since >>>I do the FileWrite() right after Receive() >>> >>> >>> >>> >>No you cannot. Winsock does not respect packet boundaries, but (see >>prior paragraph) there are no megabytes TCP packets. >> >>You have to receive all data chuncks into a buffer, and when you >>received them all then you save to file (or save every packet direct to >> >> > > > >>disk). There is no problem to know the moment of close the file because >> >> > > > >>you know the length of the data from your protocol. >> >>--- >>Rgds, Wilfried [TeamICS] >>http://www.overbyte.be/eng/overbyte/teamics.html >>http://www.mestdagh.biz >> >>Saturday, October 29, 2005, 11:01, Kei wrote: >> >> >> >> >> >>>Hi! I'm new to ICS! >>> >>> >>> >>> >> >> >> >> >>>I am designing a simple protocol that will be mainly used locally (as >>>a database server backend)..so I'm guessing I could send up to 2GB of >>>stuff without hassle (BLOBs, for example). Right now I'm just >>>experimenting with the facility for two parties to effectively "talk" >>>to each other, even with long long messages, and with binary data >>> >>> >transfer. > > >>>For example, for sending a message, the command is "msg [text]" >>> >>> >>> >>> >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> >> >> >> >>>If A is larger than the default buffer size (256 chars) then the A >>>(sender) will warn B in advance, like this: &quo
Re: [twsocket] OK to Receiving >2GB ?
Hello David, > I don't allocate the memory up to the size of the "Designated size" but > keep realloc-ing until "remaining-byte" counter reaches zero.. I'm not sure I understeand exacly what you mean. But if you are reallocating memory whole the time, you end up with a non efficient program, if not talking about memory fragmentation. What Arno meant was to receive in a fixed buffer and copy that buffer to the place you want, eg a TFileStream. For the fixed buffer you even not need to allocate memory, just put it on the stack as this: procedure TApp.OnDataAvailable(Sender: TObject; Error: word); var Buf: array[0..MAX_BUF_SIZE - 1] of char; begin // receive here > Which one is more efficient performance-wise? I think you wants to save to a file. The most efficient is to receive in a fixed buffer, then write the buffer to a file with the lower level routines FileWrite(). Open the file just once in begin, and Close it only when all data is received, or session is closed. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Sunday, October 30, 2005, 15:42, Kei wrote: > I don't allocate the memory up to the size of the "Designated size" but > keep realloc-ing until "remaining-byte" counter reaches zero.. > Do you think I should use TStream (Stream.readbuffer, readbuffer, ...) > or just a pointer of buffer (malloc, realloc, and basic pointer > operations) ? Which one is more efficient performance-wise? > David > Arno Garrels wrote: >>Do you allocate memory up to the size of data the client intends to send? >>That would be a huge waste of memory and won't work upon large data. >>Instead receive into a buffer of let's say 8 kb and write it to your >>stream at once, receive next chunk and so on. You don't have to take care >>of packet size, it's something on a lower layer encapsulated by winsock. >> >>--- >>Arno Garrels [TeamICS] >> >> >> >> >>>Thanks! >>> >>>David >>> >>>Wilfried Mestdagh wrote: >>> >>> >>> Hello David, >How do I know the maximum size possible? > > > > at receiving side, specially depending on the speed of your own program you mostly never get above 8 kb. However I have seen receiving packets 10 time as high. But you dont have to care mutch, if you receive not all, then OnDataAvailable is immediatly called again, meaning if you have a temporary buffer that is to small, however if you receive on the stack then you can make him (almost) as large you wants. >I certainly don't want to malloc 100KB for a TCP packet... > > > > See above, you dont need to. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 20:25, Kei wrote: >Hi! > > > > >I think I'm going to accept the reality that... TCP packets are splitted >into arbitrary sizes.. but!!! >How do I know the maximum size possible? What is the maximum size that a >packet can be? I certainly don't want to malloc 100KB for a TCP >packet... > > > > >Thanks! > > > > >David > > > > >Wilfried Mestdagh wrote: > > > > >>Hello David, >> >> >> >> >> >> >> >>>Hi! I'm new to ICS! >>> >>> >>> >>> >>> >>> >>Welcome to the group :) >> >> >> >> >> >> >> >>>A->>B: msg hello >>>B->>A: msg yo! how's it going? >>> >>> >>> >>> >>> >>> >>Yes that's the way to go. Design a user made proto for what you intend >>to do. >> >> >> >> >> >> >> >>>If A is larger than the default buffer size (256 chars) then the A >>>(sender) will warn B in advance >>> >>> >>> >>> >>> >>> >>You can do that, but is not nececary. Also you will maybe have a very >>mutch allocation / deallocation of memory and you can eventually end up >>with fragmented memory where you have not a nice large block in it at >>the moment you need it. But it can work, just think over carefully. A >>better idea is often to make a receive buffer that grows automatically >>if (and only if) needed, and then just reuse that buffer over and over >>again. Then you have some (re)allocation in begin but then stable. >> >> >> >> >> >> >> >>>TWSocket will automatically split it into packets, >>> >>> >>> >>> >>> >>> >>Winsock will split in packets as large as the M
Re: [twsocket] OK to Receiving >2GB ?
Kei wrote: > I don't allocate the memory up to the size of the "Designated size" but > keep realloc-ing until "remaining-byte" counter reaches zero.. > Do you think I should use TStream (Stream.readbuffer, readbuffer, ...) > or just a pointer of buffer (malloc, realloc, and basic pointer > operations) ? Which one is more efficient performance-wise? I'm not sure whether I got exactely what you mean. I mean in order to receive data in the ClientDataAvailable event a fixed sized buffer of type PChar is fine. After each successfull call to Receive I would write received data from that buffer to a file or blob stream. Since you also decrement a "remaining-byte" counter you know when everything has been received. Does this answer your question? --- Arno Garrels [TeamICS] > David > > Arno Garrels wrote: > >> Do you allocate memory up to the size of data the client intends to send? >> That would be a huge waste of memory and won't work upon large data. >> Instead receive into a buffer of let's say 8 kb and write it to your >> stream at once, receive next chunk and so on. You don't have to take care >> of packet size, it's something on a lower layer encapsulated by winsock. >> >> --- >> Arno Garrels [TeamICS] >> >> >> >> >>> Thanks! >>> >>> David >>> >>> Wilfried Mestdagh wrote: >>> >>> >>> Hello David, > How do I know the maximum size possible? > > > > at receiving side, specially depending on the speed of your own program you mostly never get above 8 kb. However I have seen receiving packets 10 time as high. But you dont have to care mutch, if you receive not all, then OnDataAvailable is immediatly called again, meaning if you have a temporary buffer that is to small, however if you receive on the stack then you can make him (almost) as large you wants. > I certainly don't want to malloc 100KB for a TCP packet... > > > > See above, you dont need to. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 20:25, Kei wrote: > Hi! > > > > > I think I'm going to accept the reality that... TCP packets are > splitted into arbitrary sizes.. but!!! > How do I know the maximum size possible? What is the maximum size > that a packet can be? I certainly don't want to malloc 100KB for a TCP > packet... > > > > > Thanks! > > > > > David > > > > > Wilfried Mestdagh wrote: > > > > >> Hello David, >> >> >> >> >> >> >> >>> Hi! I'm new to ICS! >>> >>> >>> >>> >>> >>> >> Welcome to the group :) >> >> >> >> >> >> >> >>> A->>B: msg hello >>> B->>A: msg yo! how's it going? >>> >>> >>> >>> >>> >>> >> Yes that's the way to go. Design a user made proto for what you >> intend to do. >> >> >> >> >> >> >> >>> If A is larger than the default buffer size (256 chars) then the A >>> (sender) will warn B in advance >>> >>> >>> >>> >>> >>> >> You can do that, but is not nececary. Also you will maybe have a very >> mutch allocation / deallocation of memory and you can eventually end >> up with fragmented memory where you have not a nice large block in >> it at the moment you need it. But it can work, just think over >> carefully. A better idea is often to make a receive buffer that >> grows automatically if (and only if) needed, and then just reuse >> that buffer over and over again. Then you have some (re)allocation >> in begin but then stable. >> >> >> >> >> >> >> >>> TWSocket will automatically split it into packets, >>> >>> >>> >>> >>> >>> >> Winsock will split in packets as large as the MTU (around 1500 >> bytes). Eventually data can (and will if high speed) arrive as 1 >> large packet but not necacarely. >> >> >> >> >> >> >> >>> I really want the sender side to send the 1MB file all at once, >>> since I do the FileWrite() right after Receive() >>> >>> >>> >>> >>> >>> >> No you cannot. Winsock does not respect packet boundaries, but (see >> prior paragraph) there are no megabytes TCP packets. >> >> You have to receive all data chuncks into a buffer, and when you >> received them all then you save to file (or save every packet direct >> to disk). There is no problem to know the mo
Re: [twsocket] OK to Receiving >2GB ?
I don't allocate the memory up to the size of the "Designated size" but keep realloc-ing until "remaining-byte" counter reaches zero.. Do you think I should use TStream (Stream.readbuffer, readbuffer, ...) or just a pointer of buffer (malloc, realloc, and basic pointer operations) ? Which one is more efficient performance-wise? David Arno Garrels wrote: >Do you allocate memory up to the size of data the client intends to send? >That would be a huge waste of memory and won't work upon large data. >Instead receive into a buffer of let's say 8 kb and write it to your >stream at once, receive next chunk and so on. You don't have to take care >of packet size, it's something on a lower layer encapsulated by winsock. > >--- >Arno Garrels [TeamICS] > > > > >>Thanks! >> >>David >> >>Wilfried Mestdagh wrote: >> >> >> >>>Hello David, >>> >>> >>> >>> >>> How do I know the maximum size possible? >>>at receiving side, specially depending on the speed of your own program >>>you mostly never get above 8 kb. However I have seen receiving packets >>>10 time as high. But you dont have to care mutch, if you receive not >>>all, then OnDataAvailable is immediatly called again, meaning if you >>>have a temporary buffer that is to small, however if you receive on the >>>stack then you can make him (almost) as large you wants. >>> >>> >>> >>> >>> I certainly don't want to malloc 100KB for a TCP packet... >>>See above, you dont need to. >>> >>>--- >>>Rgds, Wilfried [TeamICS] >>>http://www.overbyte.be/eng/overbyte/teamics.html >>>http://www.mestdagh.biz >>> >>>Saturday, October 29, 2005, 20:25, Kei wrote: >>> >>> >>> >>> >>> Hi! >>> >>> >>> I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... >>> >>> >>> Thanks! >>> >>> >>> David >>> >>> >>> Wilfried Mestdagh wrote: >>> >>> >>> >Hello David, > > > > > > > >>Hi! I'm new to ICS! >> >> >> >> >> >> >Welcome to the group :) > > > > > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> >> >> >> >> >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > > > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> >> >> >> >> >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > > > > > >>TWSocket will automatically split it into packets, >> >> >> >> >> >> >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > > > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> >> >> >> >> >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > > > > > >>Hi! I'm new to ICS! >> >> >> >> >> >> > > > > >>I am designing a simple protocol that will be mainly used locally (as >>a database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example)
Re: [twsocket] OK to Receiving >2GB ?
Kei wrote: > Hi Everyone > > I've solved this problem.. what I do is make a count of "How many bytes > remaining". Everytime the value returned by Receive() will be subtracted > from the "BytesRemaining". When OnDataAvailable is repeatedly called, a > buffer is kept being ReAlloc()'ed . When "ByteRemaining" reaches zero, > then a WriteFile will be performed, so that the buffer is written to > disk.. Do you allocate memory up to the size of data the client intends to send? That would be a huge waste of memory and won't work upon large data. Instead receive into a buffer of let's say 8 kb and write it to your stream at once, receive next chunk and so on. You don't have to take care of packet size, it's something on a lower layer encapsulated by winsock. --- Arno Garrels [TeamICS] > Thanks! > > David > > Wilfried Mestdagh wrote: > >> Hello David, >> >> >> >>> How do I know the maximum size possible? >>> >>> >> >> at receiving side, specially depending on the speed of your own program >> you mostly never get above 8 kb. However I have seen receiving packets >> 10 time as high. But you dont have to care mutch, if you receive not >> all, then OnDataAvailable is immediatly called again, meaning if you >> have a temporary buffer that is to small, however if you receive on the >> stack then you can make him (almost) as large you wants. >> >> >> >>> I certainly don't want to malloc 100KB for a TCP packet... >>> >>> >> >> See above, you dont need to. >> >> --- >> Rgds, Wilfried [TeamICS] >> http://www.overbyte.be/eng/overbyte/teamics.html >> http://www.mestdagh.biz >> >> Saturday, October 29, 2005, 20:25, Kei wrote: >> >> >> >>> Hi! >>> >>> >> >> >> >>> I think I'm going to accept the reality that... TCP packets are splitted >>> into arbitrary sizes.. but!!! >>> How do I know the maximum size possible? What is the maximum size that a >>> packet can be? I certainly don't want to malloc 100KB for a TCP >>> packet... >>> >>> >> >> >> >>> Thanks! >>> >>> >> >> >> >>> David >>> >>> >> >> >> >>> Wilfried Mestdagh wrote: >>> >>> >> >> >> Hello David, > Hi! I'm new to ICS! > > > > Welcome to the group :) > A->>B: msg hello > B->>A: msg yo! how's it going? > > > > Yes that's the way to go. Design a user made proto for what you intend to do. > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B in advance > > > > You can do that, but is not nececary. Also you will maybe have a very mutch allocation / deallocation of memory and you can eventually end up with fragmented memory where you have not a nice large block in it at the moment you need it. But it can work, just think over carefully. A better idea is often to make a receive buffer that grows automatically if (and only if) needed, and then just reuse that buffer over and over again. Then you have some (re)allocation in begin but then stable. > TWSocket will automatically split it into packets, > > > > Winsock will split in packets as large as the MTU (around 1500 bytes). Eventually data can (and will if high speed) arrive as 1 large packet but not necacarely. > I really want the sender side to send the 1MB file all at once, since > I do the FileWrite() right after Receive() > > > > No you cannot. Winsock does not respect packet boundaries, but (see prior paragraph) there are no megabytes TCP packets. You have to receive all data chuncks into a buffer, and when you received them all then you save to file (or save every packet direct to disk). There is no problem to know the moment of close the file because you know the length of the data from your protocol. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 11:01, Kei wrote: > Hi! I'm new to ICS! > > > > > I am designing a simple protocol that will be mainly used locally (as > a database server backend)..so I'm guessing I could send up to 2GB of > stuff without hassle (BLOBs, for example). Right now I'm just > experimenting with the facility for two parties to effectively "talk" > to each other, even with long long messages, and with binary data > transfer. For example, for sending a message, the command is "msg > [text]" > > > > A->>B: msg hello B->>A: msg yo! how's it going? > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B
Re: [twsocket] OK to Receiving >2GB ?
Hello David, Thanks for feedback. That is indeed the right way to do it. Remember that Receive() may return -1, and in that case you dont want to substrackt it from BytesRemaining, so something like this: Count := TWSocket(Sender).Receive(Buf, Min(BufSize, BytesRemaining)); if Count <= 0 then Exit; Dec(BytesRemaining, Count); ... --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Sunday, October 30, 2005, 08:28, Kei wrote: > Hi Everyone > I've solved this problem.. what I do is make a count of "How many bytes > remaining". Everytime the value returned by Receive() will be subtracted > from the "BytesRemaining". When OnDataAvailable is repeatedly called, a > buffer is kept being ReAlloc()'ed . When "ByteRemaining" reaches zero, > then a WriteFile will be performed, so that the buffer is written to disk.. > Thanks! > David > Wilfried Mestdagh wrote: >>Hello David, >> >> >> >>>How do I know the maximum size possible? >>> >>> >> >>at receiving side, specially depending on the speed of your own program >>you mostly never get above 8 kb. However I have seen receiving packets >>10 time as high. But you dont have to care mutch, if you receive not >>all, then OnDataAvailable is immediatly called again, meaning if you >>have a temporary buffer that is to small, however if you receive on the >>stack then you can make him (almost) as large you wants. >> >> >> >>>I certainly don't want to malloc 100KB for a TCP packet... >>> >>> >> >>See above, you dont need to. >> >>--- >>Rgds, Wilfried [TeamICS] >>http://www.overbyte.be/eng/overbyte/teamics.html >>http://www.mestdagh.biz >> >>Saturday, October 29, 2005, 20:25, Kei wrote: >> >> >> >>>Hi! >>> >>> >> >> >> >>>I think I'm going to accept the reality that... TCP packets are splitted >>>into arbitrary sizes.. but!!! >>>How do I know the maximum size possible? What is the maximum size that a >>>packet can be? I certainly don't want to malloc 100KB for a TCP packet... >>> >>> >> >> >> >>>Thanks! >>> >>> >> >> >> >>>David >>> >>> >> >> >> >>>Wilfried Mestdagh wrote: >>> >>> >> >> >> Hello David, >Hi! I'm new to ICS! > > > > Welcome to the group :) >A->>B: msg hello >B->>A: msg yo! how's it going? > > > > Yes that's the way to go. Design a user made proto for what you intend to do. >If A is larger than the default buffer size (256 chars) then the A >(sender) will warn B in advance > > > > You can do that, but is not nececary. Also you will maybe have a very mutch allocation / deallocation of memory and you can eventually end up with fragmented memory where you have not a nice large block in it at the moment you need it. But it can work, just think over carefully. A better idea is often to make a receive buffer that grows automatically if (and only if) needed, and then just reuse that buffer over and over again. Then you have some (re)allocation in begin but then stable. >TWSocket will automatically split it into packets, > > > > Winsock will split in packets as large as the MTU (around 1500 bytes). Eventually data can (and will if high speed) arrive as 1 large packet but not necacarely. >I really want the sender side to send the 1MB file all at once, since I >do the FileWrite() right after Receive() > > > > No you cannot. Winsock does not respect packet boundaries, but (see prior paragraph) there are no megabytes TCP packets. You have to receive all data chuncks into a buffer, and when you received them all then you save to file (or save every packet direct to disk). There is no problem to know the moment of close the file because you know the length of the data from your protocol. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 11:01, Kei wrote: >Hi! I'm new to ICS! > > > > >I am designing a simple protocol that will be mainly used locally (as a >database server backend)..so I'm guessing I could send up to 2GB of >stuff without hassle (BLOBs, for example). Right now I'm just >experimenting with the facility for two parties to effectively "talk" to >each other, even with long long messages, and with binary data transfer. >For example, for sending a message, the command is "msg [text]" > > > > A->>B: msg hello B->>A: msg yo! how's it going? >If A is larger than the def
Re: [twsocket] OK to Receiving >2GB ?
Hi Everyone I've solved this problem.. what I do is make a count of "How many bytes remaining". Everytime the value returned by Receive() will be subtracted from the "BytesRemaining". When OnDataAvailable is repeatedly called, a buffer is kept being ReAlloc()'ed . When "ByteRemaining" reaches zero, then a WriteFile will be performed, so that the buffer is written to disk.. Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>How do I know the maximum size possible? >> >> > >at receiving side, specially depending on the speed of your own program >you mostly never get above 8 kb. However I have seen receiving packets >10 time as high. But you dont have to care mutch, if you receive not >all, then OnDataAvailable is immediatly called again, meaning if you >have a temporary buffer that is to small, however if you receive on the >stack then you can make him (almost) as large you wants. > > > >>I certainly don't want to malloc 100KB for a TCP packet... >> >> > >See above, you dont need to. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 20:25, Kei wrote: > > > >>Hi! >> >> > > > >>I think I'm going to accept the reality that... TCP packets are splitted >>into arbitrary sizes.. but!!! >>How do I know the maximum size possible? What is the maximum size that a >>packet can be? I certainly don't want to malloc 100KB for a TCP packet... >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > >>Wilfried Mestdagh wrote: >> >> > > > >>>Hello David, >>> >>> >>> >>> >>> Hi! I'm new to ICS! >>>Welcome to the group :) >>> >>> >>> >>> >>> A->>B: msg hello B->>A: msg yo! how's it going? >>>Yes that's the way to go. Design a user made proto for what you intend >>>to do. >>> >>> >>> >>> >>> If A is larger than the default buffer size (256 chars) then the A (sender) will warn B in advance >>>You can do that, but is not nececary. Also you will maybe have a very >>>mutch allocation / deallocation of memory and you can eventually end up >>>with fragmented memory where you have not a nice large block in it at >>>the moment you need it. But it can work, just think over carefully. A >>>better idea is often to make a receive buffer that grows automatically >>>if (and only if) needed, and then just reuse that buffer over and over >>>again. Then you have some (re)allocation in begin but then stable. >>> >>> >>> >>> >>> TWSocket will automatically split it into packets, >>>Winsock will split in packets as large as the MTU (around 1500 bytes). >>>Eventually data can (and will if high speed) arrive as 1 large packet >>>but not necacarely. >>> >>> >>> >>> >>> I really want the sender side to send the 1MB file all at once, since I do the FileWrite() right after Receive() >>>No you cannot. Winsock does not respect packet boundaries, but (see >>>prior paragraph) there are no megabytes TCP packets. >>> >>>You have to receive all data chuncks into a buffer, and when you >>>received them all then you save to file (or save every packet direct to >>>disk). There is no problem to know the moment of close the file because >>>you know the length of the data from your protocol. >>> >>>--- >>>Rgds, Wilfried [TeamICS] >>>http://www.overbyte.be/eng/overbyte/teamics.html >>>http://www.mestdagh.biz >>> >>>Saturday, October 29, 2005, 11:01, Kei wrote: >>> >>> >>> >>> >>> Hi! I'm new to ICS! >>> >>> >>> >>> I am designing a simple protocol that will be mainly used locally (as a database server backend)..so I'm guessing I could send up to 2GB of stuff without hassle (BLOBs, for example). Right now I'm just experimenting with the facility for two parties to effectively "talk" to each other, even with long long messages, and with binary data transfer. For example, for sending a message, the command is "msg [text]" >>>A->>B: msg hello >>>B->>A: msg yo! how's it going? >>> >>> >>> >>> >>> If A is larger than the default buffer size (256 chars) then the A (sender) will warn B in advance, like this: "hey! I'm gonna send you 1 bytes of text" >>>A->>B: longmsg 1 >>>B->>A: ready msg >>>A->>B: msg blahblahblah...blah! >>> >>> >>> >>> >>> In this case, B will be notified of the text size, then when OnClientDataAvailable() event comes, it will malloc a bigger buffer, then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering the same mechanism to send binary data. But if the file is slightly >>>larger (>>10KB) then TWSocket will automatically split it into packets, >>> >
Re: [twsocket] OK to Receiving >2GB ?
Hello David, > How do I know the maximum size possible? at receiving side, specially depending on the speed of your own program you mostly never get above 8 kb. However I have seen receiving packets 10 time as high. But you dont have to care mutch, if you receive not all, then OnDataAvailable is immediatly called again, meaning if you have a temporary buffer that is to small, however if you receive on the stack then you can make him (almost) as large you wants. > I certainly don't want to malloc 100KB for a TCP packet... See above, you dont need to. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 20:25, Kei wrote: > Hi! > I think I'm going to accept the reality that... TCP packets are splitted > into arbitrary sizes.. but!!! > How do I know the maximum size possible? What is the maximum size that a > packet can be? I certainly don't want to malloc 100KB for a TCP packet... > Thanks! > David > Wilfried Mestdagh wrote: >>Hello David, >> >> >> >>>Hi! I'm new to ICS! >>> >>> >> >>Welcome to the group :) >> >> >> >>>A->>B: msg hello >>>B->>A: msg yo! how's it going? >>> >>> >> >>Yes that's the way to go. Design a user made proto for what you intend >>to do. >> >> >> >>>If A is larger than the default buffer size (256 chars) then the A >>>(sender) will warn B in advance >>> >>> >> >>You can do that, but is not nececary. Also you will maybe have a very >>mutch allocation / deallocation of memory and you can eventually end up >>with fragmented memory where you have not a nice large block in it at >>the moment you need it. But it can work, just think over carefully. A >>better idea is often to make a receive buffer that grows automatically >>if (and only if) needed, and then just reuse that buffer over and over >>again. Then you have some (re)allocation in begin but then stable. >> >> >> >>>TWSocket will automatically split it into packets, >>> >>> >> >>Winsock will split in packets as large as the MTU (around 1500 bytes). >>Eventually data can (and will if high speed) arrive as 1 large packet >>but not necacarely. >> >> >> >>>I really want the sender side to send the 1MB file all at once, since I >>>do the FileWrite() right after Receive() >>> >>> >> >>No you cannot. Winsock does not respect packet boundaries, but (see >>prior paragraph) there are no megabytes TCP packets. >> >>You have to receive all data chuncks into a buffer, and when you >>received them all then you save to file (or save every packet direct to >>disk). There is no problem to know the moment of close the file because >>you know the length of the data from your protocol. >> >>--- >>Rgds, Wilfried [TeamICS] >>http://www.overbyte.be/eng/overbyte/teamics.html >>http://www.mestdagh.biz >> >>Saturday, October 29, 2005, 11:01, Kei wrote: >> >> >> >>>Hi! I'm new to ICS! >>> >>> >> >> >> >>>I am designing a simple protocol that will be mainly used locally (as a >>>database server backend)..so I'm guessing I could send up to 2GB of >>>stuff without hassle (BLOBs, for example). Right now I'm just >>>experimenting with the facility for two parties to effectively "talk" to >>>each other, even with long long messages, and with binary data transfer. >>>For example, for sending a message, the command is "msg [text]" >>> >>> >> >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> >> >>>If A is larger than the default buffer size (256 chars) then the A >>>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>>1 bytes of text" >>> >>> >> >>A->>B: longmsg 1 >>B->>A: ready msg >>A->>B: msg blahblahblah...blah! >> >> >> >>>In this case, B will be notified of the text size, then when >>>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering >>>the same mechanism to send binary data. But if the file is slightly >>> >>> >>larger (>>10KB) then TWSocket will automatically split it into packets, >> >> >>>which I don't want it to do: >>> >>> >> >>A->>B: upload 1048576 picture.jpg >>B->>A: ready upload >>A->>B: 01001010101010.. (10720 bytes) >>A->>B: 1010101012.. (10720 bytes) >> >> >>>: >>>: >>> >>> >>A->>B: 01001010101010.. (4023 bytes) >> >> >> >>>I really want the sender side to send the 1MB file all at once, since I >>>do the FileWrite() right after Receive() >>> >>> >> >> >> >>>Could anybody please help me on this issue? >>> >>> >> >> >> >>>Thanks! >>> >>> >> >> >> >>>David >>> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
About 1500 bytes give or take a few over ethernet -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kei Sent: Saturday, October 29, 2005 3:26 PM To: ICS support mailing Subject: Re: [twsocket] OK to Receiving >2GB ? Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as >>a database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" >>to each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im >>considering the same mechanism to send binary data. But if the file is >>slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
http://www.faqs.org/rfcs/rfc879.html -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kei Sent: Saturday, October 29, 2005 2:26 PM To: ICS support mailing Subject: Re: [twsocket] OK to Receiving >2GB ? Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as >>a database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" >>to each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im >>considering the same mechanism to send binary data. But if the file is >>slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since >>I do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
Hi! I think I'm going to accept the reality that... TCP packets are splitted into arbitrary sizes.. but!!! How do I know the maximum size possible? What is the maximum size that a packet can be? I certainly don't want to malloc 100KB for a TCP packet... Thanks! David Wilfried Mestdagh wrote: >Hello David, > > > >>Hi! I'm new to ICS! >> >> > >Welcome to the group :) > > > >>A->>B: msg hello >>B->>A: msg yo! how's it going? >> >> > >Yes that's the way to go. Design a user made proto for what you intend >to do. > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance >> >> > >You can do that, but is not nececary. Also you will maybe have a very >mutch allocation / deallocation of memory and you can eventually end up >with fragmented memory where you have not a nice large block in it at >the moment you need it. But it can work, just think over carefully. A >better idea is often to make a receive buffer that grows automatically >if (and only if) needed, and then just reuse that buffer over and over >again. Then you have some (re)allocation in begin but then stable. > > > >>TWSocket will automatically split it into packets, >> >> > >Winsock will split in packets as large as the MTU (around 1500 bytes). >Eventually data can (and will if high speed) arrive as 1 large packet >but not necacarely. > > > >>I really want the sender side to send the 1MB file all at once, since I >>do the FileWrite() right after Receive() >> >> > >No you cannot. Winsock does not respect packet boundaries, but (see >prior paragraph) there are no megabytes TCP packets. > >You have to receive all data chuncks into a buffer, and when you >received them all then you save to file (or save every packet direct to >disk). There is no problem to know the moment of close the file because >you know the length of the data from your protocol. > >--- >Rgds, Wilfried [TeamICS] >http://www.overbyte.be/eng/overbyte/teamics.html >http://www.mestdagh.biz > >Saturday, October 29, 2005, 11:01, Kei wrote: > > > >>Hi! I'm new to ICS! >> >> > > > >>I am designing a simple protocol that will be mainly used locally (as a >>database server backend)..so I'm guessing I could send up to 2GB of >>stuff without hassle (BLOBs, for example). Right now I'm just >>experimenting with the facility for two parties to effectively "talk" to >>each other, even with long long messages, and with binary data transfer. >>For example, for sending a message, the command is "msg [text]" >> >> > >A->>B: msg hello >B->>A: msg yo! how's it going? > > > >>If A is larger than the default buffer size (256 chars) then the A >>(sender) will warn B in advance, like this: "hey! I'm gonna send you >>1 bytes of text" >> >> > >A->>B: longmsg 1 >B->>A: ready msg >A->>B: msg blahblahblah...blah! > > > >>In this case, B will be notified of the text size, then when >>OnClientDataAvailable() event comes, it will malloc a bigger buffer, >>then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering >>the same mechanism to send binary data. But if the file is slightly >> >> >larger (>>10KB) then TWSocket will automatically split it into packets, > > >>which I don't want it to do: >> >> > >A->>B: upload 1048576 picture.jpg >B->>A: ready upload >A->>B: 01001010101010.. (10720 bytes) >A->>B: 1010101012.. (10720 bytes) > > >>: >>: >> >> >A->>B: 01001010101010.. (4023 bytes) > > > >>I really want the sender side to send the 1MB file all at once, since I >>do the FileWrite() right after Receive() >> >> > > > >>Could anybody please help me on this issue? >> >> > > > >>Thanks! >> >> > > > >>David >> >> > > > > > > > > > > > > > > > > -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] OK to Receiving >2GB ?
Hello David, > Hi! I'm new to ICS! Welcome to the group :) > A->>B: msg hello > B->>A: msg yo! how's it going? Yes that's the way to go. Design a user made proto for what you intend to do. > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B in advance You can do that, but is not nececary. Also you will maybe have a very mutch allocation / deallocation of memory and you can eventually end up with fragmented memory where you have not a nice large block in it at the moment you need it. But it can work, just think over carefully. A better idea is often to make a receive buffer that grows automatically if (and only if) needed, and then just reuse that buffer over and over again. Then you have some (re)allocation in begin but then stable. > TWSocket will automatically split it into packets, Winsock will split in packets as large as the MTU (around 1500 bytes). Eventually data can (and will if high speed) arrive as 1 large packet but not necacarely. > I really want the sender side to send the 1MB file all at once, since I > do the FileWrite() right after Receive() No you cannot. Winsock does not respect packet boundaries, but (see prior paragraph) there are no megabytes TCP packets. You have to receive all data chuncks into a buffer, and when you received them all then you save to file (or save every packet direct to disk). There is no problem to know the moment of close the file because you know the length of the data from your protocol. --- Rgds, Wilfried [TeamICS] http://www.overbyte.be/eng/overbyte/teamics.html http://www.mestdagh.biz Saturday, October 29, 2005, 11:01, Kei wrote: > Hi! I'm new to ICS! > I am designing a simple protocol that will be mainly used locally (as a > database server backend)..so I'm guessing I could send up to 2GB of > stuff without hassle (BLOBs, for example). Right now I'm just > experimenting with the facility for two parties to effectively "talk" to > each other, even with long long messages, and with binary data transfer. > For example, for sending a message, the command is "msg [text]" A->>B: msg hello B->>A: msg yo! how's it going? > If A is larger than the default buffer size (256 chars) then the A > (sender) will warn B in advance, like this: "hey! I'm gonna send you > 1 bytes of text" A->>B: longmsg 1 B->>A: ready msg A->>B: msg blahblahblah...blah! > In this case, B will be notified of the text size, then when > OnClientDataAvailable() event comes, it will malloc a bigger buffer, > then Receive(CustomSized_Buffer, SizeHeToldMe). Similarly Im considering > the same mechanism to send binary data. But if the file is slightly larger (>>10KB) then TWSocket will automatically split it into packets, > which I don't want it to do: A->>B: upload 1048576 picture.jpg B->>A: ready upload A->>B: 01001010101010.. (10720 bytes) A->>B: 1010101012.. (10720 bytes) > : > : A->>B: 01001010101010.. (4023 bytes) > I really want the sender side to send the 1MB file all at once, since I > do the FileWrite() right after Receive() > Could anybody please help me on this issue? > Thanks! > David -- To unsubscribe or change your settings for TWSocket mailing list please goto http://www.elists.org/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
