Re: [twsocket] IPv6 implementation details
Hello, We use TStringList quicksort/binary search routine I wrote for caching--works so well that I can recommend. Also, has anybody tested the IPv6 with listening/server components? Best Regards, SZ On Wed, Jun 2, 2010 at 10:04 PM, Francois PIETTE francois.pie...@skynet.bewrote: How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Shouldn't TWSocket have his own internal cache for DNS lookup ? Or maybe instead of more complex TWSocket, maybe a component dedicated to DNS lookup could have it. As simple strategy inside TWSocket would then be enough for most applications while high load applications could benefit from a caching component. This is also, imo, good design to offload a specific task to a specific class. Of course this could be done in each application. AVL tree used in SSL is a good candidate to build a cache system. -- francois.pie...@overbyte.be http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: Hello, We use TStringList quicksort/binary search routine I wrote for caching--works so well that I can recommend. With many entries all TList-derrived classes are dog-slow compared with binary search trees for this purpose! Also, has anybody tested the IPv6 with listening/server components? I just commited a few changes. The HTTP client and server should work now. Please guys, give it a trial and report bugs or provide fixes. Log: Setter of TCustomWSocket.Addr sets the correct internal socket family if a host name equals either a valid IPv6 or IPv4 address, overriding public property SocketFamily. THttpCli added property SocketFamily. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
On Thu, Jun 3, 2010 at 3:43 PM, Arno Garrels arno.garr...@gmx.de wrote: Fastream Technologies wrote: Hello, We use TStringList quicksort/binary search routine I wrote for caching--works so well that I can recommend. With many entries all TList-derrived classes are dog-slow compared with binary search trees for this purpose! We are using Sorted=true; while adding items and no IndexOf() but our own routine based on searching in a sorted stringlist. Also, has anybody tested the IPv6 with listening/server components? I just commited a few changes. The HTTP client and server should work now. Please guys, give it a trial and report bugs or provide fixes. Log: Setter of TCustomWSocket.Addr sets the correct internal socket family if a host name equals either a valid IPv6 or IPv4 address, overriding public property SocketFamily. THttpCli added property SocketFamily. What happens if one sets 0.0.0.0 as listening IP? Does it cover the IPv6 interface as well? I mean, our customers are seeking simplicity--can they listen on both IPv's with single listening socket? Best Regards, SZ -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: We use TStringList quicksort/binary search routine I wrote for caching--works so well that I can recommend. With many entries all TList-derrived classes are dog-slow compared with binary search trees for this purpose! We are using Sorted=true; while adding items and no IndexOf() but our own routine based on searching in a sorted stringlist. Yes that is how I compared a custom TObjectList (sorted, with binary search) and my AVL TObjectTree. Tested with 1 Million inserts, I decided to kill the test application after one or two minutes while it was still inserting. Here are the results of a binary search tree running on an old PentiumD: Tree 1Mil inserts: 578 msec Tree 1 worst case searches: 0 msec Tree 10x iter over 1Mil: 625 msec Tree Clear 1Mil: 375 msec Iterating over trees is of course slower than over TList. What happens if one sets 0.0.0.0 as listening IP? Does it cover the IPv6 interface as well? I mean, our customers are seeking simplicity--can they listen on both IPv's with single listening socket? No, either IPv6 or IPv4. If you want to listen on multiple interfaces or different IP versions you have to create another listening socket as with SSL too. Also mapped IPv4 addresses are not supported (yet?) -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
We use TStringList quicksort/binary search routine I wrote for caching--works so well that I can recommend. With many entries all TList-derrived classes are dog-slow compared with binary search trees for this purpose! Confirmed. I recently had a project with such issue. The result where extremely bad with a simple TList and really super fast with Arno's AVL tree. -- francois.pie...@overbyte.be http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Hello, Has the IPv6 async DNS resolving issue been resolved? How many socket/threads? Best Regards, SZ On Sun, May 30, 2010 at 6:26 PM, Arno Garrels arno.garr...@gmx.de wrote: Angus Robertson - Magenta Systems Ltd wrote: FTP also still not supporting IPv6 FTP needs the EPRT and EPSV commands to support IP6: http://tools.ietf.org/html/draft-ietf-ftpext-ftp-over-ipv6-01 from 1998, but IP6 has been slow arriving. The latest version of Serv-U supports these IP6 commands. I did check with my ISP about IP6, they have addresses allocated by RIPE but have not yet configured their network to use IP6. I suspect it won't happen without a good commercial reason. To use and test IPv6 it is enough to have a IPv6 capable LAN or just one PC with IPv6, name resolution works fine with simple host files. My provider provides 6to4 tunnels. I opened a new svn branch with my current IPv6 changes: URL is svn://svn.overbyte.be/ics/branches/icsipv6 User and password = ics. You and anybody else is invited to implement the FTP or other still missing parts. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Arno Garrels wrote: Fastream Technologies wrote: Hello, Has the IPv6 async DNS resolving issue been resolved? How many socket/threads? There is a (hopefully) thread-safe new class which provides async lookups. One global object of this class is allocated in unit initialization with the maximum number of threads that it may spawn currently set to the number of processors.´ How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Since once the internal call to GetAddrInfo() is fired cancelation of this request won't free this thread for new task (it has to wait until GetAddrInfo() returned before it is able to pick another request from the request queue). -- Arnio Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Hello, On Wed, Jun 2, 2010 at 7:33 PM, Arno Garrels arno.garr...@gmx.de wrote: Arno Garrels wrote: Fastream Technologies wrote: Hello, Has the IPv6 async DNS resolving issue been resolved? How many socket/threads? There is a (hopefully) thread-safe new class which provides async lookups. One global object of this class is allocated in unit initialization with the maximum number of threads that it may spawn currently set to the number of processors.´ How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Since once the internal call to GetAddrInfo() is fired cancelation of this request won't free this thread for new task (it has to wait until GetAddrInfo() returned before it is able to pick another request from the request queue). -- Arnio Garrels I do not have much free time these days but why don't you use the web stress tester BCB source at http://www.fastream.com/webstresstester.php which can open tens of thousands of concurrent connections for testing this new feature? Best Regards, SZ -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
On Wed, Jun 2, 2010 at 7:33 PM, Arno Garrels arno.garr...@gmx.de wrote: Arno Garrels wrote: Fastream Technologies wrote: Hello, Has the IPv6 async DNS resolving issue been resolved? How many socket/threads? There is a (hopefully) thread-safe new class which provides async lookups. One global object of this class is allocated in unit initialization with the maximum number of threads that it may spawn currently set to the number of processors.´ How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Since once the internal call to GetAddrInfo() is fired cancelation of this request won't free this thread for new task (it has to wait until GetAddrInfo() returned before it is able to pick another request from the request queue). -- Arnio Garrels I think this approach could be ok for our reverse proxy where the number of target servers is limited and we have built-in DNS cache. But for content/forward proxy, under heavy load, this would cause a bottleneck. SZ -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: On Wed, Jun 2, 2010 at 7:33 PM, Arno Garrels arno.garr...@gmx.de wrote: Arno Garrels wrote: Fastream Technologies wrote: Hello, Has the IPv6 async DNS resolving issue been resolved? How many socket/threads? There is a (hopefully) thread-safe new class which provides async lookups. One global object of this class is allocated in unit initialization with the maximum number of threads that it may spawn currently set to the number of processors.´ How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Since once the internal call to GetAddrInfo() is fired cancelation of this request won't free this thread for new task (it has to wait until GetAddrInfo() returned before it is able to pick another request from the request queue). -- Arnio Garrels I think this approach could be ok for our reverse proxy where the number of target servers is limited and we have built-in DNS cache. But for content/forward proxy, under heavy load, this would cause a bottleneck. Question, how do you think Windows achives async behaviour? For instance when you call API WSAAsyncGetHostByName(). Without using threads under the hood, it's AFAIK imposible. -- Arno Garrels SZ -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
How fast/slow this implementation works with thousands of concurrent requests is of course untested. One DNS lookup may take one or even multiple seconds in the worst case. Shouldn't TWSocket have his own internal cache for DNS lookup ? Or maybe instead of more complex TWSocket, maybe a component dedicated to DNS lookup could have it. As simple strategy inside TWSocket would then be enough for most applications while high load applications could benefit from a caching component. This is also, imo, good design to offload a specific task to a specific class. Of course this could be done in each application. AVL tree used in SSL is a good candidate to build a cache system. -- francois.pie...@overbyte.be http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Angus Robertson - Magenta Systems Ltd wrote: FTP also still not supporting IPv6 FTP needs the EPRT and EPSV commands to support IP6: http://tools.ietf.org/html/draft-ietf-ftpext-ftp-over-ipv6-01 from 1998, but IP6 has been slow arriving. The latest version of Serv-U supports these IP6 commands. I did check with my ISP about IP6, they have addresses allocated by RIPE but have not yet configured their network to use IP6. I suspect it won't happen without a good commercial reason. To use and test IPv6 it is enough to have a IPv6 capable LAN or just one PC with IPv6, name resolution works fine with simple host files. My provider provides 6to4 tunnels. I opened a new svn branch with my current IPv6 changes: URL is svn://svn.overbyte.be/ics/branches/icsipv6 User and password = ics. You and anybody else is invited to implement the FTP or other still missing parts. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Not sure if it is the same issue but with our proxy server code with many
threads, the debugger sometimes gives AV on Wndproc when step run too.
SZ
On Sat, May 29, 2010 at 11:38 AM, Arno Garrels arno.garr...@gmx.de wrote:
Hi,
When new property SocketFamily is set to sfAny method
DnsLookup may now return a list containing both IPv4 and IPv6
strings. Looks like Windows always returns IPv6 first.
The top item is also copied to string FDnsResult.
Should I change this order?
BTW.: I wrote some helper classes to emulate the
WSAAsyncGetHostByxx functions.
While I was testing them I got strange AV's upon debugging
and stepping thru the multi-threaded code. Took me several days
to prove that this is not caused by my code but either a
debugger bug (my favorite) or some other condition (driver,
root kit or whatsoever).
I made a very simple test case which triggers the AV on different
hardware and Windows versions in both D7 and D2010 here.
However some German testers were not able to reproduce the issue,
very strange.
unit Unit1;
interface
uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, StdCtrls;
const
WM_TEST = WM_USER + 1;
type
TMyThread = class(TThread)
protected
FWindowHandle: HWND;
procedure Execute; override;
public
constructor Create(AHWnd: HWND); reintroduce;
end;
TForm1 = class(TForm)
Memo1: TMemo;
Button1: TButton;
procedure Button1Click(Sender: TObject);
private
procedure WmTest(var Msg: TMessage); message WM_TEST;
public
{ Public-Deklarationen }
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
{ TMyThread }
constructor TMyThread.Create(AHWnd: HWND);
begin
FWindowHandle := AHWnd;
inherited Create(FALSE);
end;
procedure TMyThread.Execute;
var
I : Integer;
begin
I := 0;
while not Terminated do
begin
Sleep(500);
SendMessage(FWindowHandle, WM_TEST, ThreadID, I);
Inc(I);
end;
end;
procedure TForm1.WmTest(var Msg: TMessage);
begin
Memo1.Lines.Add(IntToHex(Msg.WParam, 8) + ' ' + IntToStr(Msg.LParam));
end;
procedure TForm1.Button1Click(Sender: TObject);
var
Thread : TMyThread;
I : Integer;
begin
IsMultiThread := TRUE;
for I := 1 to 4 do // == Set a break point here and repeat pressing F8
until the AV is raised.
Thread := TMyThread.Create(Handle);
end;
end.
--
To unsubscribe or change your settings for TWSocket mailing list
please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket
Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
When new property SocketFamily is set to sfAny method DnsLookup may now return a list containing both IPv4 and IPv6 strings. Looks like Windows always returns IPv6 first. The top item is also copied to string FDnsResult. Should I change this order? An idea: Give the opportunity for the developer to select the priority. Maybe splitting sfAny to two values: one with IPv6 priority and one with IPv4 priority. Maybe sfAny should be kept and mapped to Windows own default while the two new values could override Windows default. I think the component default should be IPv4 to get best backward compatibility. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
RTT wrote: Is happening here too, but only if I continue with the step over debugging (F8) after the 4 thread are created. Never happened while they are created. No idea if that's what you want to say with repeat pressing F8 until the AV is raised. The debugger will keep stepping the WmTest function, until the AV is raised. Windows Vista, and happens in both D7 and D2009. Usually stops in the thread inc(I) line, but not always. Thanks for testing, I'm happy that someone else is able to reproduce the AV! Yes, that is exactly what I do/get as well. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
I made a very simple test case which triggers the AV on different hardware and Windows versions in both D7 and D2010 here. However some German testers were not able to reproduce the issue, very strange. I cannot reproduce the issue. I'm using Vista Ultimate 32 bit with all services packs installed except IE8. I'm using Delphi 14.0.3513.24210 french. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Currently I made one breaking change that IMO cannot be prevented without tons of workaround code. That is type change of TCustomWSocket public field/property sin from TSockAddrIn to TSockAddrIn6. TSockAddrIn6 can be passed to all old functions by typ casting it to TSockAddrIn and passing the size of TSockAddrIn. However if anybody has an idea how to easily avoid this breaking change please let me know. Keep the current property as it is and use it for anything related to IPv4. Create a new property sin6 : TSockAddrIn6 and use if for IPv6. Anyone recompiling an existing program with IPv4 will have no change to do in his code. Anyone willing to support IPv6 will have to do some changes, including this one. Instead of creating two variables, maybe only create one, the one being the largest. Then map the properties sin and sin6 to the same location using getter and setter. Propably the variable could be a variant record with both types to makes the code clearer because this would remove most typecasts. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: Maybe it is another 64-bit Windows/debugger API issue? I am using Win7 Pro x64 and had been using 64-bit XP and Vista for 4 years... I do not think it's related to 64-Bit Windows. I tested and got the AV with Delphi 7 and Delphi 2010 on XP-Pro SP3 (fully patched), PentiumD DualCore, as well as with Delphi 2010 on Win7 64-Bit, i5 QuadCore. Tested on real machines. Both have VMware and Delphi 2010 installed. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
I said so because I had faced a WinXP/Vista x64 issue (BLUE SCREEN) which MS fixed in Win7. I thought it might be something similar. BTW, did you see the message I sent to you privately? It's been a while since I heard back from you to my replies and I am beginning to think that it could be a spam filter issue on your side. Regards, SZ On Sat, May 29, 2010 at 2:28 PM, Arno Garrels arno.garr...@gmx.de wrote: Fastream Technologies wrote: Maybe it is another 64-bit Windows/debugger API issue? I am using Win7 Pro x64 and had been using 64-bit XP and Vista for 4 years... I do not think it's related to 64-Bit Windows. I tested and got the AV with Delphi 7 and Delphi 2010 on XP-Pro SP3 (fully patched), PentiumD DualCore, as well as with Delphi 2010 on Win7 64-Bit, i5 QuadCore. Tested on real machines. Both have VMware and Delphi 2010 installed. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: I said so because I had faced a WinXP/Vista x64 issue (BLUE SCREEN) which MS fixed in Win7. I thought it might be something similar. BTW, did you see the message I sent to you privately? It's been a while since I heard back from you to my replies and I am beginning to think that it could be a spam filter issue on your side. Last long weekend I was out of town for a few days and after reading your mails I thought you solved the problem yourself. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Fastream Technologies wrote: Ok, no problem. BTW, we are very interested in OpenSSL accelerators It's already included as an experimental feature since a while, however there was zero feedback from developers owning accelerator hardware yet. All you need to test is a system with an accelerator card, AFAIK Dell offers such boxes from the shelf. and IPv6 supports. Of course as much as I can find time, I wanted to help you out. Turned out to be more work than initially planned. Maybe we should create a svn branch for it? -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Turned out to be more work than initially planned. Maybe we should create a svn branch for it? What makes you think another branch would be useful ? -- francois.pie...@overbyte.be http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Francois PIETTE wrote: Turned out to be more work than initially planned. Maybe we should create a svn branch for it? What makes you think another branch would be useful ? Better testing before it's finally commited to the main branch. I did some general cleanup including removal of all old conditional code. Winsock.pas can no longer be used so I provide a custom version (v2) in OverbyteIcsWinsock.pas. That means a semi-breaking change, since instead of including winsock.pas, projects have to include/use OverbyteIcsWinsock.pas. I also moved all stuff that is needed to load the libraries to OverbyteIcsWinsock.pas. The Interface and Implemaéntation section(s) of OverbyteIcsWinsock.pas are .inc files for cleaner and easier crossplattform support. I'm now implementing your idea to prevent the type change of field sin however that also means plenty of ifs and thens FTP also still not supporting IPv6 and most likely I overlooked some required changes. Broadcasting is replaced in IPv6 by Multicasting (I'm not very familiar with this stuff) however seems to require another API to deal with interfaces. -- Arno Garrels -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
Turned out to be more work than initially planned. Maybe we should create a svn branch for it? What makes you think another branch would be useful ? Better testing before it's finally commited to the main branch. OK, go on. -- francois.pie...@overbyte.be The author of the freeware multi-tier middleware MidWare The author of the freeware Internet Component Suite (ICS) http://www.overbyte.be -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
Re: [twsocket] IPv6 implementation details
FTP also still not supporting IPv6 FTP needs the EPRT and EPSV commands to support IP6: http://tools.ietf.org/html/draft-ietf-ftpext-ftp-over-ipv6-01 from 1998, but IP6 has been slow arriving. The latest version of Serv-U supports these IP6 commands. I did check with my ISP about IP6, they have addresses allocated by RIPE but have not yet configured their network to use IP6. I suspect it won't happen without a good commercial reason. Angus -- To unsubscribe or change your settings for TWSocket mailing list please goto http://lists.elists.org/cgi-bin/mailman/listinfo/twsocket Visit our website at http://www.overbyte.be
