Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Thomas wrote: > > This could be an option too: > #ifndef IPV6_V6ONLY > #define IPV6_V6ONLY 27 > #endif It compiles with that. But it doesn't work. So for now, I think we'll just stick with IPv4-only servers for "fossil server" on windows - at least until somebody can suggest a way to fix it. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
Oh, and this is not a Windows thingy. For Linux: #include #include http://man7.org/linux/man-pages/man7/ipv6.7.html On 2017-12-29 02:23, Thomas wrote: On 2017-12-29 01:17, Richard Hipp wrote: On 12/28/17, Olivier Mascia wrote: To get a proper dual-stack socket, the socket must be created with AF_INET6 first then setsockopt(s, IPPROTO_IPV6, IPV6_V6ONLY,...) When I try to do this I get: error C2065: 'IPV6_V6ONLY': undeclared identifier MSVC 2015 MSVC 2015 here too, and no error. Maybe the SDK is too old, no idea. Check the bottom of https://msdn.microsoft.com/en-us/library/windows/desktop/ms738574(v=vs.85).aspx . This could be an option too: #ifndef IPV6_V6ONLY #define IPV6_V6ONLY 27 #endif ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 2017-12-29 01:17, Richard Hipp wrote: On 12/28/17, Olivier Mascia wrote: To get a proper dual-stack socket, the socket must be created with AF_INET6 first then setsockopt(s, IPPROTO_IPV6, IPV6_V6ONLY,...) When I try to do this I get: error C2065: 'IPV6_V6ONLY': undeclared identifier MSVC 2015 MSVC 2015 here too, and no error. Maybe the SDK is too old, no idea. Check the bottom of https://msdn.microsoft.com/en-us/library/windows/desktop/ms738574(v=vs.85).aspx . This could be an option too: #ifndef IPV6_V6ONLY #define IPV6_V6ONLY 27 #endif ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Olivier Mascia wrote: > To get a proper dual-stack socket, the socket must be created with AF_INET6 > first then setsockopt(s, IPPROTO_IPV6, IPV6_V6ONLY,...) When I try to do this I get: error C2065: 'IPV6_V6ONLY': undeclared identifier MSVC 2015 -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Replacing subversion revision number... by what?
On Wed, Dec 27, 2017 at 11:56:31PM +0100, Olivier Mascia wrote: > > Le 27 déc. 2017 à 23:24, Joerg Sonnenberger a écrit : > > > > On Wed, Dec 27, 2017 at 10:10:21PM +, bch wrote: > >> On Wed, Dec 27, 2017 at 2:06 PM Olivier Mascia wrote: > >> > >>> Hello, > >>> > >>> Coming from subversion where there is a revision number, incremented by > >>> one by each commit, > >> > >> > >> > >> Let me be the first of many to say that those centrally controlled > >> increments are not possible in a *distributed* source control system. > > > > That's not completely true. You could use the length of the commit chain > > to the root for most of the purposes of the CVS/SVN revision number. > > It's just not necessarily a unique property of a commit. > > Thanks Joerg. > > >> I'm considering replacing the subversion revision ID, for the purpose of > >> defining the file version ID (as above) at release-external build time, by > >> the count of check-ins in the root repository. That is the count returned > >> by 'fossil info' in one of the multiple lines of output (for instance > >> 'check-ins: 8801'). > > My 'count of check-ins' is your 'length of the commit chain to the root', or > are we talking of something else here? If you have a commit graph like: A | B | \ C D | | E F Both E and F have a LoCC of 4, but the count f check-ins would depend on the order of commits? Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] HTML instead of fossil sync protocol
On 28 December 2017 at 12:02, Richard Hipp wrote: > On 12/28/17, jungle Boogie wrote: >> Hi All, >> >> I cannot clone a fossil repo running on a reaspberry pi. >> --fossil-debug was originally used but since then, multiple fossil >> binaries have been built with it disabled. > > Sounds to me like you are still running the older binary that was > built using --fossil-debug. Are you 100% certain that you have > installed the new one? Have you rebooted the machine to make sure you > killed off the old one? > > I do believe the wrong binary in the wrong path was the issue. Thanks for the fast response. Thanks! ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] HTML instead of fossil sync protocol
On 12/28/17, jungle Boogie wrote: > Hi All, > > I cannot clone a fossil repo running on a reaspberry pi. > --fossil-debug was originally used but since then, multiple fossil > binaries have been built with it disabled. Sounds to me like you are still running the older binary that was built using --fossil-debug. Are you 100% certain that you have installed the new one? Have you rebooted the machine to make sure you killed off the old one? > >>fossil clone http://10.7.0.82:8080 cpp_projects.f > Round-trips: 1 Artifacts sent: 0 received: 0 > server replies with HTML instead of fossil sync protocol: > cfile bc64ba9aec57c8dc726c7d920b317f2a06080276030b7b2a31339ee822261b0c 161 > 148 > > Round-trips: 2 Artifacts sent: 0 received: 11 > server replies with HTML instead of fossil sync protocol: > config /reportfmt 579 > 2440587.5 > Round-trips: 2 Artifacts sent: 0 received: 14 > Clone done, sent: 513 received: 3344 ip: 10.7.0.82 > > This is fossil version 2.5 [1fae64de38] 2017-12-28 17:20:14 UTC > for the pi and the windows computer. > > > fossil 2.4 was downloaded from fossil-scm.org > fossil 2.4 was used to clone trunk > fossil from trunk was compiled --with-openssl=none > > In the /setup_ulist page, I do see this: > sort w/o index in [SELECT uid, login, cap, info, > date(mtime,'unixepoch'), lower(login) AS sortkey, CASE WHEN info LIKE > '%expires 20%' THEN substr(info,instr(lower(info),'expires')+8,10) END > AS exp,atime FROM user LEFT JOIN lastAccess ON login=uname WHERE login > NOT IN ('anonymous','nobody','developer','reader') ORDER BY sortkey] > > The repo has been rebuilt with the fossil binary disabled for > debugging, but this is still present. > > Any pointers are appreciated. > > > > -- > --- > inum: 883510009027723 > sip: jungleboo...@sip2sip.info > ___ > fossil-users mailing list > fossil-users@lists.fossil-scm.org > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users > -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] HTML instead of fossil sync protocol
> > Any pointers are appreciated. > > I created a brand new repo with zero commits and I was able to clone it with fossil server running. So it looks like an issue with the repo itself, perhaps enabling fossil debug on the binary caused an issue? > > -- > --- > inum: 883510009027723 > sip: jungleboo...@sip2sip.info -- --- inum: 883510009027723 sip: jungleboo...@sip2sip.info ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] HTML instead of fossil sync protocol
Hi All, I cannot clone a fossil repo running on a reaspberry pi. --fossil-debug was originally used but since then, multiple fossil binaries have been built with it disabled. >fossil clone http://10.7.0.82:8080 cpp_projects.f Round-trips: 1 Artifacts sent: 0 received: 0 server replies with HTML instead of fossil sync protocol: cfile bc64ba9aec57c8dc726c7d920b317f2a06080276030b7b2a31339ee822261b0c 161 148 Round-trips: 2 Artifacts sent: 0 received: 11 server replies with HTML instead of fossil sync protocol: config /reportfmt 579 2440587.5 Round-trips: 2 Artifacts sent: 0 received: 14 Clone done, sent: 513 received: 3344 ip: 10.7.0.82 This is fossil version 2.5 [1fae64de38] 2017-12-28 17:20:14 UTC for the pi and the windows computer. fossil 2.4 was downloaded from fossil-scm.org fossil 2.4 was used to clone trunk fossil from trunk was compiled --with-openssl=none In the /setup_ulist page, I do see this: sort w/o index in [SELECT uid, login, cap, info, date(mtime,'unixepoch'), lower(login) AS sortkey, CASE WHEN info LIKE '%expires 20%' THEN substr(info,instr(lower(info),'expires')+8,10) END AS exp,atime FROM user LEFT JOIN lastAccess ON login=uname WHERE login NOT IN ('anonymous','nobody','developer','reader') ORDER BY sortkey] The repo has been rebuilt with the fossil binary disabled for debugging, but this is still present. Any pointers are appreciated. -- --- inum: 883510009027723 sip: jungleboo...@sip2sip.info ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
Thus said Warren Young on Thu, 28 Dec 2017 08:20:21 -0700: > I'm pretty sure I've never seen Fossil crash in the years I've been > running it. I've had it dump core a few times but those were generally due to bugs that were quickly fixed. Andy -- TAI64 timestamp: 40005a4540f3 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 17:37, Olivier Mascia a écrit : > > The quick/easy/dirty/temporary fix before completing the proper support > (server-side) of IPv4+IPv6 would be to simply adjust the hint.ai_family > initialisation in http_socket.c::socket_open to read: > > #ifdef _WIN32 > hints.ai_family = AF_INET; > #else > hints.ai_family = g.fIPv4 ? AF_INET : AF_UNSPEC; > #endif > > It works nicely for me. Perfectly expected performance of a large clone > (needing 250 round-trips) in about 50 seconds. Either using explicitly IPv4 > in the URL or, with that patch, the proper DNS name. To further clarify, here is the patching I did today based on fossil version 2.5 [f4a9df4dd0] 2017-12-23 04:21:41 UTC. There is that discussed temporary limitation of the client-side (on Windows) to resolving only to IPv4 (to match the server being only IPv4) and another small change which is to call shutdown() on sockets before calling closesocket(). It is more interesting in winhttp.c than in http_socket.c, because it will help the windows TCP stack to more cleanly and generally more quickly free up kernel resources by sending the proper TCP FIN before getting rid of the socket. Kind of warning "I'm pulling the plug" before actually doing so. Don't expect measurable performance improvement from that. The only benefit would be seen under rare situations when the server machine has thousands of other sockets active (maybe other softwares than Fossil). And even so, would be seen only if those other software also behave 'by the book'. Index: src/http_socket.c == --- src/http_socket.c +++ src/http_socket.c @@ -119,10 +119,12 @@ ** is a no-op. */ void socket_close(void){ if( iSocket>=0 ){ #if defined(_WIN32) +if (shutdown(iSocket, 1/*SD_SEND*/) == 0) /* Initiates the shutdown, sending TCP 'FIN' */ + shutdown(iSocket, 0/*SD_RECEIVE*/); /* Now denies receive */ closesocket(iSocket); #else close(iSocket); #endif iSocket = -1; @@ -147,11 +149,15 @@ char zRemote[NI_MAXHOST]; socket_global_init(); socket_close(); memset(&hints, 0, sizeof(struct addrinfo)); +#ifdef _WIN32 + hints.ai_family = AF_INET; +#else hints.ai_family = g.fIPv4 ? AF_INET : AF_UNSPEC; +#endif hints.ai_socktype = SOCK_STREAM; hints.ai_protocol = IPPROTO_TCP; sqlite3_snprintf(sizeof(zPort),zPort,"%d", pUrlData->port); rc = getaddrinfo(pUrlData->name, zPort, &hints, &ai); if( rc ){ Index: src/winhttp.c == --- src/winhttp.c +++ src/winhttp.c @@ -214,10 +214,12 @@ } end_request: if( out ) fclose(out); if( in ) fclose(in); + if (shutdown(p->s, 1/*SD_SEND*/) == 0) /* Initiates the shutdown, sending TCP 'FIN' */ +shutdown(p->s, 0/*SD_RECEIVE*/); /* Now denies receive */ closesocket(p->s); file_delete(zRequestFName); file_delete(zReplyFName); file_delete(zCmdFName); fossil_free(p); @@ -277,10 +279,12 @@ } end_request: if( out ) fclose(out); if( in ) fclose(in); + if (shutdown(p->s, 1/*SD_SEND*/) == 0) /* Initiates the shutdown, sending TCP 'FIN' */ +shutdown(p->s, 0/*SD_RECEIVE*/); /* Now denies receive */ closesocket(p->s); file_delete(zRequestFName); file_delete(zReplyFName); fossil_free(p); } -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 17:18, Richard Hipp a écrit : > > There is one thread per connection in the parent process, which allows > the parent to manage multiple simultaneous incoming connections. As > each thread gets a complete HTTP request, it writes that request into > a temporary file, uses fossil_system() to invoke the "fossil http" > command to handle the request and store the reply into a second > temporary file, then the thread reads the reply and sends it back over > the wire. Finally, the two temporary files are deleted. > > The fork() approach is much nicer, but we don't have fork() on > windows. I'm open to suggests on better ways to do this. Unless your intend is to be able to support thousands of simultaneous requests, there is few to earn in changing this scheme. The scaling limitation would be that spawning of sub-processes. Overcoming that is possible, but is a major redesign of the very clear and simple winhttp.c you currently have. I wouldn't loose precious time in that area. As pointed in my other message, the main culprit of occasional sluggish windows behaviour when fossil reaches out a windows fossil server is that the server inadvertently implements an IPv4-only listening socket while the client is dual-stacked. It can loose seconds (in the order of multiple times 10 or 20 seconds) per IPv6 connection attempt not answered by the remote server before finally trying an IPv4 address and succeeding its connection. -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 16:55, Olivier Mascia a écrit : > >> I'll experiment with the fossil code here. Familiarizing with winhttp.c for >> now. > > I haven't yet come to bottom of it, but using IP (IPv4) in the URL (instead > of name) changes it all!! There seem to be a high price paid in resolving > the server name on each round trip (from client). This or, due to dual-stack > being the norm on Windows, client connections loose time trying IPv6 > (generally preferred by Windows when DNS reports both and A records for > a name) before falling back to IPv4. > > My test now runs (the network portion, excluding rebuilding metadata and next > things) within 50 seconds for all 250 round-trips and 102700 artifacts > received. It was close to 1 hour and a half before. > > I'm further analysing http_socket.c (socket_open) following this discovery. Okay, it revolves around g.fIPv4 which only appears in these lines: Search "fIPv4" (3 hits in 3 files) I:\fossil\src\http_socket.c (1 hit) Line 154: hints.ai_family = g.fIPv4 ? AF_INET : AF_UNSPEC; I:\fossil\src\main.c (1 hit) Line 159: int fIPv4; /* Use only IPv4, not IPv6. --ipv4 */ I:\fossil\src\url.c (1 hit) Line 382: if( find_option("ipv4",0,0) ) g.fIPv4 = 1; In http_socket.c it is tested to configure hints.ai_family with either AF_INET or AF_UNSPEC in order to query for only IPv4 or both IPv6 and IPv4. The server name resolution made by the client then returns both IPv6 and IPv4 because I didn't mentioned "--ipv4" option anywhere in my commands. Further, enumerating the IPs, the code attempts socket creation *and* connection, trying the next returned IP if the connect fails. Sounds good isn't it? No. Because the server side never ever initialised a listening socket for dual-stack. It listens on IPv4 only. Leading to long pauses from the client trying IPv6 connections, unresponding, before trying next one which might happen to be IPv4 and reachable. The server code do initialize its listening socket as this: s = socket(AF_INET, SOCK_STREAM, 0); That (on Windows) gives you an IPv4-limited socket. To get a proper dual-stack socket, the socket must be created with AF_INET6 first then setsockopt(s, IPPROTO_IPV6, IPV6_V6ONLY,...) should be called to remove the "only IPv6" attribute from the socket. See https://msdn.microsoft.com/en-us/library/windows/desktop/ms738574(v=vs.85).aspx. That is not the only thing to do because the test that follows on the validity of the IP address would have to be adjusted to take IPv4/IPv6 into account. So the current situation is simply that when acting as server, fossil.exe setup an IPv4-only listening socket, while by default it will attempt dual-stack client connections, loosing a huge amount of time re-establishing those connections. Add to this that recent Windows will reorder the IPs returned by getaddrinfo() as to list IPv6 first when the OS looks like having proper IPv6 internet connectivity. The quick/easy/dirty/temporary fix before completing the proper support (server-side) of IPv4+IPv6 would be to simply adjust the hint.ai_family initialisation in http_socket.c::socket_open to read: #ifdef _WIN32 hints.ai_family = AF_INET; #else hints.ai_family = g.fIPv4 ? AF_INET : AF_UNSPEC; #endif It works nicely for me. Perfectly expected performance of a large clone (needing 250 round-trips) in about 50 seconds. Either using explicitly IPv4 in the URL or, with that patch, the proper DNS name. I'll see to help with proper Windows dual-stack server socket in January. -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Warren Young wrote: >> >> Even the Windows server starts a new process to handle each request. >> It just does so using system() rather than fork(). > > That leaves me wondering what the threads are used for then, if not to > implement a thread-per-connection model. There is one thread per connection in the parent process, which allows the parent to manage multiple simultaneous incoming connections. As each thread gets a complete HTTP request, it writes that request into a temporary file, uses fossil_system() to invoke the "fossil http" command to handle the request and store the reply into a second temporary file, then the thread reads the reply and sends it back over the wire. Finally, the two temporary files are deleted. The fork() approach is much nicer, but we don't have fork() on windows. I'm open to suggests on better ways to do this. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On Dec 28, 2017, at 9:11 AM, Richard Hipp wrote: > > On 12/28/17, Warren Young wrote: >> >> drh, I seem to recall messages about non-free()’d buffers in the Fossil >> server code, where the patch was rejected with the argument that the forked >> child is about to die anyway, so why bother freeing the buffer. That >> doesn’t really apply in a single-process server, as I expect applies on the >> Windows side. Will Fossil server leak memory on Windows? > > Even the Windows server starts a new process to handle each request. > It just does so using system() rather than fork(). That leaves me wondering what the threads are used for then, if not to implement a thread-per-connection model. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Warren Young wrote: > > drh, I seem to recall messages about non-free()’d buffers in the Fossil > server code, where the patch was rejected with the argument that the forked > child is about to die anyway, so why bother freeing the buffer. That > doesn’t really apply in a single-process server, as I expect applies on the > Windows side. Will Fossil server leak memory on Windows? Even the Windows server starts a new process to handle each request. It just does so using system() rather than fork(). We have to be careful about leaks in the parent process that is sitting there listening for requests and spawning children to handle each request. But there is no need to be concerned with memory leaks in the children, because they are all short-lived. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On Dec 28, 2017, at 8:55 AM, Olivier Mascia wrote: > >> Le 28 déc. 2017 à 15:42, Olivier Mascia a écrit : >> >> I'll experiment with the fossil code here. Familiarizing with winhttp.c for >> now. > > I haven't yet come to bottom of it, but using IP (IPv4) in the URL (instead > of name) changes it all!! That means your domain resolution configuration has a problem. The first lookup will take significant time even on a small quiet LAN, but throughout the entire remaining Fossil clone operation, it will remain cached in the local machine, so it should look it up again near-instantly. (See ipconfig /flushdns) If DNS isn’t working right, Windows might be falling back to NetBIOS name discovery or some other horrid case. > due to dual-stack being the norm on Windows, client connections loose time > trying IPv6 (generally preferred by Windows when DNS reports both and A > records for a name) before falling back to IPv4. Easy to try: uncheck IPv6 in the network control panel. I’ve seen that clear problems up on Windows before. If it does fix it, it might mean you need to set up IPv6 DNS. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 15:42, Olivier Mascia a écrit : > > I'll experiment with the fossil code here. Familiarizing with winhttp.c for > now. I haven't yet come to bottom of it, but using IP (IPv4) in the URL (instead of name) changes it all!! There seem to be a high price paid in resolving the server name on each round trip (from client). This or, due to dual-stack being the norm on Windows, client connections loose time trying IPv6 (generally preferred by Windows when DNS reports both and A records for a name) before falling back to IPv4. My test now runs (the network portion, excluding rebuilding metadata and next things) within 50 seconds for all 250 round-trips and 102700 artifacts received. It was close to 1 hour and a half before. I'm further analysing http_socket.c (socket_open) following this discovery. -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On Dec 28, 2017, at 7:42 AM, Olivier Mascia wrote: > > server on macOS, client on Windows (server 2016 and server 2012 R2) is OK. > Server and client on same Windows box, through localhost is comparable (just > slightly slower) to server macOS and client Windows. That’s not good. The localhost connection should be much faster than a LAN connection, irrespective of the OSes involved. I’d expect to see an order of magnitude difference in tests not involving the disk. (Which of course doesn’t include these tests, but still!) Either Windows’ localhost implementation is terrible, or the Windows server (software or hardware) has some serious slowdown. drh’s idea of disabling antivirus is one possibility, though a recent SQLite benchmark [1] showed that SQLite tended to keep Windows Defender from doing much damage by its nature, since antivirus doesn’t re-scan the DB file each time more content is added to it. [1]: https://www.sqlite.org/fasterthanfs.html ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On Dec 28, 2017, at 6:11 AM, Olivier Mascia wrote: > > *** time skew *** server is slow by 106.6 seconds …snip… > I'll have a close look in the next weeks at the source code of the Windows > server feature of Fossil. The warning "*** time skew *** will certainly be > an interesting starting point. That has nothing to do with the Windows-specific parts of Fossil. It simply means your client and server clocks aren’t synchronized. One (or both?) of them isn’t getting a regular NTP update, or one of them is pointed at an NTP server that is itself desynchronized. Fossil reports this because the skew can be reflected in the file timestamps in such a way that it screws up build systems based on timestamps, which is most of them. > Our own software is actually much like Fossil: single process, multi-usages, > with its own HTTPS embedded server (using OpenSSL). (Albeit currently only > available on Windows, so we have good experience in that area of HTTP serving > on Windows using custom C/C++ code). The current Windows-specific parts of Fossil server use thread-per-connection, according to a recent mailing list message. As you know, that’s not ideal. For a maximum of 10 clients, it’s probably insignificant, but if you could rewrite it using WSAEventSelect() or similar, that would be more efficient. That might be really difficult, though, since the POSIX builds of Fossil server use fork() instead, which means the server code probably isn’t written in a way that would easily give you the data to ship in an event-driven manner. drh, I seem to recall messages about non-free()’d buffers in the Fossil server code, where the patch was rejected with the argument that the forked child is about to die anyway, so why bother freeing the buffer. That doesn’t really apply in a single-process server, as I expect applies on the Windows side. Will Fossil server leak memory on Windows? ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On Dec 28, 2017, at 4:26 AM, Olivier Mascia wrote: > > Is "fossil server" generally deemed sufficient for a small LAN (lab) of let's > say < 10 developers each with their own clone ? > Or should a proper HTTP server be used as front end to SCGI mode of fossil? That’s how I run Fossil on private civilized LANs. (As opposed to LANs where bad actors are expected to be present.) I don’t bother with the HTTPS front-end proxy in that case. > cloning a 2 GB repository through http over 1 Gbps LAN is curiously slow and > I wonder why. A Fossil clone is not a linear file copy operation. The server pulls some data from its database, ships it over the network, then the client has to integrate that into its local database. Back and forth it goes until the whole thing is transferred. Then the client’s going to spend a bunch more time reorganizing what it received into an optimal format. This is one reason why Fossil reports round trips during clone and sync operations. It’s good that it’s as low a value as it is, but it isn’t yet “1”, the ideal. > Not that we'll clone every day of course, but it looks so slow that it > questions stability. I’m pretty sure I’ve never seen Fossil crash in the years I’ve been running it. Keep in mind that sqlite.org runs on Fossil. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 15:04, Richard Hipp a écrit : > > On 12/28/17, Richard Hipp wrote: >> >> Running multiple copies of the new fossil-stress.tcl script >> (https://www.fossil-scm.org/fossil/file/tools/fossil-stress.tcl) from >> a linux box against a windows laptop confirms that the "fossil server" >> command on Windows sometimes latches up. > > Preliminary findings are that this latch-up is the result of an > adverse interaction with built-in anti-virus software on Win10. You > *might* have more success by disabling anti-virus on your server - if > you dare. We'll try to figure out a way to fix the problem in fossil, > in the meantime. Confirmed that server on macOS, client on Windows (server 2016 and server 2012 R2) is OK. Server and client on same Windows box, through localhost is comparable (just slightly slower) to server macOS and client Windows. I'll dig in the source code over the week-end. Generally, there are subtle TCP window issues to take into account when configuring sockets on various Windows versions, along side with setting up the 'just-right' buffer sizes on those sockets to get it right and not have traffic stalling unexpectedly. Been there in our own projects. I'll experiment with the fossil code here. Familiarizing with winhttp.c for now. -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Richard Hipp wrote: > > Running multiple copies of the new fossil-stress.tcl script > (https://www.fossil-scm.org/fossil/file/tools/fossil-stress.tcl) from > a linux box against a windows laptop confirms that the "fossil server" > command on Windows sometimes latches up. Preliminary findings are that this latch-up is the result of an adverse interaction with built-in anti-virus software on Win10. You *might* have more success by disabling anti-virus on your server - if you dare. We'll try to figure out a way to fix the problem in fossil, in the meantime. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Richard Hipp wrote: > > The "fossil server" command *should* be sufficient for this. However, > it doesn't get used that much. Most people run from a proper HTTP > server of some kind. Consequently, it is possible for bugs to linger > in "fossil server" without anybody noticing. Running multiple copies of the new fossil-stress.tcl script (https://www.fossil-scm.org/fossil/file/tools/fossil-stress.tcl) from a linux box against a windows laptop confirms that the "fossil server" command on Windows sometimes latches up. I'll try to figure out why. Or get somebody with more Windows experience to figure out why for me :-) -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 13:51, Richard Hipp a écrit : > > On 12/28/17, Olivier Mascia wrote: >> >> Is "fossil server" generally deemed sufficient for a small LAN (lab) of >> let's say < 10 developers each with their own clone ? >> Or should a proper HTTP server be used as front end to SCGI mode of fossil? > > The "fossil server" command *should* be sufficient for this. However, > it doesn't get used that much. Most people run from a proper HTTP > server of some kind. Consequently, it is possible for bugs to linger > in "fossil server" without anybody noticing. One such performance > bug, that had been in the code for over 10 years, was fixed just > before Chrismas > (https://www.fossil-scm.org/fossil/info/05ec15cad53e8176). That > particular fix applies to unix systems only, but there might be > similar problems on the windows side. > > As a test of last weeks' fix, I am currently running a clone of the > Fossil self-hosting repository using the "fossil server" command in a > datacenter near Paris on a machine that is approximately equal to a > Raspberry Pi. The clone is internet-facing > (http://www4.fossil-scm.org/) and so far seems to be holding up fine. > > I just took a peek at the code for "fossil server" on windows. > (Unlike most other commands in fossil, the "fossil server" command > requires a separate implementation in windows.) I am reminded that > windows is not the ideal platform on which to host a web server and > that compromises had to be made in the implementation. It *should* > work. And it does work fine for things like "fossil ui". But if you > start hitting it with any kind of load, I can see that it might easily > bog down. Probably it will take someone with more windows-foo than me > to fix that. > > So if you are having problems using "fossil server" on Windows, you > might do well to set up a Raspberry PI (or the equivalent) in the > corner of the room and run "fossil server" there instead, using the > latest "trunk" version of Fossil that contains the Christmas-eve patch > for the "fossil server" inefficiency. Thanks a lot Richard for this insider view on the matter. I'll not get down to Raspberry PI (which I really like by the way) but spawn a common linux VM for the task and I'm confident, reading your comments, that it will be just fine. By the way, in between, the clone completed successfully. Here is the display at end (project ids and password removed). Round-trips: 250 Artifacts sent: 0 received: 102700 *** time skew *** server is slow by 106.6 seconds Clone done, sent: 85932 received: 2249557802 ip: 10.32.1.6 Rebuilding repository meta-data... 100.0% complete... Extra delta compression... Vacuuming the database... I'll have a close look in the next weeks at the source code of the Windows server feature of Fossil. The warning "*** time skew *** will certainly be an interesting starting point. Hopefully I might be able to bring something in that area. Our own software is actually much like Fossil: single process, multi-usages, with its own HTTPS embedded server (using OpenSSL). (Albeit currently only available on Windows, so we have good experience in that area of HTTP serving on Windows using custom C/C++ code). For information, my strange experience today was running this code (on both server and cloner side): This is fossil version 2.5 [f4a9df4dd0] 2017-12-23 04:21:41 UTC Compiled on Dec 27 2017 16:08:53 using msc-19.11 (64-bit) -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
On 12/28/17, Olivier Mascia wrote: > > Is "fossil server" generally deemed sufficient for a small LAN (lab) of > let's say < 10 developers each with their own clone ? > Or should a proper HTTP server be used as front end to SCGI mode of fossil? The "fossil server" command *should* be sufficient for this. However, it doesn't get used that much. Most people run from a proper HTTP server of some kind. Consequently, it is possible for bugs to linger in "fossil server" without anybody noticing. One such performance bug, that had been in the code for over 10 years, was fixed just before Chrismas (https://www.fossil-scm.org/fossil/info/05ec15cad53e8176). That particular fix applies to unix systems only, but there might be similar problems on the windows side. As a test of last weeks' fix, I am currently running a clone of the Fossil self-hosting repository using the "fossil server" command in a datacenter near Paris on a machine that is approximately equal to a Raspberry Pi. The clone is internet-facing (http://www4.fossil-scm.org/) and so far seems to be holding up fine. I just took a peek at the code for "fossil server" on windows. (Unlike most other commands in fossil, the "fossil server" command requires a separate implementation in windows.) I am reminded that windows is not the ideal platform on which to host a web server and that compromises had to be made in the implementation. It *should* work. And it does work fine for things like "fossil ui". But if you start hitting it with any kind of load, I can see that it might easily bog down. Probably it will take someone with more windows-foo than me to fix that. So if you are having problems using "fossil server" on Windows, you might do well to set up a Raspberry PI (or the equivalent) in the corner of the room and run "fossil server" there instead, using the latest "trunk" version of Fossil that contains the Christmas-eve patch for the "fossil server" inefficiency. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] fossil server on a small private LAN
> Le 28 déc. 2017 à 12:26, Olivier Mascia a écrit : > > Dear, > > Is "fossil server" generally deemed sufficient for a small LAN (lab) of let's > say < 10 developers each with their own clone ? > Or should a proper HTTP server be used as front end to SCGI mode of fossil? > > From some quick-n-dirty tests (yeah, on a Windows server, not the best of all > ideas but simply convenient right now), it looks like cloning a 2 GB > repository through http over 1 Gbps LAN is curiously slow and I wonder why. > Not that we'll clone every day of course, but it looks so slow that it > questions stability. To give context to this situation, I'm now 45 minutes later and it still is not complete. fossil clone http://olivier@something:8081/repotest repotest-om.fossil password for olivier: * remember password (Y/n)? Y Round-trips: 138 Artifacts sent: 0 received: 82563 ... The computer running fossil server is doing "nothing", 1 or 2% cpu consumed, LAN mostly calm. The computer pulling the clone does the same. I'm not seeing significant disk IO on any of them. The LAN segment they're both on has a background noise of about 1 Mbps with short peaks of less than 50 Mbps while this is ongoing. It really looks as if either the fossil process running the clone OR the fossil process running the server is abnormally slow doing its work. I'll probably retry such a clone locally (over http but both processes on same machine, through localhost) to try to get an understanding of what's going on. I must admit it awfully looks like a pure networking issue, but I tested a file transfer minutes before rerunning this clone test and I peaked 940 Mbps on the wire easily. A Gremlin, there is. Who's got some water? -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] fossil server on a small private LAN
Dear, Is "fossil server" generally deemed sufficient for a small LAN (lab) of let's say < 10 developers each with their own clone ? Or should a proper HTTP server be used as front end to SCGI mode of fossil? From some quick-n-dirty tests (yeah, on a Windows server, not the best of all ideas but simply convenient right now), it looks like cloning a 2 GB repository through http over 1 Gbps LAN is curiously slow and I wonder why. Not that we'll clone every day of course, but it looks so slow that it questions stability. I'll spawn a linux VM or re-test from a Mac machine to get new experiences, but maybe there are anyway things to pay attention to, which I didn't (yet). -- Best Regards, Meilleures salutations, Met vriendelijke groeten, Olivier Mascia ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users