Re: [fossil-users] Synchronizing endless loop
With the help of the excellent documentation on the Fossil Sync Protocol I have found a very simple solution to fix synchronization of unversioned files. Due to how work is distributed between the server and the client, it's correct that the server ignores "uvigot" cards, but the client should stop sending them. http://www.fossil-scm.org/index.html/doc/trunk/www/sync.wiki I would like to suggest a solution in prosaic form, so you can add it to Fossil without any copyright concerns, if you think it's useful: In the client synchronization code, merge the test for the break condition whether the send limit has been reached with the entry test for the `while' loop, so that the loop is only entered if the size of the output blob is LESS THAN the send limit AND `db_step()' returns SQLITE_ROW. http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=1934 Like this, send_unversioned_file() is only called by the client if there's room for one more file, and no "uvigot" cards will be generated. By not entering the loop, the skipped unversioned file remains in the `uv_tosend' table to be sent later, and `uvDoPush' remains 1 as `db_step()' is not called before assuring there's enough room. Testing if the output blob size is LESS THAN the send limit does not allow the output blob to grow as large as the send limit, whereas before it was allowed to grow up to the send limit. But `send_unversioned_file()' takes the "uvigot" path if the output blob is GREATER THAN OR EQUAL the send limit, so the changed `while' condition uses the same limits as `send_unversioned_file()'. I'm not sure however if "max-upload" can be assumed to always be greater than zero. Through the web interface, "max-upload" can be set to a negative number, so it may be safe to make sure "max-upload" is greater than zero when reading the value from the configuration. http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=1745 --Florian ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Synchronizing endless loop
Andy Bradford: > This error would seem to indicate that your client sync code > doesn't actually know what the uvigot card is Indeed, if I get it right from following the code paths, there are separate functions to handle synchronization for the client: http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=1681-1689 and for the server: http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=1127-1140 The client code is processing the "uvigot" card: http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=2113-2183 The counterpart for the server handler is missing (that's where the "server says: bad command" message comes from), but that's probably intentional, as inferred from this comment: http://www.fossil-scm.org/index.html/artifact/0d5ab26abb?txt=1=682-685 The `send_unversioned_file()' function should only generate "uvigot" cards on the server (that is, for the client), but the function is also called from the client sync code (generating "uvigot" cards for the server), without discrimination of client or server mode. I have rebuilt the Fossil client binary with an extra flag passed to this function to indicate whether "uvigot" cards are desired or not. The error message is gone, but unversioned files are not synchronized at all, that way. So I feel that the server should handle "uvigot" cards as well, and indicate the unversioned files that the client should transfer. Let's see if I dare diving any deeper ... ;-) --Florian ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Synchronizing endless loop
I have another observation regarding the failed synchronization of unversioned files: If the `max-upload' setting is tweaked to be large enough (I set it to the default 25*1000 on both the server and the client) so that the synchronization operation does not require multiple round-trips, unversioned files are synchronized properly. There are only two artifact receipts log entries, one containing all the committed files and check-in metadata, and another one for the unversioned file. --Florian ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Synchronizing endless loop
Thank you very much for looking into my issue! I have built the server from source code with `./configure; make'. I haven't run `make clean' as I was working inside a new directory containing just the extracted Fossil source code. I have now replaced the server binary with the one matching my OS from http://pkg.freebsd.org: Fossil version 1.36 [c24373934d] 2016-10-24 14:59:33 UTC Compiled on Oct 28 2016 19:32:28 using gcc-4.2.1 20070831 patched [FreeBSD] (64-bit) The version hash is the same as for my previous server binary, and also matches that of my Windows client executable (the official download from fossil-scm.org). But duh, I need a faster Internet connection ... On my slower home line, I have always cancelled my tests after around 100 or so iterations of the synchronization operation. Today, on my ultra-fast line at work, I noticed that the `sync -u' operation completes after 130 iterations, but unversioned files are NOT synchronized. The artifact receipts log shows 130 corresponding entries, most of them consisting of only a single artifact. If the synchronization is started without `-u', there are only 18 iterations, and 18 corresponding artifact receipts log entries for the same commit. Each artifact receipts log entry has multiple (usually 5-10) artifacts. So this is a somewhat inflationary use of the term "endless loop" for my part, I'm sorry for my impatience and not waiting for my tests to finish! Updated problem summary === When adding and committing a lot of data, the synchronization operation invoked with `fossil sync -u' emits a "server says: bad command: uvigot ..." for each round-trip, and unversioned files are NOT synchronized. Moreover, the synchronization operation takes many more round-trips to complete, and most of the corresponding artifact receipts log entries consist of only one single artifact. Updated steps to reproduce == * Use Fossil version 1.36 [c24373934d] on the server and the client * Create a new empty repository on the server * Clone the empty repository from the server to the client * Add and commit a lot of files (try the Fossil /src directory) * Add an unversioned file * Run `fossil sync -u' --Florian ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Synchronizing endless loop
Thus said "Andy Bradford" on 30 Oct 2016 23:35:46 -0600: > > Sync with http://some.fossil.server/fossil.cgi/test-repo > > Round-trips: 2 Artifacts sent: 3 received: 0 > > Error: bad command: uvigot home.wiki 1477722315 > > ecb1c59ef204582770184dfc6ddc3f7451224c94 416 > > This error would seem to indicate that your client sync code doesn't > actually know what the uvigot card is (even though it does seem to > know what -u is). Did you build the client from source? If so, did you > run ``make clean'' prior to configuring and building? Correction, after looking at the code, this appears to be coming from the server: http://www.fossil-scm.org/index.html/artifact/0d5ab26abbb86680?ln=1564,1567 So the questions I asked above apply to the server version you reported. Thanks, Andy -- TAI64 timestamp: 40005816e70b ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Synchronizing endless loop
Thus said Florian Balmer on Sat, 29 Oct 2016 09:32:25 +0200: > * Clone the repository from the server to the client > * Add and commit a lot of files (try the Fossil /src directory) > * Add an unversioned file (I named it `home.wiki') > * Run `fossil sync -u' I tried this and it works fine for me. > Sync with http://some.fossil.server/fossil.cgi/test-repo > Round-trips: 2 Artifacts sent: 3 received: 0 > Error: bad command: uvigot home.wiki 1477722315 > ecb1c59ef204582770184dfc6ddc3f7451224c94 416 This error would seem to indicate that your client sync code doesn't actually know what the uvigot card is (even though it does seem to know what -u is). Did you build the client from source? If so, did you run ``make clean'' prior to configuring and building? Thanks, Andy -- TAI64 timestamp: 40005816d856 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
