Re: [Monotone-devel] Re: [ANN] mtn2git v0.1
Hi, Felipe Contreras wrote: Monotone can have multiple heads in one single branch, but from what I understand that mostly happens locally (not on the published repo). Uh.. no, I'd say quite the opposite: if multiple developers work on the same branch and do their commits, the first time both heads appear in the same branch on the same machine will be the public repository. Anyway, If that happens the commits are still there, just dangling temporarily in no branch. ..for the mtn repo imported into git that is, right? Regards Markus Wanner (ne Schiltknecht) ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] using empty() instead of size()
Hi, Zack Weinberg wrote: Sure, but C++98 requires size() to be O(1) for every container, so there's really no excuse for a 4-15% performance difference. Here are the results I got from this test run. It's a debian lenny machine, comparing these revisions: new: 805c482bc9bb80cd393be7d3ba01a65377d91d9c ref: d1bd931bd76507267b99e57be07125dbb2335fd5 (which is from nvm.cvsimport-branch-reconstruction, but that does only change rcs_import.cc, so annotate shouldn't be affected at all) annotate-a-ref.csv annotate-a-new.csv p annotate-avg-resident-MiB 30.12 23.72 0.00 annotate-avg-size-MiB 125.68 119.28 0.00 annotate-max-resident-MiB 40.38 34.22 0.00 annotate-max-size-MiB 136.90 130.86 0.00 annotate-num-samples2089.201982.80 0.16 annotate-system-time 0.82 0.85 0.11 annotate-user-time 4.35 3.49 0.00 annotate-wall-time 10.60 10.06 0.16 annotate-b-ref.csv annotate-b-new.csv p annotate-avg-resident-MiB 17.62 11.75 0.00 annotate-avg-size-MiB 110.95 103.95 0.00 annotate-max-resident-MiB 24.82 18.66 0.00 annotate-max-size-MiB 121.46 115.32 0.00 annotate-num-samples 702.60 599.80 0.17 annotate-system-time 0.19 0.18 0.26 annotate-user-time 0.68 0.51 0.00 annotate-wall-time 3.59 3.05 0.16 annotate-c-ref.csv annotate-c-new.csv p annotate-avg-resident-MiB 18.42 12.45 0.00 annotate-avg-size-MiB 111.05 104.89 0.00 annotate-max-resident-MiB 24.48 18.30 0.00 annotate-max-size-MiB 121.06 114.96 0.00 annotate-num-samples 778.40 712.40 0.05 annotate-system-time 0.25 0.24 0.67 annotate-user-time 0.95 0.74 0.00 annotate-wall-time 3.95 3.63 0.06 annotate-d-ref.csv annotate-d-new.csv p annotate-avg-resident-MiB 34.34 28.24 0.00 annotate-avg-size-MiB 130.00 124.07 0.00 annotate-max-resident-MiB 42.30 35.93 0.00 annotate-max-size-MiB 138.82 132.57 0.00 annotate-num-samples3443.203213.50 0.00 annotate-system-time 1.02 1.02 0.84 annotate-user-time 8.85 7.82 0.00 annotate-wall-time 17.48 16.29 0.00 This leads me to think that the STL implementation doesn't provide an O(1) implementation for size()... savings is avg memory consumption seems to confirm this, no? Other explanations? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] using empty() instead of size()
Hi, Derek Scherger wrote: Even if size does have to traverse a list to count the elements one would expect that traversing an *empty* list should be reasonably quick. Yeah, but it the list is large, traversing is rather expensive, compared to aborting and returning false upon seeing the first element, as empty() can do. It's the difference between wanting to know the exact size or just being interested if there are some elements or not. All the same, empty() does seem a bit easier on the eyes Yeah, I also like empty() better. and if it wins on performance too then what the heck. For what it's worth, I suspected the slightly increased binary size and additional link time required for the nvm.cbr branch's binary and rerun the benchmark again just exactly the revision before: new: 805c482bc9bb80cd393be7d3ba01a65377d91d9c ref: 639a3179473e718e8b01b9c99232fa1f81ca7f61 This time I took special care to shut down other services on the machine so that the benchmark should really be the absolutely only load. With enough test runs I cannot really confirm the speed gain. See the three results below. I don't think it's really the nvm.cbr branch, as the executable didn't grow significantly. It would at least really surprise me. I rather think it's not such a good benchmark, because it varies quite a lot and it seems to measure a lot of disc i/o due to the drop_caches. I'm not sure we really want to exclude that from our benchmarks, because normally you run operations such as annotate with quite well populated caches. Anyway, sorry for the noise. I've just dropped the comment from NEWS. Regards Markus Wanner annotate-a-ref.csv annotate-a-new.csv p annotate-avg-resident-MiB 23.53 23.49 0.70 annotate-avg-size-MiB 118.68 118.62 0.84 annotate-max-resident-MiB 34.20 34.17 0.00 annotate-max-size-MiB 130.86 130.86 nan annotate-num-samples1753.601757.40 0.87 annotate-system-time 0.86 0.87 0.67 annotate-user-time 3.49 3.50 0.62 annotate-wall-time 9.18 9.11 0.27 annotate-b-ref.csv annotate-b-new.csv p annotate-avg-resident-MiB 11.89 11.68 0.44 annotate-avg-size-MiB 103.59 103.74 0.75 annotate-max-resident-MiB 18.63 18.61 0.00 annotate-max-size-MiB 115.32 115.32 nan annotate-num-samples 602.20 540.10 0.33 annotate-system-time 0.18 0.18 0.70 annotate-user-time 0.51 0.52 0.40 annotate-wall-time 3.24 2.87 0.26 annotate-c-ref.csv annotate-c-new.csv p annotate-avg-resident-MiB 11.83 11.35 0.31 annotate-avg-size-MiB 103.50 102.42 0.41 annotate-max-resident-MiB 18.28 18.26 0.00 annotate-max-size-MiB 114.93 114.93 nan annotate-num-samples 590.60 676.10 0.26 annotate-system-time 0.24 0.27 0.12 annotate-user-time 0.75 0.75 0.77 annotate-wall-time 3.19 3.66 0.21 annotate-d-ref.csv annotate-d-new.csv p annotate-avg-resident-MiB 27.30 26.69 0.18 annotate-avg-size-MiB 123.26 122.28 0.18 annotate-max-resident-MiB 36.11 36.08 0.00 annotate-max-size-MiB 132.78 132.78 nan annotate-num-samples2699.902719.70 0.80 annotate-system-time 1.07 1.04 0.32 annotate-user-time 7.85 7.89 0.33 annotate-wall-time 13.87 14.22 0.36 annotate-a-ref.csv annotate-a-new.csv p annotate-avg-resident-MiB 23.42 23.43 0.99 annotate-avg-size-MiB 118.53 118.56 0.93 annotate-max-resident-MiB 34.19 34.17 0.00 annotate-max-size-MiB 130.86 130.86 nan annotate-num-samples1708.501741.50 0.37 annotate-system-time 0.88 0.85 0.23 annotate-user-time 3.48 3.50 0.34 annotate-wall-time 9.11 9.16 0.63 annotate-b-ref.csv annotate-b-new.csv p annotate-avg-resident-MiB 11.54 11.67 0.41
Re: [Monotone-devel] using empty() instead of size()
Hi, Stephen Leake wrote: I don't know how to interpret these numbers. What tool did you run to generate them? It's the output of compare.py from the nvm.cbench branch, which takes CSV inputs from the pythonic benchmark suite. I need to get some profile data on running mtn over NFS; something is deadly slow. Would this tool help with that? You can certainly run benchmarks over NFS as well. Although I'm not sure how helpful that is. You are already saying it's deadly slow. What difference does a benchmark value make against real world experience? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] using empty() instead of size()
Hi, Stephen Leake wrote: I need the profile info more than the total time; I want to find out which particular function in mtn is slow. Then I want to add a ticker output to it, so I will know it's not hung; then maybe speed it up somehow. Guessing you are on Linux, that sounds more like you should have a look at oprofile and try the exact mtn actions you want profiling data for. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] using empty() instead of size()
Hi, Eric Anderson wrote: Assuming that the code for getting memory usage is derived from the code I wrote a long time ago (output looks similar, so likely); I expect that the memory consumption differences are just sampling effects. It sampled every 1 second or so, and so changes to runtime can affect when the memory usage is sampled and hence the estimated memory usage. Yeah, I've been questioning the 'avg' value as well. If you now tell me that sampling is done at 1 second intervals, then I even suspect the max value to not be very meaningful. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] upgrade of included botan for mtn
Hi, I've finally taken the time to go through upgrading the included botan library to 1.7.9 first, then 1.7.12. Some renaming and name conflicts during merging were a PITA to solve and made me do it in two steps. The Global_RNG of botan has gone, so I've added a pointer to an RNG to the app_state, the key_store and the database. Most places using an RNG have access to a key_store object, so we could maybe even get rid of the pointer in the database object. Only mkstemp.cc was puzzling me: I've now changed it to assign its own RNG. Dunno if that can be optimized to use monotone's, but OTOH it maybe doesn't matter. The Memory_Exhausted exception has now gone, so we don't need a special check for that and can rely on std::bad_alloc now. Performance of the SHA-1 remains pretty much the same since we cannot use the optimized SSE2 variant (+60% sha1 throughput [1]). That alone is a good reason to push the library-build branch. Regards Markus Wanner [1]: benchmark results for sha1: (using mtn benchmark_sha1): default botan sha1: ~ 144 MiB/s botan_sha1_sse: ~ 234 MiB/s I've been unable to measure the amd64_asm variant, yet. ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] upgrade of included botan for mtn
Hi, Markus Wanner wrote: I've finally taken the time to go through upgrading the included botan library to 1.7.12. Sorry, I've forgotten the most important part: please test net.venge.monotone.botan. I'd like to land that upgrade it on mainline. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] upgrade of included botan for mtn
Hi, Stephen Leake wrote: It's failing compile on Win32 MinGW. I'll look into it. I can also test on Cygwin. Cool, thanks. You can request that buildbots build this branch via the buildbot website. Good idea. I've just triggered another amd64 and a ppc test. I used to have a buildbot for MinGW, but the machine finally died. I may be able to get another one. That would certainly be helpful. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] upgrade of included botan for mtn
get me wrong I'd love to make the SHA-1 asm much faster (perhaps copying Dean's SSE2 algorithm into inline asm?), but at the moment the SSE2 code is where to go for the fastest SHA-1 (my only thought was that OpenSSL's might be faster, but I checked and the SSE2 is faster than OpenSSL 0.9.8g's SHA-1 on my Core2 at least, hard to say about 32-bit x86 tho). And the 32-bit x86 asm SHA-1 (asm_ia32 module) should be significantly faster than the C++ on non-SSE2 x86, and maybe faster than the SSE2, but I haven't compared them yet. Aha, good to know. Thank you again. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] library build
Hi, I've taken out the library-build branch again today and played around a bit. It took me several approaches and fiddling with auto* just to get it to compile again. I still feel uncomfortable maintaining a special wrapper and multiple 3rd party libraries inside monotone with the additional option to use a system provided library. It seems obvious to me that we don't have the manpower to maintain such a multi-purpose build harness, so I'm proposing to go the easier way and make monotone dependent on 3rd party libraries, without providing them (and a kitchen-sink build harness). Instead we should IMO simply rip out the still-bundled stuff (pcre, sqlite, idna, botan) and adjust the Makefile cruft to build against system provided libraries - as the vast majority of other software does. Or how often did you encounter a 'configure' which doesn't configure anything but leaves that task for 'make', making you fiddle with aclocal in subdirectories if things don't work out as planned? Comments? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Hi, Zbigniew Zagórski wrote: From my recent tests it appears that: - lua - pcre - botan - sqlite3 build without any problem on mingw/msys with simple ./configure --prefix=/mingw ; make ; make install Cool, good to know. Jack: are you interested in making botan compliant to standard configure/make scenario ? I know it's overkill sometimes but making configuration script names configure and supporting --prefix option would be very convienient for some ad-hoc builders. That's exactly the line of thinking I'd like to encourage: if our dependent libraries don't build on other systems, help them with their build system. That's much more in the open source spirit than coming up with our own one - as we currently do. (Although I think botan is a bad example here, because its build harness looks quite thought through and tested on several platforms - certainly more than monotone currently has). PS. I would like to test this library-build branch on mingw, is it buildable at least on some common platform like linux ? Checkout the branch net.venge.monotone.library-build. Expect having to fiddle with autoconf et al. Corrections and improvements very welcome. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] library build
Hi, Zack Weinberg wrote: It might make sense to start a new project to rip out our bundled libraries altogether and then fix everything necessary to make it work with system libraries. I've started such an effort in nvm.stripped. Beginning with: * botan (because I know that from earlier upgrades) * pcre (because it was so easy) * lua (becasue I'm been afraid of the problems you mentioned) The m4 scripts for botan and lua are stolen from the library-build build branch and adjusted. I'm not an m4 hacker, so if some m4 guru could check those that'd be great. The branch builds on my debian lenny against botan 1.7.8 (up to head), pcre 7.6 and lua 5.1 as external libraries. It passes all tests of the testsuite and weights 30 MiB (vs 37 MiB on mainline and Intel 32bit). It also became clear that there are still quite a few hacks that we rely on in our local copies of the libraries. The big one was, we compile lua as C++ so its error handling turns into C++ exceptions. A system-provided liblua won't do this. That certainly needs to be investigated. I'm currently using lua.hpp, thus including the system provided lua library as C code, yes. We don't do error propagation across the lua/C++ interface correctly anyway (errors in lua hooks have a bad way of just disappearing), so I would *like* to rip that out and go back to lua's C interface Feel free... :-) I didn't do sha1 benchmarks, but those should now use the assembler optimized variants. After just having upgraded (and now landed) monotone's included botan to 1.7.12. Jack is already approaching 1.7.15 with yet another set of renaming. I'm rather going to use the nvm.stripped branch than continue to manually upgrade again. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: nvm.stripped
Hello Jack, Jack Lloyd wrote: Currently (rev 44c07854971c158f07da7f53d811d2e298c3f354) I am seeing this: $ ./configure [the usual configure output] checking for Botan version 1.7.8 or newer... no configure: error: Your botan library version not match (1.7.15). Hm.. thanks for this notification. I've never tried non-standard locations so far. Note that at compilation no cflags (such as -I) are set. Against my local workspace, botan-config does this: $ botan-config --cflags -I/home/lloyd/projects/botan/mainline/build/include $ botan-config --libs -L/home/lloyd/projects/botan/mainline/ -lm -lpthread -lrt -lbotan $ botan-config --version 1.7.15 Hm.. you manually added 'botan-config' from /home/lloyd/.. to your PATH, right? And I checked with an echo that botan_CFLAGS is being set properly. (My Botan workspace was configured and I confirmed that build/include/botan/build.h did exist, so it should have found it, if the -I flag had been passed to g++) Any ideas? Not so far, but I'll check the issue. Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: nvm.stripped
Hello Jack, Jack Lloyd wrote: On Tue, Oct 07, 2008 at 01:10:55PM +0200, Markus Wanner wrote: Hm.. you manually added 'botan-config' from /home/lloyd/.. to your PATH, right? Yes (and edited it slightly so it would produce the right output for an in-dir build) Are you sure this emits proper CFLAGS and LIBS? Is the PATH set correctly? Have you recreated configure from configure.ac of monotone before retrying? Not so far, but I'll check the issue. Sorry, I cannot reproduce this issue. When installing botan to /usr/botan/include. In that case I get these from the resulting botan-config: # PATH=/usr/local/botan/bin:$PATH botan-config --libs --cflags -L/usr/local/botan/lib -lm -lpthread -lrt -lbotan -I/usr/local/botan/include With the same PATH variable set, monotone's configure correctly finds the installed botan and spits out the following: checking for Botan version 1.7.8 or newer... yes configure: using botan compile flags: -I/usr/local/botan/include configure: using botan link flags: -L/usr/local/botan/lib -lm -lpthread -lrt -lbotan Checking with ldd (and a proper LD_LIBRARY_PATH) confirms that monotone got built against the library in /usr/local/botan/libs. The tests succeed, except for empty_environment, which fails to load the botan library due to a cleaned up LD_LIBRARY_PATH. We need to adjust that test to support builds against libraries in non-standard locations. However, I want to make sure my modifications work on Windozen as well. I've adjusted the m4 scripts a bit, mostly cosmetics, though. monotone is not also accepting the upcoming botan 1.7.16 and builds cleanly against it (mtn rev 3030f61139f208ffd7c91ad13d88d9a376bc6139). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Questions about Monotone
Hi, Daniel Carrera wrote: Anyways, at this point I would like to commit the changes related to Issue_B. But I already have a bunch of changes for Feature_A and I don't want to mix the two together. These are logically different changes and should be on different patches. I would like to know how Monotone could help me deal with this situation. This situation happens every week. I am often working on improving the crappy source code I inherited from my predecessor or fixing security issues while the boss wants to add a new feature. So I need a simple way to say put these changes in the patch but don't include these other ones. I consider this a standard feature of pretty much ever VCS, including even CVS: you simply tell it which files to commit, no? My current solution is Darcs, which indeed makes the above easy. When I run darcs record, darcs shows me the changes one at a time, and I can tell it which ones to include in the patch. Another equally good solution is: (1) record the current changes for Feature_A, (2) work on Issue_B, (3) record the changes for Issue_B, (4) UN-DO step (1), and continue working on Feature_A. This latter option is actually more useful. Ah, so you are talking about changes within the same file. Well, yeah, that's an area where monotone isn't that brilliant, yet. One way to do it would be to use a branch for Feature_A, commit your experimental changes there and then switch back to the main / stable branch for Issue_B. As long as the changes don't conflict, the merge algorithm does everything for you. Remember that branches are cheap. Another way is to split logically different modules into different files, which is advisable anyway, IMO. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Questions about Monotone
Hi, Daniel Carrera wrote: Yes, thats right. Often the changes are in the same file. It's a smallish program. Most of the functionality is in two files: a core program and a library, and then I have a bunch of front-end type files (PHP templates). So making a change often involves changing something in the core file. Yeah, monotone lacks the functionality to choose from single changes within a file. Do I need to make a separate directory to make a branch? I'll read the documentation on branches and I'll probably have a question then. No, branches are independent from directories. However, directories are also tracked within branches. I.e. you can rename and move them and merging takes care (as long as your renames don't conflict, which is a much longer story). Yes. This program is severely lacking in modularity. It was quite horrid when I first got it. The whole program was one single 1200-line if statement! Yeah, I know such things as well... :-( I've spent the last few years cleaning up the program. Moving code out of the if-statement-from-hell and putting the code into functions (*functions*!). LOL! So anyways, I agree that modularity is where I want the program to be in the future. But truthfully it will be a long time before it gets there. You can also see why I am working on two things at once. Feature_A usually is refactoring the code to make it modular, clean and correct. Understood. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.stripped
Hi, Jack Lloyd wrote: OK, if I change in botan.m4 -CFLAGS=$CFLAGS $BOTAN_CPPFLAGS +CPPFLAGS=$CPPFLAGS $BOTAN_CPPFLAGS Ah! Thank you very much for pointing this out. Looks like configure on my system simply picked the system installed botan during that stage, but then continued to use the provided botan :-( I've removed my system wide botan and can confirm that the issue is solved now. (As of e5f6d85442972da43d641bd3d7bf5d6667bbd23a). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.stripped
Hi, Derek Scherger wrote: I've made this change Cool, thanks. and also moved sqlite to the Attic ..oops, looks like we've done duplicate work. I've just committed and pushed what I have. It's not quite complete, yet (rev 4c625e162bf17c69406de23482187313dafd29cd). Have you pushed your work, yet? which went much more smoothly than I was expecting since it involved upgrading from 3.4.2 to 3.5.9. One test fails with this later change, fail_cleanly_on_unreadable_db, which I assume is an actual change in sqlite behaviour but I'm not certain of this. Yes, Ralf S. Engelschall has noted that before, see my other mail. I've tested my variant against sqlite 3.3.8 (included in debian etch), 3.5.9 (debian testing, IIRC) and 3.6.3 (current stable release). These tests fail: sqlite 3.3.8: 146 database_checkFAIL (line 23) 192 empty_environment FAIL (line 29) 203 fail_cleanly_on_unreadable_db unexpected success 216 i_selectorFAIL (line 32) (does not have the 'hex()' function, but obviously works like we expect for 'fail_cleanly_on_unreadable_db') sqlite 3.5.9: 192 empty_environment FAIL (line 29) 203 fail_cleanly_on_unreadable_db FAIL (line 62) sqlite 3.6.3: 192 empty_environment FAIL (line 29) 203 fail_cleanly_on_unreadable_db FAIL (line 62) The 'empty_environment' test fails due to LD_LIBRARY_PATH required for building against libraries in non-standard locations. I'm now looking at idna (libidn) which ought to be easy but isn't because of some changes that were made at the 2007 summit iirc. Lapo, do you recall the best effort stuff that got added to idna/toutf8.c? This is not in idna 1.5 but I'm wondering if maybe idna 1.5 might just do the right thing? Okay, so I let you take care of libidn. If you don't mind, I'll follow the sqlite things and try to fix those tests. Certainly not today, though. ... just trying to avoid duplicate work... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Monotone upgrade policy for the SQLite copy?
Hi, Ralf S. Engelschall wrote: Ok, I finally found time to finish the preparation of the SQLite upgrade. Sorry for the delay, I was heavily busy because of my day job. I've pushed the following two branches to monotone.ca today: o net.venge.monotone.rse.sqlite-upgrade ... o net.venge.monotone.rse.sqlite-upgrade-amalgamation ... I've revisited those branches, but decided to follow the spirit of nvm.stripped and made mtn link against a system provided sqlite3 library instead, at least for the nvm.stripped branch for now. We can still revert to include the amalgamation later on, I think. What's most important for me now is making monotone compatible with newer sqlite versions. Do you mind suspending these branches? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] nvm.stripped test results + adding version numbers to version --full
Hi, Jack Lloyd wrote: New version. Use the version macros for Botan instead of calling the string function. Mostly for consistency with the rest of the version # sources. Thanks, very cool. Committed as rev ce0db5b1f6d3b9ab45535784e298599733d0d6ab. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.stripped
Hi, I've now corrected or worked around issues with various sqlite versions. These tests have failed before: Markus Wanner wrote: 146 database_checkFAIL (line 23) 216 i_selectorFAIL (line 32) Revision 40dba08582bcb2eecde9c9700fcab85b11f2ff61 adds a 'hex()' function for sqlite versions below 3.3.13, which didn't have that, yet. That made these two tests succeed for sqlite 3.3.8 - 3.3.13. 192 empty_environment FAIL (line 29) The 'empty_environment' test does no longer completely wipe the environment, but maintains the given LD_LIBRARY_PATH environment variable. This only works under Linux for now. I'm lacking any other systems to test on. Added in rev f67d1482d7c2aa37adaeeb7db09161be9dc8b616 This should also help other linked libraries. 203 fail_cleanly_on_unreadable_db unexpected success Rev b268fec033da5902f6af76e8e56b2ace759e090f fixes this issue by removing he offending checks. Investigating revealed, that newer sqlite3 versions are trying to read/write a journal file in the same directory where the database resides. That of course fails for unreadable, but only searchable directories. Here's an strace of sqlite3 trying to open a journal file on a a-rw+x parent directory '/home/markus/test': open(/home/markus/test/test.db-journal, O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE|O_NOFOLLOW, 0644) = 4 open(/home/markus/test, O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied) close(4) With these fixes I've been able to compile and successfully test monotone against all of these sqlite versions: 3.3.8 (as currently included in debian etch / stable) 3.3.12 3.3.17 3.4.1 (what we shipped up to now) 3.5.9 (as currently included in debian lenny / testing) 3.6.3 (newest upstream release) I didn't care to test anything older than 3.3.8. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.stripped
Hi, Markus Wanner wrote: 3.4.1 (what we shipped up to now) Oops, revoking that. 3.4.1 and 3.4.2 currently fail on test 'database_dump_load', which is fishy because 3.4.1 was integrated before, no? I'm investigating on the issue. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Concerns about the net.venge.monotone.stripped branch
Hi, Richard Levitte wrote: Just a moment ago, I saw a change that made the montone code depend on the version of SQLITE as described by a C macro. That's fine and dandy provided you link monotone with the static library, but what happens if it's linked with the shared library and then moved to another machine with a more modern (or even slightly older!) version of SQLite? Yeah, I've been thinking about that as well. I'm not sure where to look for run time version information of the library, but I've been able to exchange *.so files without any hassle so far for versions which do not require the hex() function. I don't know how SQLite handles such cases, but I do know that not paying attention to that kind of thing have and still can lead to problems. Agreed. Thanks for pointing this out. I'll check if we can add dynamic version checks as well. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: Concerns about the net.venge.monotone.stripped branch
Hi, Bruce Stephens wrote: http://www.sqlite.org/c3ref/libversion.html suggests it should be easy enough. Thanks for the hint. Any idea what CVS tags they define? I've checked out their CVS tree, but I'm missing 'cvs ls tags' :-) And their downloads website doesn't list order versions... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Concerns about the net.venge.monotone.stripped branch
Hi, Richard Levitte wrote: In the same vein, the code to display the versions of SQLite, Botan and so on might be misleading in a shared library environment. Maybe it should display both the versions it was compiled against and the versions it's running against (if that's possible to find out, which isn't always a given). I've done that for SQLite and botan now. The mtn version --full output now looks like (as an example): monotone 0.41 (base revision: 35c9a1b5661fa713059728bec11204df5d0fe7c4) Running on : Linux 2.6.27-rc6-markus1 #4 SMP Thu Sep 18 13:30:24 CEST 2008 x86_64 C++ compiler: GNU C++ version 4.3.2 C++ standard library: GNU libstdc++ version 20080905 Boost version : 1_34_1 SQLite version : 3.5.9 (compiled against 3.3.8) Lua version : Lua 5.1.3 PCRE version: 7.8 Botan version : 1.6.3 (compiled against 1.6.3) Changes since base revision: format_version 1 new_manifest [84b53d2311043f42fcd55b15e9b548b3b9fb7e48] old_revision [67a5c30e1c38753ca92bda29120299e4872ac34b] Generated from data cached in the distribution; further changes may have been made. The cost for being independent of the actual SQLite library is having to use sqlite3_prepare() instead of sqlite3_prepare_v2(), which exists only since 3.3.14. However, as that's just 1.5 years old and because 3.3.8 is in etch, I'm trying to still support those versions. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Concerns about the net.venge.monotone.stripped branch
Hi, Markus Wanner wrote: Richard Levitte wrote: Just a moment ago, I saw a change that made the montone code depend on the version of SQLITE as described by a C macro. That's fine and dandy provided you link monotone with the static library, but what happens if it's linked with the shared library and then moved to another machine with a more modern (or even slightly older!) version of SQLite? Proper runtime checks are in place for SQLite and Botan since revision 7aa69891001cc53af518ca9bff8e1c7dae0c280d. SQLite seems to maintain a backwards compatible API for all versions 3.x.x. Compiling against 3.3.8 and running with 3.5.9 worked just fine for all unit test. Botan has a stable branch 1.6.x and a development one being 1.7.x. Supporting the moving target 1.7.x branch is a bit of a hassle, but OTOH we well be up and running right away when 1.8.x alias next stable branch comes out. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] nvm.stripped failing check for lua
Hi, Alex Sandro Queiroz e Silva wrote: The Lua authors have had no problems with breaking API and ABI compatibility with each new version so far. So programs usually work with just one version of Lua only. Between 5.0 and 5.1, yes. But I've so far thought that 5.1.x are bugfix releases only. See: http://www.lua.org/bugs.html. That's why I made nvm.stripped require lua 5.1.x (and not 5.0.x). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Concerns about the net.venge.monotone.stripped branch
Hi, Alex Sandro Queiroz e Silva wrote: The lua.h file defines LUA_RELEASE. That's the one I've been using before, which does not make much sense, because it can only serve as a build time check. It's not a runtime check like the global lua variable _VERSION. (And no, there is no _RELEASE global lua variable). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Concerns about the net.venge.monotone.stripped branch
Hi, Zack Weinberg wrote: well, you could always do the moral equivalent in C: lua_getglobal(L, _VERSION); std::string luaversion(lua_tostring(L, -1)); // copies lua_pop(L, 1); Hm.. I've tried that and stumbled over two problems. One is, that mt_version.cc doesn't have access to a Lua_state object (and gets called from sanity.cc, which doesn't either). The other one is, the patch version isn't included in the _VERSION global variable: # lua Lua 5.1.3 Copyright (C) 1994-2008 Lua.org, PUC-Rio print(_VERSION) Lua 5.1 So there's utterly no point in showing that, since we (build time) require 5.1.x anyway. I've thus reverted to show only 'Lua 5.1' in rev 6b777e3eb453b3b0be4d9aaa93e81cd56c6ca635. (As opposed to showing the patch version (i.e. 5.1.3) we've built against). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.stripped
Hi, Stephen Leake wrote: Assuming we eventually integrate this branch into mainline, how many versions of sqlite (and all the other libraries!) do we need the buildbots to test? I think it's sufficient supporting the library versions shipped with the distributions. And do we want to get into testing all possible combinations of library versions? Certainly not *all* possible combinations. I'd say we don't even need to test every single patch version of each library. After all, these (should) provide a stable API and ABI. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] dates in monotone
Hi, for the CVS import branch I needed to compare dates. Up until now, monotone parses the given date strings, but still used strings internally to represent dates. In nvm.dates I've switched to using unsigned 64bit integers to represent dates internally (as seconds since the Unix epoch). That required adding a method similar to the mktime() system call, which translates year, month, day, hour, min, sec values into such a timestamp. As the existing function, which does the reverse, it does not care about leap seconds, but only handles leap days correctly. While that's find and dandy, I'm still wondering why we (uhm.. Zack) have implemented gmtime() from scratch. The comments tell something about systems not using the Unix epoch. But which system? Are the mktime() and gmtime() system functions really that unusable for portable software? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: dates in monotone
Hi, Zack Weinberg wrote: Yes. Some (not all) Windows C runtime libraries (or possibly operating system major releases, I didn't investigate this in detail) use a different epoch and/or don't implement the distinction between UTC and local time correctly. It would also be a problem if we wanted to port to various other systems not in wide use anymore; VMS and classic MacOS come to mind. Okay, thanks. Looks like enough reason to leave our own implementations of gmtime() and mktime() in there. I will mention these OSes in the comments, then. Any objections against landing nvm.dates on mainline? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: dates in monotone
Hi, Zack Weinberg wrote: I should add that this is only a problem because the CVS file format uses Unix-epoch second counts as a storage representation. The standards say nothing about the epoch used by time_t; it's not even guaranteed to be an arithmetic type! The problematic Windows gmtime() and mktime() are correct as long as they are used only in conjunction with themselves and the rest of the time.h API -- you'll note that date_t::now() does use time() and gmtime(). It is only when interpreting integers defined by some external format to be Unix-epoch second counts that we have to do the math by hand. Understood. So please don't create new situations where Unix-epoch second counts are stored on disk or transmitted over the net. Uhm.. yeah, I've created such a situation in date_t::now(). Thanks for pointing that out. In rev af1cd9d04a3cabab86dc14c8167ff5e83b26d6a6, I've reverted that to using the system's gmtime() to gain a struct tm, which is independent of the system's epoch. Instead of creating a date string from there, I'm now using our own date_t::mktime() function, which is guaranteed to use Unix epochs. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: dates in monotone
Hi, Zack Weinberg wrote: I will not have time to look at the code until later today or perhaps even tomorrow. In the meantime: are you storing Unix-epoch second counts in the database? If so, then yes, I object, for the reasons in my second message. No, I've only changed the internal representation of date_t to use (comparable) u64 seconds since Unix epoch, instead of storing strings. Maybe I need to check for performance issues, but I don't think this part is overly performance critical. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: dates in monotone
Hi, Lapo Luchini wrote: BTW: when using 64 bits isn't it usual to store signed milliseconds as opposed to unsigned seconds? Yeah, thanks, good point. I'm using 1/100th seconds in nvm.cvsimport-branch-reconstruction already anyway... date_t would get even more usable there when using milliseconds. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Monotone Security
Hi, Zack Weinberg wrote: Yes. Distributed systems research has concluded that there ain't no such thing as a trustable global clock. (I don't have cites - this is my paraphrase of something Nathaniel said some years ago.) While that's certainly true, I also agree with Daniel that there's something wrong with revisions A - B having timestamps in reverse ordering. IMO monotone should at least warn about such obviously ill-certified revisions, better yet protect against such wrong information. That's one reason for my work on nvm.dates: being able to compare the dates, so we can do these checks. So with refusing to commit revisions with a date *before* its ancestor, monotone would help detecting clock skews. And by warning and (possibly automatically) distrusting certs from the future, monotone could prevent situations where you cannot commit because someone else has signed a revision with an erroneous date. Keep in mind that you can always decide to *not* trust a cert with an invalid date. Note that this does not have anything to do with a global time or using timestamps for internal purposes. It's rather just about extending validity checking of the given information. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: hypothetical - future-dated certs (Re: Monotone Security)
Hi, Daniel Carosone wrote: I'm not so sure.. I'd happily go for 'strange' or some other word, rather than 'wrong'. At the very least, I'd like to try and work through all the cases we can think of where this might arise, and just confirm whether a sensible or legitimate interpretation could have been intended. If we fail to come up with any, fine. If we do, perhaps we'll have a new and interesting use case to consider supporting properly.. Okay, let's see... The question remains, however: when something like this is detected, what should either the user or the software do to resolve the issue? That's the question, yes. Let's say I decide to add another date cert with a more reasonable value; this doesn't and can't get rid of the old one, so nothing is really solved. True. You'd need to explicitly distrust the old cert. In particular, my concern is that despite agreeing and acknowleding that there can't be a global clock, warnings or errors like this help to encourage in the users' minds that there is a global clock anyway. Can we really get rid of that at all? If you continue this line of thinking, why do we have date certs at all? AFAICT it's just for users convenience so far. I worry that this will just lead to confusion, rather than encouraging users to fully embrace the realities of the distributed/disconnected world. I don't think that's feasible. Users will always (want to) express themselves in normal dates and times. Otherwise you would have to encourage time definitions like: before rev foo as signed by bar. Again, fully support this, just not sure how it should be used. Giving room to experiment and discover how it should be used is a good end in itself. Good, I'm glad. ;-) I already have some code to compare dates, I'm thinking about adding that to mtn db check or something (as soon as I get time to clean up my spaghetti code...). It seems to me to be a much more relevant thing to consider as part of cert format rewrites, when we add a signing-date to certs, and that it should be applied to the signing-date rather than the basic date. At that point, signing a cert with a future date *could* be used to delay the validity of a cert until some future date. However, if something like that does turn out to be legitimately useful, my opinion is that it should be via explicit valid-from and valid-until (optional) cert fields, rather than by trying to overload the signing date. Hm.. interesting use case. I fail to see a use case for a limit with valid-until, yet, so let's discuss the valid-from thing first. I've already encountered situations, where I wanted to postpone publication of a revision. Signing with a valid-from in the future could possibly solve this, yes. But that would benefit from the very same date checking, because descendant revisions must be valid from an even later point in the future. Otherwise you'd suddenly make your revision appear, just because it's descendant's cert is already valid. So AFAICT such a feature would even *require* proper timestamp checking. Otherwise you could not enforce such restrictions. Or put another way: as soon as we start to use the timestamps for more than just users convenience, we have to check and validate this information better. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] date certs on net.venge.monotone
Hi, as mentioned in other threads already, I'm currently analyzing date certs and comparing them against the revision ancestry. The obvious first test database includes all of net.venge.monotone* (and nothing else). At the time of testing, it had 14156 revisions with 17737 relations between them stored in the table revision_ancestry. 32 of the ancestry relations are initial revisions with the null_id as parent. All revisions had at least one date cert. If there are multiple date certs for the same revision, I've decided to counting the earliest date for the revision. Checking these dates against the revision ancestry reveals, that only 58 (0.33%) of all of these dependencies turn out to have inversed date information (see [1] for a full list). For all of the remaining 17647 (99.67%) ancestory relations, the date information is plausible (i.e. the timestamp of the date cert on the ancestor revision is lower than that of the descendant revision) (Yes, this currently includes counting dependent revisions with equal timestamps as invalid, which is certainly bogus, just highly unlikely). For Botan and Pidgin repositories it's pretty similar: Botan: 3 of 1786 (0.17%) Pidgin: 36 of 28349 (0.13%) What surprised me about the pidgin repository is, that there are some date certs with six digits (!) for milliseconds in the date, i.e.: 2006-11-01T21:39:36.807940 Thus, I've added support for that format (without parsing the milliseconds part, though) in nvm.dates. OpenEmbedded (latest mtn snapshot) is slightly higher: 501 of 32686 (1.53%) But what currently puzzles me is the Xaraya repository: Xaraya: 6037 or 23413 (25%) I didn't investigate further, but the time gaps are pretty large... however, maybe choosing the earliest date just isn't clever enough. Regards Markus Wanner [1]: full list of date mismatches: mtn: checking timestamps of ancestor revisions mtn:rev 006972c572eb094a3c03d3ea6105e7f31f5bfa05 - rev c6bd5efd89266879cf9cbe7006b0fe5ca5d5b4dc mtn:but date 2003-10-20T22:15:28 ! - 2003-10-20T21:38:09 mtn:(difference: 2239 seconds) mtn:rev 02ef37090f20abd12a6aba5cd0244ca78a05ab11 - rev a43b9f2bec32d4a63f073e414f067862d676f04c mtn:but date 2004-05-03T00:48:36 ! - 2004-05-03T00:05:41 mtn:(difference: 2575 seconds) mtn:rev 05a0f9ce71013e8bc5c04c825a53d17a5e3007a9 - rev 3011908f7f234e48e5d90160cb406f02f09c19d2 mtn:but date 2007-09-02T05:29:33 ! - 2007-09-02T03:35:35 mtn:(difference: 6838 seconds) mtn:rev 05ee0ead1ba7806116bd8f887b8206790a2a7eec - rev 692d74c8ef9a112ba23ece0018daf82ad4382942 mtn:but date 2005-06-25T01:00:01 ! - 2005-06-25T00:06:00 mtn:(difference: 3241 seconds) mtn:rev 080864450f24829523bed008fbad268a6c7eb05c - rev b664839edf950807096ce987038b1d8ea0ebd0e9 mtn:but date 2004-08-09T17:21:34 ! - 2004-08-05T23:09:44 mtn:(difference: 324710 seconds) mtn:rev 0826abb7a5e5aacbcda1d3634178b718aada8378 - rev 00900acfa10d85988692ee7571028e7364afef79 mtn:but date 2005-07-24T02:09:56 ! - 2005-07-23T23:30:22 mtn:(difference: 9574 seconds) mtn:rev 087994568abcbf9aee0347356bad0e4af09152d2 - rev a9cd9f0a9d3d93dcd451ac49254d92603ec1272f mtn:but date 2005-05-03T11:00:00 ! - 2005-05-03T10:55:12 mtn:(difference: 288 seconds) mtn:rev 13df407d1c24805c9a350dc49257c48a9ff3f2d4 - rev 91891a99890d2a9e497f3055f1f787d62d408d67 mtn:but date 2005-10-19T07:52:39 ! - 2005-10-19T06:23:00 mtn:(difference: 5379 seconds) mtn:rev 14312f90136babc24be6a746397f99b121e10353 - rev 650d4a285dbd92d434399202ec7fd3d412ba444d mtn:but date 2005-11-21T01:18:57 ! - 2005-11-21T00:53:23 mtn:(difference: 1534 seconds) mtn:rev 14e42d73aa188b25687c93aecf799fc4b7b9f1e6 - rev f128c0fb1cd5bab53172c3fb443d3a7207182844 mtn:but date 2006-03-13T06:28:22 ! - 2006-03-13T00:31:35 mtn:(difference: 21407 seconds) mtn:rev 1ca6b8755c5fed18ca636aee06149963c0b2eef4 - rev 3e8a78b0f8151f58578134cad523b7c664229760 mtn:but date 2005-05-03T07:55:58 ! - 2005-05-03T07:46:23 mtn:(difference: 575 seconds) mtn:rev 1d11d03b9643ccf9305e332b598e59622231f998 - rev fa4cedc2a54a54225587fd0f788f63fa887cc6e9 mtn:but date 2006-08-06T09:04:46 ! - 2006-08-06T08:57:07 mtn:(difference: 459 seconds) mtn:rev 25609512782fe474133707a82d73349aa56f1f5c - rev 40bbc03627a68c9e4afd5fddd9fc9401e6e26b2f mtn:but date 2007-12-16T17:10:30 ! - 2007-12-16T17:08:30 mtn:(difference: 120 seconds) mtn:rev 296d8fb7e07fbead3128638ca638e525e1ab38fc - rev 5c9e19109f4627ad83b51037afec5157e4be06ca mtn:but date 2005-07-27T11:41:38 ! - 2005-07-27T11:19:22 mtn:(difference: 1336 seconds) mtn:rev 2b1e8f23addc3eb4aaf326f5632ce1e6fef5dad1 - rev 059e2e79d9de7a2a30a9a67947308df4cc970c62 mtn:but date 2005-07-14T00:44:41 ! - 2005-07-13T23:58:09 mtn:(difference: 2792 seconds) mtn:rev 2b9274ae48761b2ba80aaf59e7c69cbc3ebd8797 - rev
Re: [Monotone-devel] Re: hypothetical - future-dated certs (Re: Monotone Security)
Hi, Daniel Carosone wrote: That's kind of my point about the separate date certs we have currently. You propose a mechanism whereby an out-of-order or future-dated date cert would be considered invalid and untrusted -- instead of now where it's trusted but essentially ignored (other than for display and as a selector target). It doesn't seem like much benefit, which is why I think it only becomes interesting when we're looking at signing-dates on certs that make other kinds of statements (like asserting branch membership). Looks like we are just approaching the problem from different angles. I've been analyzing what's required to convert to atomic certs (or super-certs or whatever you'd like to call it). I'm thinking that we need to clean up our current certs to be able to convert them. I could attach a special netsync-instruction type of cert that says don't send until date X, or I could attach a branch cert that says add to branch Y after date X (and then rely on netsync branch pattern selection. In practice and in general, I don't think this really cuts it as a solution. In particular, I probably do want to sync these revisions in a limited context (such as between my own machines just for backup purposes).[*] Secondly, the decision for when I'm ready to publicise that work is rarely date-based. I fully agree here. I'm in the need for such a feature often enough as well. But it looks like it depends on atomic certs *and* policy branches, so we are not likely to get that feature in the near future, IMO. :-( To me, a better use for valid-from might be on testresults and job scheduling. Say I want to automatically build downloadable snapshots for multiple architectures of the most recent revision to have passed a set of tests, but the scheduling of when during the day to start the build depends on some complex pattern that I can't easily represent in the multiple platform's scheduling tool. So, I make the build depend on a summary test cert, that passes if all the other prerequisite tests pass. As those test certs come in, I issue summary test certs for revs during the day that have passed the set - and I future-date them for the time this evening that the build should start. The build slaves just check on a basic interval for new work. When the cert validity time comes around, the build slaves suddenly see all the summary certs for the day, and build the latest (by ancestry) rev with a valid cert. Hm.. do we really want to dig into scheduling of build bots? I don't quite think that fits the scope of monotone. Certainly not yet. Oh, as a somewhat dumb use for a valid-until field: a suspend cert for a head I want to come back to -- but not until next month. You can 'un-suspend' the branch at any time by simply committing a new revision. So I don't think this us an utterly needed feature. Or another: a tag cert that names the current base for patch submissions for the next quarterly integration branch. Next quarter, the tag expires and a new one is made. Hm.. that might count. Although, doesn't policy branches provide a more flexible mechanism to do these things? (I.e. general purpose cert revocation, instead of only time based). Or another: maybe I'm doing a ZipperMerge, with an explicitly-named intermediate zipper branch in the middle; perhaps I expect to take a day or two to progress through the merge. In any case, it's really only the last few zipper merge certs that I care about - the current central head, and whichever left or right node I next want to bring in. I could set these certs to expire quickly, and hold off syncing until then - if expired certs aren't sent via netsync, they no longer need to be passed around the rest of the world. Again, sorry, but I don't think saving a few kbytes for these certs is worth the effort (and confusion) introduced by certs expiring by date. [*] I have in mind a different mechanism for this, based on a concept of key-based netsync communities (to mostly replace the current netsync permission hooks), which I've never taken the time to write down - in part because I haven't really thought about how it intersects with branch policy, and in part because I'd really like to think of a nice way to make it work for revision encryption too.. Sounds very much like something that could work together with policy branches, no? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] date certs on net.venge.monotone
Hi, Daniel Carosone wrote: If we're only storing up to seconds, it could happen readily for a multi-way merge that created several intermediate merge nodes in the one command. Sure. See the long list of revisions I've posted, there's one such case within net.venge.monotone*: ee293e066aa9e8222b7ce3b854affe49ddf9317c 4dd538bd7bb906041b817b8bc6a93e87fb3759d5 Both of these revisions have the timestamp 2008-04-30T15:32:06. But AFAICT this is the only occurrence within the monotone repository so far. And the Oscar for the quickest merge goes to: Author: [EMAIL PROTECTED] :-) Regards Markus Wanner - Revision: 4dd538bd7bb906041b817b8bc6a93e87fb3759d5 Ancestor: ee293e066aa9e8222b7ce3b854affe49ddf9317c Ancestor: ee5efcb81167e04e6c915ce21bb32b832b7ad964 Author: [EMAIL PROTECTED] Date: 2008-04-30T15:32:06 - Revision: ee293e066aa9e8222b7ce3b854affe49ddf9317c Ancestor: 301352168b24a1633d45b92051592ed44142471b Ancestor: e2ed2c1ee19f1f3ad1616838e6cbd1be66fc020a Author: [EMAIL PROTECTED] Date: 2008-04-30T15:32:06 ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] WARNING: ~/.monotone/keys CONSIDERED HARMFUL
Hi, Timothy Brownawell wrote: It's entirely possible to destroy the private half, since only one person should ever have that. And being sure in this case sounds more like a matter of contracts rather than technology. Something can be useful even when technology can't guarantee it. Good point. However, it *should* not be necessary to create and drop keys for that, but instead just adjust keys to trust, possibly limited in time. Hopefully we get there with your policy branches, one day... :-) I must admit that I still didn't look at your branch... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] WARNING: ~/.monotone/keys CONSIDERED HARMFUL
Hi, Robert White wrote: Try things like wanting to be able to revoke/destroy one key when the contract is over etc. I fail to see how that's even possible in a distributed environment. The only thing one single party can do is distrust a key. There's no way to make sure the other party destroys its key - and won't have a backup somewhere. Seriously, just because you don't understand any use case where you would want more than one key doesn't mean they don't exist... Just because you are using monotone another way than intended (same key id for different keys) doesn't mean it's harmful for all others as well, as you make it sound. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Summit thoughts - and changes?
Hi, Matthew A. Nicholson wrote: I think this is a great idea. Why don't we set a date for the next virtual summit. Perhaps the first full weekend in December (the 5th-7th)? Or is that too close to thanksgiving (US) or christmas? Good idea. However, chances to get a complete weekend every now and then are pretty low here. I could only attend half a day or so. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: WARNING: ~/.monotone/keys CONSIDERED HARMFUL
Hi, Brian May wrote: You need to use email addresses in order to answer the question Who signed this revision? Huh? No, your key id can be pretty much any string you want it to be. Monotone certainly doesn't bind that to email addresses exclusively. It's just common practice. I could imagine setting up a database with false email addresses for the keys, so anybody who syncs from my database gets these keys and may get mislead as to how made the changes. Why mislead? I could use the key id Markus_Wanner. If you get the public part of that key and trust it is really me, you can perfectly well use that to validate my signatures. As far I am aware there isn't any user visible way at the moment to test two keys are the same, we rely on the keyid which could be falsified. That's certainly an agreed short-coming, yes. They is also no way to tell what key signed a particular revision, apart from its keyid. Well, that is based on the assumption that no two keys have the same key id. As long as that's true, you don't need any other way. The only part of this I am not sure of - is how monotone behaves if it notices a keyid has a different key during a sync operation. My guess is, that the key won't be transferred (or checked), but the certs transferred are then checked against the wrong key and dismissed as untrusted. Then the next part required would be some way of securely mapping the key id to a user - they are a number of different ways this could be done. Once we use hashes for keyids that is. Huh? How should that be possible? Isn't it sufficient exchanging known public keys during netsync? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] WARNING: ~/.monotone/keys CONSIDERED HARMFUL
Hi, Daniel Carrera wrote: Was there a good reason why monotone didn't use GnuPG for signatures? I have a feeling it was related to speed or something. This question also caught my interest. This is in the FAQ: Thanks for pointing this out. The last point is interesting, and it seems sensible to me. Hm.. I don't see how running our own PKI should be different. Our web-of-trust is just very simple (and maybe doesn't deserve the term web): every server allows certain keys commit access to certain branches, only read access to other keys. Considering that we may be sharing these policies between servers manually, isn't that a sort of a web of trust? If you don't call it that now, you probably will with policy branches. identification: Who are you? authentication: Are you allowed to do this? They are not the same thing, and they are not even related. In principle you could trust and authorize someone without knowing their name. Without knowing a name, yes. But is the name really the answer to the question who are you? (This is getting philosophical...) I'd argue that to authenticate *someone* to do something, you always need to identify the *someone* first. That doesn't necessarily mean getting his name. You can easily authenticate by confirming that it's the same person who has written revision 276264b0... for example. That's the first revision in net.venge.mononone, being version 0.4 of monotone, i.e. the person commonly known as Graydon Hoare. Trust me or not... ;-). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] WARNING: ~/.monotone/keys CONSIDERED HARMFUL
Hi, Daniel Carrera wrote: My position is that what the PGP web of trust provides identification but not authorization and so it does not help Monotone. According to your own definition below, PGP only provides authentication, not identification. I'm thinking of identification in broader terms than just assigning a name and/or email address. For example, visual identification also counts (but is obviously pretty useless for monotone). To authorize someone to do something, you certainly need authentication. Otherwise, how do you know who you authorize to do something? It isn't a web of trust. But as you imply, what we call it doesn't matter. Just call it policy or whatever. If I'm publishing my server's policy and someone else adopts it, that someone else trusts me enough to use the same ACL on his monotone server. This doesn't need policy branches, but it's already possible with today's monotone, and not even improbable. I for example have granted write permissions on my server to somebody I don't know in person. I only know others trust him enough to have commit access on net.venge.monotone. Why should that not be a web of trust? A pretty manual one (because we still need to exchange policies manually), yes, but it certainly still is a web of trust, IMO. Policy branches are (partly) about automating that aspect. The real issue is whether there is any benefit to using PGP from inside Monotone. I opine that there isn't. The things Monotone needs (authorization), PGP does not facilitate. The things that PGP provides that Monotone lacks (web of trust, identification), are not things Monotone needs. Just my opinion though. As pointed out above, you certainly need authentication within monotone. Using GPG from monotone would allow to authenticate someone by his GPG key instead of by his monotone key. And GPG keys are much more wide spread than monotone keys, which might be a reason to at least support GPG. Anyway, that's certainly not on top of my priority list for monotone ;-) Trying to not start an argument about the definition of identification, notice that what PGP provides which Monotone doesn't is assurance that the name and email actually match the key. I'd state that PGP provides authentication - pretty independently of the name and email. As an example, I've just recently changed my name due to marriage, but the PGP key and my identity remained the same. I simply added my new name, now having Markus Schiltknecht and Markus Wanner as names for my identity. the guy who owns this key .. that's exactly the identity part of the story. Just replace this key with a GPG key and you could theoretically throw away monotone keys. (Not that I'd want to do that) should be allowed into the server. For that purpose, PGP doesn't appear to provide anything that Monotone's light-weight alternative doesn't already provide. I absolutely agree to that from a technical point of view. But it requires people to create monotone keypairs, whereas by supporting GPG, they could use their existing GPG keypairs. Now, on identification: I think the following might be a relevant example: Imagine an ID card that has a picture of you but no name. You and I might disagree on whether we would call this identification. I'd certainly call that (visual) identification, yes. But it might clear up confusion if I say that this is an example of what Schneier means by no identification. In Schneier's lingo, this ID card may provide authentication but not identification. In my recent emails I have tried to follow Schneier's lingo. Well, yeah, in that sense, both, the GPG key as well as the monotone key only provide authentication but not identification, because it's always possible that the key got stolen. But how's that point relevant between GPG keys and monotone keys? It's not like monotone keys are any safer in that regard than GPG keys. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] 3rd party libraries
Hi,
On IRC we recently had a discussion about using boost::date_time instead
of rolling our own in dates.{cc,hh} - as I've been doing in nvm.dates.
I've been under the impression, that we do not want to rely on boost
libraries. But in nvm.stripped I'm working on linking monotone against
other libraries as well, so it seems we need to revisit the question of
how to work with 3rd party modules in general.
There are three major options, all of which we've already used:
* Dynamic linking to external libraries. We've always linked against
libz, for example. Pros: low maintenance cost for us, enjoy newer
versions by replacing the lib - Cons: dependencies on other libs, mtn
needs to (dynamically) cope with various versions of the lib.
* Source inclusion: we've done that for botan, sqlite, lua, pcre and
others. Pros: need to support only one version, independence of upstream
(i.e. netxx is dead) - Cons: high maintenance cost for us, packagers
don't like it.
* Avoid the library: we are currently doing that with boost, for
example. Pros: no dependency, no license issues, code exactly what we
need - Cons: hight maintenance and development costs for us.
I think pros and cons are pretty clear, sure you can still add some
general ones here and there. Just wanting to make sure we speak about
the same things.
Now, let's get to the single libraries we use:
* libz: dynamic linking hasn't posed any problems and the library seems
to be available on pretty much any system.
* botan: source inclusion *has* posed problems in the past, namely that
we couldn't use asm optimizations due running our own build harness for
botan. Additionally, maintaining the build harness was a hassle and
something I'm glad to get rid of. The library is still pretty new and
not available on all systems. We are currently using a development
branch (1.7), which is a bit unfortunate. However, I've adjusted
nvm.stripped, so that it compiles against 1.6.x as well. I'm all for
dynamic linking of that library.
* sqlite: source inclusion worked fine so far, but it hasn't really
been updated for quite awhile. nvm.stripped provides dynamic linking,
but SQLite also offers a pretty simple amalgamation release, which we
could use as well. Offering choice between system library and bundled
one should be simple enough to maintain, IMO.
* lua: pretty similar to sqlite, except the source inclusion variant is
a bit more complicated. I'm all for dynamic linking and don't see much
of a reason for source inclusion.
* pcre, idna: I'm not too sure about these, but dynamic linking seems
to make more sense. I didn't hit any stumbling block so far in nvm.stripped.
* boost: we've done quite a lot of work to get rid of boost and
replaced it with our own code. It began with boost::fs (see [1]) and
continued with test, regex, date_time and probably others. Now we only
rely on some boost header files, but we don't source include these
ATM. Reading the ML-archive, there are several complaints about
difficulties building boost, it's bjam based build harness is a bit
uncommon. Botan also provides it's own (perl based) configure script.
And autoconf is far from being everybody's friend. Boost is old enough
to be shipped with most major distributions.
The reason for removing boost::regexp were, that it's bloated. pcre is
much smaller. I'm unsure about boost::test and boost::date_time.
If we really decide to not want any boost library, I'd like to
a) be clearer about the reasons and
b) source include the required boost header files, so as to really be
independent of boost.
If not, we need to decide if we want to re-add boost::date_time to
replace our own date handling code. mtn-dates.o from nvm.dates is
currently 89 KiB on my 64bit machine, while the boost::date_time .so
file weights only 60 KiB (presumably lots of header only stuff). From a
quick glance at the docu, it does much more than what's currently
covered by nvm.dates. But improved date parsing would be nice to have.
Not sure if it provides the same independence from the system's epoch as
we currently do.
I've dropped boost::graph from nvm.cvsimport-branch-reconstruction and
coded my own graph stuff (i.e. a dijkstra search algo) for better
performance and flexibility (different in-memory representation from
boost's).
In the archive I've read that boost::program_options didn't do what we
want. Looking at the boots documentation, I also see boost::asio - could
that perhaps be a better replacement for the (unmaintained) netxx?
Anybody used boost::asio before?
Thoughts? Comments?
Regards
Markus Wanner
[1]: Nathaniel: boost::fs is slow:
http://article.gmane.org/gmane.comp.version-control.monotone.devel/4262
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] date certs on net.venge.monotone
Hi, Ethan Blanton wrote: That is correct. It actually caused problems for us at some point, when something related to monotone started rejecting those fractional second values. The oldest revisions in that Pidgin data started out as CVS, were migrated to SVN, and then subsequently migrated to Monotone. There's plenty of weird and wonderful-ness in there. ;-) Aha, thanks for confirmation. Which brings me to a (yet another) reason that I don't think this mucking about with date certs is a good idea -- there are almost certainly *correct* date certs in the Pidgin repository which are out of order. I'm not sure what you mean be *correct* dates certs being out of order. I've counted exactly 36 discrepancies in the pidgin repository, see my mail starting this thread: pidgin: 36 of 28349 (0.13%) During our migration, there were a number of revisions which, for one reason or another, had problems which made them difficult to migrate (svn is capable of creating revisions under normal circumstances which it cannot subsequently handle gracefully). Yeah, svn is capable of doing lots of nasty things... There are sequences of such revisions which were collapsed to one patch-up revision in the migration; the date stamp on the patch-up revision is the date of its creation, not the date of the original changes. This causes one date in the middle of a sequence of valid revisions to be months or even years newer than its ancestors or descendants. Agreed. Generally speaking, date information in monotone is very much meta-information. Introducing the idea of global clocks in this manner takes it from being meta-information to being revision information. I'm just proposing to check the date and warn the user if it obviously doesn't match. It would help making that meta-information more reliable, nothing more, nothing less. I think this is a bad idea. If I've had a patch sitting on my disk for 2 months, and I just now get around to committing it, it may be a reasonable and correct thing for me to want to do to use a --date argument to the commit of the creation of the patch, not the commit of the patch. In that case you should certainly also commit against the parent revision you've created the patch against. So that would not result in timestamp inconsistencies. I guess, basically, I'm not a fan of validating anything for consistency which does not have a clear definition of consistency. In the absence of global clocks, dates do not have consistency. The fact that monotone currently recognizes and ignores this information is a feature, not a bug. Monotone does not ignore the information, but create date certs from it. Ignoring it would mean dropping date certs entirely. Please keep in mind that for us human beings there *is* a global clock and we like to express ourselves in terms of date and time. Of course, monotone cannot rely on the system clock being in sync with the global clock. But as pointed out, the assumption is good enough to provide that convenience feature called date cert. To help the user making that convenience feature more reliable, it would help if monotone checked date certs, IMO, without hurting anything else. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] date certs on net.venge.monotone
Hi, Stephen Leake wrote: If there is no global shared clock, then developers working in different time zones can easily commit revisions to a shared server that have timestamps using local times that appear to be out of order with respect to each other. Any sane OS today knows the difference between local time and UTC. Measurement shows that this is not an issue (except maybe for Xaraya, which needs further investigation) On the other hand, there are such things as global clocks; we use them all the time at NASA. If each developer on a team commits to configuring their computer such that the functions local_to_UTC and UTC_to_local work properly, My assumption is that most clocks are close enough to the global clock to be useful for mtns purpose. Looking at the dates in nvm, this assumption holds true for the vast majority of committed revisions. monotone could support an option to use UTC times for timestamps. Monotone currently uses UTC dates for its certs, which might not be overly user friendly. However, we do not take leap seconds into account within our own date calculation routines. TAI might be a better global time choice, since it doesn't do leap seconds, and hence is monotonic. Conversion between UTC and local time is pretty simple on most systems, where as UTC to TAI and back is not as common. IMO the user shouldn't need to care about anything except local time. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] date certs on net.venge.monotone
Hi, Thomas Moschny wrote: Monotone stores timestamps in the date certs in UTC, of course. Nevertheless, clocks for each developer might differ a bit such that commits appear to be made in non-chronological order. That's theoretically possible, though pretty rare in practice. So, I don't think we should warn there. Even make hints about clock skews. Why is that so unwanted for monotone? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] date certs on net.venge.monotone
Hi, Nathaniel Smith wrote: Make warns about clock skew (between your local host and its NFS server) because such clock skew breaks Make. Agreed. Right now, clock skew (between all the hosts used by everyone in a distributed development project) does not break monotone; This depends on whether you count invalid date certs as breakage or not. You obviously don't, I do. it continues to provide its basic functionality of managing revisions just fine. Your proposal would compromise this core functionality (either by causing revisions to disappear, or just printing annoying warning messages), so as to better provide the functionality of letting people know that someone else's clock that they can't fix is broken. The main motivation is making dates cert information more reliable. But it looks like that's not considered core functionality of monotone. I understand the impulse, Phew, good to hear. but I don't see how this is the right place for it -- if I really want to know about clock skew, there are better ways to find out, and if don't have access to those better ways, I can just run that script you wrote over my monotone history anyway. My question is: how do I avoid silly situations like this one here: # mtn -d $BOTAN_DB log \ --brief --last=3 \ --from 1cc7ac30ff0c2871e24e3a76d147cab907b5429e o 1cc7ac30ff0c2871e24e3a76d147cab907b5429e [EMAIL PROTECTED] | 2006-09-26T19:33:01 net.randombit.botan o fb8114127ac9e23c9870bba631a79c41ece29355 [EMAIL PROTECTED] | 2006-09-27T03:44:57 net.randombit.botan o 3c74c0b06690f73b9d34435e77432e6776c1f7e3 [EMAIL PROTECTED] 2006-09-27T01:29:54 net.randombit.botan Why does monotone show a date at all, if it's so unreliable, unchecked and not considered core functionality by monotone? Anyway, I think there's enough resistance to drop the idea for now. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] nvm.dates
Hi, I'm considering nvm.dates ready to land on mainline. All it does is using a 64 bit integer to represent dates internally, so that date_t now provides comparison and difference operators on these timestamps. It does not alter the textual representation in the database nor for netsync. As there seems to be a majority voting against further checking of the date cert information, I'm not continuing my efforts into that direction. Instead I've turned that portion of code into an additional --full helper option for mtn db info, which now displays some statistic analysis of those date certs, see below for examples - no warnings, no further checking, only db info is affected. I've committed that into a separate branch nvm.dates.statistics. Please review as well, I consider that to be ready to land on mainline, too. Regards Markus Wanner Here's the relevant additional part of db info --full for some of the larger monotone databases. Admittedly, the seconds values are often too large to really be human readable, but I'm not overly keen about writing beautification code for that. monotone counts: ancestry edges : 17,827 timestamp correctness between revisions correct dates : 17,733 edges equal dates : 1 edges incorrect dates : 61 edges based on root : 32 edges missing date(s) : 0 edges timestamp differences between revisions: mean: -28,365 sec min : -1,944,060 sec max : 60,951,415 sec 1st percentile : 6 sec 5th percentile : 48 sec 10th percentile : 134 sec 25th percentile : 1,032 sec 50th percentile : 10,026 sec 75th percentile : 72,933 sec 90th percentile : 263,267 sec 95th percentile : 613,691 sec 99th percentile : 3,904,735 sec botan = counts: ancestry edges : 1,797 timestamp correctness between revisions correct dates : 1,793 edges equal dates : 0 edges incorrect dates : 3 edges based on root : 1 edges missing date(s) : 0 edges timestamp differences between revisions: mean: 115,312 sec min : -29,516 sec max : 19,441,763 sec 1st percentile : 9 sec 5th percentile : 24 sec 10th percentile : 43 sec 25th percentile : 160 sec 50th percentile : 736 sec 75th percentile : 6,181 sec 90th percentile : 96,268 sec 95th percentile : 355,795 sec 99th percentile : 2,287,001 sec openembedded counts: ancestry edges : 32,693 timestamp correctness between revisions correct dates : 32,185 edges equal dates : 1 edges incorrect dates : 500 edges based on root : 7 edges missing date(s) : 0 edges timestamp differences between revisions: mean: 24,547 sec min : -49,295,411 sec max : 49,409,456 sec 1st percentile : -6,927 sec 5th percentile : 35 sec 10th percentile : 65 sec 25th percentile : 263 sec 50th percentile : 1,757 sec 75th percentile : 8,855 sec 90th percentile : 39,635 sec 95th percentile : 81,190 sec 99th percentile : 410,294 sec pidgin == counts: ancestry edges : 28,353 timestamp correctness between revisions correct dates : 28,313 edges equal dates : 11 edges incorrect dates : 25 edges based on root : 4 edges missing date(s) : 0 edges timestamp differences between revisions: mean: 52,390 sec min : -10,292,985 sec max : 63,591,821 sec 1st percentile : 14 sec 5th percentile : 62 sec 10th percentile : 129 sec 25th percentile : 583 sec 50th percentile : 3,000 sec 75th percentile : 16,084 sec 90th percentile : 62,077 sec 95th percentile : 116,494 sec 99th percentile : 675,252 sec xaraya == counts: ancestry edges : 23,413 timestamp correctness between revisions correct dates : 17,306 edges equal dates : 5,781 edges incorrect dates : 256 edges based on root : 67 edges missing date(s) : 3 edges timestamp differences between revisions: mean: -88,445 sec min : -80,315,704 sec max : 80,488,504 sec 1st percentile : -345,601 sec 5th percentile : 0 sec 10th percentile : 0 sec 25th percentile : 0 sec 50th percentile : 86,400 sec 75th percentile : 345,601 sec 90th percentile : 1,036,800 sec 95th percentile : 2,335,023 sec 99th percentile : 19,489,817 sec This also solves the mystery of the xaraya repository: I've counted equal dates as invalid before. Obviously they often use some sort of automatic or scripted merge or propagation of revisions, which then end up having equal timestamps. The analysis confirms my assumption that developer machine's clocks are mostly close enough to the commonly known global clock. Percentages of invalid date cert information for the checked repositories are: monotone: 0.34% botan:0.17% oe: 1.53% pidgin: 0.09% xaraya: 1.09% ___ Monotone-devel mailing list Monotone
Re: [Monotone-devel] 3rd party libraries
Hi, Zack Weinberg wrote: In this message I just want to address the boost situation. Thank you very much, that was what I've been looking for: a compilation of reasons for having dropped boost. Makes sense to me. (And justifies that my re-implementation of parts of boost::date_time in our dates.cc wasn't just a waste of time). .. the random number generator... Uh.. do we use a RNG from boost as well? We certainly use one from botan. I'll look into that... some day Also, I've read the boost::asio docs and I actually think that would be a good thing to be using. The abstractions look sane, the author appears to understand what you need to do to get a stable C++ ABI, and it would certainly be an improvement over the unmaintained 'netxx' thing we have now. But I'd much rather we used it in its standalone libasio incarnation, that doesn't pull in any other boost modules. Cool, thanks for this opinion. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: date certs on net.venge.monotone
Hi, Bruce Stephens wrote: Because (presumably) it's useful, even though it's not as reliable as you might like. I note that (AFAIK) no other VCS verifies dates either, and they all still keep them. And emails regularly contain incorrect dates, but dates are still on the whole useful. Agreed. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hi, Zack Weinberg wrote: I'm all for dynamic linking of botan, too, and being able to use the accelerated engines. Cool. I'm trying to keep up with the development of botan, test against new releases and potentially adopt to its changes. I'm eagerly awaiting 1.8, though. :-) So I believe this is the one that started graydon on the bundle everything path, with specific point revs of sqlite 2.x working for us and others not, and over at my day job we've been having similar problems with more recent releases of it. So it may well be worth continuing to bundle a known good version of the amalgamation. If we do, though, we should be much more proactive about testing newer releases so we don't wind up where we are now (i.e. stuck multiple feature releases behind the curve) again. Agreed. By now I know about enough to test and integrate new sqlite releases as well. I'll try to keep monotone's sqlite integration code up-to-date as well. I plan to add an optional amalgamation variant to nvm.stripped. I think I mentioned this before, but if we're going to stop bundling lua we have to overhaul the error-handling interface between lua and our code, because we're currently relying on lua's ability to be compiled as C++ and use C++ exceptions to report errors. Of course we need to overhaul it *anyway* because right now lots of errors just silently get lost, but remember that this is on the critical path for unbundling it. Okay, thanks for the reminder. To be compatible to system libraries, compiling lua with plain C (instead of C++) makes sense, IMO. Thus I'll look into proper lua error handling for C compiled lua. Regarding file size, are you sure you're not counting debug information? On my (also 64-bit - amd64) machine, the file is 129KB, but 'size' says it's only got about 7KB of actual code in it. That's a n.v.m build; I haven't built your .dates branch but I don't see anything in there that would bulk it up that much. Oh, right, that was counting debug info as well. It's rather in the order of 11 KiB. Improved date parsing certainly would be nice to have but I think if we're going to do anything there, it oughta be that we figure out how to use gnulib's getdate.y, which is the gold standard as far as user-supplied date parsing. Okay, thanks for that hint. I won't have time to look into that anytime soon, though. nvm.dates is currently sufficient for my needs WRT dates. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: date certs on net.venge.monotone
Hi, Lapo Luchini wrote: I think Nathaniel's point of you can't fix is important… let's exemplify it: One can always distrusting invalid information. Currently this involves distrusting the whole revision, but that doesn't necessarily have to be bound together. mtn could easily provide a command to re-certify a revision with a sane date, for example. day 1. user NTP commits with correct date 2008-10-24 day 2. user MadTime commits with incorrect date 2009-10-25 (and you can do nothing about it, as it's incorrect in the future, so the check you propose would happily pass) day 3. user NTP tries to commit with correct date 2008-10-26 gets a warning he can do nothing about as it wasn't his fault really # mtn pull madtime.badhost.com ... WARNING: rev XY is certified to be from the future (or your system's clock is out of sync). ... # date-- checking against wrist-watch # mtn cert XY date `date -u +%Y-%M-%dT%H:%M:%S` Of course, that gets pretty tedious when multiple revisions are involved. But that's a question of good or bad UI, not one of internal workings and checking. And it would require policy branches to be anywhere close to usable. So, yes, at the moment it would involve unwanted UI uglification. So I'm postponing this proposal until we have policy branches :-) Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hi, Matthew Nicholson wrote: Yeah, I recently noticed the same issue. To properly unwind the stack on error in a c++ app, lua needs to be compiled with a c++ compiler. Hm.. thanks for this hint. It uses setjmp and longjmp for error handling when compiled as C and exceptions when compiled as C++. If we don't require stack unwinding on error (I have not looked at the code), then dynamic linking is fine, but I think this is a candidate for bundling. I'll check this issue. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hello Jack, Jack Lloyd wrote: I would really like to get 1.8.0 out the door before the end of the year. We'll see. Sounds fine to me. Thanks for the overview. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Lua loading dynamic libraries not possible in monotone?
Hi, Alex Sandro Queiroz e Silva wrote: Markus Hanauska wrote: The platform itself supports this. When downloading and building Lua command line interpreter, I can run the code and it works just fine. Yeah, it has been intentionally disabled here: - Revision: db3a7b999139a8c005b76b172496e15b276e39db Ancestor: caaff5d316a3bd5636529a94fd075aba76963583 Author: [EMAIL PROTECTED] Date: 2006-08-15T16:08:10 Branch: net.venge.monotone Deleted entries: m4/popt.m4 Modified files: ChangeLog configure.ac lua/luaconf.h m4/boost.m4 options.hh ChangeLog: * configure.ac: Delete MTN_POPT_DEPENDENCIES. Add strptime to main AC_CHECK_FUNCS list (this is the only thing that MTN_POPT_DEPENDENCIES checked for, that was used outside popt). Also add lrint to this list. Delete AC_SEARCH_LIBS for libdl. * lua/luaconf.h: Include config.h. Set LUA_USE_POSIX if _WIN32 is not defined. Add versions of lua_number2int and lua_number2integer that use the C99 lrint() function; remove the unreliable union hack. At end of file, forcibly undefine all LUA_DL_* macros to ensure that lualib.cc uses the stub implementation. * m4/popt.m4: Delete file. * m4/boost.m4: Create a new macro to factor out code in common among most of the MTN_BOOST_LIB_* macros. * options.hh: Fix typo. If I remember correctly, it was turned off for security reasons. If that's the case, it's another thing I'd have to check when building monotone against the lua system library, because that one quite certainly has dynamic module loading enabled. However, I don't quite understand why it should be a security issue. All hooks are user defined, so what should preventing dynamic loading protect against? Maybe it's rather a simplification for portability? Zack? @Markus: maybe you want to try my net.venge.monotone.stripped branch, which links against a system provided lua 5.1 library. That one should be capable of loading dynamic libraries. (And I'm curious if you can get nvm.stripped to compile on MacOS X. ;-) ) Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hi,
Stephen Leake wrote:
I'm confused.
Your question confuses me too ;-)
You seem to be implying that a dynamically linked object file cannot
be compiled with a C++ compiler. Is that correct?
An object file *is* compiled, it can only be linked against other
compiled objects.
The difference here is between compiling the lua source code with a C or
a C++ compiler. Only the later can throw exceptions, the former only
knowns longjmp/setjmp functions to provide similar behaviour.
This is the relevant part of luaconf.h:
#if defined(__cplusplus)
/* C++ exceptions */
#define LUAI_THROW(L,c) throw(c)
#define LUAI_TRY(L,c,a) try { a } catch(...) \
{ if ((c)-status == 0) (c)-status = -1; }
#define luai_jmpbuf int /* dummy variable */
#elif defined(LUA_USE_ULONGJMP)
/* in Unix, try _longjmp/_setjmp (more efficient) */
#define LUAI_THROW(L,c) _longjmp((c)-b, 1)
#define LUAI_TRY(L,c,a) if (_setjmp((c)-b) == 0) { a }
#define luai_jmpbuf jmp_buf
#else
/* default handling with long jumps */
#define LUAI_THROW(L,c) longjmp((c)-b, 1)
#define LUAI_TRY(L,c,a) if (setjmp((c)-b) == 0) { a }
#define luai_jmpbuf jmp_buf
#endif
What I don't quite understand is, how that's relevant to monotone.
That's a slightly different issue, but...
So far I thought that the exceptions from C++ compiled lua were
propagated to monotone. But that doesn't seem to be the case. No matter
if lua is compiled to use C++ exceptions internally or not, we are
calling lua functions via lua_pcall() in Lua::call(). That's a
protected call which returns a boolean upon failure.
Thus, the only thing that seems dubious to me is that there is only one
failure case. And i.e. for use_inodeprints(), that's not even
different from a function returning false:
bool
lua_hooks::hook_use_inodeprints()
{
bool use = false, exec_ok = false;
exec_ok = Lua(st)
.func(use_inodeprints)
.call(0, 1)
.extract_bool(use)
.ok();
return use exec_ok;
}
Which means, that the following two lua functions result in the same
behavior for monotone:
function use_inodeprints()
return true
end
function use_inodeprints()
error(this error does not really show up anywhere noticeable)
end
Only if you turn on '--debug' you'll see it in the log lines.
This is in contrast to compilation errors, i.e. for such a thing:
function use_inodeprints()
-- lua comments normally look like this line
# this is not a lua comment and leads to a compilation error
end
This raises a lua error while loading rcfile ... exception.
Am I missing something?
If not, I'd say this is irritating at least and vote for propagating lua
errors back to the monotone user, just like loading failures.
There are not many tests for such thing, are there? I didn't find
neither unit nor lua tests.
Regards
Markus Wanner
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hi, Stephen Leake wrote: Obviously we don't want to re-compile the system library. But is there some reason it wasn't compiled with C++ in the first place? Uh.. because if compiled by a C compiler, it's still usable from C++, but not the other way around, AFAICT. Besides, I'd guess there are more C users of that library than C++. There seems to be an assumption that system libraries are _not_ compiled with C++. For most libraries, there's no such choice. Lua is an exception in that regard. And the default is compiling in C. Being able to compile in C++ is just an additional option or feature. I suspect that's because there is no system library naming convention that distinguishes between otherwise identical libraries, one compiled with C++, one with C. Object symbols are also different. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] compilation failure on Win32 MinGW
Hi, Stephen Leake wrote: Stephen Leake [EMAIL PROTECTED] writes: compile on Win32 MinGW is failing with: in branch net.venge.monotone make[3]: *** No rule to make target `../monotone/botan/big_base.cpp', needed by `botan/lib3rdparty_a-big_base.o'. Stop. Hm.. did you run 'make distclean' and probably re-run the autoconf toolchain? I've upgraded the included botan version which carried some renames and such. But I didn't test on Win32. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] nvm.dates
Hi, thank you for your review. Stephen Leake wrote: So I think keeping date unsigned is ok, but please document this rationale in dates.hh. Okay, will do. One reason for using unsigned is that a single $DATE $MAX is sufficient in valid(), whereas with a signed integer we'd have two comparisons. Not sure if it matters that much at all. This change certainly makes it easier to add operations on dates in the future, which we might want to do. I can see that some 'import from other CM tool' functions might want to check dates. Yes, CVS is one candidate, but we already have a special parse_date in rcs_import.cc... I didn't look into merging those, even though it would not be overly complicated, because they both are basically: YY(YY) MM DD HH MM SS But differ in the separators... I see no reason to object to landing it on mainline. Cool, thanks. I see no reason to object to landing this on mainline either. Some of the stats could be useful; frequency of checkins might correlate with something :). Hehe... yeah. :-) Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] 3rd party libraries
Hi, Matthew Nicholson wrote: So this was not completely clear from the other responses. To help visualize this, draw an imaginary line between lua code and c++ code called from lua and monotone's other c++ code. Errors (C++ exceptions) are not supposed to cross this line. Hm.. that's a good point as well. So we don't only need to pay attention to the lua - mtn boundary, but also the other way around, when lua calls back mtn. Thanks for that hint. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.dates
Hi, Lapo Luchini wrote: Well the question is: should we allow commits (or rather, imports) to have a date before year 1970? (personally I'd use signed for the sake of it, but won't really ever notice, as I won't be importing anything from the 60s anyway ^_^) We didn't ever support dates before 1970, which is a pretty common boundary. I don't see much reason to add support for that now, either. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] compilation failure on Win32 MinGW
Hello Stephen, Stephen Leake wrote: compile on n.v.m Win32 MinGW is failing with: ../monotone/botan/es_dev.cpp:7:24: sys/select.h: No such file or directory Hm.. thanks for that report. Must have to do with our self-made build harness for botan and the upgrade to botan 1.7.12 (which already seems old by now). The problem is in Makefile.am BOTAN_SOURCES; previously, 'es_dev.cpp' was not included there; it was added if needed in lib3rdparty_a_SOURCES by an 'if WIN32_PLATFORM' conditional. That conditional is still there, and does the same thing. So I think we just need to delete 'es_dev.cpp' from BOTAN_SOURCES. That seems correct, thanks. Hmm. That almost works; now I get the same error in 'es_egd.cpp'. This was previously not compiled on Win32, but also not on other OS as well, apparently. It could be added to the same 'if WIN32_PLATFORM' conditional. That gives no errors on Win32 (obviously). According to the botan module build information, es_egd is only supposed to be built for unix systems. That didn't get clear from your description, but you did keep es_egd.cpp for Unixen in your patch. So did I. Then I get: ../monotone/botan/primes.cpp:608: warning: integer constant is too large for long type These constants need a suffix 'LL' to be interpreted as type 'long long'. Yes, this is known an has been discussed with Jack. He didn't want that patched in upstream but instead requires an -fpermissive compiler flag. I'm pretty clearly against diverting from upstream, but this currently means compiling all of the 3rd party libraries with -fpermissive. I'm just wondering why this fails for MinGW. Is your gcc invoked with that flag or not? See below for the patch for these changes. Any objection to committing this? I've committed the changes to Makefile.am with minor adjustments and some comments on these exceptions. Thanks for your report and please test again. All in all I'm really looking forward to nvm.stripped to get rid of these custom build harness issues. Do you mind test building that branch on MinGW? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Lua loading dynamic libraries not possible in monotone?
Hi, Markus Hanauska wrote: Okay, I will try that out. What else is special about this stripped branch? Any other changes (despite the fact that it uses external lua) compared to the standard version? It links monotone against the system's libraries, instead of bundling required libraries. You thus need development packages for: PCRE Lua Botan SQLite AFAICR I've already changed the INSTALL document somewhat, please consult that. Can anyone show a real-life attack for this? After all the Lua code (and the libraries it might use) have the same restrictions (e.g. file permissions, other system restriction) as someone would have on command line anyway. To me a security problem only arises if an attacker could that way execute code he could otherwise not execute. However Lua will not be able to access a dynamic library a user isn't able to access anyway, either by using a stand-alone Lua copy or by compiling a tiny piece of C code that links against the library and execute it on the machine. Agreed. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: Can't find lua lib
Hi, Markus Hanauska wrote: Where does it look for this library? Where do I have to install it? I downloaded the lib via fink, it is there. I even softlink'ed it to /usr/lib, still no success. Any ideas? Fink? That sounds like MacOS X, right? Where is it, when you say it's *there*? You might want to try various LUA_CPPFLAGS (or was it LUA_CFLAGS, I don't remember...). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: nvm.dates review
available, but that's not urgent.) -unsigned int const MIN = 60; -unsigned int const HOUR = MIN * 60; -unsigned int const DAY = HOUR * 24; -unsigned int const YEAR = DAY * 365; -unsigned int const LEAP = DAY * 366; +u64 const SEC = u64_C(1000); +u64 const MIN = 60 * SEC; +u64 const HOUR = 60 * MIN; +u64 const DAY = 24 * HOUR; +u64 const YEAR = 365 * DAY; +u64 const LEAP = 366 * DAY; Why was this necessary? The number of milliseconds in a year exceeds 2^32. Other operations must be done in 64bit as well, so I concluded it's safer to use 64bit operations and types everywhere. Another advantage of not using struct tm is, you could report the milliseconds too. Uh.. we certainly don't want that for date certs, do we? Any other use cases? This function should maybe have the word millisecs somewhere in its name. Agreed, renamed to millisecs_since_unix_epoch(). It might be more efficient to divide out the milliseconds at the very beginning. (And come to think of it, you could do seconds, minutes, and hours that way too, and deal in day counts for the rest of the code, which might be more comprehensible than how I wrote it originally.) I simply required the ability to do some +/- calculations on these dates. I do clearly not have the intention to write a general purpose and high performance date_time handling library for C++. It's fractional seconds (you could have an arbitrary number); why ignore them, when you support millisecond precision? Because we don't need them. It's simple enough to add as soon as we do. None the less, I've now added milliseconds support for our_gmtime and our_mktime. If you get the time to improve the parser as well, please go ahead. However, as you say yourself, we should maybe better look into gnulib's parser. I expect this will be going away again after you take out the four-digit year check. Not sure I get around increasing that limit anytime soon, but I promise to take care of it before year 2050, which should still be soon enough. ;-) Again, thank you for your review. Please bear in mind what the purpose for this change was/is: adding the ability to get timestamp differences, adding and subtracting differences to them. IMO this has already taken way too long and we should better spend our dev-time on more relevant things. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: nvm.dates review
Hi, Markus Wanner wrote: Agreed, renamed to millisecs_since_unix_epoch(). Hm.. this isn't used anywhere, since date_t::operator-() is sufficient in all cases so far. So we might even drop it. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: Summit thoughts - and changes?
Hi, Lapo Luchini wrote: Other people? Comments? December doesn't work for me anymore... sorry. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] compilation failure on Win32 MinGW
Hi, Stephen Leake wrote: Is botan supposed to compile with anything other than Gnu compilers? Yes. Agreed, but we should be clear on why upstream isn't adding LL. Please check the botan-devel archive. I committed this change over the weekend, because I wanted to fix those warnings in nvm.resolve_conflicts. I'll take them back out for now. Good. but this currently means compiling all of the 3rd party libraries with -fpermissive. Which means we're giving up a lot of compiler checks; that doesn't seem like a good idea. Agreed. nvm.stripped helps that. AFAICT Jack thinks of it as gcc being overly pedantic. In that case, I absolutely agree with him. In general the autoconf stuff can handle per-file compiler options. But that may be a pain to implement for this. Yes, having done the recent integrated-botan upgrades, I strongly recommend against such a thing. Botan itself doesn't use autoconf. It doesn't actually fail, since it's just a warning. But in my experience, C warnings are bad, and should be fixed. So I always compile under Emacs, which highlights warnings in the compiler output. Ah! So, please be more specific. The warning is there on gcc on all platforms and should be ignored. With all the LLs in place, and removing -fpermissive, it compiles without warning or error. Sure. What is the compiler actually doing for these constants (without LL)? The warning implies it is truncating them to 32 bits, which would be really bad. It doesn't truncate them, it's just an annoying and pedantic warning, AFAICT. Do we have unit tests that verify the compiler is doing the right thing? No, why should we? It's a botan thing. Please concentrate on monotone ;-) I've committed the changes to Makefile.am with minor adjustments and some comments on these exceptions. Thanks for your report and please test again. It compiles fine now. Cool, thanks for testing. All in all I'm really looking forward to nvm.stripped to get rid of these custom build harness issues. Do you mind test building that branch on MinGW? Configure dies, with error: Your lua library is not useable. I don't have a MinGW Lua package installed. We'll need to update the MinGW installation instructions on the Wiki. But apparently the Wiki is moribund at the moment? nvm.stripped hasn't landed, yet, so I don't think updating the Wiki is the first thing to do. We (you?) first need to figure out how to compile nvm.stripped on MinGW, then we can document. Do you have a lua library at all? Maybe it should say: not found, instead of not usable? The m4 scripts aren't overly clever, yet. I was hoping some m4 wizard could pick up and help, because I'm not particularly found in m4. I'll have to get my MinGW buildbot back on line; the video output died, so I need to get a new computer. Having a MinGW buildbot would certainly be cool. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] mtn wiki
Hi, the wiki has partly been migrated to another wiki system on the last summit. That's great, but the efforts somehow stalled. We are now left with a wiki marked as do not edit for already more than half a year. I only had a quick glance at the newish wiki branch, so I'd like to know from those in the know: * what's missing on the new ikiwiki? * what does it take to switch to the new one? * what does it give us, that the former one didn't? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] mtn buildbots
Hi again, besides the wiki, what's up with the buildbots? Most of those seem to be more redish than green, but mostly because the bots are down. Maybe that's a complaint towards buildbot itself, but I find it troublesome to distinguish between monotone failures and buildbot ones. Additionally, I'd like to start thinking about testing of nvm.stripped. For that, the bots would need to have the required libraries installed. Preferably with some comments about which versions of the libraries are installed. I'd like to add some bots with different library versions. Virtual ones and most probably all Debian, but none the less... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] compilation failure on Win32 MinGW
Hi, Stephen Leake wrote: There's no search function at http://lists.randombit.net/pipermail/botan-devel/, so that's rather difficult. Oh, hm.. Jack? Then I suggest we put a comment in the code at that point saying we are ignoring the warning, so this doesn't come up again. That would only lead to tedious merge failures for upgrades. Better strip that 3rd party library... I guess we assume botan has its own test suite; those of us who are paranoid should run it in addition to the mtn test suite :). That's close to impossible to do for the integrated botan. You'd have to port the test suite. Again, strip it! Well, we could add a future process section to the wiki. But this is moot, since the wiki is currently not editable. Good point. What's up with the wiki? Let's start another thread for that question, though... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Re: Summit thoughts - and changes?
Hi, Lapo Luchini wrote: What about 17th-18th January 2009? Works for me as well. Although I cannot promise to be available all weekend long. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] heads up on nvm.stripped
Hi, I've upgraded and tested nvm.stripped with the recently announced botan 1.7.21 and SQLite 3.6.5. The code to switch sha1 engines within monotone caused problems with the newer botan release. As we do not use that code anywhere except for the benchmark_sha1 command, I've simply removed it. It rather looks like botan offers something similar to benchmark various sha1 engines. I didn't look into that, though, but don't think we need to duplicate that work. nvm.stripped now compiles fine and passes all tests again. Regards Markus Wanner monotone 0.41 (base revision: c18c4c9d2fbcdaaf08fb448837b778e6f30968a6) Running on : Linux 2.6.24-rc7-cametha #1 SMP Mon Jan 14 15:31:35 CET 2008 i686 C++ compiler: GNU C++ version 4.3.2 C++ standard library: GNU libstdc++ version 20080905 Boost version : 1_34_1 SQLite version : 3.6.5 (compiled against 3.6.5) Lua version : Lua 5.1 PCRE version: 7.6 2008-01-28 (compiled against 7.6) Botan version : 1.7.21 (compiled against 1.7.21) ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] heads up on nvm.stripped
Hi, Jack Lloyd wrote: I've attached a patch for sha1.cc demoing the new benchmark code. Very cool, thank you. FYI, my Core (1) Duo reports: mtn: Benchmarking botan's SHA-1 core mtn: SHA-1 provider 'core': 117.172 MiB/s mtn: SHA-1 provider 'ia32': 135.325 MiB/s For informative purposes only, since it currently only works on an unreleased version (future 1.7.22, nrb head), and API-wise things are still in flux in this area. That's fine. I fear the #ifdef's are not quite enough. We'd have to check for the botan version dynamically. So I'd say we better wait a bit until that API stabilizes. In the future it may be useful to extend Monotone to run a quick test and then choose the fastest SHA-1 provider to use for the rest of the program run. 20 milliseconds of benchmark time at startup could pay for itself quickly in many cases. ..or even store the setting in _MTN/options because such a benchmark might not pay-off every time mtn is invoked. Alternatively have it stored in monotonerc and set as a system config during installation of monotone. Storing in _MTN/options sounds easier to automate, so I'm currently favoring that variant. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] mtn status speed
Hello Jack, Jack Lloyd wrote: I could not say for certain, but from looking at the code implementing the status command it appears to respect the masks provided on the command line. Tracking the open syscalls using strace (http://linux.die.net/man/1/strace) also seemed to confirm this, mtn didn't open or stat any files outside of the paths specified. One odd thing I noticed is that mtn will stat each file twice, right in a row. Yeah, this is known and has been discussed before. Unfortunately I don't remember the reason for calling stat() twice. Caching of the stat() result has been considered, but argued against, because stat() should be fast and simple enough. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Monotone Crash
Hi, Ulf Ochsenfahrt wrote: I was just syncing when I got a monotone crash, as you can see from the log below. It's not reproducible - second sync worked. I'm posting it here to preserve the log in case someone encounters this again. No core dump, apparently. I remember having seen this as well, but haven't been able to reproduce it either. A glibc error is rather bizarre, IMO. Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Status of nvm.stripped
Hi, Thomas Keller wrote: Understood - then please merge it in after 0.42 is released. I absolutely agree with Zack, that merging just *after* a release is the way to go. But please do not merge, before further testing on various platforms. We can do that before landing it on mainline. I've set up some virtual machines, which I intend to use as buildbots, but didn't get around installing all the required libraries and tools. Another issue that came up is, that we'd have to test against various versions of the dependent libraries. I'm not sure how much coverage we can really achieve with reasonable efforts. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Status of nvm.stripped
Hi, Stephen Leake wrote: If you are refering to the conflicts resolution stuff in nvm.resolve_conflicts, that has already been merged into main. Oh, I missed that. Sorry for the noise, then. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] monotone-0.41 compile failure with glibc(disabled-nls)
Daniel Black wrote: As per https://bugs.gentoo.org/show_bug.cgi?id=238540 monotone fails to compile when glibc is compiled with --disable-nls. This occures even when monotone is configured with ./configure --disable-nls. Hm.. I'm unable to reproduce this issue on my virtual Gentoo 2008.0 box. I'm using these USE flags in /etc/make.conf: # manually prevent unneeded stuff USE=-kde -qt3 -qt4 -gtk -gnome -nls I've rebuilt 'world' once, then recompiled glibc, gcc and gettext again, but still don't see the compilation errors. The tests suite runs through fine as well. What bugs me is, that ./configure still says: ... checking whether NLS is requested... yes ... (as long as I don't manually give --disable-nls). So I'm guessing my glibc is still compiled with nls. How do I check? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] throw usage(); or N(); for argument checking?
Hi, Timothy Brownawell wrote: N(false, message) results in throw informative_faulure(message), the question is what to throw rather than whether to throw. I'm with Timothy on that. These states are exceptional enough and much simpler to code. While control loops (why a loop, simple conditions would do, no?) clutter .. ehm.. exception handling a lot. A command that does throw usage() gives the same result as calling mtn help command, printing full usage info to stderr, where N() results in mtn: misuse: message on stderr and will put a note in any debug log. As long as the message is maintained in case of throw usage(), I'm fine. I dislike tools which just throw the complete usage page at me and let me figure myself. Some hint on what's wrong certainly helps. And that hint should survive, IMO. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] throw usage(); or N(); for argument checking?
Hi, LeJacq, Jean Pierre wrote: Two considerations: 1. N() can always be modified to provide an alternate implementation, though I acknowledge that changing from throw semantics would be problematic. For example, adding logging before the throw. Isn't this exactly what Timothy proposes? Clearer separation of the two. 2. N() is not the best choice for handling this to begin with. Agreed. Timothy proposed using throw usage(); instead, which is according to these two requests, AFAICT. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] Monotone 0.42: peer [...] IO failed in confirmed state (success)?
Hello Timothy, Markus Wanner wrote: Ralf S. Engelschall wrote: | mtn: peer monotone.ca IO failed in confirmed state (success) | mtn: successful exchange with monotone.ca I can reproduce this on a i386 FreeBSD 6.4 and will try to hunt the bug - sometime this month. Thank you for reporting this. This has to do with your changes to netsync between 0.41 and 0.42 (b0fcf9be.. by Timothy 2008-10-18, per mtn annotate). AFAICT select() is saying the file descriptor is ready for reading, while reading from the file descriptor returns 0 bytes. According to the man page for read(), returning less than the requested amount of bytes is fine, only return values 0 signify an error. The select_tut man page clearly indicates, that after getting zero bytes from read() or recv(), one should not call select() on the fd anymore. I'm not overly familiar with netsync, but tried to fix it anyway. Please review rev. f18abebd.. Unit tests work fine on Debian sid, FreeBSD 6.4 and Gentoo 2008.0 and the misleading error reported by Ralf has disappeared. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] monotone-0.41 compile failure with glibc(disabled-nls)
Hi, Philipp Gröschler wrote: If you have app-portage/gentoolkit installed, try # equery uses sys-libs/glibc to see which use flags apply to your current glibc build. Thanks for this hint, glibc really is compiled with NLS support. How do I build the glibc *without* it? Does the following help? # USE=-nls emerge glibc Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] monotone-0.41 compile failure with glibc(disabled-nls)
Hi, Philipp Gröschler wrote: You can check with # USE=-nls emerge -av glibc so that portage prints out what it would do. If it accepts the use flag, the output should look like Thanks, that looks good. If NLS still would be compiled in, try # eselect profile list and chose hardened/linux/x86 (or whatever architecture is appropriate). Not 100% sure if that helps, but on when I tried this on my virtual box a few days ago, NLS was disabled globally. Disabling it for only the glibc could get you into serious trouble, I guess. Sometimes Gentoo is a little bit goatish ;-) Hehe.. yeah. I'll try the hardened variant, cannot hurt to have such a hardened buildbot... Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] git fast-export
Hi, Derek Scherger wrote: I've spent a bit of holiday hacking time working on a git_export command for monotone, more as an experiment than anything else. I've committed the result to net.venge.monotone.fast-export for people to have a look at. Awesome! Three exported branch names net.prjek:tester, net.prjet:tester/drop-for-propagate and prjek.net:tester where changed (with sed) during the import process because git does not allow colon's (and various other characters) in branch/ref names. I simply changed : and / in these names to . although the / should have worked it did cause an error of some sort. Maybe respect those limitations within mtn git_export? The conversion was verified by checking out each of the 276 branches and 126 tags from both git and mtn and comparing the resulting workspaces. The script I used to do this verification was a bit dumb and failed to checkout a few revisions so these weren't compared. Using only the branch name failed in some cases because there were multiple heads and using only a tag name failed in some cases because the tagged revisions had no branch certs. All of the branches and tags that did checkout were identical according to diff -qr so I'm reasonably confident that the new exporter basically works. In my experience with cvs_import, it's rather often the history, than the end result, which looks strange. However, that's certainly harder to check and git seems a tiny bit closer to the spirit of mtn, than CVS. I suspect that the various other git fast-import conversion scripts that exist for monotone are probably slower and less robust than this implementation (unless they work similarly from rosters) which uses the monotone internals to do the work. I spent a bit of time initially trying to export revisions using the revision data structures but this didn't work very well. Git only deals with files and trying to order a mix of renames of directories and files from monotone correctly from revisions was difficult. Ultimately I didn't use the revision data structures at all but built up a similar files-only based revision representation by comparing rosters. Much like what is done for make_cset, but ignoring directories and producing only file deletions, renames and additions. This works much better, correctly handles pivot_root and a few other odd things that working with revisions proved difficult. Sounds good. This exporter does not (yet) handle all rename ordering issues that are possible. For example rename a b followed by rename b c will probably fail on import unless it is executed as rename b c followed by rename a b. Similarly rename a b followed by rename b a which is indeed possible, will probably fail on import and requires the introduction of a third temporary file. These problems can be fixed in the exporter and can also be fixed in the exported data by re-ordering renames as required. Hm.. maybe I just don't understand git fast-import format, but isn't the ordering derived from the revision graph? This feels a bit like throwing in the proverbial towel and I hope this doesn't elicit any ill-will from the current monotone crowd. I'm not really planning on converting my personal stuff from monotone any time soon but knowing it can be done without losing information is nice. You are definitely underestimating the mtn world domination plan (tm). I'm still happy to contribute to monotone but with 2 small kids my free/hacking time is pretty limited. Very well understood. Some questions aside: is there a git fast-export? Can you comment on writing mtn git_import? Problems we are facing there? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] Re: nvm.dates review of the review
Hello Zack, Zack Weinberg wrote: I've now pushed some revs to nvm.dates.. Very cool! Thank you. I have some further changes in mind for date handling, but I'm not going to get to them for a couple days. If you would like to merge nvm.dates and nvm.dates.statistics into mainline now, please go ahead. Barry any objections I plan to land both tomorrow. I made the ::from_string() factory into a constructor but left ::now() and the other constructors alone. That's enough consistency for me. Cool, makes sense. (I've adjusted nvm.dates.statistics accordingly). I changed our_mktime to our_timegm. The system function mktime() operates in the local timezone so it is clearer not to use that name at all for a function that operates in GMT. Some (unfortunately not all) systems have timegm() that operates in GMT. Looks fine. Practically speaking, is far enough in the future, I imagine by then we'll all be using something totally different. But making the code work as far out as you possibly can is good anyway, because it forces you to make the algorithm more robust. I made our_gmtime work as far out as I could (not to 2147483647 -- it turns out that we overflow a signed 64-bit millisecond count before then) and in the process was forced to come up with a technique for calculating the year that is just plain better, even in the date range we'll actually be using. Cool. Another advantage of not using struct tm is, you could report the milliseconds too. Uh.. we certainly don't want that for date certs, do we? Any other use cases? I haven't done this part yet, but we do actually want milliseconds in date certs, or at least we want to accept and preserve them when they show up. I don't remember the exact details, but conversion from some foreign formats produces date certs with milliseconds in them, which is why the from-string constructor currently accepts and ignores digits after the decimal point. IIRC that's tailor, which wrote date certs with milliseconds. We certainly should to preserve this information, yes. Are you saying you want to default to storing milliseconds in the date cert? I personally don't think that's necessary, but I don't mind much. What I *do* mind is that I prefer not to see the milliseconds part in the normal case, because it's basically noise which just clutters the output. I want to mention four more things that we should try to get out of our date handling. With your changes and mine on top of them, we're very close, but some more work is still needed. I'm planning to do all of this, but if you feel inspired to do 'em first, please go ahead. Very good compilation. Sadly, I won't have time to look into this again anytime soon. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] git fast-export
Hi, Thomas Keller wrote: Of course this is bikeshed discussion and still doesn't solve the issue where to put these - having two groups already with only one command (database, automation) is already stupid enough - and a command group cannot be specified in the commandline either, so an import and export group is equally stupid. Sorry if that didn't get clear, but I'd vote for the smallest change for now. (IMO, that means leaving cvs_import and git_export, but put them into a single command group). Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] git fast-export
Hi, Thomas Keller wrote: I scanned over your patch and noticed only a little quirk: you've introduced yet another command group (git); wouldn't it be better to collect all the different import/export functions under one common group, like the existing rcs (revision control systems) or, to avoid the RCS analogy, vcs? Maybe ovcs, for Other, Overaged, Obscure or Outrageous VCS? Other than that, I agree with Thomas that a single command group for all import/export/push/pull/sync mechanisms to/from other VCSen is plenty. It's not like we have such an overwhelming amount of conversion commands. (I'd also love to see the underscore in cvs_import and git_export vanish. But that's another issue..) Other than that I think you're right that people should have the choice what to use, and at least some need for a mtn - git conversion tool popped up a few times on the list. Since git seems to become the new big thing its formats will probably be more wildly used than ours, so even people who stick to monotone will have something from this. (F.e. code_swarm [0] has a git-mode to render the pretty videos, though the XML format [1] it needs as input is easy enough to create even with a custom lua command...) I absolutely agree. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] git fast-export
Hi, Derek Scherger wrote: It's used for revs that have no other branch certs to use. This is a bit off-topic here, but: is there a use case for revisions without *any* branch cert? It's somewhat similar to the invalid date certs: it's certainly easier to just store the data and not care. However, most other VCSen I've seen so far do not allow such a thing. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] nvm.dates landed
Hi, Markus Wanner wrote: Barry any objections I plan to land both tomorrow. net.venge.monotone.dates.statistics (and therefore net.venge.monotone.dates) have just landed on mainline. Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] monotone-0.41 compile failure with glibc(disabled-nls)
Hi, Thomas Moschny wrote: It turns out that this can be solved by updating our copy of gettext.h with a more recent version. If no one objects, I'm going to commit that. Sure, please go ahead. I'm still unable to reproduce *this* bug (on Gentoo Hardened, that is), but it turns out there are enough other bugs ;-) Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] memory access violation in win32 0.42 when canceling blocked `mtn diff --external` using Ctrl+C
Hi, Daniel Atallah wrote: c:\Program Files\monotone\mtn.exe: fatal: memory access violation this is almost certainly a bug in monotone. please send this error message, the output of 'c:\Program Files\monotone\mtn.exe version --full', and a description of what you were doing to monotone-devel@nongnu.org Hm.. I get similar errors on Gentoo Hardenen during unit tests. Might not be related at all, but I'm still wondering where these come from. Clearly this isn't a major problem, but I figured I should report it. Thanks for your report. Did you (or anybody else on MinGW) run the test suite for 0.42? rant I'm trying to get MinGW installed, but Windoze is so damn uncomfortable! /rant Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] memory access violation in win32 0.42 when canceling blocked `mtn diff --external` using Ctrl+C
Hi, Zbigniew Zagórski wrote: Yep, i've ran test suite before publishing binaries. There were no ... err, almost no failures. At least no significant ones. Cool, thanks. Failures: util_mtnopt because windows doesn't have /bin/sh usually installed automate_lua because 3.4 on polish locale is 3,4 grrr They are: 1. minor, 2. unrelated I forgot to report them never mind having time to fix. Can you reproduce what the OP reported? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] compatibility of 0.40 and 0.41 question
Hi, Hugo Cornelis wrote: I am surprised that a 0.41 monotone executable cannot trivially pull a 0.40 database. Please note, that these are two slightly different things: pulling is done via netsync, different mtn versions very rarely break netsync compatibility. Another issue is different mtn versions reading the same database file. There have not been any database schema changes since 0.39, so 0.40, 0.41 and 0.42 should all be able to read the same database file without any changes. So the question is a very general one: what is exactly the policy for different monotone versions pulling, pushing and syncing databases? Please read UPGRADE and/or NEWS. AFAICT there have not been any netsync incompatibilities since at least 0.25. For database schema changes, a simple mtn db migrate should do the trick. What does mtn version --full and mtn db info give? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] bug in monotone
Hi, Thomas Keller wrote: Anyways, the error is quite common and unfortunately won't be fixed any time soon. Let's at least say: we should try to improve the error message. Do we have some sort of TODO file or something? Or is this a summit'09 task: to organize todo items? Regards Markus Wanner ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
