Re: [fossil-users] Fossil performance and optimisation
On Aug 11, 2017, at 2:55 PM, Damien Sykes-Lindleywrote: > > so that sounds can easily be edited and changed without loss of quality they > are uncompressed PCM data. I suppose I could convert them to FLAC I must not have been clear: when you cannot use a text-based file format with Fossil, you at least want to use an *uncompressed* binary file format. So, stick with PCM, WAV, AIFF, etc. Absolutely do not switch to FLAC. If you move to a game engine that wants a compressed audio format — some use Ogg Vorbis, for example — my recommendation is that you still store the data in Fossil in uncompressed form, but add some build rule that converts it to the compressed form for testing and delivery. Not only will this make Fossil — or really, *any* VCS — happier, it lets you change your compression settings, format, etc. without bloating the repository with multiple versions. I speak from experience. My largest repository has essentially the same graphics for a web app in: 1. 8-bit GIF format from the days before PNG was in all browsers 2. 8-bit PNG format from the days before IE got the ability to cope with alpha channels in PNG 3. A different set of 8-bit PNGs when we changed our UI color scheme enough that the old 8-bit matting caused color fringing when the old PNGs were composited atop the new app background. 4. 24-bit PNG format once we finally dropped support for those old versions of IE If I’d had the foresight to write a script to convert them from the high-res PSD or SVG forms to GIFs or PNGs, I’d still only have one version of most of these graphics. > As for the executable, sometimes that gets included due to the fact that we > forget to delete it after testing an executable copy and don't use Fossil's > ignore feature You can “shun” those after the fact to strip them out of the repo. You have to get all clones to cooperate for this to work, but it’s doable. > compiled language like C++ where there are many assets to manage can make > compilation a real ballache - maybe that's because I'm so new to such > languages. If your build system is not reliable, I’d say that’s where you should spend your efforts, not on trying to get Fossil to cope with checkins containing unstripped binaries generated from other files that are efficiently stored in the repository. You probably need some amount of scripting that lets you clone a repository and then run the script to get things into a buildable shape, automatically and reliably. Anything less becomes the PITA you’ve run into. Windows is a bit of an outlier here: on all the other major platforms, we have powerful, scriptable build systems, which we had to build in order to deal with the complexity of platform differences. A positive side effect of that is that we can script our way out of most build complexities. I pity the Windows dev that is forced to choose between batch files and PowerShell to achieve the same end. > I had no idea about the checksum setting. Despite three year usage, it seems > I've only scratched the surface of Fossil. Again, another reason why I put > some of my thoughts out there. Some users, and of course the developers > themselves, who are much more knowledgeable on these things than I am. We now await a repeat of your tests. :) ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Action-based automation and scripting
On Fri, Aug 11, 2017 at 10:27 PM, Warren Youngwrote: > 3. libfossil: > > http://fossil.wanderinghorse.net/repos/libfossil/index.cgi/wiki/home > > Beware that it no longer works with Fossil 2.x repositories that use > non-SHA-1 hashes. That’d the the first thing someone would need to fix, if > they wanted to adopt the project. > Anyone interested in doing so, please feel free to get in touch with me on-list or off, but be aware that it won't be a cake-walk - you will have to get your hands dirty and figure out how fossil works internally (much of which is documented in the libfossil API docs, however). If a new owner needs a place to host it, i would be happy to continue hosting it for them. -- - stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil performance and optimisation
Hi Warren, Very interesting. As I said I don't understand how these things work internally, I just thought I'd put it out here and see what the more knowledgeable people thought of these comparisons. I just know that it records artefact changes, I've no idea how the compression or local/remote sync works or what repercussions would occur as a result in either system. I ought to clarify that the games I work on are based on sound rather than graphics, and so that sounds can easily be edited and changed without loss of quality they are uncompressed PCM data. I suppose I could convert them to FLAC, but given that the game engine doesn't support this format they would have to be reconverted back to PCM before every test. As for the executable, sometimes that gets included due to the fact that we forget to delete it after testing an executable copy and don't use Fossil's ignore feature, although to be fair with some of my non-game projects I do include executables and binary libraries for the simple fact that using a mainstream, compiled language like C++ where there are many assets to manage can make compilation a real ballache - maybe that's because I'm so new to such languages. So if I include the executable I can always get the latest without having to spend hours trying to figure out how to recompile the thing. Same with libraries - some libraries that I use are proprietary and so including them in source form isn't an option. Especially in an interpreted language where I can simply run the code from the source script, if it needs the libraries to run I always include them. Again, maybe a mistake on my part but I always like to be prepared when I am checking out a new branch or version to test or work with. The Fossil and Git commit tests were both locals with no remotes attached, so one didn't have any speed advantages against the other due to network usage. I had no idea about the checksum setting. Despite three year usage, it seems I've only scratched the surface of Fossil. Again, another reason why I put some of my thoughts out there. Some users, and of course the developers themselves, who are much more knowledgeable on these things than I am. Cheers. Damien. -Original Message- From: Warren Young Sent: Friday, August 11, 2017 9:08 PM To: Fossil SCM user's discussion Subject: Re: [fossil-users] Fossil performance and optimisation On Aug 11, 2017, at 7:10 AM, Damien Sykes-Lindleywrote: I couldn't help noticing there seemed to be a silence on speed comparisons. There have been many threads on this over the years. Just for a start, search the list archives for “NetBSD”. After cloning and working with several publicised Fossil repositories, I can't help but notice that the majority of them are rather small. Yes, that’s best practice for most any DVCS. Even the Linux project takes this philosophy: http://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html That is, the single monolithic repo you see when cloning “the Linux Git repo” is something of an illusion, which the developers of Linux don’t actually deal with very much. Most of the projects that I am involved with are games...Of course these will contain binary files That “of course” needn’t be a foregone conclusion. Many asset formats are available in text forms, which are friendly for use in version control systems. For example, you may be able to store 3D models in the repository in COLLADA format and some 2D assets in SVG. For the bitmapped textures, it’s better to store those as uncompressed bitmap formats, then compress them during the build process to whatever format you’ll use within the game engine and for distribution. A 1-pixel change to a Windows BMP file causes a much smaller change to the size of a Fossil repository than does a 1-pixel change to a JPEG or PNG, because that 1 pixel difference can throw off the whole rest of the compression algorithm, causing much of the rest of the file to change. This can be tricky to manage. You might think TIFF is a good file format for this purpose, but you’re forgetting all the metadata in it that changes simply when a file is opened and re-saved. (Timestamps, GUIDs, etc.) It’s better to go with a bare “box of pixels” format like Windows BMP. All of this does make the checkout size bigger, but Fossil’s delta compression has two positive consequences here: 1. The Fossil repository size will probably be as small or even smaller. A zlib-compressed Windows BMP file is going to be about the same size as a PNG file with the same content. 2. If those files are changed multiple times between initial creation and product ship time, the delta compression will do a far better job if the input data isn’t already compressed. This is how you get the high compression ratios you see on most Fossil repositories by visiting their /stat page. My biggest repository is
Re: [fossil-users] 500 errors on Chisel.
Any update on the migration? Thanks, Jacob. On Sat, Jun 3, 2017 at 11:22 AM Jacob MacDonaldwrote: > Appreciate the update! > > On Sat, Jun 3, 2017 at 11:16 AM Roy Keene wrote: > >> Jacob, >> >> This is caused by Fossil using a newer version of SQLite3 than the >> system SQLite3 which "Edit" uses. The system SQLite3 DB can't open the >> Fossil repository to make some modifications due to the schema being >> unsupported on some repositories. >> >> I can't make Fossil use the system SQLite3 since it requires newer >> versions (presumably to support this schema). >> >> I plan to upgrade the SQLite3 used by Flint/PHP next week as I migrate to >> a new host which should address the issue. >> >> A better fix would be to not talk directly to the SQLite3 DB but make the >> changes by calling the Fossil binary, but that hasn't been investigated >> too closely. >> >> Thanks, >> Roy Keene >> >> On Sat, 3 Jun 2017, Jacob MacDonald wrote: >> >> > For the past few weeks, I've been unable to edit my repositories on >> > Chisel (page throws a 500). Unfortunately, since I only have access to >> > my account password and not each repo's password, that means I can't get >> > write access to any of my hosted code. >> > Is this a known issue/something I'm doing wrong? >> > >> > I've seen a bit of chatter re: Chisel on the list, but if there's a >> > better place for the question, please let me know. >> > >> > Thanks! >> > >> > Jacob. >> > >> > >> ___ >> fossil-users mailing list >> fossil-users@lists.fossil-scm.org >> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users >> > ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Action-based automation and scripting
On Aug 11, 2017, at 7:03 AM, Damien Sykes-Lindleywrote: > > Is it possible for Fossil to run scripts when performing operations like > creating new repositories, importing, exporting, committing etc? There are several ways: 1. Tcl/TH1 script feature: https://www.fossil-scm.org/index.html/doc/trunk/www/th1.md You probably want to be looking at th1Hooks. 2. React after the fact by monitoring /timeline.rss. This is sufficient for CI/CD integration, for example. https://www.fossil-scm.org/index.html/help?cmd=/timeline.rss 3. libfossil: http://fossil.wanderinghorse.net/repos/libfossil/index.cgi/wiki/home Beware that it no longer works with Fossil 2.x repositories that use non-SHA-1 hashes. That’d the the first thing someone would need to fix, if they wanted to adopt the project. 4. Fossil’s JSON API: https://goo.gl/GSBUa2 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil performance and optimisation
On Aug 11, 2017, at 7:10 AM, Damien Sykes-Lindleywrote: > > I couldn't help noticing there seemed to be a silence on speed comparisons. There have been many threads on this over the years. Just for a start, search the list archives for “NetBSD”. > After cloning and working with several publicised Fossil repositories, I > can't help but notice that the majority of them are rather small. Yes, that’s best practice for most any DVCS. Even the Linux project takes this philosophy: http://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html That is, the single monolithic repo you see when cloning “the Linux Git repo” is something of an illusion, which the developers of Linux don’t actually deal with very much. > Most of the projects that I am involved with are games...Of course these will > contain binary files That “of course” needn’t be a foregone conclusion. Many asset formats are available in text forms, which are friendly for use in version control systems. For example, you may be able to store 3D models in the repository in COLLADA format and some 2D assets in SVG. For the bitmapped textures, it’s better to store those as uncompressed bitmap formats, then compress them during the build process to whatever format you’ll use within the game engine and for distribution. A 1-pixel change to a Windows BMP file causes a much smaller change to the size of a Fossil repository than does a 1-pixel change to a JPEG or PNG, because that 1 pixel difference can throw off the whole rest of the compression algorithm, causing much of the rest of the file to change. This can be tricky to manage. You might think TIFF is a good file format for this purpose, but you’re forgetting all the metadata in it that changes simply when a file is opened and re-saved. (Timestamps, GUIDs, etc.) It’s better to go with a bare “box of pixels” format like Windows BMP. All of this does make the checkout size bigger, but Fossil’s delta compression has two positive consequences here: 1. The Fossil repository size will probably be as small or even smaller. A zlib-compressed Windows BMP file is going to be about the same size as a PNG file with the same content. 2. If those files are changed multiple times between initial creation and product ship time, the delta compression will do a far better job if the input data isn’t already compressed. This is how you get the high compression ratios you see on most Fossil repositories by visiting their /stat page. My biggest repository is rocking along at 39:1 compression ratio, and it hasn’t been rebuilt and recompressed lately. > (generally an executable Why would you include generated files in a version control repository? Fossil is not a networked file system. If you try to treat it like one, it will take its revenge on you. > dependency libraries In source code form only, perhaps. Even then, it’s better to hold those in separate repositories. It would be nice if Fossil had a sub-modules feature like Git to help with this, so that opening the main repository also caused sub-Fossils to be cloned and opened in subdirectories. Meanwhile, you have to do manual “fossil open --nested” commands, but it’s a one-time hassle. Nested checkins would also be nice. That is, if a file changes in a nested checkout, a “fossil ci” from the top level should offer to check in the changes on the sub-project. > Also note that all commits were tests only and so weren't synced to remotes. > Naturally this means that commits are even slower when syncing. It also means that local differences are a smaller percentage of the total time taken for many operations, since the time may be swamped by network I/O. For instance, I notice in your tests that you seem to be comparing “fossil ci” to “git commit”, where the fair test would be against “git commit -a && git push”. > 1. Git seems to do better at compressing and opening smaller repositories, > while Fossil triumphs over larger ones. Be careful with such comparisons. Fossil repositories aren’t kept optimally small, since that would increase the time for checkins and such. Every now and then, even after an initial import, you want to look into “fossil rebuild” and some of its more advanced options. This is what I was getting at about with my comments about the 39:1 compression ratio I’m currently seeing on my largest Fossil repository. I expect I could make it smaller, if I did such a rebuild. I have no idea if Git has some similar “rebuild” feature, though I will speculate that the per-file filesystem overheads will eat away at a lot of any advantages Git has. Be sure you’re calculating size-on-disk, not the total size of the files alone. That is, a 1 byte file on a filesystem with a 4K block size takes 4K plus a directory entry, not 1 byte. Fossil, by keeping all artifacts in a single file, does not have this overhead. The “rebuild” problem is
Re: [fossil-users] Action-based automation and scripting
On Fri, Aug 11, 2017 at 4:25 PM, Damien Sykes-Lindley < dam...@dcpendleton.plus.com> wrote: > Oh wow. That must be so frustrating. Keyboards, both musical and computer, > are the centre of my life right now, so if that ever happened to me I would > be completely devastated. > Hope things work out for you. > The first six months were depressing, but humans can adapt to anything. You eventually get used to not having to get up every morning, not having to go to bed early, and basically just being a lazy old fart! :) -- - stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Action-based automation and scripting
Hi Stephan, Oh wow. That must be so frustrating. Keyboards, both musical and computer, are the centre of my life right now, so if that ever happened to me I would be completely devastated. Hope things work out for you. Best, Damien. From: Stephan Beal Sent: Friday, August 11, 2017 2:18 PM To: Fossil SCM user's discussion Subject: Re: [fossil-users] Action-based automation and scripting On Fri, Aug 11, 2017 at 3:03 PM, Damien Sykes-Lindleywrote: Also, like Git has LibGit for managing repositories, does Fossil have such a library? I know someone was working on LibFossil - does this give total access or is it still a work in progress? That was me. An unusually extreme case of RSI ("Sulcus-ulnaris Syndrome" in both arms) has removed me completely from the programming scene the past nearly-3 years, so libfossil is deal until/unless someone wants to take it over (which involves bringing it up to date with the SHA changes). i'm currently waiting on the paperwork to go through for my early retirement due to inability to work (my hands and elbows can't handle much activity anymore), and my programming days are over until/unless my hands eventually fully recover (it's not clear whether they ever will or not). -- - stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Action-based automation and scripting
On Fri, Aug 11, 2017 at 3:03 PM, Damien Sykes-Lindley < dam...@dcpendleton.plus.com> wrote: > Also, like Git has LibGit for managing repositories, does Fossil have such > a library? I know someone was working on LibFossil - does this give total > access or is it still a work in progress? > That was me. An unusually extreme case of RSI ("Sulcus-ulnaris Syndrome" in both arms) has removed me completely from the programming scene the past nearly-3 years, so libfossil is deal until/unless someone wants to take it over (which involves bringing it up to date with the SHA changes). i'm currently waiting on the paperwork to go through for my early retirement due to inability to work (my hands and elbows can't handle much activity anymore), and my programming days are over until/unless my hands eventually fully recover (it's not clear whether they ever will or not). -- - stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Fossil performance and optimisation
Hi, I was introduced to Fossil about three years ago. It is the first VCS I have actually used extensively for my own work and as such I haven't really had much to compare it with. However, now I'm in the realm of contributing code I am having to learn how to make use of other systems as well. For the purposes of my experiments, I have been comparing Fossil alongside Git, as this seems to be the popular alternative and there is a comparison sheet describing many areas of difference between Fossil and Git. For the most part, Fossil has many advantages over Git. Though I couldn't help noticing there seemed to be a silence on speed comparisons. There is a separate article on performance statistics that again doesn't even mention speed. After cloning and working with several publicised Fossil repositories, I can't help but notice that the majority of them are rather small. The system.data.sqlite repository is probably the biggest I've seen. As of 11th August 2017: Flint: 468KB, 65 commits Hydra: 648KB, 37 commits Fossilbook: 37.2MB, 82 commits Fossil: 56.1MB, 10696 commits SQLite: 92MB, 19275 commits SLT: 142.5MB, 165 commits TCL: 212.3MB, 21227 commits system.data.sqlite: 395.4MB, 3072 commits Most of the projects that I am involved with are games designed to run on Windows systems. These are anywhere from a small arcade game (10-100MB), or a fully fledged action packed or role playing adventure game (the biggest so far being 5GB). Of course these will contain binary files (generally an executable, dependency libraries and, most of all, media). It is the big ones where Fossil seems to be tripping over its own feet... My machine: Windows 7 64-bit quad-core hyperthreaded, 32GB ram, 31.42TB total storage. Fossil version: 2.3 [f7914bfdfa] 2017-07-21 03:19:30 UTC Note: None of the below repositories have any ticket or custom wiki entries, and all only have the main trunk branch. Also note that all commits were tests only and so weren't synced to remotes. Naturally this means that commits are even slower when syncing. I have included export times because I have also done some experimentation with import/export. While the below tests don't seem to be too taxing in regards to exporting, try git-cloning https://github.com/torvalds/linux (roughly a 15 minute operation) and import that to Fossil. On my machine it hadn't finished importing after three days, and the bigger it got the more it slowed. Took an hour to go from 2 to 3GB, and roughly 10 hours to go from 4 to 5GB. Of course you wouldn't find me so much as touching the Linux code - it is very different to Windows and even using it has turned out to be a pain in the rear and a bullet in the leg. I did this merely to test Fossil's handling of large repositories. Similar slowdowns also seem to be occurring when building zip files from repositories. In fact there have been various scenarios when I have wanted to visit a certain checkin, and have had to open that checkin as a separate checkout and zip it with another utility, because Fossil gives me out of memory errors. With Fossil, we get the following results: Repo1: 76.5MB, 3 total commits Clone: 14S (approx. 5.1MBPS) Open: 3S (Latest commit, 322 files, 89.1MB, approx. 107 files or 29.7MB per second) Commit: 12S (Deleted 5 small files) Export: 7S (approx. 10.93MBPS) Repo2: 554.4MB, 315 total commits Clone: 2M27S (Download=1M10S, finalise=1M17S, approx. 7.92MBPS) Open: 10S (Latest commit, 446 files, 385MB, approx. 44 files or 38.5MB per second) Commit: 8S (Deleted one small file) Export: 1M1S (Approx. 9.09MBPS) Repo3: 792MB, 340 total commits Clone: 2M16S (Download=1M40S, finalise=36S, approx. 7.92MBPS) Open: 20S (Latest commit, 1087 files, 906MB, Approx. 54 files or 45.3MB per second) Commit: 25S (Changed a few audio files) Export: 1M24S (Approx. 9.43MBPS) Repo4: 2.4GB, 7 commits Clone: 6M30S (Download=5M, finalise=1M30S, approx. 8MBPS) Open: 43S (Latest commit, 574 files, 2.36GB, approx. 13 files or 181.5MB per second) Commit: 1M18S (Edited perhaps 10 lines of code) Export: 3M55S (Approx. 10MBPS) Repo5: 5.6GB, 24 total commits Clone: 9M1S (Download=6M38S, Finalise=2M23S, approx. 14.07MBPS) Open: 1M35S (Latest commit, 1380 files, 4.18GB, approx. 14 files or 45MB per second) Commit: 1M30S (Added 2 source code files totalling 62.2KB) Export: 11M49S (Approx. 9.17MBPS) Importing the same repositories into Git and doing similar commits yields the following results. Unfortunately I don't have a private Git server from which to test clones, however Fossil's cloning is pretty fast anyway so I'm not too concerned about this. Repo1: 76.5MB, 3 total commits Git size: 75.1MB Open: 3S (Latest commit, 322 files, 89.1MB, approx. 107 files or 29.7MB per second) Commit: Instantaneous (Edited code) Repo2: 554.4MB, 315 total commits Git size: 655.5MB Open: 5S (Latest commit, 446 files, 385MB, approx. 88 files or 77MB per second) Commit: instantaneous (Source tree directory
[fossil-users] Action-based automation and scripting
Hi there, Is it possible for Fossil to run scripts when performing operations like creating new repositories, importing, exporting, committing etc? I'm guessing there will be somewhere, since sometimes people want to have an automated commit build, mail announcements and the likes, without having to manually edit and rebuild Fossil to make it work like that. I just can't seem to find it documented. Also, like Git has LibGit for managing repositories, does Fossil have such a library? I know someone was working on LibFossil - does this give total access or is it still a work in progress? Cheers. Damien. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Captchas
Hi there, A while ago, I wrote about the possibility of providing an accessible alternative to an ASCII-art CAPTCHA system due to the fact that visually impaired users can only read these, through screen reading technology, as a series of dashes, dots and bars. Although the CAPTCHA system is no longer needed for the most part, anonymous logins still require the use of one of these CAPTCHAs. An accessible CAPTCHA could be providing an alternative to the visual CAPTCHA (such as audio), using scrolling plaintext, or requiring the answering of a question (either calculating a formula, trivia etc). Though I appreciate that implementing some of those options may be hard (certainly hard for me who has limited mainstream programming/software engineering knowledge), I would hope that at least the question option would be easier to implement while also maintaining a reasonable size. I didn’t quite get the final upshot of that suggestion. Are there plans to implement this? Cheers. Damien. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users