On Sun, May 21, 2017 at 11:25 PM, Stephan Eggermont <step...@stack.nl> wrote:
> At the PharoDays I was painfully reminded that SSDs perform really badly > when using small files. The Bloc tutorial used a github filetree repo and > that has a lot of files. The whole folder is 116 MB in 16K files. Copying > that amount of data should not be noticable, taking about a third of a > second. With it being in so many files, it took more than half a minute, or > a hundred times as long. > > That is too much overhead. How can we improve the file format in a way > that keeps the cross-platform exchange advantages and a reasonable way to > view diffs and propose small changes using the github web tools? > > Cuis uses a different format with git. How does that compare? What is used > in Squeak? > > So this was copying files at the OS level? Which OS? For comparison, pulling whole of pharo-core across network and unpacking... $ time git clone --depth=1 https://github.com/pharo-project/pharo-core.git real 0m44.306s user 0m5.384s sys 0m5.908s Count and size of files $ find pharo-core/ | wc -l 152547 [files] $ du -sh pharo-core 639MB The smaller size of the archive without compression was a surprise. i.e. two-thirds empty space in filesystem blocks... $ time tar cf pharo-core.tar pharo-core/ real 0m8.216s user 0m2.284s sys 0m3.776s $ ls -lh pharo-core.tar 212MB File by file copy to a USB stick. Wow! Yes, thats over an hour. $ time cp -R pharo-core /media/ben/USBSTICK/pharo-core real 68m49.168s user 0m1.852s sys 0m39.768s Many forums indicated the following should be faster, but it was not... $ time ((tar cf - pharo-core) | (tar xf - -C /media/ben/USBSTICK/)) real 77m47.896s user 0m6.360s sys 0m38.900s Saving to an archive on USB stick is much quicker... $ time tar cf /media/ben/USBSTICK/pharo-core.tar pharo-core real 1m35.723s user 0m3.188s sys 0m5.972s Saving to a compressed archive on USB stick is muchmuch quicker... $ time tar zcf /media/ben/USBSTICK/pharo-core.tgz pharo-core real 0m11.778s user 0m7.908s sys 0m3.604s Saving to a compressed archive on USB stick, then unpacking that to local drive... $ time (tar zcf /media/ben/USBSTICK/pharo-core.tgz pharo-core \ && tar zxf /media/ben/USBSTICK/pharo-core.tgz -C restore/) real 0m49.934s user 0m10.552s sys 0m9.192s So 38 seconds to unpack, which is very similar to original git clone. But I guess your primary drive is SSD? So you don't get to bypass the issue using archvies? What result do you get cloning pharo-core like my first test case? cheers -ben
