GfW is an abbreviation for Git for Windows' - I think I'd introduced it earlier in one of the posts.
The "Native" size for Windows, in backwards compatibility terms (which is at the "compiled binary" level") is 32 bits, i.e. old code runs without change. Hence the addition of "long long" by Microsoft on Windows. Linux took the opposite view of backwards compatibility (which is at the source code level), so to a full step up to 64 bit long (it's in the POSIX spec). Horses for courses. Clydesdale vs Arabian/Thoroughbred. On 04/03/2022 06:35, skybuck2000 wrote: > > > On Thursday, March 3, 2022 at 11:42:54 AM UTC+1 Konstantin Khomoutov > wrote: > > On Wed, Mar 02, 2022 at 06:03:46PM +0000, Skybuck Flying wrote: > > >> Search for "4GB" in this blog post [1] which discusses the GfW > 2.35 release > >> notes. > >> > >> 1. https://github.blog/2022-01-24-highlights-from-git-2-35/ > > > Thanks for the update, however I don't see how this could effect > git > > checkout of linux kernel, since most linux source files are only > a few > > kilobytes... > > Because Git has several data storage/transfer optimization tricks. > One of them > is using of so-called "pack files" (basically that's the > behind-the-scenes > mechanism that allows Git to remain fast and have modest storage > requirements > while its data model states that each commit is a snapshot of the > whole > project). Pack files are used for both storing stuff on the > filesystem, and > for data transfer; they basically are gzipped archives with added > index files > for fast random access to the contained data. > > > Depends on how these pack files are used, if only parts of them are > read/written then it does not need to be a problem :) > > Also I did not see any big spikes in memory consumption, at least when > I was paying attention. > > > > My take is that processing of such a pack file hits some limit of > the current > GfW implementation, which has nothing to do with the OS limits > (given that > up-to-date API calls are used). > > > What is GfW ? > > Firefox also fails regularly when many tabs open, it only does this > when pagefile.sys is off. > > So as far as I am concerned it clearly has something to do with the > memory system in windows which is probably less well tested when > pagefile.sys is off because most systems/default installation have a > pagefile.sys, therefore I recommend any serious programmer working on > serious project to turn off pagefile.sys or swap space and observe the > memory system ! I am sure you will be surprised ! =D > > > > While I think such problems were actually solved some time ago, > some may still > remain. In any case, it's hard to say something w/o proper bug > report - either > in the project's tracker of at least on the git-for-windows > mailing list. > > > Why would that make any difference ? Are you hoping for developer > feedback on this issue ? > > I did mention the error message but in case you missed it here it is > one more time: > > fatal: Out of memory, malloc failed (tried to allocate 214601 bytes) > fatal: index-pack failed > > So from this I can see two things: > > 1. It was trying to allocate a 214 kilobyte block which is quite large > and makes the fragmentation hypothesis somewhat believable since > windows uses 4 kilobyte pages as far as I know and these are stored in > "lists". So to forfill this request it needs 214601 / 4096 = 53 > pages. Now maybe a statiscian can somehow calculate the chance of 53 > consequetive pages being present without any gaps/missing ones, > especially when the system is under heavy use trying to create many > little C files or whatever it was doing. From what I remember it was > resolving deltas... so it seems git was "replaying" changes and these > could be made out of many many many little changes, hence this makes > the memory fragmentation hypothesis believeable. > > Also what kind of file would need 214 kilobytes ? Perhaps linux has a > file of that size, maybe some documentation or so, though I have also > seen big lookup tables in other software like bitcoin 256psek or > something and pascalcoin and regex/unicode lookup table which is > slowing down or freezing analysis tools so this is not good. > > 2. Indeed it's doing something with an index-pack file... so this is > probably what you based your analysis on so you did read the bug/error > report that I posted :) > So this could mean it was extracting something or maybe indeed packing > something up, but it wasn't that big... and it was computing some > index file. > > For now the easy assumption is to assume the system simply run out of > memory, but this is a suspicious/somewhat bad conclusion because the > chart in the video does not really back that hypothesis up, and also > as far as I can re-call the browser was not crashing and seemed to be > doing fine, now maybe it simply had enough memory to do it's thing, > but usually the browser can suddenly take up a lot of memory and crash > quite easily... then again... since git is using many little files... > maybe it just hit it sooner and is more sensitive to out of memory or > something, though it could also be memory fragmentation or a > combination. For now the memory fragmentation or in combination with > system cache taking up precious memory is believeable because other > people have noticed strange behaviour with windows 7. > > > By the way, do you use a 64-bit install? If, for some reason, > you're using a > 32-bit version, the limit of circa 4 GB (actaually lower) will be > "native". > > > new@new-PC MINGW64 / > $ file $( which git ) > /mingw64/bin/git: PE32+ executable (console) x86-64, for MS Windows > > new@new-PC MINGW64 / > $ git --version --build-options > git version 2.29.2.windows.2 > cpu: x86_64 > built from commit: 3464b98ce6803c98bf8fb34390cd150d66e4a0d3 > sizeof-long: 4 > sizeof-size_t: 8 > shell-path: /bin/sh > > Something confusing and conflicting information, however if a 64 bit > version is available I usually download the 64 bit version. The OS is > 64 bit so it doesn't really make sense to run 32 bit software and > artificially limit it's capabilities. However I did not make git so > maybe deep inside git it might be using some 32 bit components and/or > limitations, this I don't know. > > The PE32+ statement is somewhat confusing, but I guess it means an > extended version of PE32 and is therefore PE64 ? or is this a wrong > assumption/guess ? > > At least mingw seems to be 64 bit, x86_64 seems to indicate a 64 bit > version of x86 called x64... sizeof-long: 4 is a bit worriesome on > delphi this would be 8, not sure what C defines as a long, but a long > long is probably 8, so a long would be 4. size_t is something I don't > understand and don't care about lol. > > So all in all not too terribly clear, but the 64 says enough for me ! > ;) So 64 bit it is most likely ! ;) > > Bye, > Skybuck =D > -- > You received this message because you are subscribed to a topic in the > Google Groups "Git for human beings" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/git-users/-CVnTANaU-k/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > git-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/git-users/5b4d4a0b-3d2b-473f-90f0-7d691d687696n%40googlegroups.com > <https://groups.google.com/d/msgid/git-users/5b4d4a0b-3d2b-473f-90f0-7d691d687696n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/git-users/6b25f9cc-4a8a-1549-1519-5f9c049dc7bc%40iee.email.
