Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Dale, Well ehhh. I feel somewhat ashamed right now When I followed your last instructions I stumbled upon a few hardlinks in the tar-file from my original source. And, as I understood Git treats these hardlinks as separate files. And yes, in the tar-file from Git I found a few exact copy's for several binaries. But now all with a unique inode-number, wheras these files in the original tar-file had the same inode . So, case closed I think. Now I now my target will still function 100% with its filesystem from Git, but several files will be present as 100% duplicates, thus using up more memory. But luckily I've got more than enough flash ! Perhaps In the near future I'm gonna try one of the following : 1) Use pre- post-scripts in Git to prepare the source (undo it from hardlinks, replace them with soft ones) 2) Look for a third party tool that does this for me 3) Take a look at SVN or Mercurial to see if this could be more suitable for this specific goal (although I realy would like to stick to Git !) Many, many thanks for all your help ! Kindest regards, Peter Op woensdag 21 augustus 2013 19:44:53 UTC+2 schreef Dale Worley: From: peter ing...@gmail.com javascript: The files that I suspect are all compiled without stripping (debug symbols present in the file). When I delete all these files the size of my tar-bal from the original filesystem is nearly the same as the one made from the git checkout. Here some things to check: Produce tar files rather than tar.gz files and see if the size difference is still present. The ordering of files in a tar file might affect how efficient the compression is. Get a directory listing of both tar files (-tv) and see if they contain exactly the same files, and with exactly the same lengths. Do a tree file comparison between the files that go into the original tar and the ones that go into the new tar. Dale -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
From: peter ing...@gmail.com When I followed your last instructions I stumbled upon a few hardlinks in the tar-file from my original source. And, as I understood Git treats these hardlinks as separate files. And yes, in the tar-file from Git I found a few exact copy's for several binaries. But now all with a unique inode-number, wheras these files in the original tar-file had the same inode . OK, that explains why when you deleted a bunch of binary files, it almost exactly fixed the space consumption -- those deleted files were the other copies of the hardlinked files. 1) Use pre- post-scripts in Git to prepare the source (undo it from hardlinks, replace them with soft ones) 2) Look for a third party tool that does this for me 3) Take a look at SVN or Mercurial to see if this could be more suitable for this specific goal (although I realy would like to stick to Git !) I doubt that there's any source-control system that handles hardlinks as such. Hardlinks are very Unix-specific and considered somewhat treacherous, so there aren't a lot of situations where they're used. If you really, really want to use hardlinks in your working copies, you'll want to have a script you can run post-checkout that reconstructs the hardlinks correctly. And I suppose have a version-controlled control file that lists what sets of files are alternative links to the same underlying file. Dale -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
If you're willing to switch to soft links, many SCM tools (including git) will handle those correctly. Plus tar will save the link rather than more copies of the file. wor...@alum.mit.edu wrote: From: peter ing...@gmail.com When I followed your last instructions I stumbled upon a few hardlinks in the tar-file from my original source. And, as I understood Git treats these hardlinks as separate files. And yes, in the tar-file from Git I found a few exact copy's for several binaries. But now all with a unique inode-number, wheras these files in the original tar-file had the same inode . OK, that explains why when you deleted a bunch of binary files, it almost exactly fixed the space consumption -- those deleted files were the other copies of the hardlinked files. 1) Use pre- post-scripts in Git to prepare the source (undo it from hardlinks, replace them with soft ones) 2) Look for a third party tool that does this for me 3) Take a look at SVN or Mercurial to see if this could be more suitable for this specific goal (although I realy would like to stick to Git !) I doubt that there's any source-control system that handles hardlinks as such. Hardlinks are very Unix-specific and considered somewhat treacherous, so there aren't a lot of situations where they're used. If you really, really want to use hardlinks in your working copies, you'll want to have a script you can run post-checkout that reconstructs the hardlinks correctly. And I suppose have a version-controlled control file that lists what sets of files are alternative links to the same underlying file. Dale -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Paul, Yep, I already modified my filesystem to soft-links. So now all goes well with git. Peter -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Dale, Well, I've been reading the articles you advised. And I've done some extensive testing with various situations. But till now, all without the result I would like. My files turned out not to be 'contaminated' with large holes, so every experiment with 'sparse options' (while copying, tar-ing git-checkout) yielded the same result. But just now I found out one curious thing ! After a series of trials, isolating (if possible) the files that are the cause of the problem I found one difference between these files and the rest in the filesystem. The files that I suspect are all compiled without stripping (debug symbols present in the file). When I delete all these files the size of my tar-bal from the original filesystem is nearly the same as the one made from the git checkout. So, the quest continues .. Regards, Peter Here's one explanation: http://en.wikipedia.org/wiki/Sparse_files#Sparse_files_in_Unix Also, read the du and cp manual pages, looking for the words holes and sparse, to see situations where this matters. Dale -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Martin, Thanks, but my problem is not the difference between the size of the source in the git-repository and the tar-file made from the same source. Obviously tther will be differences depending on the compressing-algorithm used by tar and git. My problem is the difference between 2 tar-files. One made from the source before committing and pushing, and the second tar, made from the same source after cloneing and checking-out. I would expect them to be the same size (apart from small differences due to .gitignore etc). But an 20% increase is too much ! Regards, Peter Op maandag 19 augustus 2013 22:59:24 UTC+2 schreef Martin Møller Skarbiniks Pedersen: On 19 August 2013 21:10, peter boudewijns ing...@gmail.com javascript: wrote: Hi All, [...] When making a compressed tarball from the files from the repository (after clone/checkout) I get a very much larger tar.gz-file. Size goes up from 16M to 21M (!?) Not so strange. git is very good at compressing. One my of bare git repository is 32M but a tar.gz file of all files excluding the .git directory is 92M. /Martin -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Philip, Tnx for your tip. But I made 100% sure NOT to include the .git directory. And still I get a the difference between 2 tar-files. One made from the source before committing and pushing, and the second tar, made from the same source after cloneing and checking-out. I would expect them to be the same size (apart from small differences due to .gitignore etc). But an 20% increase is too much ! Regards, Peter Op maandag 19 augustus 2013 21:47:15 UTC+2 schreef Philip Oakley: - Original Message - *From:* peter boudewijns javascript: *To:* git-...@googlegroups.com javascript: *Sent:* Monday, August 19, 2013 8:10 PM *Subject:* [git-users] Strange effect when tar-ing a cloned repository Hi All, I've been trying to put my filesystem for a very small busybox-based distro into a git-repository. And with succes. The only strange thing I can not get my head around is the following : When making a compressed tarball from the files from the repository (after clone/checkout) I get a very much larger tar.gz-file. Size goes up from 16M to 21M (!?) Has anyone got a clue ? Thanks PeTer The usual reason is that you 'forgot' that the git repo itself is inside the hidden directory .git at the top level. So you have both your working tree of regular files, and then you have the hidden repo storage - so you have everything twice, and the history as well! Have a look at the 'git archive' command if you want just your your work tree, without the whole repo history. Philip -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
did you already try a cleanup? git gc -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi tombert, No, not yet. At this moment I'm busy makeing a detailed list from all file- directory-sizes before and after git-commit/git-checkout. Thereafter I'll surely try 'git gc' ! Regards, Peter Op dinsdag 20 augustus 2013 09:28:23 UTC+2 schreef tombert: did you already try a cleanup? git gc -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Dale, I've been running some tests to see if I could find the origin of my size-difference. As it turned out, when examining the filesystem's size on the target with 'su', this was about 10M bigger in case of the git-cloned filesystem (I make a jffs2-file to flash the target, thereafter examine the target via a tty). The entire difference could be pinned down in just 1 directory, 'sbin'. So, I made complete listings (ls) from /sbin, both the original as the git-cloned version. And they are exactly the same !? But with su -hs the /sbin directory yields the 10M difference .. ? I do not know enough about the way Linux writes its files, and how it determines the size of the files. But it seems to me the git-cloned files contain empty space that occupies filesystem-space, but is not counted when calculating the actual filesize . Both versions function 100% on the target, so why worry ? But I still would like to know whats going on And yes, I also used git gc (--aggressive), but this yields no improvement at the client side upon cloneing/checking out. To be continued Regards, Peter My guess is that the cloned repository isn't compressed in exactly the same way as the original repository. The first step would be to find out the amount of disk space occupied by the original and the cloned repositories (using du -s) rather than depending on the size of the .tar files. If you want the repository to be small, look into git gc --aggressive. Dale Dale Worley -- Today is: 12.19.16.17.0 9 Ahaw 18 Mak Only 1100 more shopping days until the end of the World. -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
Hi Dale, Thanks for the explanation and tips. I'm gonna study the articles this evening. So perhaps I can find the conclusive answer to my 'problem'. Kindest regards, Peter Here's one explanation: http://en.wikipedia.org/wiki/Sparse_files#Sparse_files_in_Unix Also, read the du and cp manual pages, looking for the words holes and sparse, to see situations where this matters. Dale -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
From: peter boudewijns ing...@gmail.com I've been trying to put my filesystem for a very small busybox-based distro into a git-repository. And with succes. The only strange thing I can not get my head around is the following : When making a compressed tarball from the files from the repository (after clone/checkout) I get a very much larger tar.gz-file. Size goes up from 16M to 21M (!?) Has anyone got a clue ? My guess is that the cloned repository isn't compressed in exactly the same way as the original repository. The first step would be to find out the amount of disk space occupied by the original and the cloned repositories (using du -s) rather than depending on the size of the .tar files. If you want the repository to be small, look into git gc --aggressive. Dale Dale Worley -- Today is: 12.19.16.17.0 9 Ahaw 18 Mak Only 1100 more shopping days until the end of the World. -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
- Original Message - From: peter boudewijns To: git-users@googlegroups.com Sent: Monday, August 19, 2013 8:10 PM Subject: [git-users] Strange effect when tar-ing a cloned repository Hi All, I've been trying to put my filesystem for a very small busybox-based distro into a git-repository. And with succes. The only strange thing I can not get my head around is the following : When making a compressed tarball from the files from the repository (after clone/checkout) I get a very much larger tar.gz-file. Size goes up from 16M to 21M (!?) Has anyone got a clue ? Thanks PeTer The usual reason is that you 'forgot' that the git repo itself is inside the hidden directory .git at the top level. So you have both your working tree of regular files, and then you have the hidden repo storage - so you have everything twice, and the history as well! Have a look at the 'git archive' command if you want just your your work tree, without the whole repo history. Philip -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [git-users] Strange effect when tar-ing a cloned repository
On 19 August 2013 21:10, peter boudewijns ing...@gmail.com wrote: Hi All, [...] When making a compressed tarball from the files from the repository (after clone/checkout) I get a very much larger tar.gz-file. Size goes up from 16M to 21M (!?) Not so strange. git is very good at compressing. One my of bare git repository is 32M but a tar.gz file of all files excluding the .git directory is 92M. /Martin -- You received this message because you are subscribed to the Google Groups Git for human beings group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
