Re: [git-users] Remove files and folders permanently from commit history and pack files
> From: cemico
> thank you for your very detailed information.
> But i can't get it to work.
Try the following script. It's not efficient, but if the man pages
are correct, if you give it a blob hash (the full hash), it will show
what file names it has in what commits. Then for each commit, it will
show what refs have that commit as an ancestor. And at the bottom, it
lists all the refs that (indirectly) point to the blob in question.
You have to clear out the reflogs yourself.
But that should give you enough information to make the blob
un-referenced and then "git gc --prune=now --aggressive" should be
able to remove it.
I suppose it could be improved by searching through the reflogs and
seeing which of them point to commits that contain the blob.
Dale
--
#! /bin/bash
# Find a given blob, given its hash.
BLOB="${1:?First argument is the blob hash.}"
# Temporary file name base.
T=${TMPDIR:-/tmp}/${0##*/}.$$
# Make a copy of stdout, since we will be redirecting stdout within the script.
exec 3>&1
# Find the commits in which the blob exists.
# List all commits.
git rev-list --all |
while read COMMIT
do
# COMMIT_ADDED records whether this commit hash has been stored or not.
COMMIT_ADDED=no
# List all files within the commit.
git ls-tree --full-tree $COMMIT |
while read MODE TYPE HASH NAME
do
# If this file matches the blob, report it.
if [ "$HASH" = "$BLOB" ]
then
echo >&3 Commit $COMMIT Name "$NAME"
if [ "$COMMIT_ADDED" = no ]
then
# Send the commit onward.
echo $COMMIT
COMMIT_ADDED=yes
fi
# Beware that the same blob may appear multiple times, so we
# have to continue searching through this tree.
fi
done
done >$T.commits
# Find the refs that lead to the commits.
# List all the refs.
git show-ref |
while read COMMIT REF
do
# REF_ADDED records whether this ref has been stored or not.
REF_ADDED=no
# List all the commits in the ref that contain the blob.
git rev-list $REF |
grep -f $T.commits |
while read COMMIT
do
echo >&3 Commit $COMMIT Ref "$REF"
if [ "$REF_ADDED" = no ]
then
# Send the ref onward.
echo "$REF"
REF_ADDED=yes
fi
done
done >$T.refs
# Write out the refs.
sed <$T.refs -e 's/^/Ref /'
--
--
You received this message because you are subscribed to the Google Groups "Git
for human beings" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to git-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
Hi Dale, thank you for your very detailed information. But i can't get it to work. That's an odd "feature" of git in my opinion. I got a lokal git repository and i'm working alone on it. It should be possible and way way easier to completely remove accidentally committed files and folders from the history and pack files. Now i can only delete the whole repo and start all over again. Loosing my complete commit history. And in the future i can only pray not to commit any false files and folders or i can start all over again ;( Best regards Am Mittwoch, 16. Juli 2014 16:30:43 UTC+2 schrieb Dale Worley: > > > From: cemico > > > > > How do i remove let's say a "concept_art" folder from all commits (i > have > > about 50 commits by now) and from the pack file? > > > > I tried many things now. e.g: > > - > http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > > - http://rtyley.github.io/bfg-repo-cleaner/ > > - filter-branch with "rm" > > - git gc with prune=now and aggresive > > > > and the thing i thought which must really work: > > > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > > First, be aware that what you want to do is intrinsically difficult. > Git is oriented toward allowing you to manipulate commits, etc. But > removing objects from storage is not a task that it is designed to > make easy. I have had a little experience and I think I can provide > some useful guidance. > > You already know how to recreate all your commits with certain > directories or files removed: > > git filter-branch --index-filter $SCRIPT HEAD > > You also know how to remove a stored object if it is no longer > referenced: > > git gc --aggressive --prune=now > > What is difficult is ensuring that you have removed all references to > the object. In general, this involves removing records of past states > of the HEAD, and there are a lot of those records. You have to remove > the record of the previous version of the branch you are manipulating. > (You have to do this before git-filter-branch, so that > git-filter-branch actually runs, and after it, to remove the value > that git-filter-branch put there.) Your command looks to me like it > should suffice: > > rm -rf .git/refs/original/ > > Where I think you have a problem is with the reflog. In the > repository on which I use git-filter-branch, I have configured > > git config core.logallrefupdates false > > To ensure that no reflog entries are generated. In your case, you use > > git reflog expire --all > > But as far as I can tell from the manual page, that means "examine all > reflogs to find expired entries". You probably need to add > "--expire=now" so that all entries are considered expired. > > You can probably examine .git/logs to see if there are any reflog > entries, and what their values are. My suspicion is that one value > points to a commit that points to the files you want to excise. > > Dale > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
> From: cemico > > How do i remove let's say a "concept_art" folder from all commits (i have > about 50 commits by now) and from the pack file? > > I tried many things now. e.g: > - http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > - http://rtyley.github.io/bfg-repo-cleaner/ > - filter-branch with "rm" > - git gc with prune=now and aggresive > > and the thing i thought which must really work: > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch First, be aware that what you want to do is intrinsically difficult. Git is oriented toward allowing you to manipulate commits, etc. But removing objects from storage is not a task that it is designed to make easy. I have had a little experience and I think I can provide some useful guidance. You already know how to recreate all your commits with certain directories or files removed: git filter-branch --index-filter $SCRIPT HEAD You also know how to remove a stored object if it is no longer referenced: git gc --aggressive --prune=now What is difficult is ensuring that you have removed all references to the object. In general, this involves removing records of past states of the HEAD, and there are a lot of those records. You have to remove the record of the previous version of the branch you are manipulating. (You have to do this before git-filter-branch, so that git-filter-branch actually runs, and after it, to remove the value that git-filter-branch put there.) Your command looks to me like it should suffice: rm -rf .git/refs/original/ Where I think you have a problem is with the reflog. In the repository on which I use git-filter-branch, I have configured git config core.logallrefupdates false To ensure that no reflog entries are generated. In your case, you use git reflog expire --all But as far as I can tell from the manual page, that means "examine all reflogs to find expired entries". You probably need to add "--expire=now" so that all entries are considered expired. You can probably examine .git/logs to see if there are any reflog entries, and what their values are. My suspicion is that one value points to a commit that points to the files you want to excise. Dale -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
I thank you for the infos and i'll try that repack and unpack thing. I need to keep the pack file small, because if i have, like i do, 50 Projects on my server and every small wordpress project has 2-3 Gigabyte disc space, my server runs out of space soon. In the future i have to .gitignore my concept and gallery folders, but for the present projects, i need a way to "shrink the pack". Am Mittwoch, 16. Juli 2014 10:00:52 UTC+2 schrieb Magnus Therning: > > On Wed, Jul 16, 2014 at 12:49:20AM -0700, cemico wrote: > > As far as i understood, the pack file get's generated through the "git > gc" > > garbage collection command. > > But i tried really much by now and didn't find the right way. > > That's the only point i'm missing: > > How to rewrite the pack file, ignoring every file and folder that's not > in > > the commit history. > > Ah, that's the pack file you are talking about :) > > Why do you care about it at all? It's internal to git and AFAIK there > is no need to ever worry about it since git does the best it can to > keep it small. If you really need then it sounds like you should > explore `git repack` and `git unpack-objects`. I haven't played > around with that myself yet, but hopefully someone else will be able > to help if you have further questions. > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.orgjabber: mag...@therning.org > > twitter: magthe http://therning.org/magnus > > Any fool can write code that a computer can understand. Good programmers > write code that humans can understand. > -- Martin Fowler > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
On Wed, Jul 16, 2014 at 12:49:20AM -0700, cemico wrote: > As far as i understood, the pack file get's generated through the "git gc" > garbage collection command. > But i tried really much by now and didn't find the right way. > That's the only point i'm missing: > How to rewrite the pack file, ignoring every file and folder that's not in > the commit history. Ah, that's the pack file you are talking about :) Why do you care about it at all? It's internal to git and AFAIK there is no need to ever worry about it since git does the best it can to keep it small. If you really need then it sounds like you should explore `git repack` and `git unpack-objects`. I haven't played around with that myself yet, but hopefully someone else will be able to help if you have further questions. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus Any fool can write code that a computer can understand. Good programmers write code that humans can understand. -- Martin Fowler pgpfrwspFuSph.pgp Description: PGP signature
Re: [git-users] Remove files and folders permanently from commit history and pack files
As far as i understood, the pack file get's generated through the "git gc" garbage collection command. But i tried really much by now and didn't find the right way. That's the only point i'm missing: How to rewrite the pack file, ignoring every file and folder that's not in the commit history. Am Mittwoch, 16. Juli 2014 09:46:43 UTC+2 schrieb Magnus Therning: > > On Wed, Jul 16, 2014 at 12:40:41AM -0700, cemico wrote: > > After running the commands with filter-branch etc., my folder (let's say > > "concept") i want to remove from all commits, really gets deleted from > all > > commits. > > even the physical folder gets deleted (good!) > > but the "pack" file stays the exact same way too big 1,3 GB > > I want to remove folders and files > > - from the complete commit history > > - physically > > - and most importantly from the pack file > > > > Best regards and thank you for your quick answers > > Well, AFAICS you don't do anything at all with any "pack" file (I'm > not even sure what such a file is). Surely you'd have to modify it > using some command in order to shrink to a size you're more happy > with, no? > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.orgjabber: mag...@therning.org > > twitter: magthe http://therning.org/magnus > > Code as if whoever maintains your program is a violent psychopath who > knows > where you live. > -- Anonymous > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
On Wed, Jul 16, 2014 at 12:40:41AM -0700, cemico wrote: > After running the commands with filter-branch etc., my folder (let's say > "concept") i want to remove from all commits, really gets deleted from all > commits. > even the physical folder gets deleted (good!) > but the "pack" file stays the exact same way too big 1,3 GB > I want to remove folders and files > - from the complete commit history > - physically > - and most importantly from the pack file > > Best regards and thank you for your quick answers Well, AFAICS you don't do anything at all with any "pack" file (I'm not even sure what such a file is). Surely you'd have to modify it using some command in order to shrink to a size you're more happy with, no? /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus Code as if whoever maintains your program is a violent psychopath who knows where you live. -- Anonymous pgpSO95qnNNkY.pgp Description: PGP signature
Re: [git-users] Remove files and folders permanently from commit history and pack files
After running the commands with filter-branch etc., my folder (let's say "concept") i want to remove from all commits, really gets deleted from all commits. even the physical folder gets deleted (good!) but the "pack" file stays the exact same way too big 1,3 GB I want to remove folders and files - from the complete commit history - physically - and most importantly from the pack file Best regards and thank you for your quick answers Am Mittwoch, 16. Juli 2014 09:34:52 UTC+2 schrieb Magnus Therning: > > On Wed, Jul 16, 2014 at 12:11:49AM -0700, cemico wrote: > > 1. Sorry, i don't really understand what the question is. What does > >rebasing and marking with "e" mean? Is it possible to achieve > >what i want with that rebasing and "e"ing? > > Using `git rebase -i` you can do changes to history manually. That > doesn't really scale to even a small repo like your though. For > instance, to modify the latest changeset run `git rebase -i HEAD^`, > then read the instructions as the editor pops up and then replace > 'pick' with 'e' (or 'edit') for the changeset. Save the file and exit > the editor. When your done making your changes you just commit and > then run `git rebase --continue`. Of course you do this on a backup > of your repo :) > > (The stuff above is from memory so please read the full help for `git > rebase` before blaming me for any misinformation. ;) > > > 2. We did this: > > > > git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch > DIRECTORY/' --prune-empty --tag-name-filter cat -- --all > > > > and afterwards: > > > > rm -rf .git/refs/original/ && git reflog expire --all && git gc > --aggressive --prune > > > > That did not help, so we tried: > > > http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > > http://rtyley.github.io/bfg-repo-cleaner/ > > > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > > > > with no effort > > I've done exactly that sort of thing in the past, using the > same sources for information, so what is it that does happen when > running the command above? Does `git filter-branch` complete > successfully? > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.orgjabber: mag...@therning.org > > twitter: magthe http://therning.org/magnus > > Heuristic is an algorithm in a clown suit. It’s less predictable, it’s > more > fun, and it comes without a 30-day, money-back guarantee. > -- Steve McConnell, Code Complete > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
On Wed, Jul 16, 2014 at 09:34:32AM +0200, Magnus Therning wrote: > On Wed, Jul 16, 2014 at 12:11:49AM -0700, cemico wrote: > > 2. We did this: > > > > git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch > > DIRECTORY/' --prune-empty --tag-name-filter cat -- --all > > > > and afterwards: > > > > rm -rf .git/refs/original/ && git reflog expire --all && git gc > > --aggressive --prune > > > > That did not help, so we tried: > > http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > > http://rtyley.github.io/bfg-repo-cleaner/ > > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > > > with no effort > > I've done exactly that sort of thing in the past, using the > same sources for information, so what is it that does happen when > running the command above? Does `git filter-branch` complete > successfully? It's also worth having a look at `--tree-filter`, it's slower than `--index-filter` but I find it easier to work with. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus Goto labels should be left-aligned in all caps and should include the programmer's name, home phone number, and credit card number. -- Abdul Nizar pgpLRWTK6AEei.pgp Description: PGP signature
Re: [git-users] Remove files and folders permanently from commit history and pack files
On Wed, Jul 16, 2014 at 12:11:49AM -0700, cemico wrote: > 1. Sorry, i don't really understand what the question is. What does >rebasing and marking with "e" mean? Is it possible to achieve >what i want with that rebasing and "e"ing? Using `git rebase -i` you can do changes to history manually. That doesn't really scale to even a small repo like your though. For instance, to modify the latest changeset run `git rebase -i HEAD^`, then read the instructions as the editor pops up and then replace 'pick' with 'e' (or 'edit') for the changeset. Save the file and exit the editor. When your done making your changes you just commit and then run `git rebase --continue`. Of course you do this on a backup of your repo :) (The stuff above is from memory so please read the full help for `git rebase` before blaming me for any misinformation. ;) > 2. We did this: > > git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch > DIRECTORY/' --prune-empty --tag-name-filter cat -- --all > > and afterwards: > > rm -rf .git/refs/original/ && git reflog expire --all && git gc --aggressive > --prune > > That did not help, so we tried: > http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > http://rtyley.github.io/bfg-repo-cleaner/ > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > with no effort I've done exactly that sort of thing in the past, using the same sources for information, so what is it that does happen when running the command above? Does `git filter-branch` complete successfully? /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus Heuristic is an algorithm in a clown suit. It’s less predictable, it’s more fun, and it comes without a 30-day, money-back guarantee. -- Steve McConnell, Code Complete pgptS2TY16SWH.pgp Description: PGP signature
Re: [git-users] Remove files and folders permanently from commit history and pack files
1. Sorry, i don't really understand what the question is. What does rebasing and marking with "e" mean? Is it possible to achieve what i want with that rebasing and "e"ing? 2. We did this: git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch DIRECTORY/' --prune-empty --tag-name-filter cat -- --all and afterwards: rm -rf .git/refs/original/ && git reflog expire --all && git gc --aggressive --prune That did not help, so we tried: http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ http://rtyley.github.io/bfg-repo-cleaner/ http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch with no effort best regards Am Mittwoch, 16. Juli 2014 06:37:37 UTC+2 schrieb Magnus Therning: > > On Tue, Jul 15, 2014 at 07:46:52AM -0700, cemico wrote: > > Hello Tower team, > > > > I have a git project in which all files are of about 700 MB size. > > In the .git folder lies a "pack" file which is 1.13 GB (Gigabyte) > > > > I accidently commited files and folders, which don't have to be under > > version control. > > > > How do i remove let's say a "concept_art" folder from all commits (i > have > > about 50 commits by now) and from the pack file? > > > > I tried many things now. e.g: > > - > http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > > - http://rtyley.github.io/bfg-repo-cleaner/ > > - filter-branch with "rm" > > - git gc with prune=now and aggresive > > > > and the thing i thought which must really work: > > > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > > > > None of the above work as expected or better: None work like i need it > to > > work. > > Some delete all commits (the most do that), some don't delete the files > > from all commits, all(!) don't make the "pack" file smaller after > deleting > > all files and folders i don't want. > > > > What i need is: > > - Exclude files and folders from my repo > > - Delete all references to theses files and folders from all commits > > - But don't delete any old commit > > - Rewrite the pack file, so that not any of the excluded files and > folders > > are in it > > > > What is the way to achieve that? > > `filter-branch` is the tool to alter history in a repo, but it's not > easy to wield. It'd be easier to help you if you answer these > questions: > > 1. Say that you were to rewrite a changeset manually using `git rebase -i` >and marking it 'e', what shell commands would you use to modify the >changeset? > > 2. Exactly what arguments did you pass to `git filter-branch`? > > /M > > -- > Magnus Therning OpenPGP: 0xAB4DFBA4 > email: mag...@therning.orgjabber: mag...@therning.org > > twitter: magthe http://therning.org/magnus > > Most software today is very much like an Egyptian pyramid with > millions of bricks piled on top of each other, with no structural > integrity, but just done by brute force and thousands of slaves. > -- Alan Kay > -- You received this message because you are subscribed to the Google Groups "Git for human beings" group. To unsubscribe from this group and stop receiving emails from it, send an email to git-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [git-users] Remove files and folders permanently from commit history and pack files
On Tue, Jul 15, 2014 at 07:46:52AM -0700, cemico wrote: > Hello Tower team, > > I have a git project in which all files are of about 700 MB size. > In the .git folder lies a "pack" file which is 1.13 GB (Gigabyte) > > I accidently commited files and folders, which don't have to be under > version control. > > How do i remove let's say a "concept_art" folder from all commits (i have > about 50 commits by now) and from the pack file? > > I tried many things now. e.g: > - http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ > - http://rtyley.github.io/bfg-repo-cleaner/ > - filter-branch with "rm" > - git gc with prune=now and aggresive > > and the thing i thought which must really work: > http://git-scm.com/book/en/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch > > None of the above work as expected or better: None work like i need it to > work. > Some delete all commits (the most do that), some don't delete the files > from all commits, all(!) don't make the "pack" file smaller after deleting > all files and folders i don't want. > > What i need is: > - Exclude files and folders from my repo > - Delete all references to theses files and folders from all commits > - But don't delete any old commit > - Rewrite the pack file, so that not any of the excluded files and folders > are in it > > What is the way to achieve that? `filter-branch` is the tool to alter history in a repo, but it's not easy to wield. It'd be easier to help you if you answer these questions: 1. Say that you were to rewrite a changeset manually using `git rebase -i` and marking it 'e', what shell commands would you use to modify the changeset? 2. Exactly what arguments did you pass to `git filter-branch`? /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: mag...@therning.org jabber: mag...@therning.org twitter: magthe http://therning.org/magnus Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves. -- Alan Kay pgpzlV6VzP_f4.pgp Description: PGP signature
