From: "Jeff King" <>: Friday, August 22, 2014 12:21 AM
On Thu, Aug 21, 2014 at 06:49:10PM -0400, Jeff King wrote:

The few things I don't anonymize are:

  1. ref prefixes. We see the same distribution of refs/heads vs
     refs/tags, etc.

2. refs/heads/master is left untouched, for convenience (and because it's not really a secret). The implementation is lazy, though, and would leave "refs/heads/master-supersecret", as well. I can tighten
     that if we really want to be careful.

3. gitlinks are left untouched, since sha1s cannot be reversed. This
     could leak some information (if your private repo points to a
     public, I can find out you have it as submodule). I doubt it
     matters, but we can also scramble the sha1s.

Here's a re-roll that addresses the latter two. I don't think any are a big deal, but it's much easier to say "it's handled" than try to figure
out whether and when it's important.

This also includes the documentation update I sent earlier. The
interdiff is a bit noisy, as I also converted the anonymize_mem function to take void pointers (since it doesn't know or care what it's storing,
and this makes storing unsigned chars for sha1s easier).

Just a bit of bikeshedding for future improvements..

The .gitignore is another potential user problem area that may benefit form not being anonymised when problems strike. For example, there's a current problem on the git-users list!topic/git-users/JJFIEsI5HRQ about "git clean vs git status re .gitignore", which would then also beg questions about retaining file extensions/suffixes (.txt, .o, .c, etc).

I've had a similar problem with an over zealous file compare routine where the same too much vs too little was an issue.

One thought is that the user should be able to, as an option, select the number of initial characters retained from filenames, and similarly, the option to retain the file extension, and possibly directory names, such that the full .gitignore still works in most cases, and the sort order works (as far as it goes on number of characters).

All things for future improvers to consider.


-- >8 --
Subject: teach fast-export an --anonymize option

Sometimes users want to report a bug they experience on
their repository, but they are not at liberty to share the
contents of the repository. It would be useful if they could
produce a repository that has a similar shape to its history
and tree, but without leaking any information. This
"anonymized" repository could then be shared with developers
(assuming it still replicates the original problem).

This patch implements an "--anonymize" option to
fast-export, which generates a stream that can recreate such
a repository. Producing a single stream makes it easy for
the caller to verify that they are not leaking any useful
information. You can get an overview of what will be shared
by running a command like:

 git fast-export --anonymize --all |
 perl -pe 's/\d+/X/g' |
 sort -u |

which will show every unique line we generate, modulo any
numbers (each anonymized token is assigned a number, like
"User 0", and we replace it consistently in the output).

In addition to anonymizing, this produces test cases that
are relatively small (compared to the original repository)
and fast to generate (compared to using filter-branch, or
modifying the output of fast-export yourself). Here are
numbers for git.git:

 $ time git fast-export --anonymize --all \
        --tag-of-filtered-object=drop >output
 real    0m2.883s
 user    0m2.828s
 sys     0m0.052s

 $ gzip output
 $ ls -lh output.gz | awk '{print $5}'

Signed-off-by: Jeff King <>
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to
More majordomo info at

Reply via email to