RE: GIT archive invocation with SHA sum

2012-08-17 Thread Manfred Rudigier
Junio C Hamano [mailto:gits...@pobox.com] writes:

>> I have read that this restriction was made for some kind of security
>> reason, and one possibility would be to add a config option on the
>> server repo to turn off this safety check. As we manage the git repos
>> ourselves, this would be perfectly ok for us.
>>
>> What is the current status of this - is there already such a config option?
>
>No, there isn't.
>
>I am no going to say that such a change will never happen, but you may want
>to rethink the way you transfer the source material to your build system,
>especially your build system is rebuilding each and every versions of the
>software.
>
>I can understand that the use of "git archive" can give you a way to start from
>an empty directory and build afresh for every version, but it is stupid to
>_transfer_ the full copy of the source material
>300 times over the network in order to rebuild 300 versions a day, all of which
>will be quite similar to each other (after all, the difference between them 
>will
>be limited by the productivity of your workforce).

Our build system creates an embedded firmware image similar to OpenEmbedded or 
OpenWrt. There all components (including boot loader, kernel, busybox and many 
other third party components) are compiled during nightly builds. Each 
component has its own GIT repository on the server and our build system 
references those components via SHA-1 sums. Many of those components do almost 
NEVER change (we do not update the Linux kernel every day), so most of this 
code must only be fetched once using git archive.

>Fortunately, it is quite simple to fix.  You can maintain a single bare 
>repository
>on the build machine (i.e. the one you currently run "git archive --remote"
>on), and run "git fetch --mirror" against the machine that hosts the repository
>(i.e. the one you currently run "git archive --remote" against) to keep it up 
>to
>date.  That way, you should be able to drastically cut the amount of the
>network traffic from the source repository and the build server.

I know that I could workaround this by mirroring all repositories locally, but 
it is much more convenient to use git archive right away. I agree that once the 
repos are mirrored, there is less traffic on the network, but traffic has not 
been an issue for us so far. Also, a complete mirror of all involved 
repositories will be much larger than just the exported tar archives (e.g. 
Linux kernel repo is 770MB, while the tar.gz is just 96MB).

It was a really nice feature of GIT and it is just too bad that it is no longer 
available.

Regards,
Manfred
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: GIT archive invocation with SHA sum

2012-08-17 Thread Junio C Hamano
Manfred Rudigier  writes:

> we have recently upgraded our Ubuntu server to 12.04, which comes
> with a newer version of GIT. However, we have noticed that git
> archive -remote does not work anymore by specifying the SHA sum of
> a commit.
> ...
> I have read that this restriction was made for some kind of
> security reason, and one possibility would be to add a config
> option on the server repo to turn off this safety check. As we
> manage the git repos ourselves, this would be perfectly ok for us.
>
> What is the current status of this - is there already such a config option?

No, there isn't.

I am no going to say that such a change will never happen, but you
may want to rethink the way you transfer the source material to your
build system, especially your build system is rebuilding each and
every versions of the software.

I can understand that the use of "git archive" can give you a way to
start from an empty directory and build afresh for every version,
but it is stupid to _transfer_ the full copy of the source material
300 times over the network in order to rebuild 300 versions a day,
all of which will be quite similar to each other (after all, the
difference between them will be limited by the productivity of your
workforce).

Fortunately, it is quite simple to fix.  You can maintain a single
bare repository on the build machine (i.e. the one you currently run
"git archive --remote" on), and run "git fetch --mirror" against the
machine that hosts the repository (i.e. the one you currently run
"git archive --remote" against) to keep it up to date.  That way,
you should be able to drastically cut the amount of the network
traffic from the source repository and the build server.

Then, you copy out from that bare repository (i.e. the one you have
on your build server that you "git fetch --mirror" into) using "git
archive" into a fresh directory to build.  Because this invocation
of "git archive" is purely local to the machine, you do not need any
configuration option to bust the "you can only see the branches
available to you" security measure to copy things out.

If your build server is building only the tagged versions for
releases (I doubt that is the case, but I am writing this just for
completeness), "git archive --remove" should be able to reach them,
so you do not need such a configuration option, either.

Hrm?
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html