[gentoo-dev] [PATCH git-r3] Support 'smart' choice of '--depth' for 'git fetch' command.

2013-09-06 Thread Michał Górny
A shallow git clone is a clone of the repository which keeps only
the latest commit in the branch. Aside from many disadvantages
and potential issues, they are very popular as they allow our users to
save both space and bandwidth. Therefore, they are supported natively
and used by default in git-r3.

While it is straightforward to perform the first shallow clone, updating
shallow clones is a more problematic matter. Shallow clones can be
either updated using plain 'git fetch' or 'git fetch --depth 1'.

Using 'git fetch' is the recommended way. In this case, new commits are
fetched alike in regular git repo. However, in a very outdated clone
this may involve fetching a number of objects that are no longer
relevant. Worse than that, if there are no common ancestors between
the clone and upstream (this may happen due to branch switch or force
push upstream), 'git fetch' starts fetching all the commits like with
regular clone.

'git fetch --depth 1' re-creates a shallow clone starting with newest
commit. That is, it not only fetches new commits but also discards all
the old commits. However, either due to protocol limitations or
intentional behavior, 'git fetch --depth 1' can re-fetch objects that
were fetched already, resulting in a major bandwidth loss and slow down.

Due to this issue, git-r3 uses plain 'git fetch' on subsequent updates
to the repository. However, as noted above, this may cause
re-downloading the repository history unintentionally. This patch aims
to solve this.

The patch introduces a 'smart fetch' concept that choses between 'git
fetch' and 'git fetch --depth 1' based on transferred object count.
Since the object count is transferred as progress information by
the remote server and is not really obligatory, the smart fetch function
works the following way:

1. plain 'git fetch' is launched to fetch new commits,

2. the progress output is teed to a sed pipe that looks for the object
count and writes it to a state file,

3. a parallel process waits for the state file to be written. when it is
written, it launches 'git fetch --dry-run --depth 1' to obtain
the respective object count.

4. if the count of (--depth 1) fetch = 0.75*count of plain fetch,
the plain fetch is interrupted and shallow fetch is started instead.

The usual overhead of this is extra connection on client side and commit
counting and possibly some compression on server side (git is killed
via SIGPIPE when sed gets the commit count).

If '--depth 1' seems beneficial, the first fetch is usually killed
before the compression finishes on server side, therefore avoiding waste
of bandwidth on client.

The factor of 0.75 was chosen arbitrarily and may change. It should be
noted that we operate purely on object counts and not sizes. Therefore,
we need to assume that objects fetched by '--depth 1' may be
significantly larger than those by incremental fetch.

What are your thoughts?
---
 gx86/eclass/git-r3.eclass | 98 +--
 1 file changed, 95 insertions(+), 3 deletions(-)

diff --git a/gx86/eclass/git-r3.eclass b/gx86/eclass/git-r3.eclass
index 8585252..a076da7 100644
--- a/gx86/eclass/git-r3.eclass
+++ b/gx86/eclass/git-r3.eclass
@@ -247,6 +247,93 @@ _git-r3_set_submodules() {
done  (echo ${data} | git config -f /dev/fd/0 -l)
 }
 
+# @FUNCTION: _git-r3_smart_fetch
+# @USAGE: git-fetch-args...
+# @DESCRIPTION:
+# Try fetching without '--depth' and switch to '--depth 1' if that
+# will involve less objects fetched.
+_git-r3_smart_fetch() {
+   debug-print-function ${FUNCNAME} $@
+
+   local sed_regexp='.*Counting objects: \([0-9]*\), done\..*'
+
+   # start the main fetch
+   local cmd=( git fetch --progress ${@} )
+   echo ${cmd[@]} 2
+
+   # we copy the output to the 'sed' pipe for parsing. whenever sed finds
+   # the process count, it quits quickly to avoid delays in writing it.
+   # then, we start a dummy 'cat' to keep the pipe alive
+
+   ${cmd[@]} 21 \
+   | tee (
+   sed -n -e /${sed_regexp}/{s/${sed_regexp}/\1/p;q} \
+${T}/git-r3_main.count
+   exec cat /dev/null
+   ) 
+   local main_pid=${!}
+
+   # start the helper process
+   _git-r3_sub_fetch() {
+   # wait for main fetch to get object count; if the server doesn't
+   # output it, we won't even launch the parallel process
+   while [[ ! -s ${T}/git-r3_main.count ]]; do
+   sleep 0.25
+   done
+
+   # ok, let's see if parallel fetch gives us smaller count
+   # --dry-run will prevent it from writing to the local clone
+   # and sed should terminate git with SIGPIPE
+   local sub_count=$(git fetch --progress --dry-run --depth 1 
${@} 21 \
+   | sed -n -e /${sed_regexp}/{s/${sed_regexp}/\1/p;q})
+   local 

Re: [gentoo-dev] Re: [PATCH git-r3] Support 'smart' choice of '--depth' for 'git fetch' command.

2013-09-06 Thread Michał Górny
Dnia 2013-09-06, o godz. 16:55:41
Duncan 1i5t5.dun...@cox.net napisał(a):

 Michał Górny posted on Fri, 06 Sep 2013 14:27:17 +0200 as excerpted:
 
  A shallow git clone is a clone of the repository which keeps only the
  latest commit in the branch. Aside from many disadvantages and potential
  issues, they are very popular as they allow our users to save both space
  and bandwidth. Therefore, they are supported natively and used by
  default in git-r3.
 
 (Without reading thru all that source ATM...) What's the magic 
 incantation for make.conf (or env/cat/pkg files) to force full repos, 
 client side?

The eclass is fully documented, and therefore has a man page.
And the mailing list is not your private support channel. Please learn
to either:

a) write on topic,

b) create a new topic if you believe you have anything useful to say,

c) ask on IRC, forums or even through private mail but *please* do not
spam the mailing list.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature


Re: [gentoo-dev] net-proxy herd is still looking for help (was: Re: [gentoo-dev] net-proxy herd is empty)

2013-09-06 Thread Bertrand Jacquin

On 2013-09-01 16:01, Tom Wijsman wrote:

On Sun, 3 Mar 2013 21:00:19 +0200
Pavlos Ratis daster...@gentoo.org wrote:


On Sun, Mar 3, 2013 at 6:16 PM, Tom Wijsman tom...@gentoo.org wrote:
 On Sun, 03 Mar 2013 17:00:57 +0100
 Pacho Ramos pa...@gentoo.org wrote:

 As wschlich no longer has enough time for that packages, this herd
 is now empty. If you want to help, please join the herd. If nobody
 joins, I will proceed with dropping it and moving its packages
 maintainer-needed letting everybody want the packages they prefer.

 Not joining until others join because I don't want to be the sole
 herd member, but I do want to help out with occasional bumps and
 such if there clearly is a case of lack of manpower.

 I'm also interested in stepping up as a maintainer for
 net-proxy/privoxy.

I am interested in joining the herd. Now we can be 2 members. :)


While I joined under the above premise to help out I have became the
main maintainer but am unable to cover everything this herd does;
Pavlos didn't have much time either to together cover everything.

So, we currently have around 48 bugs open at the moment; there are
also probably some versions bumps waiting as well, as well as some of
the other maintaining tasks that possibly come along.

If you want to help, please join the herd. If nobody joins, we will
likely proceed with dropping it in a month and moving its packages
to maintainer-needed letting everybody want the packages they prefer.


Can be in as a proxied maintainer as net-proxy is something I'm 
interested in as other net- stuff.



Another option is to combine the net herds together into a bigger net
herd, as I have previously suggested; we might need to look into that.




[gentoo-dev] Re: [PATCH git-r3] Support 'smart' choice of '--depth' for 'git fetch' command.

2013-09-06 Thread Duncan
Michał Górny posted on Fri, 06 Sep 2013 14:27:17 +0200 as excerpted:

 A shallow git clone is a clone of the repository which keeps only the
 latest commit in the branch. Aside from many disadvantages and potential
 issues, they are very popular as they allow our users to save both space
 and bandwidth. Therefore, they are supported natively and used by
 default in git-r3.

(Without reading thru all that source ATM...) What's the magic 
incantation for make.conf (or env/cat/pkg files) to force full repos, 
client side?

I do enough git whatchanged checking and occasional bisection to make 
shallow clones something I don't want to even /think/ about.  If I put 
the magic incantation in make.conf now, I won't have to worry about it 
when packages start switching over.

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman




Re: [gentoo-dev] Re: Improve the security of the default profile

2013-09-06 Thread Rick Zero_Chaos Farina
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 09/05/2013 07:06 AM, Mike Frysinger wrote:
 On Thursday 05 September 2013 06:13:28 Agostino Sarubbo wrote:
 during an irc debate, me and other people just noticed that the default
 profile could use more flags to enhance the security.

 An hint is here:
 https://wiki.ubuntu.com/ToolChain/CompilerFlags

 Please argue about what we _don't_ use.
 
 the only thing we don't use by default is SSP.  and we have hardened for that.
 
 fairly certain the other flags we've been using in Gentoo for years.
 -mike
 

Since I don't see this in the profile and I know almost nothing about
how the toolchain works, perhaps you could grace us with how to see the
fact that we've been using in Gentoo for years :-)

Thanks,
Zero
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSKqIYAAoJEKXdFCfdEflKCwkP/Rdlo2rk+g8qyfB9SlsFgoP0
4b+/qkB8WmwNBEURhR7kwF/SJa6kh0BOcorz33e/YO4jayn/yW1ve36HrKOGR52G
56oNWWtRYzsiscObpOVxf+JM9EMm2RVrhfM1Z9FIP8pTFS8gj31fR8caPJssjUGv
xl0wSUahs1+q44xOX+NB+7y47nhrjwfq2OTUHsekMdOWt43MoLp86qEMJKlPFG9a
djEpkshTpE2pZZMQ8jGGASmITcWlHhuipeWkwDCblcxMMCWgFr+CfovEqJXeoz5I
jI4rtpe4QNl7QA+eXY1fygiAiVgx15BYq2SIBC51AluvVgaYRw8ANr8qSUhCakXM
Af49vhzp8/Id3/aytOrllprucPHTICMARKbYhAJyGtfJtKkQ3iGHHOlrIN2ufnrO
gO/EZUqb+NRlHrv845a0HQA3zmYDNBJw5zu6GymV4aMsUcVQE/uSbqAZ7BxuWlV2
LxLvE9pn48WvcvBYp4R36DRQg955D34GKI1VRojgESsyLIgq4Q0wLjarY1fsG4O/
iUZRyXOI5erVCiOGey42kCr19fw1ta35XtKrEQPwWJkb2na1RB7PHbGBdVBlU/Lq
mLAWFSCwocg+wNBuBWcpJlFdLV4eQYxSqyTqeFdxYBv9qxvqqLzkGUxqDy8L4bAT
KglCdavI5Y2UBcFuv4/w
=yb4E
-END PGP SIGNATURE-