Re: Bloom filters for have/want negotiation

2015-09-11 Thread Shawn Pearce
On Fri, Sep 11, 2015 at 2:13 PM, Michael Haggerty wrote: > I have been thinking about Wilhelm Bierbaum's talk at the last GitMerge > conference [1] in which he describes a scheme for using Bloom filters to > make the initial reference advertisement less expensive. ... > But it got me thinking abou

Re: Bloom filters for have/want negotiation

2015-09-12 Thread Shawn Pearce
On Sat, Sep 12, 2015 at 12:01 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> The worst case is due to a bug in the negotiation. With nothing >> common, the client just goes on forever until it reaches roots >> (something is wrong with MAX_IN_VAIN). We saw 56,318

Re: [RFC/PATCH v1] Add Travis CI support

2015-09-26 Thread Shawn Pearce
On Fri, Sep 25, 2015 at 11:52 AM, Jeff King wrote: > On Fri, Sep 25, 2015 at 11:29:31AM -0700, Junio C Hamano wrote: > >> > So I wonder if it would be >> > helpful to have a microformat that the client would use to look at this. >> > E.g., it would fetch the cert tree, then confirm that the curre

Re: [PATCH] index-pack: avoid excessive re-reading of pack directory

2015-06-09 Thread Shawn Pearce
On Fri, Jun 5, 2015 at 5:29 AM, Jeff King wrote: > > However, some code paths make a large number of > has_sha1_file checks which are _not_ expected to return 1. > The collision test in index-pack.c is such a case. On a > local system, this can cause a performance slowdown of > around 5%. But on a

Re: [PATCH] index-pack: avoid excessive re-reading of pack directory

2015-06-10 Thread Shawn Pearce
On Wed, Jun 10, 2015 at 7:00 AM, Jeff King wrote: > On Tue, Jun 09, 2015 at 08:46:24PM -0700, Shawn Pearce wrote: > >> > This patch introduces a "quick" flag to has_sha1_file which >> > callers can use when they would prefer high performance at >> >

git push keeps writing after server failure

2015-06-12 Thread Shawn Pearce
I did something stupid like trying to push a copy of WebKit[1] into my GitHub account. This is ~5.2 GiB of data, which GitHub prefers not to accept. Ok ... $ git push --all g...@github.com:spearce/wk.git Counting objects: 2752427, done. Delta compression using up to 12 threads. Compressing objects

Re: RFC/Pull Request: Refs db backend

2015-06-23 Thread Shawn Pearce
On Tue, Jun 23, 2015 at 4:47 AM, Jeff King wrote: > > One of the problems we've had with large-ref repos is that the reflog > storage is quite inefficient. Yup. We ran into this with Gerrit Code Review years ago. The refs/changes/... namespace created by Gerrit Code Review is 1 ref per snapshot p

Re: [PATCH 3/7] pack-protocol.txt: Mark all LFs in push-cert as required

2015-07-03 Thread Shawn Pearce
On Fri, Jul 3, 2015 at 11:07 AM, Jeff King wrote: > I wondered briefly whether this impacted the keepalives we added to > `upload-pack` in 05e9515; those are implemented as 0-byte data packets, > which we send during the potentially long counting/delta-compression > phase before we send out pack d

Re: [PATCH 3/7] pack-protocol.txt: Mark all LFs in push-cert as required

2015-07-06 Thread Shawn Pearce
On Wed, Jul 1, 2015 at 1:49 PM, Junio C Hamano wrote: > Dave Borowitz writes: > >>> I am moderately negative about this; wouldn't it make the end result >>> cleaner to fix the implementation? >> >> I'm not sure I understand your suggestion. Are you saying, you would >> prefer to make LFs optional

send-pack silently tries to recreate funny remote refs

2015-07-06 Thread Shawn Pearce
Today I learned that git push will silently try to recreate a remote funny ref if you push to it. "Hypothetically" ... lets say I have a reimplementation of Git called JGit. Lets say its on a server somewhere on the network. Lets say its a bit more liberal in what it accepts... $ git push origin

Re: Git Smart HTTP with HTTP/2.0

2015-07-11 Thread Shawn Pearce
On Sat, Jul 11, 2015 at 12:00 AM, Ilari Liusvaara wrote: > On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote: >> As we known, HTTP/2.0 has been released. All Git-Smart-HTTP are currently >> implemented using HTTP/1.1. > > Nit: It is HTTP/2. > >> Frequently used Git developers often feel

Re: Git Smart HTTP with HTTP/2.0

2015-07-11 Thread Shawn Pearce
On Sat, Jul 11, 2015 at 11:26 AM, Ilari Liusvaara wrote: > On Sat, Jul 11, 2015 at 10:23:09AM -0700, Shawn Pearce wrote: >> >> > Websockets over HTTP/2 (a.k.a. "websockets2") has not been defined yet. >> > With Websockets(1), it would probably already be p

Re: [PATCH 2/4] receive-pack: implement advertising and receiving push options

2016-07-10 Thread Shawn Pearce
On Fri, Jul 8, 2016 at 5:31 PM, Stefan Beller wrote: > + > + /* NEEDSWORK: expose the limitations to be configurable. */ > + int max_options = 32; > + > + /* > +* NEEDSWORK: expose the limitations to be configurable; > +* Once the limit can be lifted, include a wa

Re: [PATCH 2/4] receive-pack: implement advertising and receiving push options

2016-07-11 Thread Shawn Pearce
On Sun, Jul 10, 2016 at 11:05 AM, Stefan Beller wrote: > On Sun, Jul 10, 2016 at 10:06 AM, Shawn Pearce wrote: >> On Fri, Jul 8, 2016 at 5:31 PM, Stefan Beller wrote: >>> + >>> + /* NEEDSWORK: expose the limitations to be configurable. */ >&

Re: [PATCH] contrib/persistent-https: update ldflags syntax for Go 1.7+

2016-07-18 Thread Shawn Pearce
On Mon, Jul 18, 2016 at 9:32 PM, Parker Moore wrote: >> The label does not even identify the version of the source in any way, so I >> am not sure how people are depending on that feature anyway ;-) > > Would it be a better solution simply to remove this build flag? > Alternatively, if Git wished

Re: [RFC] Add support for downloading blobs on demand

2017-01-13 Thread Shawn Pearce
On Fri, Jan 13, 2017 at 7:52 AM, Ben Peart wrote: > > Goal > > > To be able to better handle repos with many files that any individual > developer doesn’t need it would be nice if clone/fetch only brought down > those files that were actually needed. > > To enable that, we are proposing addin

Re: Git clonebundles

2017-02-04 Thread Shawn Pearce
On Mon, Jan 30, 2017 at 11:00 PM, Stefan Saasen wrote: > > Bitbucket recently added support for Mercurial’s clonebundle extension > (http://gregoryszorc.com/blog/2015/10/22/cloning-improvements-in-mercurial-3.6/). > Mercurial’s clone bundles allow the Mercurial client to seed a repository > using

Re: [PATCH 2/2] connect: know that zero-ID is not a ref

2016-09-02 Thread Shawn Pearce
On Fri, Sep 2, 2016 at 10:15 AM, Jonathan Tan wrote: > > + if (is_null_oid(&old_oid)) { > + if (strcmp(name, "capabilities^{}")) Its not the zero ID that is special, its the "capabilities^{}" name that is special when its the first entry in the stream. In the w

Re: [PATCH 2/2] connect: know that zero-ID is not a ref

2016-09-02 Thread Shawn Pearce
On Fri, Sep 2, 2016 at 12:56 PM, Stefan Beller wrote: > On Fri, Sep 2, 2016 at 12:39 PM, Shawn Pearce wrote: >> On Fri, Sep 2, 2016 at 10:15 AM, Jonathan Tan >> wrote: >>> >>> + if (is_null_oid(&old_oid)) { >>> +

Re: [PATCH 2/2] connect: know that zero-ID is not a ref

2016-09-02 Thread Shawn Pearce
On Fri, Sep 2, 2016 at 1:13 PM, Jeff King wrote: > > Hmm. So since this is backwards-compatible, I'm not overly concerned > with changing the client. But I wonder if you considered that the > documentation is wrong, and that JGit should stop sending the extra > capabilities line? No, JGit needs t

Re: [PATCH 13/16] init: allow alternate backends to be set for new repos

2016-01-06 Thread Shawn Pearce
On Wed, Jan 6, 2016 at 4:52 AM, Duy Nguyen wrote: > On Wed, Dec 23, 2015 at 8:34 PM, Michael Haggerty > wrote: >> On 12/03/2015 01:35 AM, David Turner wrote: >>> diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt >>> index 8174d27..9ea6753 100644 >>> --- a/Documentation/git-ini

RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

2016-02-10 Thread Shawn Pearce
Sorry, no code, only words today. Previously people have proposed a few different options for resumable clone, like "clone.bundle" (currently used by Android as a hack) or "skip start of regenerated pack". Here is another option. We could implement resumable clone by making a bit of a hybrid of th

Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

2016-02-10 Thread Shawn Pearce
On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce wrote: > > ... Thoughts? Several of us at $DAY_JOB talked about this more today and thought a variation makes more sense: 1. Clients attempting clone ask for /info/refs?service=git-upload-pack like they do today. 2. Servers that support res

Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

2016-02-11 Thread Shawn Pearce
On Wed, Feb 10, 2016 at 7:43 PM, Junio C Hamano wrote: > Jonathan Nieder writes: > >> I really like this design. I'm tempted to implement it (since it >> lacks a bunch of the downsides of clone.bundle). > > Just to see people are not stepping on each others toe, implementing > slightly different

[PATCH] remote-curl: Include curl_errorstr on SSL setup failures

2016-02-13 Thread Shawn Pearce
such as when the SSL setup fails. Don't include HTTP 0 in the message. Signed-off-by: Shawn Pearce --- remote-curl.c | 16 ++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/remote-curl.c b/remote-curl.c index c704857..f611432 100644 --- a/remote-curl.c +++ b/rem

Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

2016-02-13 Thread Shawn Pearce
On Wed, Feb 10, 2016 at 1:49 PM, Jeff King wrote: > On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote: > >> On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce wrote: >> > >> > ... Thoughts? >> >> Several of us at $DAY_JOB talked about this more t

Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

2016-02-14 Thread Shawn Pearce
On Sun, Feb 14, 2016 at 9:05 AM, Jeff King wrote: > On Sat, Feb 13, 2016 at 06:14:31PM -0800, Shawn Pearce wrote: > >> > And with "resumable=", the client does not have to hit the server >> > to do a redirect; it can go straight to the final URL, saving a >&g

Re: [PATCH] remote-curl: Include curl_errorstr on SSL setup failures

2016-02-14 Thread Shawn Pearce
On Sun, Feb 14, 2016 at 8:50 AM, Jeff King wrote: > On Sat, Feb 13, 2016 at 05:39:34PM -0800, Shawn Pearce wrote: > >> For curl error 35 (CURLE_SSL_CONNECT_ERROR) users need the >> additional text stored in CURLOPT_ERRORBUFFER to debug why >> the connection did not star

Re: Is it possible to give specific PR number

2016-03-19 Thread Shawn Pearce
On Wed, Mar 16, 2016 at 8:20 AM, Igor Korot wrote: > Hi, > Is it possible to tell Git to have a PR with a specific number? Git does not have PRs. Are you referring to a GitHub Pull request? If so you should ask GitHub support. GitHub is a commercial entity that is separate from the Git open sou

Re: [PATCH] http: Support sending custom HTTP headers

2016-04-25 Thread Shawn Pearce
On Mon, Apr 25, 2016 at 6:13 AM, Johannes Schindelin wrote: > To make communication for `git fetch`, `git ls-remote` and friends extra > secure, we introduce a way to send custom HTTP headers with all > requests. Hmm. Its not Apr 1 2016. So I guess you are serious. :) > This allows us, for examp

RefTree: Alternate ref backend

2015-12-17 Thread Shawn Pearce
I started playing around with the idea of storing references directly in Git. Exploiting the GITLINK tree entry, we can associate a name to any SHA-1. By storing all references in a single tree, atomic transactions are possible. Its a simple compare-and-swap of a single 40 byte SHA-1. This of cour

Re: RefTree: Alternate ref backend

2015-12-17 Thread Shawn Pearce
On Thu, Dec 17, 2015 at 1:57 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> For example, recent git.git has a structure like this: >> >> $ git ls-tree -r refs/txn/committed >> 12 blob 22e42fc826b437033ca444e09368f53a0b16

Re: RefTree: Alternate ref backend

2015-12-17 Thread Shawn Pearce
On Thu, Dec 17, 2015 at 2:10 PM, Jeff King wrote: > On Thu, Dec 17, 2015 at 01:02:50PM -0800, Shawn Pearce wrote: > >> I started playing around with the idea of storing references directly >> in Git. Exploiting the GITLINK tree entry, we can associate a name to >> any SH

Re: RefTree: Alternate ref backend

2015-12-22 Thread Shawn Pearce
On Tue, Dec 22, 2015 at 7:41 AM, Michael Haggerty wrote: > On 12/17/2015 10:02 PM, Shawn Pearce wrote: >> I started playing around with the idea of storing references directly >> in Git. Exploiting the GITLINK tree entry, we can associate a name to >> any SHA-1. >> >

Re: RefTree: Alternate ref backend

2015-12-22 Thread Shawn Pearce
On Tue, Dec 22, 2015 at 9:17 AM, Michael Haggerty wrote: > > etc. But we store branches into the main "refs/remotes/origin/" > namespace, leaving no reserved space for the remote "HEAD" (not to > mention other namespaces that might appear on the remote, such as > "refs/changes/*", "refs/pull/*", a

Re: RefTree: Alternate ref backend

2015-12-22 Thread Shawn Pearce
On Tue, Dec 22, 2015 at 11:09 AM, Junio C Hamano wrote: > Shawn Pearce writes: > >>> But really, aside from slightly helping >>> disambiguate references from paths in the command line, what is it good >>> for? >> >> Nothing really; today refs/ prefix

Re: git objects

2015-12-24 Thread Shawn Pearce
On Thu, Dec 24, 2015 at 2:17 PM, Thiago Farina wrote: > > When ever I make a commit (assume I'm changing a single file) and do a > 'git push origin master', git says 'Counting objects: 6, done.' > > Does git makes 6 objects everytime? What are those objects? 1 commit object; 1 blob object for the

Re: [PATCH] fetch-pack: always allow fetching of literal SHA1s

2017-05-09 Thread Shawn Pearce
On Tue, May 9, 2017 at 3:16 PM, Jeff King wrote: > On Tue, May 09, 2017 at 11:20:42AM -0700, Jonathan Tan wrote: > >> fetch-pack, when fetching a literal SHA-1 from a server that is not >> configured with uploadpack.allowtipsha1inwant (or similar), always >> returns an error message of the form "S

Re: [PATCH] fetch-pack: always allow fetching of literal SHA1s

2017-05-09 Thread Shawn Pearce
On Tue, May 9, 2017 at 9:33 PM, Jeff King wrote: > On Tue, May 09, 2017 at 09:22:11PM -0700, Shawn Pearce wrote: > >> > Hmm. That makes sense generally, as the request should succeed. But it >> > seems like we're creating a client that will sometimes succeed and

reftable: new ref storage format

2017-07-12 Thread Shawn Pearce
We've been having scaling problems with insane number of references (>866k), so I started thinking a lot about improving ref storage. I've written a simple approach, and implemented it in JGit. Performance is promising: - 62M packed-refs compresses to 27M - 42.3 usec lookup You can read a re

Re: reftable: new ref storage format

2017-07-13 Thread Shawn Pearce
On Thu, Jul 13, 2017 at 12:32 PM, Jeff King wrote: > On Wed, Jul 12, 2017 at 05:17:58PM -0700, Shawn Pearce wrote: > >> ### Problem statement >> >> Some repositories contain a lot of references (e.g. android at 866k, >> rails at 31k). The existing packed-refs for

Re: reftable: new ref storage format

2017-07-13 Thread Shawn Pearce
On Thu, Jul 13, 2017 at 1:35 PM, Jeff King wrote: > On Thu, Jul 13, 2017 at 12:56:54PM -0700, Stefan Beller wrote: > > I agree that a full binary search of a reftable is harder because of the > prefix compression (it may still be possible by scanning backwards, but > I think there are ambiguities

Re: reftable: new ref storage format

2017-07-14 Thread Shawn Pearce
On Fri, Jul 14, 2017 at 7:27 AM, Dave Borowitz wrote: > On Thu, Jul 13, 2017 at 8:11 PM, Shawn Pearce wrote: >> In another (Gerrit Code Review), we disable reflogs for >> the insane refs/changes/ namespace, as nearly every reference is >> created once, and never modified.

Re: reftable: new ref storage format

2017-07-15 Thread Shawn Pearce
On Fri, Jul 14, 2017 at 1:08 PM, Jeff King wrote: > On Thu, Jul 13, 2017 at 05:11:52PM -0700, Shawn Pearce wrote: > > I think the "stack" implementation is what makes me most uncomfortable > with this proposal. Atomicity with filesystem operations and especially > readdir

Re: reftable: new ref storage format

2017-07-16 Thread Shawn Pearce
On Sun, Jul 16, 2017 at 10:33 AM, Michael Haggerty wrote: > Thanks for your reftable proposal. Thanks for your time reading the proposal. I really was looking forward to your insights on this project. > It would solve a lot of scalability > problems that we currently have, and do it in a way tha

Re: reftable: new ref storage format

2017-07-16 Thread Shawn Pearce
On Sun, Jul 16, 2017 at 12:43 PM, Shawn Pearce wrote: > On Sun, Jul 16, 2017 at 10:33 AM, Michael Haggerty > wrote: > >> * The tuning parameter number_of_restarts currently trades off space >> (for the full refnames and the restart_offsets) against the need to >> read

Re: reftable: new ref storage format

2017-07-16 Thread Shawn Pearce
On Sun, Jul 16, 2017 at 2:13 PM, Dave Borowitz wrote: > On Sun, Jul 16, 2017 at 3:43 PM, Shawn Pearce wrote: >> True... but... in my "android" example repository we have 866,456 live >> refs. A block size of 64k needs only 443 blocks, and a 12k index, to >> get

reftable [v2]: new ref storage format

2017-07-17 Thread Shawn Pearce
This is an updated draft after discussion on list with Peff, Michael Haggerty, and Dave Borowitz. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Biggest changes from the first proposal are: - reflog is now i

Re: SHA1 collisions found

2017-02-28 Thread Shawn Pearce
On Tue, Feb 28, 2017 at 11:34 AM, Linus Torvalds wrote: > On Tue, Feb 28, 2017 at 11:07 AM, Junio C Hamano wrote: >> >> In a way similar to 8415558f55 ("sha1dc: avoid c99 >> declaration-after-statement", 2017-02-24), we would want this on >> top. > > There's a few other simplifications that could

Re: RFC v3: Another proposed hash function transition plan

2017-03-09 Thread Shawn Pearce
On Mon, Mar 6, 2017 at 4:17 PM, Jonathan Nieder wrote: > Linus Torvalds wrote: >> On Fri, Mar 3, 2017 at 5:12 PM, Jonathan Nieder wrote: > >>> This document is still in flux but I thought it best to send it out >>> early to start getting feedback. >> >> This actually looks very reasonable if you

Re: [PATCH] http.postbuffer: make a size_t

2017-03-30 Thread Shawn Pearce
On Thu, Mar 30, 2017 at 1:29 PM, David Turner wrote: > Unfortunately, in order to push some large repos, the http postbuffer > must sometimes exceed two gigabytes. On a 64-bit system, this is OK: > we just malloc a larger buffer. I'm slightly curious what server you are pushing to that needs the

Re: reftable [v2]: new ref storage format

2017-07-17 Thread Shawn Pearce
On Mon, Jul 17, 2017 at 11:53 AM, Stefan Beller wrote: > On Mon, Jul 17, 2017 at 8:01 AM, Shawn Pearce wrote: > >> A ref block is written as: >> >> 'r' >> uint24 ( block_len ) >> ref_record* >> uint32( restart_offset )

Re: reftable [v2]: new ref storage format

2017-07-18 Thread Shawn Pearce
On Mon, Jul 17, 2017 at 12:51 PM, Junio C Hamano wrote: > Shawn Pearce writes: >> You can read a rendered version of this here: >> https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md > > Just a few comments. > >> A variable nu

Re: reftable [v3]: new ref storage format

2017-07-22 Thread Shawn Pearce
3rd iteration of the reftable storage format. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Significant changes from v2: - efficient lookup by SHA-1 for allow-tip-sha1-in-want. - type 0x4 for FETCH_HEAD, MER

Re: reftable [v2]: new ref storage format

2017-07-23 Thread Shawn Pearce
My apologies for not responding to this piece of feedback earlier. On Wed, Jul 19, 2017 at 7:02 AM, Ævar Arnfjörð Bjarmason wrote: > On Tue, Jul 18 2017, Shawn Pearce jotted: >> On Mon, Jul 17, 2017 at 12:51 PM, Junio C Hamano wrote: >>> Shawn Pearce writes: >>>>

Re: reftable: new ref storage format

2017-07-23 Thread Shawn Pearce
+git@vger.kernel.org. I originally sent the below reply privately by mistake. On Mon, Jul 17, 2017 at 6:43 PM, Michael Haggerty wrote: > On Sun, Jul 16, 2017 at 12:43 PM, Shawn Pearce wrote: >> On Sun, Jul 16, 2017 at 10:33 AM, Michael Haggerty >> wrote: > > On second

Re: reftable: new ref storage format

2017-07-23 Thread Shawn Pearce
On Sun, Jul 23, 2017 at 3:56 PM, Shawn Pearce wrote: > On Mon, Jul 17, 2017 at 6:43 PM, Michael Haggerty > wrote: >> On Sun, Jul 16, 2017 at 12:43 PM, Shawn Pearce wrote: >>> On Sun, Jul 16, 2017 at 10:33 AM, Michael Haggerty >>> wrote: > >> * What woul

Re: reftable [v3]: new ref storage format

2017-07-24 Thread Shawn Pearce
On Mon, Jul 24, 2017 at 3:22 PM, Stefan Beller wrote: > On Sat, Jul 22, 2017 at 11:29 AM, Shawn Pearce wrote: >> 3rd iteration of the reftable storage format. >> >> You can read a rendered version of this here: >> https://googlers.googlesource.com/sop/jgit/+/refta

Re: reftable [v3]: new ref storage format

2017-07-28 Thread Shawn Pearce
On Thu, Jul 27, 2017 at 7:28 AM, Michael Haggerty wrote: > On Sat, Jul 22, 2017 at 11:29 AM, Shawn Pearce wrote: >> 3rd iteration of the reftable storage format. >> >> [...] >> ### Objectives >> >> - Near constant time lookup for any single reference, ev

Re: reftable [v3]: new ref storage format

2017-07-28 Thread Shawn Pearce
On Fri, Jul 28, 2017 at 7:18 PM, Michael Haggerty wrote: > On Fri, Jul 28, 2017 at 3:12 PM, Shawn Pearce wrote: >> I'm with you this far, and like the {min,max}_update_index in the >> header. I'm concerned about update_index in 32 bits. At some point you >> ne

Re: reftable [v4]: new ref storage format

2017-07-30 Thread Shawn Pearce
4th iteration of the reftable storage format. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Significant changes from v3: - Incorporated Michael Haggerty's update_index concept for reflog. - Explicitly docume

Re: reftable [v4]: new ref storage format

2017-07-31 Thread Shawn Pearce
On Mon, Jul 31, 2017 at 12:01 PM, Stefan Beller wrote: > On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >> 4th iteration of the reftable storage format. >> >> You can read a rendered version of this here: >> https://googlers.googlesource.com/sop/jgit/+/refta

Re: reftable [v4]: new ref storage format

2017-07-31 Thread Shawn Pearce
On Mon, Jul 31, 2017 at 12:42 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> ### Peeling >> >> References in a reftable are always peeled. > > This hopefully means "a record for an annotated (or signed) tag > records both the tag object and the object

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Tue, Aug 1, 2017 at 6:54 AM, Dave Borowitz wrote: > On Sun, Jul 30, 2017 at 11:51 PM, Shawn Pearce wrote: >> - Ref-like files (FETCH_HEAD, MERGE_HEAD) also use type 0x3. > >> - Combine reflog storage with ref storage for small transactions. >> - Separate reflog s

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Mon, Jul 31, 2017 at 4:43 PM, Shawn Pearce wrote: > On Mon, Jul 31, 2017 at 12:42 PM, Junio C Hamano wrote: >> >> As a block cannot be longer than 16MB, allocating uint32 to a >> restart offset may be a bit overkill. I do not know if it is worth >> attempting to pa

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Mon, Jul 31, 2017 at 11:41 PM, Michael Haggerty wrote: > On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >> 4th iteration of the reftable storage format. >> [...] > > Before we commit to Shawn's reftable proposal, I wanted to explore > what a contrasting d

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Mon, Jul 31, 2017 at 11:41 PM, Michael Haggerty wrote: > On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >> 4th iteration of the reftable storage format. >> [...] > > Before we commit to Shawn's reftable proposal, I wanted to explore > what a contrasting d

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Tue, Aug 1, 2017 at 4:27 PM, Shawn Pearce wrote: > On Mon, Jul 31, 2017 at 11:41 PM, Michael Haggerty > wrote: >> On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >>> 4th iteration of the reftable storage format. >>> [...] >> >> Before we commit

Re: reftable [v4]: new ref storage format

2017-08-01 Thread Shawn Pearce
On Tue, Aug 1, 2017 at 6:51 PM, Michael Haggerty wrote: > On Tue, Aug 1, 2017 at 4:27 PM, Shawn Pearce wrote: >> On Mon, Jul 31, 2017 at 11:41 PM, Michael Haggerty >> wrote: >>> On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >>>> 4th iter

Re: reftable [v4]: new ref storage format

2017-08-02 Thread Shawn Pearce
On Wed, Aug 2, 2017 at 2:28 AM, Jeff King wrote: > On Tue, Aug 01, 2017 at 07:38:37PM -0700, Shawn Pearce wrote: > >> > OBJS blocks can also be >> > unbounded in size if very many references point at the same object, >> > thought that is perhaps only a theoret

Re: reftable [v4]: new ref storage format

2017-08-02 Thread Shawn Pearce
On Wed, Aug 2, 2017 at 6:50 PM, Junio C Hamano wrote: > Junio C Hamano writes: > >> I like the general idea, what the file format can represent and how >> it does so, but I am a bit uneasy about how well this "stacked" part >> would work for desktop clients. > > Two more random things before I fo

Re: reftable [v4]: new ref storage format

2017-08-03 Thread Shawn Pearce
On Wed, Aug 2, 2017 at 1:28 PM, Jeff King wrote: > On Wed, Aug 02, 2017 at 12:50:39PM -0700, Junio C Hamano wrote: > >> With the traditional "packed-refs plus loose" layout, no matter how >> many times a handful of selected busy refs are updated during the >> day, you'd need to open at most two fi

Re: reftable [v4]: new ref storage format

2017-08-03 Thread Shawn Pearce
On Thu, Aug 3, 2017 at 11:38 AM, Michael Haggerty wrote: > On Tue, Aug 1, 2017 at 7:38 PM, Shawn Pearce wrote: >> On Tue, Aug 1, 2017 at 6:51 PM, Michael Haggerty >> wrote: >>> On Tue, Aug 1, 2017 at 4:27 PM, Shawn Pearce wrote: >>>> On Mon, Jul 31

Re: reftable [v4]: new ref storage format

2017-08-03 Thread Shawn Pearce
On Thu, Aug 3, 2017 at 3:48 PM, Michael Haggerty wrote: > I've revised the blockless reftable proposal to address some feedback: I've been thinking more about your blockless proposal. I experimentally modified my reftable implementation to omit padding between blocks, bringing it a tiny bit clos

Re: reftable [v4]: new ref storage format

2017-08-05 Thread Shawn Pearce
On Tue, Aug 1, 2017 at 6:51 PM, Michael Haggerty wrote: > On Tue, Aug 1, 2017 at 4:27 PM, Shawn Pearce wrote: >> On Mon, Jul 31, 2017 at 11:41 PM, Michael Haggerty >> wrote: >>> On Sun, Jul 30, 2017 at 8:51 PM, Shawn Pearce wrote: >>>> 4th iter

Re: reftable [v5]: new ref storage format

2017-08-05 Thread Shawn Pearce
5th iteration of the reftable storage format. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Significant changes from v4: - Supported Michael Haggerty's multi-level index. - Restart table now appears at start

Re: reftable [v5]: new ref storage format

2017-08-06 Thread Shawn Pearce
On Sun, Aug 6, 2017 at 9:56 AM, Ævar Arnfjörð Bjarmason wrote: > On Sun, Aug 06 2017, Shawn Pearce jotted: > >> 5th iteration of the reftable storage format. > > I haven't kept up with all of the discussion, sorry if these comments > repeat something that's alread

Re: reftable [v6]: new ref storage format

2017-08-06 Thread Shawn Pearce
6th iteration of the reftable storage format. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Changes from v5: - extensions.refStorage = reftable is used to select this format. - Log records can be explicitly

Re: reftable [v5]: new ref storage format

2017-08-07 Thread Shawn Pearce
On Sun, Aug 6, 2017 at 4:37 PM, Ben Alex wrote: > Just on the LmdbJava specific pieces: > > On Mon, Aug 7, 2017 at 8:56 AM, Shawn Pearce wrote: >> >> Looks pretty complete. Its a Java wrapper around the C implementation >> of LMDB, which may be sufficient for

Re: reftable [v6]: new ref storage format

2017-08-07 Thread Shawn Pearce
On Mon, Aug 7, 2017 at 11:27 AM, Stefan Beller wrote: > On Sun, Aug 6, 2017 at 6:47 PM, Shawn Pearce wrote: >> 6th iteration of the reftable storage format. >> >> You can read a rendered version of this here: >> https://googlers.googlesource.com/sop/jgit/+/refta

Re: reftable [v5]: new ref storage format

2017-08-08 Thread Shawn Pearce
On Tue, Aug 8, 2017 at 12:52 AM, Jeff King wrote: > On Mon, Aug 07, 2017 at 03:40:48PM +, David Turner wrote: > >> > -Original Message----- >> > From: Shawn Pearce [mailto:spea...@spearce.org] >> > In git-core, I'm worried about the caveats relat

Re: reftable [v6]: new ref storage format

2017-08-08 Thread Shawn Pearce
On Tue, Aug 8, 2017 at 12:01 PM, Junio C Hamano wrote: > I notice that you raised the location of restart table within a > block in this iteration (or maybe it happened in v5). > > This forces you to hold all contents in core before the first byte > is written out. You start from the first entry

Re: reftable [v6]: new ref storage format

2017-08-08 Thread Shawn Pearce
On Tue, Aug 8, 2017 at 12:25 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> For `log_type = 0x4..0x7` the `log_chained` section is used instead to >> compress information that already appeared in a prior log record. The >> `log_chained` always includes `old

Re: reftable [v6]: new ref storage format

2017-08-08 Thread Shawn Pearce
On Tue, Aug 8, 2017 at 4:34 PM, Junio C Hamano wrote: > Shawn Pearce writes: > >> Given that the index can now also be multi-level, I don't expect to >> see a 2G index. A 2G index forces the reader to load the entire 2G to >> take advantage of the restart table. It m

Re: reftable [v6]: new ref storage format

2017-08-15 Thread Shawn Pearce
On Mon, Aug 14, 2017 at 5:13 AM, Michael Haggerty wrote: > On 08/07/2017 03:47 AM, Shawn Pearce wrote: >> 6th iteration of the reftable storage format. > > Thanks! > >> Changes from v5: >> - extensions.refStorage = reftable is used to select this format. >>

Re: reftable [v7]: new ref storage format

2017-08-15 Thread Shawn Pearce
7th iteration of the reftable storage format. You can read a rendered version of this here: https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md Changes from v6: - Blocks are variable sized, and alignment is optional. - ref index is required on variable sized

Re: reftable [v7]: new ref storage format

2017-08-16 Thread Shawn Pearce
On Tue, Aug 15, 2017 at 11:15 PM, Stefan Beller wrote: > On Tue, Aug 15, 2017 at 7:48 PM, Shawn Pearce wrote: >> 7th iteration of the reftable storage format. >> >> You can read a rendered version of this here: >> https://googlers.googlesource.com/sop/jgit/+/refta

Re: reftable [v7]: new ref storage format

2017-08-17 Thread Shawn Pearce
On Tue, Aug 15, 2017 at 7:48 PM, Shawn Pearce wrote: > 7th iteration of the reftable storage format. > > You can read a rendered version of this here: > https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md FYI, JGit has adopted this v7 draft as

Re: Git GSoC 2014

2014-02-15 Thread Shawn Pearce
On Sat, Feb 15, 2014 at 4:17 AM, Thomas Rast wrote: > David Kastrup writes: > >> Thomas Rast writes: >> >>> Motivation: I believe that migrating to libgit2 is the better approach, >>> medium term, than rewriting everything ourselves to be nice, clean and >>> thread-safe. I took a shot a while a

Re: [PATCH 0/6] inotify support

2014-02-19 Thread Shawn Pearce
On Sun, Jan 12, 2014 at 3:03 AM, Nguyễn Thái Ngọc Duy wrote: > It's been 37 weeks since Robert Zeh's attempt to bring inotify support > to Git [1] and unless I missed some mails, no updates since. So here's > another attempt with my preferred approach (can't help it, playing > with your own ideas

Re: Git in GSoC 2014

2014-02-26 Thread Shawn Pearce
gt; > Yeah, I'd agree. Want to write it up? > >> I wonder if anybody from JGit would also be interested on mentoring >> for the equivalent task (index v4 on JGit). I've CC'ed Shawn Pearce. > > A project that added to both libgit2 and JGit would be cool, but I do

Re: [PATCH] repack: add `repack.honorpackkeep` config var

2014-03-03 Thread Shawn Pearce
On Fri, Feb 28, 2014 at 10:05 PM, Jeff King wrote: > On Fri, Feb 28, 2014 at 10:09:08AM -0700, Nasser Grainawi wrote: > >> > Exactly. The two features (bitmaps and .keep) are not compatible with >> > each other, so you have to prioritize one. If you are using static .keep >> > files, you might wan

Re: [RFC/WIP] Pluggable reference backends

2014-03-10 Thread Shawn Pearce
On Mon, Mar 10, 2014 at 4:00 AM, Michael Haggerty wrote: > I have started working on pluggable ref backends. In this email I > would like to share my plans and solicit feedback. Yay! JGit already has pluggable ref backends, so it is good to see this starting in git-core. FWIW the Gerrit Code R

Re: [RFC/WIP] Pluggable reference backends

2014-03-10 Thread Shawn Pearce
On Mon, Mar 10, 2014 at 2:07 PM, Michael Haggerty wrote: > On 03/10/2014 04:52 PM, Jeff King wrote: >> On Mon, Mar 10, 2014 at 07:30:45AM -0700, Shawn Pearce wrote: >> >>>> * Store references in a SQLite database, to get correct transaction >>>> hand

Re: egit vs. git behaviour (was: [RFC/WIP] Pluggable reference backends)

2014-03-12 Thread Shawn Pearce
On Wed, Mar 12, 2014 at 3:26 AM, Andreas Krey wrote: > On Mon, 10 Mar 2014 19:39:00 +0000, Shawn Pearce wrote: >> Yes, this was my real concern. Eclipse users using EGit expect EGit to >> be compatible with git-core at the filesystem level so they can do >> something in

No progress from push when using bitmaps

2014-03-12 Thread Shawn Pearce
Today I tried pushing a copy of linux.git from a client that had bitmaps into a JGit server. The client stalled for a long time with no progress, because it reused the existing pack. No progress appeared while it was sending the existing file on the wire: $ git push git://localhost/linux.git mas

Re: No progress from push when using bitmaps

2014-03-13 Thread Shawn Pearce
On Thu, Mar 13, 2014 at 2:26 PM, Jeff King wrote: > On Wed, Mar 12, 2014 at 05:21:21PM -0700, Shawn Pearce wrote: > >> Today I tried pushing a copy of linux.git from a client that had >> bitmaps into a JGit server. The client stalled for a long time with no >> progres

Re: [PATCH v3 0/8] Hiding refs

2014-03-14 Thread Shawn Pearce
On Fri, Mar 14, 2014 at 5:37 AM, Duy Nguyen wrote: > On Wed, Mar 12, 2014 at 3:36 AM, Jeff King wrote: >> If the client is limited to setting a few flags, then something like >> http can get away with: >> >> GET >> foo.git/info/refs?service=git-upload-pack&advertise-symrefs&refspec=refs/heads/

Re: [PATCH v3 0/8] Hiding refs

2014-03-14 Thread Shawn Pearce
On Fri, Mar 14, 2014 at 4:30 PM, Duy Nguyen wrote: > On Fri, Mar 14, 2014 at 11:45 PM, Shawn Pearce wrote: >> >> You missed the SSH case. It doesn't have this slot to hide the data into. > > Right now we run this for ssh case: "ssh git-upload-pack > ".

Re: [PATCH 18/18] signed push: final protocol update

2014-08-22 Thread Shawn Pearce
On Fri, Aug 22, 2014 at 10:59 AM, Junio C Hamano wrote: > Junio C Hamano writes: > >> There are a few gotchas I can certainly use help on, especially from >> a smart-http expert ;-). >> >> * "pushed-to " will identify the site and the repository, so >>you cannot MITM my push to an experiment

  1   2   3   >