From: Josh Steadmon
Currently the client advertises that it supports the wire protocol
version set in the protocol.version config. However, not all services
support the same set of protocol versions. When connecting to
git-receive-pack, the client automatically downgrades to v0
From: Josh Steadmon
This is an alternate approach to the previous series. We add a registry
of supported wire protocol versions that individual commands can use to
declare supported versions before contacting a server. The client will
then advertise all supported versions, while the server
From: Josh Steadmon
V2 of this series pulls the compiler flags out of the Makefile, to be
provided by the user depending on the combination of compiler and
fuzzing engine in use. This also makes it more compatible with
OSS-Fuzz's build process.
Josh Steadmon (2):
fuzz: Add basic fuzz testing
From: Josh Steadmon
Breaks the majority of check_packed_git_idx() into a separate function,
load_idx(). The latter function operates on arbitrary buffers, which
makes it suitable as a fuzzing test target.
Signed-off-by: Josh Steadmon
---
.gitignore | 1 +
Makefile| 1 +
fuzz
From: Josh Steadmon
fuzz-pack-headers.c provides a fuzzing entry point compatible with
libFuzzer (and possibly other fuzzing engines).
Signed-off-by: Josh Steadmon
---
.gitignore | 2 ++
Makefile| 32
fuzz-pack-headers.c | 14
s
archives from remotes to always be generated as TAR files, regardless of
the actual filename (unless an explicit --format is provided).
This patch fixes that behavior; archival format is determined properly
from the output filename, even when --remote is used.
Signed-off-by: Josh Steadmon
Helpe
to always be generated as TAR files, regardless of
the actual filename (unless an explicit --format is provided).
This patch fixes that behavior; archival format is determined properly
from the output filename, even when --remote is used.
Signed-off-by: Josh Steadmon
Helped-by: Jeff King
---
to always be generated as TAR files, regardless of
the actual filename (unless an explicit --format is provided).
This patch fixes that behavior; archival format is determined properly
from the output filename, even when --remote is used.
Signed-off-by: Josh Steadmon
Helped-by: Jeff King
---
When passing both --remote and --output to git-archive, initialize the
archivers before attempting to determine the format from the output
filename. Without initialization, the format cannot be determined.
Signed-off-by: Josh Steadmon
---
builtin/archive.c | 5 -
1 file changed, 4
∫ file ~/good.zip
/home/steadmon/good.zip: Zip archive data, at least v1.0 to extract
∫ git archive --output ~/bad.zip --remote=. HEAD
∫ file ~/bad.zip
/home/steadmon/bad.zip: POSIX tar archive
(apply patch and build)
∫ ./bin-wrappers/git archive --output ~/fixed.zip --remote=. HEAD
∫ file
to always be generated as TAR files, regardless of
the actual filename (unless an explicit --format is provided).
This patch fixes that behavior; archival format is determined properly
from the output filename, even when --remote is used.
Signed-off-by: Josh Steadmon
Helped-by: Jeff King
---
This is a minor iteration on v2, to change an error message to a BUG.
Josh Steadmon (1):
protocol: advertise multiple supported versions
builtin/archive.c | 3 ++
builtin/clone.c| 4 ++
builtin/fetch-pack.c | 4 ++
builtin/fetch.c| 5 ++
builtin/ls-remote.c
-by: Josh Steadmon
---
builtin/archive.c | 3 ++
builtin/clone.c| 4 ++
builtin/fetch-pack.c | 4 ++
builtin/fetch.c| 5 ++
builtin/ls-remote.c| 5 ++
builtin/pull.c | 5 ++
builtin/push.c | 4 ++
builtin/send-pack.c| 3 ++
connect.c
OSS-Fuzz requires C++-specific flags to link fuzzers. Passing these in
CFLAGS causes lots of build warnings. Using separate CXXFLAGS avoids
this.
Signed-off-by: Josh Steadmon
---
Makefile | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index
-off-by: Josh Steadmon
---
remote-curl.c | 4 +++-
t/lib-httpd.sh | 1 +
t/lib-httpd/apache.conf | 4
t/lib-httpd/error-smart-http.sh | 3 +++
t/t5551-http-fetch-smart.sh | 5 +
5 files changed, 16 insertions(+), 1 deletion(-)
create mode
HTTP servers return 400 if you send headers before the GET request.
Signed-off-by: Josh Steadmon
---
Documentation/technical/protocol-v2.txt | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/Documentation/technical/protocol-v2.txt
b/Documentation/technical/protocol-v2.txt
# Supporting HTTP remotes in "git archive"
We would like to allow remote archiving from HTTP servers. There are a
few possible implementations to be discussed:
## Shallow clone to temporary repo
This approach builds on existing endpoints. Clients will connect to the
remote server's
From: Brandon Williams
Update the config documentation to note the value `2` as an acceptable
value for the protocol.version config.
Signed-off-by: Brandon Williams
Signed-off-by: Josh Steadmon
---
Documentation/config.txt | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation
On 2018.09.27 11:20, Stefan Beller wrote:
> On Wed, Sep 26, 2018 at 6:25 PM Josh Steadmon wrote:
> >
> > This is the second version of my series to add a new protocol v2 command
> > for archiving, with support for HTTP(S).
> >
> > NEEDSWORK: a server built
On 2018.09.13 09:47, Junio C Hamano wrote:
> Josh Steadmon writes:
>
> > Signed-off-by: Josh Steadmon
> > ---
> > builtin/archive.c | 8 +++-
> > http-backend.c | 10 +-
> > transport-helper.c | 5 +++--
> > 3 files changed, 19
On 2018.09.27 15:20, Junio C Hamano wrote:
> Jonathan Nieder writes:
>
> > 1. Clients sending version=2 when they do not, in fact, speak protocol
> > v2 for a service is a (serious) bug. (Separately from this
> > series) we should fix it.
> >
> > 2. That bug is already in the wild,
On 2018.10.12 16:32, Jonathan Nieder wrote:
> Josh Steadmon wrote:
>
> > For now, I'm going to try adding an --allowed_versions flag for the
> > remote helpers, but if anyone has a better idea, let me know.
>
> I forgot to mention: the stateless-connect remote help
On 2018.10.10 11:14, Junio C Hamano wrote:
> Josh Steadmon writes:
>
> > +FUZZ_OBJS += fuzz-pack-headers.o
> > +
> > +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS))
> > +
> > ...
> > +### Fuzz testing
> > +#
> > +.PHONY: fuzz-clean fuz
On 2018.10.12 23:53, Junio C Hamano wrote:
> * js/remote-archive-v2 (2018-09-28) 4 commits
> (merged to 'next' on 2018-10-12 at 5f34377f60)
> + archive: allow archive over HTTP(S) with proto v2
> + archive: implement protocol v2 archive command
> + archive: use packet_reader for
On 2018.10.12 15:30, Stefan Beller wrote:
> On Thu, Oct 11, 2018 at 6:02 PM wrote:
> >
> > From: Josh Steadmon
> >
> > Currently the client advertises that it supports the wire protocol
> > version set in the protocol.version config. However, not all services
>
e:
> >
> > ∫ git version
> > git version 2.19.1.568.g152ad8e336-goog
> > ∫ cd ~/src/git
> > ∫ git archive --output ~/good.zip HEAD
> > ∫ file ~/good.zip
> > /home/steadmon/good.zip: Zip archive data, at least v1.0 to extract
> > ∫ git archive --ou
On 2018.10.19 19:59, Jeff King wrote:
> On Fri, Oct 19, 2018 at 04:19:28PM -0700, stead...@google.com wrote:
>
> > diff --git a/builtin/archive.c b/builtin/archive.c
> > index e74f675390..dd3283a247 100644
> > --- a/builtin/archive.c
> > +++ b/builtin/archive.c
> > @@ -45,7 +45,10 @@ static int
es, regardless of
> > the actual filename (unless an explicit --format is provided).
> >
> > This patch fixes that behavior; archival format is determined properly
> > from the output filename, even when --remote is used.
> >
> > Signed-off-by: Josh Steadmon
> > Helped
On 2018.10.22 20:06, Jeff King wrote:
> On Mon, Oct 22, 2018 at 04:51:27PM -0700, Josh Steadmon wrote:
>
> > > > +test_expect_success GZIP 'git archive with --output and --remote uses
> > > > expected format' '
> > > > + git archive --output=d5.t
On 2018.10.24 10:50, Johannes Schindelin wrote:
> Maybe a lot of explanation, but definitely a good one. The explanation and
> the patch look good to me.
>
> Thanks,
> Dscho
Agreed, as a newbie I definitely appreciate detailed explanations. Looks
good to me as well.
Reviewed-by: Josh Steadmon
On 2018.10.23 13:09, Junio C Hamano wrote:
> stead...@google.com writes:
>
> > diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh
> > index 2a97b27b0a..cfd5ca492f 100755
> > --- a/t/t5000-tar-tree.sh
> > +++ b/t/t5000-tar-tree.sh
> > @@ -39,6 +39,8 @@ test_lazy_prereq TAR_NEEDS_PAX_FALLBACK '
On 2018.11.13 12:12, Junio C Hamano wrote:
> stead...@google.com writes:
>
> > OSS-Fuzz requires C++-specific flags to link fuzzers. Passing these in
> > CFLAGS causes lots of build warnings. Using separate CXXFLAGS avoids
> > this.
>
> We are not a C++ shop, so allow me to show ignorance about
On 2018.11.13 23:30, Junio C Hamano wrote:
> stead...@google.com writes:
>
> > When a smart HTTP server sends an error message via pkt-line,
> > remote-curl will fail to detect the error (which usually results in
> > incorrectly falling back to dumb-HTTP mode).
>
> OK, that is a valid thing to
On 2018.11.13 13:01, Junio C Hamano wrote:
> stead...@google.com writes:
>
> > Currently the client advertises that it supports the wire protocol
> > version set in the protocol.version config. However, not all services
> > support the same set of protocol versions. When connecting to
> >
On 2018.11.13 19:28, SZEDER Gábor wrote:
> On Mon, Nov 12, 2018 at 01:49:05PM -0800, stead...@google.com wrote:
>
> > diff --git a/protocol.c b/protocol.c
> > index 5e636785d1..54d2ab991b 100644
> > --- a/protocol.c
> > +++ b/protocol.c
>
> > +void
Fix several bugs identified in v3, clarify commit message, and clean up
extern keyword in protocol.h.
Josh Steadmon (1):
protocol: advertise multiple supported versions
builtin/archive.c| 3 +
builtin/clone.c | 4 ++
builtin/fetch-pack.c | 4 ++
builtin/fetch.c
of registered versions. Servers
select the first allowed version from this advertisement.
While we're at it, remove unnecessary externs from function declarations
in protocol.h.
Signed-off-by: Josh Steadmon
---
builtin/archive.c| 3 +
builtin/clone.c | 4 ++
builtin/fetch-pack.c
t;
> > This patch adds a check in discover_refs() for server-side error
> > messages, as well as a test case for this issue.
> >
> > Signed-off-by: Josh Steadmon
> > ---
>
> Forgot to mention one procedural comment.
>
> As you can see in the To: line o
On 2018.11.13 09:26, Jeff King wrote:
> On Mon, Nov 12, 2018 at 02:44:56PM -0800, stead...@google.com wrote:
>
> > When a smart HTTP server sends an error message via pkt-line,
> > remote-curl will fail to detect the error (which usually results in
> > incorrectly falling back to dumb-HTTP mode).
On 2018.11.14 02:00, Jeff King wrote:
> On Tue, Nov 13, 2018 at 07:49:15PM -0500, Jeff King wrote:
>
> > Yes, the packet_read_line_buf() interface will both advance the buf
> > pointer and decrement the length. So if we want to "peek", we have to
> > do so with a copy (there's a peek function if
gets.
> As the process around the pending patches is not yet fully explored,
> leave that out.
>
> Based-on-work-by: SZEDER Gábor
> Signed-off-by: Stefan Beller
> Signed-off-by: Junio C Hamano
> ---
>
> I dialed back on the workflow, as we may want to explore it first
> before writing it down.
>
> Stefan
Looks good to me.
Reviewed-by: Josh Steadmon
>
> Add a test case to make sure that this use case does not regress again.
>
> Signed-off-by: Johannes Schindelin
Looks good to me.
Reviewed-by: Josh Steadmon
OSS-Fuzz requires C++-specific flags to link fuzzers. Passing these in
CFLAGS causes lots of build warnings. Using separate FUZZ_CXXFLAGS
avoids this.
Signed-off-by: Josh Steadmon
---
Since there's nothing else using CXXFLAGS, let's just make it explicit
that these apply to the fuzzers.
Range
On 2018.11.14 19:22, Junio C Hamano wrote:
> Josh Steadmon writes:
>
> > Fix several bugs identified in v3, clarify commit message, and clean up
> > extern keyword in protocol.h.
>
> It is good to descirbe the change relative to v3 here, which would
> help those who
On 2018.11.14 11:38, Junio C Hamano wrote:
> Josh Steadmon writes:
>
> > On 2018.11.13 13:01, Junio C Hamano wrote:
> >> I am wondering if the code added by this patch outside this
> >> function, with if (strcmp(client_ad.buf, "version=0") sprinkled all
(the latest version available on my workstation). I would
particularly appreciate advice on how to make the Makefile more
portable.
[1]: https://llvm.org/docs/LibFuzzer.html
[2]: https://github.com/google/oss-fuzz
Josh Steadmon (2):
fuzz: Add basic fuzz testing target.
fuzz: Add fuzz testing
Signed-off-by: Josh Steadmon
---
.gitignore | 2 ++
Makefile| 30 +-
fuzz-pack-headers.c | 14 ++
3 files changed, 45 insertions(+), 1 deletion(-)
create mode 100644 fuzz-pack-headers.c
diff --git a/.gitignore b/.gitignore
index
Signed-off-by: Josh Steadmon
---
.gitignore | 1 +
Makefile| 5 -
fuzz-pack-idx.c | 13 +
packfile.c | 44 +---
packfile.h | 13 +
5 files changed, 56 insertions(+), 20 deletions(-)
create mode 100644
On 2018.10.03 15:47, Stefan Beller wrote:
> On Wed, Oct 3, 2018 at 2:34 PM Josh Steadmon wrote:
> >
> > Is there a method or design for advertising multiple acceptable versions
> > from the client?
>
> I think the client can send multiple versions,
On 2018.10.02 15:28, Stefan Beller wrote:
> On Tue, Oct 2, 2018 at 3:00 PM Josh Steadmon wrote:
> >
> > For services other than git-receive-pack, clients currently advertise
> > that they support the version set in the protocol.version config,
> > regardless of wheth
/20180927012455.234876-1-stead...@google.com/
Josh Steadmon (1):
protocol: limit max protocol version per service
connect.c | 11 ---
protocol.c| 13 +
protocol.h| 7 +++
remote-curl.c | 11 ---
4 files changed, 28 insertions(+), 14 deletions
, it will downgrade from v2 to
v1.
Signed-off-by: Josh Steadmon
---
connect.c | 11 ---
protocol.c| 13 +
protocol.h| 7 +++
remote-curl.c | 11 ---
4 files changed, 28 insertions(+), 14 deletions(-)
diff --git a/connect.c b/connect.c
index 94547e5056
On 2018.10.05 12:25, Stefan Beller wrote:
> > > > I suppose if we are strict about serving from a single endpoint, the
> > > > version registry makes more sense, and individual operations can declare
> > > > acceptable version numbers before calling any network code?
> > >
> > > Ah yeah, that
Signed-off-by: Josh Steadmon
---
builtin/archive.c | 12 +++-
http-backend.c | 13 -
transport-helper.c | 7 ---
3 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/builtin/archive.c b/builtin/archive.c
index f91d222677..78a259518d 100644
--- a/builtin
NEEDSWORK: this is not backwards-compatible with older clients that set
GIT_PROTOCOL=version=2 or configure protocol.version=2.
Signed-off-by: Josh Steadmon
---
Documentation/technical/protocol-v2.txt | 21 +++-
builtin/archive.c | 45 +
builtin/upload
Move assertions outside of the check_tar function so that all top-level
code is wrapped in a test_expect_* assertion.
Signed-off-by: Josh Steadmon
---
t/t5000-tar-tree.sh | 26 ++
1 file changed, 10 insertions(+), 16 deletions(-)
diff --git a/t/t5000-tar-tree.sh b/t
Using packet_reader will simplify version detection and capability
handling, which will make implementation of protocol v2 support in
git-archive easier.
This refactoring does not change the behavior of "git archive".
Signed-off-by: Josh Steadmon
---
builtin/arch
r, but
please let me know if you have any suggestions for v3 of this series.
Josh Steadmon (4):
archive: follow test standards around assertions
archive: use packet_reader for communications
archive: implement protocol v2 archive command
archive: allow archive over HTTP(S) with prot
other implementations
> might violate these. However, violating these particular rules seems
> like an odd choice for a server to make.
>
> [1] Documentation/technical/http-protocol.txt, l. 166-167
> [2] Documentation/technical/protocol-v2.txt, l. 63-64
> [3] Documentation/technica
On 2018.11.16 11:45, Junio C Hamano wrote:
> Josh Steadmon writes:
>
> >> What I was alludding to was a lot simpler, though. An advert string
> >> "version=0:version=1" from a client that prefers version 0 won't be
> >> !strcmp("version=0
On 2018.11.16 03:44, Jeff King wrote:
[...]
> Amusingly, this does break the test you just added, because it tries to
> issue an ERR after claiming "text/html" (and after my patch, we
> correctly fall back to dumb-http).
Heh yeah, I copied the script from a dumb-http test without reading the
* v2 smart http; do not consume version packet, which will
>* be handled elsewhere.
> --
> 2.19.1.1636.gc7a073d580
>
Looks good to me.
Reviewed-by: Josh Steadmon
does not change behavior, so it's better to have simpler
code.
While we're at it, remove unnecessary externs from function declarations
in protocol.h.
Signed-off-by: Josh Steadmon
---
builtin/archive.c | 3 +
builtin/clone.c | 4 ++
builtin/fetch-pack.c| 4
the first allowed version from the advertised
list.
Every command that acts as a client or server must now register its
supported protocol versions.
Josh Steadmon (1):
protocol: advertise multiple supported versions
builtin/archive.c | 3 +
builtin/clone.c | 4
fuzz-commit-graph identified a case where Git will read past the end of
a buffer containing a commit graph if the graph's header has an
incorrect chunk count. A simple bounds check in parse_commit_graph()
prevents this.
Signed-off-by: Josh Steadmon
Helped-by: Derrick Stolee
---
commit-graph.c
On 2018.12.05 23:48, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Dec 05 2018, Josh Steadmon wrote:
>
> > Breaks load_commit_graph_one() into a new function,
> > parse_commit_graph(). The latter function operates on arbitrary buffers,
> > which makes it suitable as a f
).
Signed-off-by: Josh Steadmon
---
.gitignore | 1 +
Makefile| 1 +
commit-graph.c | 63 +
fuzz-commit-graph.c | 18 +
4 files changed, 66 insertions(+), 17 deletions(-)
create mode 100644 fuzz-commit-graph.c
diff
Add a new fuzz test for the commit graph and fix a buffer read-overflow
that it discovered.
Josh Steadmon (2):
commit-graph, fuzz: Add fuzzer for commit-graph
commit-graph: fix buffer read-overflow
.gitignore | 1 +
Makefile| 1 +
commit-graph.c | 76
and
corrupt_and_zero_graph_then_verify test functions.
Josh Steadmon (3):
commit-graph, fuzz: Add fuzzer for commit-graph
commit-graph: fix buffer read-overflow
Makefile: correct example fuzz build
.gitignore | 1 +
Makefile| 3 +-
commit-graph.c
that would be duplicated by the caller.
Adds fuzz-commit-graph.c, which provides a fuzzing entry point
compatible with libFuzzer (and possibly other fuzzing engines).
Signed-off-by: Josh Steadmon
---
.gitignore | 1 +
Makefile| 1 +
commit-graph.c | 53
Signed-off-by: Josh Steadmon
---
Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index 6b72f37c29..bbcfc2bc9f 100644
--- a/Makefile
+++ b/Makefile
@@ -3104,7 +3104,7 @@ cover_db_html: cover_db
# An example command to build against libFuzzer from
fuzz-commit-graph identified a case where Git will read past the end of
a buffer containing a commit graph if the graph's header has an
incorrect chunk count. A simple bounds check in parse_commit_graph()
prevents this.
Signed-off-by: Josh Steadmon
---
commit-graph.c | 14
fuzz-commit-graph identified a case where Git will read past the end of
a buffer containing a commit graph if the graph's header has an
incorrect chunk count. A simple bounds check in parse_commit_graph()
prevents this.
Signed-off-by: Josh Steadmon
---
commit-graph.c | 14
Signed-off-by: Josh Steadmon
---
Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index 6b72f37c29..bbcfc2bc9f 100644
--- a/Makefile
+++ b/Makefile
@@ -3104,7 +3104,7 @@ cover_db_html: cover_db
# An example command to build against libFuzzer from
to
fix this.
Josh Steadmon (3):
commit-graph, fuzz: Add fuzzer for commit-graph
commit-graph: fix buffer read-overflow
Makefile: correct example fuzz build
.gitignore | 1 +
Makefile| 3 +-
commit-graph.c | 67
that would be duplicated by the caller.
Adds fuzz-commit-graph.c, which provides a fuzzing entry point
compatible with libFuzzer (and possibly other fuzzing engines).
Signed-off-by: Josh Steadmon
---
.gitignore | 1 +
Makefile| 1 +
commit-graph.c | 53
ave it up to your
judgement as to whether they're valid or not.
Reviewed-by: Josh Steadmon
This series adds a new protocol v2 command for archiving, and allows
this command to work over HTTP(S). This was previously discussed in [1].
I've CCed everyone who participated in that discussion.
[1]:
f-by: Josh Steadmon
---
builtin/archive.c| 45 +++-
builtin/upload-archive.c | 44 ---
t/t5000-tar-tree.sh | 5 +
3 files changed, 77 insertions(+), 17 deletions(-)
diff --git a/builtin/archive.c b/builtin/archi
Signed-off-by: Josh Steadmon
---
builtin/archive.c | 8 +++-
http-backend.c | 10 +-
transport-helper.c | 5 +++--
3 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/builtin/archive.c b/builtin/archive.c
index 73831887d..5fa75b3f7 100644
--- a/builtin/archive.c
Using packet_reader will simplify version detection and capability
handling, which will make implementation of protocol v2 support in
git-archive easier.
Signed-off-by: Josh Steadmon
---
builtin/archive.c | 23 ++-
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git
81 matches
Mail list logo
