Re: Thoughts on refactoring the transport (+helper) code
On Thu, Aug 13, 2015 at 11:42:50AM -0400, Dave Borowitz wrote:
In my ideal world:
-smart_options would never be NULL, and would instead be called
options with a smart bit which is unset for dumb protocols.
-Command line option processing code in {fetch,clone,push}.c would set
fields in options (formerly known as smart_options) rather than
passing around string constants.
-TRANS_OPT_* string constants would only be used for remote helper
protocol option names, and no more hard-coding these names.
-The flags arg to the push* callbacks would go away, and callbacks
would respect options instead.
-The helper code would not send options immediately, but instead send
just the relevant options immediately before a particular command
requires them. Hopefully we could then eliminate the set_option
callback entirely. (Two things I ran into that complicated this: 1)
backfill_tags mutates just a couple of options before reusing the
transport, and 2) the handling of push_cas_option is very
special-cased.)
AFAIK, here are what one can encounter with remote helpers:
Some remote helpers are always smart, some are always dumb, and some
may be smart or dumb, depending on the URL.
I don't know how useful the last one is (smart or dumb depending on
URL) is. I have never seen such remote helper (HTTP doesn't count,
from Git PoV it is always dumb).
All smart helpers take common options associated with git smart
transport (e.g. thin, follow tags, push certs, etc..).
Dumb transports may take some of these kind of of options (esp.
smart HTTP), but it is highly dependent on the helper.
Then transports can have connection-level options (e.g. HTTPS
cert options). These can be present regardless of wheither
transport is smart or dumb.
The receive-pack / send-pack paths fall into connection-level
options, even if those are presently in smart transport common
options. Since those things make sense for some smart transports
(e.g. ssh://), but not others (e.g. git://).
-Ilari
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] http: add support for specifying the SSL version
On Thu, Aug 13, 2015 at 06:10:48PM +0200, Elia Pinto wrote: 2015-08-13 18:01 GMT+02:00 Torsten Bögershausen tbo...@web.de: + from https://en.wikipedia.org/wiki/Transport_Layer_Security#SSL_1.0.2C_2.0_and_3.0 sslv2 and sslv3 are deprecated. Should there be a motivation in the commit message why we want to support them ? They are those provided by the documentation (TLS in particular). We let the underlying library to say what is deprecated or not. In this case the call fail. The statement from the relevant SDO is much stronger than deprecated, it is not to be used under any cirmumstances. Option like this looks only useful for connecting to really broken servers, damn security. It could be useful for connecting to buggy servers after TLS 1.3 comes out and is implemented, as there are lots of servers (IIRC, on order of 10%) that can't deal with TLS 1.3 properly (but very few, IIRC 0.1%, that can't deal with TLS 1.2 correctly[1]). Also, is this option settable globally for all HTTP servers? One definitely does not want that to be possible. Configurations like this need to be per-server if they exist at all. [1] Where correctly includes secure downnegotiation, as TLS is intended to do when faced with version mismatch. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Git Smart HTTP with HTTP/2.0
On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote: As we known, HTTP/2.0 has been released. All Git-Smart-HTTP are currently implemented using HTTP/1.1. Nit: It is HTTP/2. Frequently used Git developers often feel Git HTTP protocol is not satisfactory, slow and unstable.This is because the HTTP protocol itself decides Note that there are already two versions of HTTP transport, the old dumb one and the newer smart one. The smart one is difficult to speed up (due to nature of the negotiations), but usually is pretty reliable (the efficiency isn't horrible). Now, the old dumb protocol is pretty unreliable and slow. HTTP/2 probably can't do anything with the reliability problems, but probably could improve the speed a bit. Websockets over HTTP/2 (a.k.a. websockets2) has not been defined yet. With Websockets(1), it would probably already be possible to tunnel the native git smart transport protocol over it. Probably not worth it. When HTTP/2.0 is published. We might be able to git developers jointly, based on HTTP/2.0 Git-Smart-HTTP service and client support. HTTP/2.0: https://tools.ietf.org/html/rfc7540 Well, it is published already. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Git Smart HTTP with HTTP/2.0
On Sat, Jul 11, 2015 at 10:23:09AM -0700, Shawn Pearce wrote: On Sat, Jul 11, 2015 at 12:00 AM, Ilari Liusvaara ilari.liusva...@elisanet.fi wrote: On Sat, Jul 11, 2015 at 11:10:48AM +0800, ForceCharlie wrote: Frequently used Git developers often feel Git HTTP protocol is not satisfactory, slow and unstable.This is because the HTTP protocol itself decides Note that there are already two versions of HTTP transport, the old dumb one and the newer smart one. The smart one is difficult to speed up (due to nature of the negotiations), but usually is pretty reliable (the efficiency isn't horrible). The negotiation in smart-HTTP actually has some bad corner cases. Each round of negotiation requires a new POST resupplying all previously agreed upon SHA-1s, and a batch of new SHA-1s. We have observed many rounds where this POST is MiBs in size because the peers can't quite agree and have to keep digging through history. Oh yeah that... Well, that is artifact of HTTP semantics. Now, the old dumb protocol is pretty unreliable and slow. HTTP/2 probably can't do anything with the reliability problems, but probably could improve the speed a bit. Websockets over HTTP/2 (a.k.a. websockets2) has not been defined yet. With Websockets(1), it would probably already be possible to tunnel the native git smart transport protocol over it. Probably not worth it. Another option is to tunnel using gRPC (grpc.io). libcurl probably can't do this. And linking grpc.io library into git-core is crazy. So its probably a non-starter. But food for thought. Wouldn't it link into git-remote-http (and on the server side, one could use pipes to talk to git)? But supporting websockets in git-remote-http could get annoying, especially for wss:// (https://). Dunno how bad gRPC would be. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: support git+mosh for unreliable connections
On Wed, Apr 15, 2015 at 08:13:51PM +0530, Pirate Praveen wrote: Q: Are the mosh principles relevant to other network applications? We think so. The design principles that Mosh stands for are conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully. Can the ideas be used to resume a pull, push or clone operation? Especially serializing and checkpointing. Well, it is possible to write a remote helper and serverside program that internally handles connection unreliability, so Git itself (upload-archive, upload-pack, receive-pack, archive, fetch-pack and send-pack) sees a reliable (full-duplex, half-closeable, stream) channel. Suitably done, that can resume (from Git POV, nothing special happened) across things like IP address changes. However, that is quite difficult to do in practice. Not because interface to Git is complicated, but because the transport problem itself is complicated (however, it still seems way easier than making Git internally be able to resume interrupted operations). Mosh needs to solve at least most of that, it just doesn't provode the right kind of interface. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Fwd: git-remote-fd problem
On Mon, Dec 29, 2014 at 10:47:58AM +0100, Jiri Sevcik wrote:
The remote-fd expects the transport to pass half-closes. So you can't
close all at once.
Let there be pipes W and R and transport connection C.
- W-read should be closed after being passed to remote-fd.
- R-write should be closed after being passed to remote-fd.
- Upon receiving no more data from C, close W-write.
- Upon receiving EOF from R-read, close it and signal no more data
to C.
Hi, I followed your advices, correctly close pipes but git clone still
doesnt finish and hanging on.
Code is in an attachement (its part of big system).
Few ideas:
- Check that git clone (and its subprocesses) don't inherit
w_pipe[0] (/proc/pid/fd on Linux might be handy). If they do, that
prevents this program from closing the pipe.
- Setting environment variable GIT_TRANSLOOP_DEBUG to 1 might make
git spew lots of messages to stderr about reads, writes and closes.
#create pipes
w_pipe = os.pipe()
r_pipe = os.pipe()
client_process = subprocess.Popen(/usr/bin/git clone fd::{0},{1}
/tmp/gittest.format(r_pipe[0], w_pipe[1]), shell=True)
#closing pipes
os.close(r_pipe[0])
os.close(w_pipe[1])
epoll = select.epoll()
epoll.register(w_pipe[0], select.EPOLLIN)
epoll.register(proc.fd, select.EPOLLIN)
remoteGit = proc.runDaemon(git-upload-pack /tmp/testgit)
while True:
events = epoll.poll(1)
for fd, event in events:
if fd == w_pipe[0]:
if event select.EPOLLIN:
rd = os.read(w_pipe[0], 1)
if rd:
#write data to remove git server
remoteGit.writeToChannel(rd)
else:
proc.writeError(Local socket write error)
return 1
else:
proc.writeError(Local socket error)
return 1
elif fd == proc.fd:
if event select.EPOLLIN:
#read data from remote git server
data = remoteGit.getAll()
remoteGit.stderrWrite()
if not data:
#remote server send EOF, close local pipe
#but git clone is still running
os.close(r_pipe[1])
return 0
want = len(data)
writed = 0
offset = 0
while(writed != want):
#write data from remote git server to local pipe
wr = os.write(r_pipe[1], data[offset:])
if(wr 0):
return 1
writed += wr
offset += wr
else:
return -1
-Ilari
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git-remote-fd problem
On Wed, Sep 10, 2014 at 03:29:00PM +0200, Jiri Sevcik wrote: Hi! I have problem with using git-remote-fd function. I create two local pipes for communication for locally running process git pull. Then I start git-upload-pack on remote side (this process is started in different part of my code and it works correctly). Communication runs successfully and when remote side finish, I want to close local pipes to finish local process. I call close function to all of these pipes, but unfortunatelly local process doesnt finish (in this moment transfered data aresaved in /.git/object/pack folder in .pack files) and hanging on. But if I kill child of this local process, downloaded files are unpacked and files are created sucessfully. I think this way is not so correct. Can someone help me and tell me what am I doing wrong? Thank You. Code is in an attachement (its part of big system). The remote-fd expects the transport to pass half-closes. So you can't close all at once. Let there be pipes W and R and transport connection C. - W-read should be closed after being passed to remote-fd. - R-write should be closed after being passed to remote-fd. - Upon receiving no more data from C, close W-write. - Upon receiving EOF from R-read, close it and signal no more data to C. If you have server end, the same applies, but with remote-fd replaced by upload-pack/upload-archive/receive-pack. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Anomalous AUTHOR and DOCUMENTATION sections in manpages
On Wed, Jan 22, 2014 at 12:22:23PM +0100, Michael Haggerty wrote: I just noticed that there are exactly four Git manpages with an AUTHOR section and five with a DOCUMENTATION section: These sections are inconsistent with the other manpages and seem superfluous in a project that has, on the one hand, a public history and, on the other hand, hundreds of contributors. Would the mentioned authors (CCed) consent to the removal of these sections? Sure (it has been copyedited a lot anyway). -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git:// protocol over SSL/TLS
On Fri, Dec 27, 2013 at 02:21:31PM -0800, Junio C Hamano wrote: Konstantin Khomoutov flatw...@users.sourceforge.net writes: The Git protocol does not implement it itself but you can channel it over a TLS tunnel (via stunnel for instance). Unfortunately, this means a specialized software and setup on both ends so if the question was about a general client using stock Git then the answer is no, it's impossible. Hmph, I somehow had an impression that you wouldn't need anything more complex than a simple helper that uses git-remote-ext on the client side. On the remote end, you'd need to have something that terminates the incoming SSL/TLS and plugs it to your git daemon. If you have some tool that can do cleartext I/O from stdin/stdout and establishes ciphertext connection itself, you can use it with git-remote-ext. It was written for cases exactly like that. To do git:// inside, use the %G pseudo-argument. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: GSS-Negotiate authentication requires that all data fit into postbuffer
On Sun, Oct 06, 2013 at 12:42:36AM +, brian m. carlson wrote: GSS-Negotiate authentication always requires a rewind with CURL. The remote in question only supports Negotiate authentication, so prompting for a password in this case isn't going to help. I'm probably going to look into this over the next couple of days, but two things need to be done here: 1) do not prompt for a password if only Negotiate authentication is requested, since it just won't work, and 2) recreate the data as needed if we have to rewind, since otherwise pushing a fresh copy of the Linux kernel repo simply isn't going to work as the buffer will have to be too large. An alternative is to send a small amount of data, smaller than the postbuffer, in the first chunk and only fail to rewind if the second or subsequent chunks need rewinding. Isn't 'Expect: 100-Continue' meant for stuff like this (not that it is always supported properly)? -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Exact format of tree objets
On Tue, Jun 11, 2013 at 01:25:14PM -0300, Chico Sokol wrote: Is there any official documentation of tree objets format? Are tree objects encoded specially in some way? How can I parse the inflated contents of a tree object? Tree object consists of entries, each concatenation of: - Octal mode (using ASCII digits 0-7). - Single SPACE (0x20) - Filename - Single NUL (0x00) - 20-byte binary SHA-1 of referenced object. At least following octal modes are known: 4: Directory (tree). 100644: Regular file (blob). 100755: Executable file (blob). 12: Symbolic link (blob). 16: Submodule (commit). The entries are always sorted in (bytewise) lexicographical order, except directories sort like there was impiled '/' at the end. So e.g.: ! 0 9 a a- a- (directory) a (directory) a0 ab b z. The idea of sorting directories specially is that if one recurses upon hitting a directory and uses '/' as path separator, then the full filenames are in bytewise lexicographical order. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: propagating repo corruption across clone
On Sun, Mar 24, 2013 at 02:31:33PM -0400, Jeff King wrote: Fscking the incoming objects does work, but of course it comes at a cost in the normal case (for linux-2.6, I measured an increase in CPU time with index-pack --strict from ~2.5 minutes to ~4 minutes). And I think it is probably overkill for finding corruption; index-pack already recognizes bit corruption inside an object, and check_everything_connected can detect object graph problems much more cheaply. AFAIK, standard checks index-pack has to do + checking that the object graph has no broken links (and every ref points to something valid) will catch everything except: - SHA-1 collisions between corrupt objects and clean ones. - Corrupted refs (that still point to something valid). - Born-corrupted objects. One thing I didn't check is bit corruption inside a packed object that still correctly zlib inflates. check_everything_connected will end up reading all of the commits and trees (to walk them), but not the blobs. Checking that everything is connected will (modulo SHA-1 collisions) save you there, at least if packv3 is used as transport stream. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is anyone working on a next-gen Git protocol?
On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote: Has anyone started working on a next-gen Git protocol as a result of this discussion? If not I thought I'd give it a shot if/when I have time. Unfortunately, client signaling the version is nasty to do in ways that wouldn't cause current servers to hang up or do other undesirable things. git://: Git-daemon will hang up[1] if it receives command it won't understand (and one can't add arguments either). ssh://: Commands are NAKed in non-standard ways (e.g. Gitolite vs. shell) and one can't add arguments. file://: That's easy. CONNECT: The helper needs to be told that v2 is supported (helper doing the rest). Maybe with git://, one could hack the stuff in similar way as virtual hosting was added. But that won't work with SSH (nor one can use environment with SSH). :-/ [1] And there is no guarantee that the server end of git:// is git-daemon. There's at least one git:// server implemetation that responds to unknown commands by ERR packet followed by hangup. -Ilari -- To unsubscribe from this list: send the line unsubscribe git in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
