Re: Proposal: http2 wire format

On Mon, Mar 26, 2018 at 7:51 PM, Craig Ringer wrote: > There's been no visible consideration of overheads and comparison with > existing v3 protocol. Personally I'm fine with adding some protocol overhead > in bytes terms; low latency links have the bandwidth not to care much > compared to payload

Re: Proposal: http2 wire format

On 2018-03-29 17:52:07 -0400, Peter Eisentraut wrote: > On 3/29/18 14:20, Andres Freund wrote: > > On 2018-03-28 20:34:13 -0400, Peter Eisentraut wrote: > >> On 3/28/18 12:09, Andres Freund wrote: > >>> Yea, not the most descriptive... Returning multiple different resultsets > >>> from a function /

Re: Proposal: http2 wire format

On 3/29/18 14:20, Andres Freund wrote: > On 2018-03-28 20:34:13 -0400, Peter Eisentraut wrote: >> On 3/28/18 12:09, Andres Freund wrote: >>> Yea, not the most descriptive... Returning multiple different resultsets >>> from a function / procedure. Inability to do so is a serious limitation >>> of po

Re: Proposal: http2 wire format

On 2018-03-28 20:34:13 -0400, Peter Eisentraut wrote: > On 3/28/18 12:09, Andres Freund wrote: > > Yea, not the most descriptive... Returning multiple different resultsets > > from a function / procedure. Inability to do so is a serious limitation > > of postgres in comparison to some other languag

Re: Proposal: http2 wire format

> > > * room for other resultset formats later. Like Damir, I really want to > add > > protobuf or json serializations of result sets at some point, mainly so > we > > can return "entity graphs" in graph representation rather than left-join > > projection. > > -1. I don't think this belongs in post

Re: Proposal: http2 wire format

On 3/28/18 12:09, Andres Freund wrote: > Yea, not the most descriptive... Returning multiple different resultsets > from a function / procedure. Inability to do so is a serious limitation > of postgres in comparison to some other language with procedures. This is already possible as far as the pro

Re: Proposal: http2 wire format

Hi, On 2018-03-28 09:59:34 -0400, Tom Lane wrote: > Andres Freund writes: > > A few random, very tired, points: > > > - consolidated message for common tasks: > > - (bind, [describe?,] execute) to reduce overhead of prepared > > statement execution (both in messages, as well as branches) >

Re: Proposal: http2 wire format

Hi, On 2018-03-28 16:29:37 +0800, Craig Ringer wrote: > > - allow *streaming* of large datums > > Yes, very much +1 there. That's already on the wiki. Yeah: > > * Permit lazy fetches of large values, at least out-of-line TOASTED values > http://www.postgresql.org/message-id/53ff0ef8@2ndquadr

Re: Proposal: http2 wire format

Andres Freund writes: > A few random, very tired, points: > - consolidated message for common tasks: > - (bind, [describe?,] execute) to reduce overhead of prepared > statement execution (both in messages, as well as branches) > - (anonymous parse, bind, describe, execute) to make it chea

Re: Proposal: http2 wire format

> A few random, very tired, points: > > - consolidated message for common tasks: > - (bind, [describe?,] execute) to reduce overhead of prepared > statement execution (both in messages, as well as branches) > - (anonymous parse, bind, describe, execute) to make it cheaper to > send sta

Re: Proposal: http2 wire format

On 28 March 2018 at 16:02, Andres Freund wrote: > On 2018-03-26 22:44:09 +0200, Damir Simunic wrote: > > > *NONE* of the interesting problems are solved by HTTP2. You *still* > > > need a full blown protocol ontop of it. So no, this doesn't change > that. > > > > If you had to nominate only one o

Re: Proposal: http2 wire format

On 28 March 2018 at 00:42, Damir Simunic wrote: > > > I'm rapidly losing interest. Unless this goes back toward the concrete and > practical I think it's going nowhere. > > > > Your message is exactly what I was hoping for. Thanks for your guidance > and support, really appreciate you. > > Let me

Re: Proposal: http2 wire format

On 2018-03-26 22:44:09 +0200, Damir Simunic wrote: > > *NONE* of the interesting problems are solved by HTTP2. You *still* > > need a full blown protocol ontop of it. So no, this doesn't change that. > > If you had to nominate only one of those problems, which one would you > consider the most int

Re: Proposal: http2 wire format

> > > I'm rapidly losing interest. Unless this goes back toward the concrete and > practical I think it's going nowhere. Your message is exactly what I was hoping for. Thanks for your guidance and support, really appreciate you. Let me now get busy and earn your continued interest and suppo

Re: Proposal: http2 wire format

On 26 March 2018 at 22:56, Tom Lane wrote: > Damir Simunic writes: > >> On 26 Mar 2018, at 11:06, Vladimir Sitnikov < > sitnikov.vladi...@gmail.com> wrote: > >>> If anyone finds the idea of Postgres speaking http2 appealing > > TBH, this sounds like a proposal to expend a whole lot of work (much

Re: Proposal: http2 wire format

Greetings, * Craig Ringer (cr...@2ndquadrant.com) wrote: > On 26 March 2018 at 21:05, Damir Simunic > wrote: > > The same goes for the ‘authorization’ header. Postgres does not support > > Bearer token authorization today. But maybe you’ll be able to define a > > function that knows how to deal w

Re: Proposal: http2 wire format

On 26 March 2018 at 21:05, Damir Simunic wrote: > > On 26 Mar 2018, at 11:06, Vladimir Sitnikov > wrote: > > > > Hi, > > > > >If anyone finds the idea of Postgres speaking http2 appealing > > > > HTTP/2 sounds interesting. > > What do you think of https://grpc.io/ ? > > > > Have you evaluated it

Re: Proposal: http2 wire format

On Mon, Mar 26, 2018 at 1:05 PM, Damir Simunic wrote: > Would it be the only protocol supported? What if I wanted JSON or CSV > returned, or just plain old Postgres v3 binary format, since I already have > the parser written for it? Wouldn’t you need to first solve the problem of > content negoti

Re: Proposal: http2 wire format

> Currently it is implemented via different v3 messages (parse, bind, execute, > row description, data row, etc etc). > > The claim is *any* implementation "on top of HTTP/2" would basically require > to implement those "parse, bind, execute, row data, etc" *messages*. Why? Wouldn’t you be abl

Re: Proposal: http2 wire format

On 26/03/18 21:57, Damir Simunic wrote: On 26 Mar 2018, at 15:42, Alvaro Hernandez > wrote: On 26/03/18 13:11, Damir Simunic wrote: On 26 Mar 2018, at 11:13, Vladimir Sitnikov mailto:sitnikov.vladi...@gmail.com>> wrote: Damir> * What are the criteria for getting

Re: Proposal: http2 wire format

Hi Andres, > > At least I do *NOT* want many protocols in core. We've a hard enough > time to keep up with integrating patches and maintenance to not just > willy nilly integrate multiple new features with unclear lifetimes. Admire your effort in applying all these patches—this commitfest thing

Re: Proposal: http2 wire format

Damir>Wouldn’t that be protocol semantics? Framing is already taken care of by the wire protocol. Apparently I'm using the wrong word. I do mean protocol semantics. Damir>But can you pull off grpc.. Would it be the only protocol supported? Of course there will be lots of "older clients" For inst

Re: Proposal: http2 wire format

It could make sense to arrange a Google Hangouts conversation (or alike). Hangouts allows to record sessions with up to 10 speakers and unlimited listeners. The recording can be shared via YouTube. Damir>Funny you agree with that—for someone having the experience of writing a driver and having a l

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 18:09, Vladimir Sitnikov > wrote: > > Damir>Postgres doesn’t know about grpc, s > > I'm afraid you are missing the point. > I would say PostgreSQL doesn't know about HTTP/2. > It is the same as "PostgreSQL doesn't know about grpc". > > Here's a quote from your pg_h2 repo:

Re: Proposal: http2 wire format

Hi, On 2018-03-26 20:36:09 +0200, Damir Simunic wrote: > If so, I’m not suggesting we get rid of FEBE, but leave it as is and > complement it with a widely understood and supported protocol, that in fact > takes compatibility way more seriously than FEBE. Just leave v3 frozen. Seems > like ulti

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 15:42, Alvaro Hernandez wrote: > > > > On 26/03/18 13:11, Damir Simunic wrote: >>> On 26 Mar 2018, at 11:13, Vladimir Sitnikov >> > wrote: >>> >>> Damir> * What are the criteria for getting this into the core? >>> Craig>Mine would be:

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 18:19, Vladimir Sitnikov > wrote: > > Tom>But starting from the assumption that HTTP2 solves our problems seems to > me to be "Here's a hammer. > > Agree. Funny you agree with that—for someone having the experience of writing a driver and having a long list of things th

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 16:56, Tom Lane wrote: > > Damir Simunic writes: >>> On 26 Mar 2018, at 11:06, Vladimir Sitnikov >>> wrote: If anyone finds the idea of Postgres speaking http2 appealing > > TBH, this sounds like a proposal to expend a whole lot of work (much of it > outside the cor

Re: Proposal: http2 wire format

On 26/03/18 13:11, Damir Simunic wrote: On 26 Mar 2018, at 11:13, Vladimir Sitnikov mailto:sitnikov.vladi...@gmail.com>> wrote: Damir> * What are the criteria for getting this into the core? Craig>Mine would be: +1 There's a relevant list as well: https://github.com/pgjdbc/pgjdbc/blob/mast

Re: Proposal: http2 wire format

Tom>But starting from the assumption that HTTP2 solves our problems seems to me to be "Here's a hammer. Agree. Just a side note: if v4 is ever invented I wish client language support is considered. It does take resources to implement message framing, and data parsing (e.g. int, timestamp, struct,

Re: Proposal: http2 wire format

Damir>Postgres doesn’t know about grpc, s I'm afraid you are missing the point. I would say PostgreSQL doesn't know about HTTP/2. It is the same as "PostgreSQL doesn't know about grpc". Here's a quote from your pg_h2 repo: >What we need is to really build a request object and correctly extract >

Re: Proposal: http2 wire format

Damir Simunic writes: >> On 26 Mar 2018, at 11:06, Vladimir Sitnikov >> wrote: >>> If anyone finds the idea of Postgres speaking http2 appealing TBH, this sounds like a proposal to expend a whole lot of work (much of it outside the core server, and thus not under our control) in order to get fr

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 11:06, Vladimir Sitnikov > wrote: > > Hi, > > >If anyone finds the idea of Postgres speaking http2 appealing > > HTTP/2 sounds interesting. > What do you think of https://grpc.io/ ? > > Have you evaluated it? > It does sound like a ready RPC on top of HTTP/2 with support

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 11:13, Vladimir Sitnikov > wrote: > > Damir> * What are the criteria for getting this into the core? > Craig>Mine would be: > > +1 > > There's a relevant list as well: > https://github.com/pgjdbc/pgjdbc/blob/master/backend_protocol_v4_wanted_features.md > >

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 12:47, Craig Ringer wrote: > > On 26 March 2018 at 17:34, Damir Simunic wrote: > > > > As you move forward with the PoC, consider: even if you decide not to > > become protocol-layer experts, you'll still need to become familiar > > with application-layer security in HTTP

Re: Proposal: http2 wire format

On 26 March 2018 at 17:34, Damir Simunic wrote: > > > As you move forward with the PoC, consider: even if you decide not to > > become protocol-layer experts, you'll still need to become familiar > > with application-layer security in HTTP. > > Good point. Application layer security is indeed a

Re: Proposal: http2 wire format

> On 26 Mar 2018, at 11:34, Craig Ringer wrote: > > On 26 March 2018 at 17:01, Damir Simunic > wrote: > > > > - Doesn't break new clients connecting to old servers > > > > Old server sends “Invalid startup packet” and closes the connection; client’s > T

Re: Proposal: http2 wire format

On 26 March 2018 at 17:01, Damir Simunic wrote: > > > - Doesn't break new clients connecting to old servers > > > > Old server sends “Invalid startup packet” and closes the connection; > client’s TLS layer reports an error. Does that count as not breaking new > clients? > > libpq would have to d

Re: Proposal: http2 wire format

Hi, > On 26 Mar 2018, at 06:47, Jacob Champion wrote: > > On Sun, Mar 25, 2018 at 8:11 PM, Craig Ringer wrote: >> As others have noted, you'll want to find a way to handle this in the least >> SSL-implementation-specific manner possible. IMO if it can't work with >> OpenSSL, Windows's SSL imple

Re: Proposal: http2 wire format

Damir> * What are the criteria for getting this into the core? Craig>Mine would be: +1 There's a relevant list as well: https://github.com/pgjdbc/pgjdbc/blob/master/backend_protocol_v4_wanted_features.md Vladimir

Re: Proposal: http2 wire format

Hi, >If anyone finds the idea of Postgres speaking http2 appealing HTTP/2 sounds interesting. What do you think of https://grpc.io/ ? Have you evaluated it? It does sound like a ready RPC on top of HTTP/2 with support for lots of languages. The idea of reimplementing the protocol for multiple l

Re: Proposal: http2 wire format

Hi, > On 26 Mar 2018, at 05:11, Craig Ringer wrote: > > On 26 March 2018 at 06:00, Damir Simunic wrote: > > > - Overhead for all clients. It may be tiny, but it needs to be > > measured and that cost needs to be weighed against the benefits. > > Maybe a cache miss in the context of a networ

Re: Proposal: http2 wire format

On Sun, Mar 25, 2018 at 8:11 PM, Craig Ringer wrote: > As others have noted, you'll want to find a way to handle this in the least > SSL-implementation-specific manner possible. IMO if it can't work with > OpenSSL, Windows's SSL implementation and OS X's SSL framework it's a > non-starter. +1. >

Re: Proposal: http2 wire format

On 26 March 2018 at 06:00, Damir Simunic wrote: > > - Overhead for all clients. It may be tiny, but it needs to be > > measured and that cost needs to be weighed against the benefits. > > Maybe a cache miss in the context of a network connection is > > negligible, but we do need to know. > >

Re: Proposal: http2 wire format

> On 25 Mar 2018, at 19:42, David Fetter wrote: > > On Sat, Mar 24, 2018 at 06:52:47PM +0100, Damir Simunic wrote: >> Hello hackers, >> >> I’d like to propose the implementation of new wire protocol using http2 >> framing. > > Welcome to the PostgreSQL community! This is a very interesting i

Re: Proposal: http2 wire format

On Sat, Mar 24, 2018 at 06:52:47PM +0100, Damir Simunic wrote: > Hello hackers, > > I’d like to propose the implementation of new wire protocol using http2 > framing. Welcome to the PostgreSQL community! This is a very interesting idea. Please send a patch to this mailing list on this thread.

Proposal: http2 wire format

Hello hackers, I’d like to propose the implementation of new wire protocol using http2 framing. It appears to me that http2 solves many of the issues on the TODO list under “Wire Protocol Changes / v4 Protocol,“ without any obvious downsides. The implementation I have in mind has zero impa