Re: [racket-dev] net/http-client
On Wed, Sep 4, 2013 at 11:03 AM, Jay McCarthy jay.mccar...@gmail.com wrote: On Wed, Aug 28, 2013 at 3:30 PM, Greg Hendershott greghendersh...@gmail.com wrote: This looks great!! A couple suggestions: 1. Support for Expect: 100-continue request headers would be helpful, and I think not too messy to add. The big use case I'm aware of is Amazon S3. If you make a PUT or POST request, it might need to redirect you to another URI (outage, balancing, whatever reason). Expecting and handling 100-continue lets you avoid transmitting potentially large amount of data that would be discarded, and you have to send it all over again in the request to the redirect URI. For (say) a 1 GB upload to S3, this matters. Although I don't know for sure if other upload-ish web APIs offer same, I'd guess some do as this is the use case for 100-continue generally. How I implemented this in my HTTP package was to have a `start-request` function that sends the request line and headers, peeks the response status line, then returns a `boolean?` whether the PUT/POST/PATCH/whatever data should be transmitted. [1] I think your `http-conn-send!` could do similar? Do you think it is appropriate to expect the http-client user to put in the Expect: 100-continue or better to always send it if there is a data component? Great question. I took the approach of requiring the client to supply it if they care. Instead supplying it always/automatically does seem neat. But safe? Reading http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html I'm a little nervous about this section: A server that does not understand or is unable to comply with any of the expectation values in the Expect field of a request MUST respond with appropriate error status. The server MUST respond with a 417 (Expectation Failed) status if any of the expectations cannot be met or, if there are other problems with the request, some other 4xx status. This suggests a scenario where a server might error a request solely because of the presence of an Expect: 100-continue header. That doesn't strike me as reasonable behavior, but I could imagine some server doing it. So I suppose best to default to supplying it automatically, but provide a way for the client to disable that. How to specify disabling? With headers like Server: you have a default but let the user's supplied headers override. But supplying Expect: (i.e. Expect: blank) would feel weird, to a user. And actually sending that -- if it feels weird to a server causing it to error, well that's the whole thing we're trying to avoid, see above. So (tl;dr) perhaps add an optional function parameter that defaults to #t, e.g. `[expect-100-continue? #t]` ? 2. Support for Content-Encoding response headers would also be helpful. Using the same make-pipe approach as you're doing with chunked transfer encoding. [2] Maybe this is mission creep: For HTTP 1.1. you _must_ support Transfer-Encoding: chunked, whereas Content-Encoding is just optional. However it's a good option; using compression can really help out on time as well as bandwidth charges. I just pushed support for this. Nice, thanks! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] net/http-client
On Wed, Aug 28, 2013 at 3:30 PM, Greg Hendershott greghendersh...@gmail.com wrote: This looks great!! A couple suggestions: 1. Support for Expect: 100-continue request headers would be helpful, and I think not too messy to add. The big use case I'm aware of is Amazon S3. If you make a PUT or POST request, it might need to redirect you to another URI (outage, balancing, whatever reason). Expecting and handling 100-continue lets you avoid transmitting potentially large amount of data that would be discarded, and you have to send it all over again in the request to the redirect URI. For (say) a 1 GB upload to S3, this matters. Although I don't know for sure if other upload-ish web APIs offer same, I'd guess some do as this is the use case for 100-continue generally. How I implemented this in my HTTP package was to have a `start-request` function that sends the request line and headers, peeks the response status line, then returns a `boolean?` whether the PUT/POST/PATCH/whatever data should be transmitted. [1] I think your `http-conn-send!` could do similar? Do you think it is appropriate to expect the http-client user to put in the Expect: 100-continue or better to always send it if there is a data component? 2. Support for Content-Encoding response headers would also be helpful. Using the same make-pipe approach as you're doing with chunked transfer encoding. [2] Maybe this is mission creep: For HTTP 1.1. you _must_ support Transfer-Encoding: chunked, whereas Content-Encoding is just optional. However it's a good option; using compression can really help out on time as well as bandwidth charges. I just pushed support for this. IIRC those were the two main things that motivated me to make my HTTP package at all, to support e.g. my AWS package. If http-client added them, I might not need my package anymore. (OK, it might take me awhile to phase it out until I'm ready to de-support older versions of Racket, but, I and others wouldn't need it for new projects.) [1]: https://github.com/greghendershott/http/blob/master/http/request.rkt#L142-L189 [2]: By the way, do you want to pass some `limit` optional arg in the various uses of `make-pipe`? Otherwise IIUC this will suck everything into RAM, which might not be so great with very large request or response entities. Matthew changed this a few days ago. Jay On Fri, Aug 23, 2013 at 2:48 PM, Jay McCarthy jay.mccar...@gmail.com wrote: Based on a request back in early July to remove the restrictions that net/url puts on HTTP communication (vis a vis URL encoding), I have just pushed a new HTTP client as net/http-client. The push also changes net/url to use net/http-client so that we just have 1 HTTP request producer, rather than 3. It passes all of the net/* tests, but these don't use features like proxying, HTTP/1.1, etc. I'm slightly nervous that it doesn't do those correct, but not super nervous, because I just cut-and-pasted the code. The main approach of the library is best explained by this contract: [http-sendrecv (-* ((or/c bytes? string?) (or/c bytes? string?)) (#:ssl? (or/c boolean? ssl-client-context? symbol?) #:port (between/c 1 65535) #:method (or/c bytes? string? symbol?) #:headers (listof (or/c bytes? string?)) #:data (or/c false/c bytes? string?)) (values bytes? (listof bytes?) input-port?))] Compared to net/url, - It supports bytes and strings everywhere - It supports data on every method and not just POST - It always returns the status line, headers, and content (as a port) I feel that the only thing it could do better is support two more options for #:data: - A input-port? to read from and copy to the HTTP connection - A (- output-port? void) function to call with the HTTP connection's output port to stream the data But I'd like a second opinion before adding them. Jay -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] net/http-client
This looks great!! A couple suggestions: 1. Support for Expect: 100-continue request headers would be helpful, and I think not too messy to add. The big use case I'm aware of is Amazon S3. If you make a PUT or POST request, it might need to redirect you to another URI (outage, balancing, whatever reason). Expecting and handling 100-continue lets you avoid transmitting potentially large amount of data that would be discarded, and you have to send it all over again in the request to the redirect URI. For (say) a 1 GB upload to S3, this matters. Although I don't know for sure if other upload-ish web APIs offer same, I'd guess some do as this is the use case for 100-continue generally. How I implemented this in my HTTP package was to have a `start-request` function that sends the request line and headers, peeks the response status line, then returns a `boolean?` whether the PUT/POST/PATCH/whatever data should be transmitted. [1] I think your `http-conn-send!` could do similar? 2. Support for Content-Encoding response headers would also be helpful. Using the same make-pipe approach as you're doing with chunked transfer encoding. [2] Maybe this is mission creep: For HTTP 1.1. you _must_ support Transfer-Encoding: chunked, whereas Content-Encoding is just optional. However it's a good option; using compression can really help out on time as well as bandwidth charges. IIRC those were the two main things that motivated me to make my HTTP package at all, to support e.g. my AWS package. If http-client added them, I might not need my package anymore. (OK, it might take me awhile to phase it out until I'm ready to de-support older versions of Racket, but, I and others wouldn't need it for new projects.) [1]: https://github.com/greghendershott/http/blob/master/http/request.rkt#L142-L189 [2]: By the way, do you want to pass some `limit` optional arg in the various uses of `make-pipe`? Otherwise IIUC this will suck everything into RAM, which might not be so great with very large request or response entities. On Fri, Aug 23, 2013 at 2:48 PM, Jay McCarthy jay.mccar...@gmail.com wrote: Based on a request back in early July to remove the restrictions that net/url puts on HTTP communication (vis a vis URL encoding), I have just pushed a new HTTP client as net/http-client. The push also changes net/url to use net/http-client so that we just have 1 HTTP request producer, rather than 3. It passes all of the net/* tests, but these don't use features like proxying, HTTP/1.1, etc. I'm slightly nervous that it doesn't do those correct, but not super nervous, because I just cut-and-pasted the code. The main approach of the library is best explained by this contract: [http-sendrecv (-* ((or/c bytes? string?) (or/c bytes? string?)) (#:ssl? (or/c boolean? ssl-client-context? symbol?) #:port (between/c 1 65535) #:method (or/c bytes? string? symbol?) #:headers (listof (or/c bytes? string?)) #:data (or/c false/c bytes? string?)) (values bytes? (listof bytes?) input-port?))] Compared to net/url, - It supports bytes and strings everywhere - It supports data on every method and not just POST - It always returns the status line, headers, and content (as a port) I feel that the only thing it could do better is support two more options for #:data: - A input-port? to read from and copy to the HTTP connection - A (- output-port? void) function to call with the HTTP connection's output port to stream the data But I'd like a second opinion before adding them. Jay -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] net/http-client
On Mon, Aug 26, 2013 at 6:46 AM, Sam Tobin-Hochstadt sa...@cs.indiana.edu wrote: On Fri, Aug 23, 2013 at 2:48 PM, Jay McCarthy jay.mccar...@gmail.com wrote: I feel that the only thing it could do better is support two more options for #:data: - A input-port? to read from and copy to the HTTP connection - A (- output-port? void) function to call with the HTTP connection's output port to stream the data But I'd like a second opinion before adding them. Those both sound great, but why force the second one return `void?` For example, I might want to stream the data directly into a JSON parser. The function would be given to the client so it can write the data to the server. For example: (... #:data #Here's the post data) would be the same as (... #:data (\ (op) (display #Here's the post data op))) It's the HTTP connection's OUTPUT (to the server) port, not its INPUT (from the server) port. The return happens internally to the http-client and it doesn't have any reason to use any value produced. Jay -- Jay McCarthy j...@cs.byu.edu Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay The glory of God is Intelligence - DC 93 _ Racket Developers list: http://lists.racket-lang.org/dev
