On 03-Feb-23 03:27, Daniel Stenberg via curl-library wrote:
There's always an upper bound - no one has 2^63 bytes of swap space, memory, or disk space to store an extremely large frame. And it's not likely in the foreseeable future.On Fri, 3 Feb 2023, Vitalii B. Avramenko via curl-library wrote:Such partial data may be OK for HTTP protocol when we know for sure that we have "request/response" pattern and we can detect the end of data by HTTP protocol itself, for example, with `Content-Length` header. But with websocket generally speaking we don't have any way to know where is end of frame with `CURLOPT_WRITEFUNCTION`.Yes we do: curl_ws_meta() is provided to give you exactly that information!we need a guarantee that `CURLOPT_WRITEFUNCTION` will call our callback when full frame is downloaded only, or at least we need the option that will allow us to request such behavior (something like `CURLOPT_WEBSOCKET_FULL_FRAMES_ONLY`).I have been thinking about adding a mode for the websocket API that delivers full frames only, but I have hesitated a bit since frames can be up to 2^63 bytes big we need to decide on how to handle (too) big frames for such a mode.What do you think is a reasonable behavior for a full-frame mode when it receives (ridiculously) large frames?
I think it's up to the application to decide what it's willing to handle. I don't think there's a universal answer of how. Maybe it calls getrlimit (RLIMIT_DATA) - or RLIMIT_FSIZE. Or it looks at free space on its output disk. Or bases it on estimated processing time. Or ...
For full frames, if you can't set an upper bound, your protocol user needs to rethink its usage. If your application really can deal with huge (beyond practical VM sized) data, it pretty much has to handle in in a stream - so FULL_FRAMES would be inappropriate.
So, here's a simple answer: Provide a setting for the maximum acceptable full frame size. On a FULL_FRAMES_ONLY connection, curl buffers any frame up to that size and provides it in the callback. Anything bigger (or curl can't allocate the buffer memory, times out waiting for it, etc) and curl returns an error (FRAME_TOO_BIG), aborts the connection and calls writefunction with NULL in the *ptr argument, and the actual size in 'size".
This provides the application with sufficient information to log the failure or even retry the request.
And to simplify the API, perhaps the setting should be "CURLOPT_WEBSOCKET_FULL_FRAMES_UPTO, <size>", and let zero be the current incremental delivery mode.
Vitalii can set <size> to a few GB if he can handle it. Or if he is willing to go until the OOM killer hits him, he can set size to 2^63-1 and see where fate takes him. Having lived thru "32 bits is so big that limits aren't necessary", I don't think that's a wise approach...
Timothe Litt ACM Distinguished Engineer -------------------------- This communication may not represent the ACM or my employer's views, if any, on the matters discussed.
OpenPGP_signature
Description: OpenPGP digital signature
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
