[I] An SSE streamer for LLM , v2 [openserverless]

[via GitHub]

sciabarracom opened a new issue, #82:
URL: https://github.com/apache/openserverless/issues/82

   Spec for the streamer.
   
   The streamer is available as `stream.<apihost>` over https.
   
   Optionally we may have also `<user>.<apihost>/stream` that is acutually 
proxying to `stream.<apihost>/<user>`
   
   An user will stream an action `<package>/<action>` with:
   
   
   ```
   const url = "/stream/<package>/<action>"
   // or "https://stream.<apihost>/<user>/<package>/<action>"
   const evtSource = new EventSource(url, {
     withCredentials: true,
     // other parameters here
   });
   evtSource.onmessage = (event) => {
     const data = event.data
     // do something with the event data
   };
   ```
   
   The steamer once received the request, assuming it is a json post,
    - parses the json
    - opens a socket for listening in a random port 
    - invoke the action `<user>/<package>/<action>` asynchronously passing to 
the data additionally in the args the extra parameters `STREAM_HOST` and 
`STREAM_PORT`
   - wait in the socket the user writes the content to be streamed make a buffer
   - writes the content in the strem when it arrives
   - closes the socket once done
   
   The user will use the stremer as follows:
   
   ```python
   def main(args):
       streamer = (args.get("STREAM_HOST"), args.get("STREAM_PORT"))
       if not socket[0] or not socket[1]:
           return {"body": "pleae invoke with the stremer"}
       // invoke a call to a streaming api server like OpenAI
       with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
           s.connect(streamer)
           while <there are data>:
                   data = <read data>
                   s.sendall(data)
       return {"body": "done"}
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@openserverless.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org