sciabarracom opened a new issue, #79: URL: https://github.com/apache/openserverless/issues/79
## Why you need it? LLm applications require a way to stream the results as they are generated ## How it could be? We need a stereaming proxy. It should work this way. Since actions while produce input cannot communicate back with the frontend, they should write the temporary content in redis. The proxy is mounted on `/stream` of each user with an ingress, and when it receive a request `/stream/<key>` it will stream the content as it arrives in the key `<user>:<key>` ## Other related information We should explore the best way to execute the request. We should be able to execute a request liket `how can I become rich` and forget. Normally openwhisk returns an activation id, and the result should be written in redis using this id for the streaming . So an asyncrhonous request should be more or less: 1. webaction invoking an asyncrhonous request and returning the activation id 2. action talking to an LLM and writing the result to redis in the key corresponding to the activtion id -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
