mdedetrich commented on PR #2409: URL: https://github.com/apache/pekko/pull/2409#issuecomment-3468007365
> This sounds like the resulting bytes will be different depending on how the input was split, which could be nondeterministic (e.g. depending on network conditions). I guess it's fine since this will most likely be used for 'ephemeral' streams, we can 'blame' the nondeterminism on the component that does the splitting, and someone who cares about determinism could insert a component to convert the input into fixed-size chunks. Yes this is precisely my thinking as well. Hypothetically speaking someone could just feed 1gbyte sized chunks as individual stream elements but if you are doing that then your already in a troublesome spot, regardless of compression or not. > This is a trade-off between latency and compression performance: in a naive implementation, if we for instance 'keep asking elements to buffer' on the boundary of 2 elements of a gRPC stream, it would delay sending+processing the first element while waiting for the next element to show up - which could take a long time on 'event streams', for example. It's possible to avoid this problem: we could 'keep asking for elements to buffer' only when there's currently no demand from downstream, but compress and send 'everything we have' as soon as demand is signaled. That might be significantly more complex, though. Good point, I did miss that. I am currently doing an initial implementation that follows this logic, but it can be improved on later to bypass the accumulation of data if downstream is demanding data right now. I assume that you would use `hasBeenPulled(port)` to figure out if there is downstream demand? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
