GitHub user mdedetrich added a comment to the discussion: Artifical back pressure on JDBC projections
> Fan-in/fan-out is still about connected streams, here I do not think the > streams are connected. Yes thats the entire issue I am pointing out > What you linked in R2DBC is offset store, not journal queries, but journal > queries also operate on `Future` too. True, not that familiar with projection but I remember the few times I worked on it I noticed that it was entirely `Future` based > They still convert to a `SourceProvider` down the line and then the > backpressure starts to work. The end-to-end streaming is likely not an option > at a query level, because queries do flip flop between live data and > backtracking between batches, so while I guess you could do live query as a > stream, backtracking would still need to be done in batches at certain > intervals. As I said I don't really know the details as I haven't worked with projection that much specifically, but if we are talking about pekko/reactive streams they are only as good as the weakest link, or to be put differently the minute you have some part of the stream thats disconnected (i.e. not a stream anymore) then you can lose a lot of the benefits, and jdbc is definitely not a streaming safe API. In any case, I am not sure what is meant be backtracking here (ill look into it) but on the surface I am not entirely convinced that it can't be represented as a stream that can be automatically backpressured. In any case, r2dbc projection not being a pure stream is kind of an issue design wise anyways GitHub link: https://github.com/apache/pekko/discussions/2431#discussioncomment-14894479 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
