Hi Mads, I agree with your proposal to have a transition period of having both and setting them by a config. I don't think it would make sense to have the optimizer choose from the two as it's the same platform and I assume the old Dataset implementation will be much slower as we have seen in the past.
Regarding supporting streams in general we need to come up with a plan on how this different execution model can be incorporated and if it would make sense to have pipelines that mix the streaming with the batch semantics. Maybe we bring this discussion in a separate thread. Best, Zoi On 2026/01/19 08:56:29 Mads Sejer Pedersen wrote: > Hi team, > > I've recently started work on our Flink platform, since Flink DataSets are > deprecated we should also slowly move away from Flink DataSets. > > I've just created a PR for bounded Flink streams, these look very much like > our old ones, but now comes the question of how we are going to integrate > these, such that our optimizer can use them. I suggest we have a > transitionary period where we support both DataSets & bounded streams, > perhaps you would choose either by config, or we allow the optimizer to chose. > > Lastly comes the question of whether and how we are going to support > continuous Flink DataStreams, these monitor and continuously perform > operations on new data, which is a break from the current model of Wayang. >
