Just to contribute to the discussion, when we tried to do the migration we faced some problems that could make migration quite difficult. 1 - It's difficult to test because of https://issues.apache.org/jira/browse/FLINK-18647 2 - missing mapPartition 3 - missing <X> DataSet<X> runOperation(CustomUnaryOperation<T, X> operation)
On Thu, Jul 30, 2020 at 12:40 PM Arvid Heise <ar...@ververica.com> wrote: > +1 of getting rid of the DataSet API. Is DataStream#iterate already > superseding DataSet iterations or would that also need to be accounted for? > > In general, all surviving APIs should also offer a smooth experience for > switching back and forth. > > On Thu, Jul 30, 2020 at 9:39 AM Márton Balassi <balassi.mar...@gmail.com> > wrote: > > > Hi All, > > > > Thanks for the write up and starting the discussion. I am in favor of > > unifying the APIs the way described in the FLIP and deprecating the > DataSet > > API. I am looking forward to the detailed discussion of the changes > > necessary. > > > > Best, > > Marton > > > > On Wed, Jul 29, 2020 at 12:46 PM Aljoscha Krettek <aljos...@apache.org> > > wrote: > > > >> Hi Everyone, > >> > >> my colleagues (in cc) and I would like to propose this FLIP for > >> discussion. In short, we want to reduce the number of APIs that we have > >> by deprecating the DataSet API. This is a big step for Flink, that's why > >> I'm also cross-posting this to the User Mailing List. > >> > >> FLIP-131: http://s.apache.org/FLIP-131 > >> > >> I'm posting the introduction of the FLIP below but please refer to the > >> document linked above for the full details: > >> > >> -- > >> Flink provides three main SDKs/APIs for writing Dataflow Programs: Table > >> API/SQL, the DataStream API, and the DataSet API. We believe that this > >> is one API too many and propose to deprecate the DataSet API in favor of > >> the Table API/SQL and the DataStream API. Of course, this is easier said > >> than done, so in the following, we will outline why we think that having > >> too many APIs is detrimental to the project and community. We will then > >> describe how we can enhance the Table API/SQL and the DataStream API to > >> subsume the DataSet API's functionality. > >> > >> In this FLIP, we will not describe all the technical details of how the > >> Table API/SQL and DataStream will be enhanced. The goal is to achieve > >> consensus on the idea of deprecating the DataSet API. There will have to > >> be follow-up FLIPs that describe the necessary changes for the APIs that > >> we maintain. > >> -- > >> > >> Please let us know if you have any concerns or comments. Also, please > >> keep discussion to this ML thread instead of commenting in the Wiki so > >> that we can have a consistent view of the discussion. > >> > >> Best, > >> Aljoscha > >> > > > > -- > > Arvid Heise | Senior Java Developer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Toni) Cheng
