Just 2 cents on just one of contributors: while SQL semantic can express various use cases data scientists encounter, I also agree someone who are end users who are more familiar with code instead of SQL can feel it is not flexible.
But counterless efforts have been incorporated into Spark SQL (and catalyst) so I guess it is clear Spark SQL and Structured Streaming are the things if your workload fits into them, but on the other hand, if it doesn't, just keep using RDD. RDD is still the thing underlying Spark SQL, so I don't expect it is deprecated unless Spark renews the underlying architecture. -Jungtaek Lim 2018년 10월 29일 (월) 오전 12:06, Adrienne Kole <adrienneko...@gmail.com>님이 작성: > Thanks for bringing this issue to the mailing list. > As an addition, I would also ask the same questions about DStreams and > Structured Streaming APIs. > Structured Streaming is high level and it makes difficult to express all > business logic in it, although Databricks are pushing it and recommending > for usage. > Moreover, there are some works are going on continuous streaming. > So, what is the Spark's future vision, support all or concentrate on one, > as all those paradigms have separate processing semantics? > > > Cheers, > Adrienne > > On Sun, Oct 28, 2018 at 3:50 PM Soheil Pourbafrani <soheil.i...@gmail.com> > wrote: > >> Hi, >> There are some functions like map, flatMap, reduce and ..., that >> construct the base data processing operation in big data (and Apache >> Spark). But Spark, in new versions, introduces the high-level Dataframe API >> and recommend using it. This is while there are no such functions in >> Dataframe API and it just has many built-in functions and the UDF. It's >> very inflexible (at least to me) and I at many points should convert >> Dataframes to RDD and vice-versa. My question is: >> Is RDD going to be outdated and if so, what is the correct road-map to do >> processing using Apache Spark, while Dataframe doesn't support functions >> like Map and reduce? How UDF functions process the data, they will apply to >> every row, like map functions? Are converting Dataframe to RDD comes with >> many costs? >> >
