Hi, can you please give more details around this? What is the requirement? What is the SPARK version you are using? What do you mean by multiple sources? What are these sources?
Regards, Gourav Sengupta On Wed, Aug 25, 2021 at 3:51 AM Artemis User <arte...@dtechspace.com> wrote: > Thanks Daniel. I guess you were suggesting using DStream/RDD. Would it > be possible to use structured streaming/DataFrames for multi-source > streaming? In addition, we really need each stream data ingestion to be > asynchronous or non-blocking... thanks! > > On 8/24/21 9:27 PM, daniel williams wrote: > > Yeah. Build up the streams as a collection and map that query to the > start() invocation and map those results to awaitTermination() or whatever > other blocking mechanism you’d like to use. > > On Tue, Aug 24, 2021 at 4:37 PM Artemis User <arte...@dtechspace.com> > wrote: > >> Is there a way to run multiple streams in a single Spark job using >> Structured Streaming? If not, is there an easy way to perform inter-job >> communications (e.g. referencing a dataframe among concurrent jobs) in >> Spark? Thanks a lot in advance! >> >> -- ND >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> -- > -dan > > >
