Hi, What about memory sink? That could work.
Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Mon, Dec 11, 2017 at 3:28 AM, satyajit vegesna < satyajit.apas...@gmail.com> wrote: > Hi All, > > I would like to infer JSON schema from a sample of data that i receive > from, Kafka Streams(specific topic), and i have to infer the schema as i am > going to receive random JSON string with different schema for each topic, > so i chose to go ahead with below steps, > > a. readStream from Kafka(latest offset), from a single Kafka topic. > b. Some how to store the JSON string into val and infer the schema. > c. stop the stream. > d.Create new readStream(smallest offset) and use the above inferred schema > to process the JSON using spark provided JSON support, like from_json, > json_object and others and run my actuall business logic. > > Now i am not sure how to be successful with step(b). Any help would be > appreciated. > And would also like to know if there is any better approach. > > Regards, > Satyajit. >
