If I understand correctly, you need to create a UDF (if you are using java
Extend appropriate UDF e.g. UDF1, UDF2 ..etc depending on number of
arguments and have this static list as a member variable in your class.
You can use this udf as filter in your stream directly.
On Tue, Feb 21, 2017 at 8:59 PM, nitishdeshpande [via Apache Spark User
List] <ml-node+s1001560n28412...@n3.nabble.com> wrote:
> I have a situation where I have to filter data-points in a stream based on
> some condition involving a reference to external data. I have loaded up the
> external data in a Dataframe (so that I get to query on it using SQL
> interface). But when I tried to query on Dataframe I see that we cannot
> access it inside the transform (filter) function. (sample code below)
>
> // DStream is created and temp table called 'locations' is registered
> dStream.filter(dp => {
> val responseDf = sqlContext.sql("select location from
> locations where id='001'")
> responseDf.show() //nothing is displayed
> // some condition evaluation using responseDf
> true
> })
>
> Am I doing something wrong? If yes, then what would be a better approach
> to load external data in-memory and query it during stream transformation
> stage.
>
> Link to S.O question http://stackoverflow.com/questions/42362012/spark-
> streaming-using-external-data-during-stream-transformation
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-
> Streaming-Using-external-data-during-stream-transformation-tp28412.html
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=c21hcnRzaG9iaHVAZ21haWwuY29tfDF8MzU4MDIyNzAx>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Using-external-data-during-stream-transformation-tp28412p28415.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
