bq. be able to lookup from inside MapPartitions based on a key Please describe your use case in bit more detail. One possibility is to use NoSQL database such as HBase. There're several choices for Spark HBase connector.
Cheers On Tue, Dec 22, 2015 at 4:51 PM, Zhan Zhang <zzh...@hortonworks.com> wrote: > SQLContext is in driver side, and I don’t think you can use it in > executors. > How to provide lookup functionality in executors really depends on how you > would use them. > > Thanks. > > Zhan Zhang > > On Dec 22, 2015, at 4:44 PM, SRK <swethakasire...@gmail.com> wrote: > > > Hi, > > > > Can SQL Context be used inside mapPartitions? My requirement is to > register > > a set of data from hdfs as a temp table and to be able to lookup from > inside > > MapPartitions based on a key. If it is not supported, is there a > different > > way of doing this? > > > > Thanks! > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Can-SqlContext-be-used-inside-mapPartitions-tp25771.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
