Hi Hasan,

I think that you problem comes from using deserialized RDD, which already
lost its connection with SparkContext.
Similar case could be found here:
http://stackoverflow.com/questions/29567247/serializing-rdd

If you only really need SparkContext you could probably use the one
provided to PersitentModelLoader, which would be implemented by your model.
Alternatively you could also implement PersistentModel to return false from
save method. In this case your algorithm would be retrained on deploy, what
would also provide you with the instance of SparkContext.

Regards,
Marcin


czw., 22.09.2016 o 13:34 użytkownik Hasan Can Saral <hasancansa...@gmail.com>
napisał:

> Hi!
>
> I am trying to query Event Server with PEventStore api in predict method
> to fetch events per entity to create my features. PEventStore needs sc, and
> for this, I have:
>
> - Extended PAlgorithm
> - Extended LocalFileSystemPersistentModel and
> LocalFileSystemPersistentModelLoader
> - Put a dummy emptyRDD into my model
> - Tried to access sc with model.dummyRDD.context to receive this error:
>
> org.apache.spark.SparkException: RDD transformations and actions can only
> be invoked by the driver, not inside of other transformations; for example,
> rdd1.map(x => rdd2.values.count() * x) is invalid because the values
> transformation and count action cannot be performed inside of the rdd1.map
> transformation. For more information, see SPARK-5063.
>
> Just like this user got it here
> <https://groups.google.com/forum/#!topic/predictionio-user/h4kIltGIIYE> in
> predictionio-user group. Any suggestions?
>
> Here's a more of my predict method:
>
> def predict(model: SomeModel, query: Query): PredictedResult = {
>
>   def predict(model: SomeModel, query: Query): PredictedResult = {
>
>
>   val appName = sys.env.getOrElse[String]("APP_NAME", ap.appName)
>
>       var previousEvents = try {
>         PEventStore.find(
>           appName = appName,
>           entityType = Some(ap.entityType),
>           entityId = Some(query.entityId.getOrElse(""))
>         )(model.dummyRDD.context).map(event => {
>
>           Try(new CustomEvent(
>             Some(event.event),
>             Some(event.entityType),
>             Some(event.entityId),
>             Some(event.eventTime),
>             Some(event.creationTime),
>             Some(new Properties(
>               *...*
>             ))
>           ))
>         }).filter(_.isSuccess).map(_.get)
>       } catch {
>         case e: Exception => // fatal because of error, an empty query
>           logger.error(s"Error when reading events: ${e}")
>           throw e
>       }
>
>      ...
>
> }
>
>

Reply via email to