Daniel, Thank you for your mentorship. I am using both of your methods
- pio shell (:paste and :load) - ensime and I am cruising. I appreciate your help getting me past a roadblock and will continue developing with sublime! Thanks *Shane Johnson | 801.360.3350* LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook <https://www.facebook.com/shane.johnson.71653> On Mon, May 22, 2017 at 5:58 PM, Daniel Gabrieli <[email protected]> wrote: > Hi Shane, > > I am not an expert but also came to PIO from a jupyter notebook background. > > Yes, you can print and log dataframes as PIO is running. But I've been > using the spark-shell interactively with the :paste and :load commands > heavily. If you can get the dataframes loaded into the spark-shell then > you can interactively explore them directly in the REPL, which is nice. > From pio you can also write your dataframe to csv files and then load it > into the spark-shell to work with that way. > > As far as text editor support, I am using Sublime with Ensime > http://ensime.org/editors/sublime/ . Ensime will catch type errors > without having to rebuild the program, which speeds up the cycle a bit. > Sublime doesn't have as many features as IntelliJ but it is probably easier > to set. > > > > > On Mon, May 22, 2017 at 4:31 PM Shane Johnson < > [email protected]> wrote: > >> My apologies, my last e-mail was sent on accident. >> >> *Primary Question:* >> >> I am following this tutorial (http://predictionio.incubator.apache.org/ >> resources/intellij/) and am getting stuck running `pio train`. Error >> code is above. >> >> Exception in thread "main" java.lang.NoSuchMethodError: >> com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/ >> objenesis/strategy/InstantiatorStrategy;)V >> >> *Secondary Question:* >> >> What is a good way to print out the contents of a RDD or dataframe when >> working with PredictionIO. I am new to Scala and PredictionIO. My >> experience is in Jupyter Notebooks and python where you can view the data >> frames and objects as you are proving out the model. Is there a good method >> to do this within the predictionIO development cycles. I was thinking >> IntelliJ would be the answer but perhaps there is an easy way to print out >> the dataframes in the logs as well and view it when you run `pio build`. >> >> Your mentorship is very much appreciated. >> >> class TrainingData( >> val ratings: RDD[Rating] >> ) extends Serializable { >> override def toString = { >> s"ratings: [${ratings.count()}] (${ratings.take(2).toList}...)" >> } >> } >> >> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >> <https://www.facebook.com/shane.johnson.71653> >> >> On Mon, May 22, 2017 at 5:23 PM, Shane Johnson < >> [email protected]> wrote: >> >>> Team, >>> >>> I am trying to get IntelliJ working with predictionIO. I have been >>> following this turorial and am getting close to getting the PIO Train to >>> work but it looks like it is failing as it goes to persist the model. This >>> in turn seems to be causing the `pio deploy` in intelliJ to fail as well. >>> >>> What I really am trying to do is get into an environment where I can >>> test the code and view the dataframes as they are being created. I am new >>> to PredictionIO and Scala. I am trying to find a way to view the contents >>> of class TrainingData( >>> >>> val ratings: RDD[Rating] >>> ) extends Serializable { >>> override def toString = { >>> s"ratings: [${ratings.count()}] (${ratings.take(2).toList}...)" >>> } >>> } >>> >>> >>> [INFO] [Engine] Extracting datasource params... >>> [INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be >>> used. >>> [INFO] [Engine] Datasource params: (,DataSourceParams(MyApp1,None)) >>> [INFO] [Engine] Extracting preparator params... >>> [INFO] [Engine] Preparator params: (,Empty) >>> [INFO] [Engine] Extracting serving params... >>> [INFO] [Engine] Serving params: (,Empty) >>> [INFO] [Remoting] Starting remoting >>> [INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp:// >>> [email protected]:57099] >>> [WARN] [AbstractLifeCycle] FAILED [email protected]:4040: >>> java.net.BindException: Address already in use >>> [WARN] [AbstractLifeCycle] FAILED org.spark-project.jetty. >>> server.Server@53c1179a: java.net.BindException: Address already in use >>> [WARN] [Utils] Service 'SparkUI' could not bind on port 4040. Attempting >>> port 4041. >>> [INFO] [Engine$] EngineWorkflow.train >>> [INFO] [Engine$] DataSource: org.example.recommendation. >>> DataSource@19bedeb8 >>> [INFO] [Engine$] Preparator: org.example.recommendation. >>> Preparator@7ef8eda7 >>> [INFO] [Engine$] AlgorithmList: List(org.example. >>> recommendation.ALSAlgorithm@7f287b98) >>> [INFO] [Engine$] Data sanity check is on. >>> [INFO] [Engine$] org.example.recommendation.TrainingData does not >>> support data sanity check. Skipping check. >>> [INFO] [Engine$] org.example.recommendation.PreparedData does not >>> support data sanity check. Skipping check. >>> [WARN] [BLAS] Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemBLAS >>> [WARN] [BLAS] Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefBLAS >>> [WARN] [LAPACK] Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemLAPACK >>> [WARN] [LAPACK] Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefLAPACK >>> [INFO] [Engine$] org.apache.spark.mllib.recommendation.ALSModel does >>> not support data sanity check. Skipping check. >>> [INFO] [Engine$] EngineWorkflow.train completed >>> [INFO] [Engine] engineInstanceId=6fb987b7-7674-4b95-a938-a1190e215638 >>> [INFO] [CoreWorkflow$] Inserting persistent model >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy( >>> Lorg/objenesis/strategy/InstantiatorStrategy;)V >>> at com.twitter.chill.KryoBase.setInstantiatorStrategy(KryoBase.scala:86) >>> at com.twitter.chill.EmptyScalaKryoInstantiator. >>> newKryo(ScalaKryoInstantiator.scala:59) >>> at com.twitter.chill.ScalaKryoInstantiator.newKryo( >>> ScalaKryoInstantiator.scala:94) >>> at org.apache.predictionio.workflow.KryoInstantiator. >>> newKryo(CreateServer.scala:60) >>> at org.apache.predictionio.workflow.KryoInstantiator. >>> newKryo(CreateServer.scala:58) >>> at com.twitter.chill.KryoPool$2.newInstance(KryoPool.java:59) >>> at com.twitter.chill.KryoPool$2.newInstance(KryoPool.java:57) >>> at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:37) >>> at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114) >>> at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:64) >>> at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:55) >>> at org.apache.predictionio.workflow.CoreWorkflow$. >>> runTrain(CoreWorkflow.scala:81) >>> at org.apache.predictionio.workflow.CreateWorkflow$.main( >>> CreateWorkflow.scala:250) >>> at org.apache.predictionio.workflow.CreateWorkflow.main( >>> CreateWorkflow.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke( >>> NativeMethodAccessorImpl.java:62) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke( >>> DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) >>> >>> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >>> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >>> <https://www.facebook.com/shane.johnson.71653> >>> >> >>
