Hi Shane, I am not an expert but also came to PIO from a jupyter notebook background.
Yes, you can print and log dataframes as PIO is running. But I've been using the spark-shell interactively with the :paste and :load commands heavily. If you can get the dataframes loaded into the spark-shell then you can interactively explore them directly in the REPL, which is nice. >From pio you can also write your dataframe to csv files and then load it into the spark-shell to work with that way. As far as text editor support, I am using Sublime with Ensime http://ensime.org/editors/sublime/ . Ensime will catch type errors without having to rebuild the program, which speeds up the cycle a bit. Sublime doesn't have as many features as IntelliJ but it is probably easier to set. On Mon, May 22, 2017 at 4:31 PM Shane Johnson <[email protected]> wrote: > My apologies, my last e-mail was sent on accident. > > *Primary Question:* > > I am following this tutorial ( > http://predictionio.incubator.apache.org/resources/intellij/) and am > getting stuck running `pio train`. Error code is above. > > Exception in thread "main" java.lang.NoSuchMethodError: > com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy( > Lorg/objenesis/strategy/InstantiatorStrategy;)V > > *Secondary Question:* > > What is a good way to print out the contents of a RDD or dataframe when > working with PredictionIO. I am new to Scala and PredictionIO. My > experience is in Jupyter Notebooks and python where you can view the data > frames and objects as you are proving out the model. Is there a good method > to do this within the predictionIO development cycles. I was thinking > IntelliJ would be the answer but perhaps there is an easy way to print out > the dataframes in the logs as well and view it when you run `pio build`. > > Your mentorship is very much appreciated. > > class TrainingData( > val ratings: RDD[Rating] > ) extends Serializable { > override def toString = { > s"ratings: [${ratings.count()}] (${ratings.take(2).toList}...)" > } > } > > *Shane Johnson | 801.360.3350 <(801)%20360-3350>* > LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook > <https://www.facebook.com/shane.johnson.71653> > > On Mon, May 22, 2017 at 5:23 PM, Shane Johnson < > [email protected]> wrote: > >> Team, >> >> I am trying to get IntelliJ working with predictionIO. I have been >> following this turorial and am getting close to getting the PIO Train to >> work but it looks like it is failing as it goes to persist the model. This >> in turn seems to be causing the `pio deploy` in intelliJ to fail as well. >> >> What I really am trying to do is get into an environment where I can test >> the code and view the dataframes as they are being created. I am new to >> PredictionIO and Scala. I am trying to find a way to view the contents of >> class >> TrainingData( >> >> val ratings: RDD[Rating] >> ) extends Serializable { >> override def toString = { >> s"ratings: [${ratings.count()}] (${ratings.take(2).toList}...)" >> } >> } >> >> >> [INFO] [Engine] Extracting datasource params... >> [INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be >> used. >> [INFO] [Engine] Datasource params: (,DataSourceParams(MyApp1,None)) >> [INFO] [Engine] Extracting preparator params... >> [INFO] [Engine] Preparator params: (,Empty) >> [INFO] [Engine] Extracting serving params... >> [INFO] [Engine] Serving params: (,Empty) >> [INFO] [Remoting] Starting remoting >> [INFO] [Remoting] Remoting started; listening on addresses :[akka.tcp:// >> [email protected]:57099] >> [WARN] [AbstractLifeCycle] FAILED [email protected]:4040: >> java.net.BindException: Address already in use >> [WARN] [AbstractLifeCycle] FAILED >> org.spark-project.jetty.server.Server@53c1179a: java.net.BindException: >> Address already in use >> [WARN] [Utils] Service 'SparkUI' could not bind on port 4040. Attempting >> port 4041. >> [INFO] [Engine$] EngineWorkflow.train >> [INFO] [Engine$] DataSource: >> org.example.recommendation.DataSource@19bedeb8 >> [INFO] [Engine$] Preparator: >> org.example.recommendation.Preparator@7ef8eda7 >> [INFO] [Engine$] AlgorithmList: >> List(org.example.recommendation.ALSAlgorithm@7f287b98) >> [INFO] [Engine$] Data sanity check is on. >> [INFO] [Engine$] org.example.recommendation.TrainingData does not support >> data sanity check. Skipping check. >> [INFO] [Engine$] org.example.recommendation.PreparedData does not support >> data sanity check. Skipping check. >> [WARN] [BLAS] Failed to load implementation from: >> com.github.fommil.netlib.NativeSystemBLAS >> [WARN] [BLAS] Failed to load implementation from: >> com.github.fommil.netlib.NativeRefBLAS >> [WARN] [LAPACK] Failed to load implementation from: >> com.github.fommil.netlib.NativeSystemLAPACK >> [WARN] [LAPACK] Failed to load implementation from: >> com.github.fommil.netlib.NativeRefLAPACK >> [INFO] [Engine$] org.apache.spark.mllib.recommendation.ALSModel does not >> support data sanity check. Skipping check. >> [INFO] [Engine$] EngineWorkflow.train completed >> [INFO] [Engine] engineInstanceId=6fb987b7-7674-4b95-a938-a1190e215638 >> [INFO] [CoreWorkflow$] Inserting persistent model >> Exception in thread "main" java.lang.NoSuchMethodError: >> com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V >> at com.twitter.chill.KryoBase.setInstantiatorStrategy(KryoBase.scala:86) >> at >> com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:59) >> at >> com.twitter.chill.ScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:94) >> at >> org.apache.predictionio.workflow.KryoInstantiator.newKryo(CreateServer.scala:60) >> at >> org.apache.predictionio.workflow.KryoInstantiator.newKryo(CreateServer.scala:58) >> at com.twitter.chill.KryoPool$2.newInstance(KryoPool.java:59) >> at com.twitter.chill.KryoPool$2.newInstance(KryoPool.java:57) >> at com.twitter.chill.ResourcePool.borrow(ResourcePool.java:37) >> at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:114) >> at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:64) >> at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:55) >> at >> org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:81) >> at >> org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:250) >> at >> org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) >> >> *Shane Johnson | 801.360.3350 <(801)%20360-3350>* >> LinkedIn <https://www.linkedin.com/in/shanewjohnson> | Facebook >> <https://www.facebook.com/shane.johnson.71653> >> > >
