Yarn has to be started explicitly. Usually it is part of Hadoop and is
started with Hadoop. Spark only contains the client for Yarn (afaik).
From: Miller, Clifford
Reply: user@predictionio.apache.org
Date: May 29, 2018 at 6:45:43 PM
To: user@predictionio.apache.org
Subject: Re: PIO
I use 'pio train -- --master yarn'
It works for me to train universal recommender
On Tue, May 29, 2018 at 8:31 PM, Miller, Clifford <
clifford.mil...@phoenix-opsgroup.com> wrote:
> To add more details to this. When I attempt to execute my training job
> using the command 'pio train -- --master
I recall at one point Spark switched to use per-thread classpath so that
each job would have its own isolated classpath. That was probably around
Spark 1.5 though, so not likely the exact same case here. From what version
of Spark to what version did you upgrade to?
On Tue, May 29, 2018 at 2:39
Yes, the spark-submit --jars is where we started to find the missing class.
The class isn’t found on the remote executor so we looked in the jars
actually downloaded into the executor’s work dir. the PIO assembly jars are
there are do have the classes. This would be in the classpath of the
Hi Nasos,
I believe that the template will need to be updated to use the
"org.apache.predictionio" group name. "io.prediction" was before the
project became an Apache one.
Regards,
Donald
On Fri, May 25, 2018 at 5:04 AM, Nasos Papageorgiou <
at.papageorg...@gmail.com> wrote:
> Hi all,
>
> I am
Sorry, what I meant was the actual spark-submit command that PIO was using.
It should be in the log.
What Spark version was that? I recall classpath issues with certain
versions of Spark.
On Thu, May 24, 2018 at 4:52 PM, Pat Ferrel wrote:
> Thanks Donald,
>
> We have:
>
>- built pio with
Thank you very much for your explanation. It all makes sense of course, i
guess as soon as i'll setup a cluster everything will be better (and more
manageable/predictable). Messing with hadoop, hdfs and hbase all in the
same machine seems not the way to go even in developement stage.
I'm also
No, this is as expected. When you run pseudo-distributed everything
internally is configured as if the services were on separate machines. See
clustered instructions here: http://actionml.com/docs/small_ha_cluster This
is to setup 3 machines running different parts and is not really the best
To add more details to this. When I attempt to execute my training job
using the command 'pio train -- --master yarn' I get the exception that
I've included below. Can anyone tell me how to correctly submit the
training job or what setting I need to change to make this work. I've made
not
i was able to solve the issue deleting hbase folder in hdfs with "hdfs dfs
-rm -r /hbase" and restarting hbase.
now app creation in pio is working again.
I still wonder why this problem happen though, i'm running hbase in
pseudo-distributed mode (for testing purposes everything, from spark to
10 matches
Mail list logo
