A source build did not fix the problem, has anyone run PIO 0.12.1 on a
Spark cluster? The issue seems to be how to pass the correct code to Spark
to connect to HBase:
[ERROR] [TransportRequestHandler] Error while invoking RpcHandler#receive()
for one-way message.
[ERROR] [TransportRequestHandler]
Same CLI works using local Spark master, but fails using remote master for
a cluster due to a missing class def for protobuf used in hbase. We are
using the binary dist 0.12.1. Is this known? Is there a work around?
We are now trying a source build in hope the class will be put in the
assembly
I noticed the appName is different for DataSource (“shop _live”) and
Algorithm (“shop_live”). AppNames must match.
Also the eventNames are different, which should be ok but it’s still a
question. Why input something that is not used? Given the meaning of the
events, I’d use them all for
Hello again,
After moving hbase to dataproc cluster from docker ( probs dns/hostname
resolution issues ) no more hbase error but still training stops:
[INFO] [RecommendationEngine$]
_ _ __ __ _
/\ | | (_) | \/ | |
/ \ ___| |_ _ ___
Hi,
Ok so full command now is:
pio train --scratch-uri hdfs://pio-cluster-m/pio -- --executor-memory 4g
--driver-memory 4g --deploy-mode cluster --master yarn
errors stopped after removing –executor-cores 2 --driver-cores 2
I found this error: Uncaught exception:
Hi wojciech,
I also faced many problems while setting yarn with PredictionIO. This may
be the case where yarn is tyring to findout pio.log file on hdfs cluster.
You can try "--master yarn --deploy-mode client ". you need to pass this
configuration with pio train
e.g., pio train -- --master yarn
