I think it's Yarn based, setup through Ambari.
> On Mar 31, 2017, at 6:29 AM, Donald Szeto <[email protected]> wrote: > > Hi Malay, > > Is your Spark cluster a standalone deployment or based on YARN? > > Regards, > Donald > >> On Thu, Mar 30, 2017 at 11:48 PM Malay Tripathi <[email protected]> >> wrote: >> Hello, >> >> I am running pio train on an edge node of distributed 8 node spark cluster & >> 3 node Hbase. >> When I run "pio train" the job runs but it runs on local spark & not >> submitted to cluster. >> If I do "pio train --master spark://localhost:7077" or "pio train --master >> yarn-cluster" I get below error - >> File does not exist: >> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar >> >> java.io.FileNotFoundException: File does not exist: >> hdfs://mdc2vra179.federated.fds:8020/user/da_mcom_milan/.sparkStaging/application_1489598450058_0024/template-scala-parallel-universal-recommendation-assembly-0.5.0-deps.jar >> >> at >> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1427) >> >> at >> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419) >> >> >> >> mdc2vra179 is my Hbase cluster node, also running Namenode. Not sure why my >> spark expecting a jar file on hbase/Namenode. >> >> $PIO_HOME/conf/pio-env.sh- >> SPARK_HOME=$PIO_HOME/vendors/spark >> >> >> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase/conf >> >> PIO_FS_BASEDIR=$HOME/.pio_store >> >> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines >> >> >> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp >> >> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta >> >> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH >> >> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event >> >> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE >> >> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model >> >> >> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS >> >> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-prod >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176 >> >> >> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300 >> >> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models >> >> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase >> >> >> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase >> >> >> >> Thanks, >> >> Malay
