Great! Ignite YARN cluster successfully started, but ignite spark shell doesn't see server nodes. You need to configure IP finder, by default YARN cluster using VmIpFinder, but ignite-spark MulticastIpFinder. Could you change configuration for YARN cluster (IGNITE_XML_CONFIG=/ignite/ release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml) to the following?
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd";> <bean class="org.apache.ignite.configuration.IgniteConfiguration"> <property name="discoverySpi"> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> <property name="ipFinder"> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"/> </property> </bean> </property> </bean> </beans> and also start IgniteContext with the same configuration? val ic = new IgniteContext[Integer, Integer](sc,"path to configuration file" ) On Thu, Aug 11, 2016 at 1:54 PM, percent620 <[email protected]> wrote: > 1、Adjusted cluster.properites as below > $cat cluster16.properties > # The number of nodes in the cluster. > IGNITE_NODE_COUNT=4 > # The number of CPU Cores for each Apache Ignite node. > IGNITE_RUN_CPU_PER_NODE=1 > # The number of Megabytes of RAM for each Apache Ignite node. > IGNITE_MEMORY_PER_NODE=2048 > # The version ignite which will be run on nodes. > IGNITE_VERSION=1.6.0 > IGNITE_WORK_DIR=/u01/yueyi/apache-ignite-fabric-1.6.0-bin/ > IGNITE_XML_CONFIG=/ignite/release16/apache-ignite-fabric-1.6.0-bin/config/ > default-config.xml > IGNITE_RELEASES_DIR=/ignite/release16/ > #IGNITE_USERS_LIBS=/u01/yueyi/apache-ignite-fabric-1.6.0-bin/libs/ > #IGNITE_HOSTNAME_CONSTRAINT=vmsecdomain010194070026.cm10 > IGNITE_PATH=/ignite/release16/apache-ignite-fabric-1.6.0-bin.zip > > 2、hdfs directory > $./hdfs dfs -ls /ignite/release16 > drwxr-xr-x - hbase hbase 0 2016-08-11 12:44 > /ignite/release16/apache-ignite-fabric-1.6.0-bin > -rw-r--r-- 3 hbase hbase 175866626 2016-08-11 18:31 > /ignite/release16/apache-ignite-fabric-1.6.0-bin.zip > > 3、yarn console > INFO: Application master registered. > Aug 11, 2016 6:32:57 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > Aug 11, 2016 6:32:57 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > Aug 11, 2016 6:32:57 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > Aug 11, 2016 6:32:57 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > 16/08/11 18:32:57 INFO impl.AMRMClientImpl: Received new token for : > xx1:29077 > 16/08/11 18:32:57 INFO impl.AMRMClientImpl: Received new token for : > xxx2:57492 > 16/08/11 18:32:57 INFO impl.AMRMClientImpl: Received new token for : > xxx3:59929 > 16/08/11 18:32:57 INFO impl.AMRMClientImpl: Received new token for : > xx4:23159 > > 5、contain logs as below > 1) > [18:33:13] Security status [authentication=off, tls/ssl=off] > [18:33:18] To start Console Management & Monitoring run > ignitevisorcmd.{sh|bat} > [18:33:18] > [18:33:18] Ignite node started OK (id=a060a3ee) > [18:33:18] Topology snapshot [ver=3, servers=3, clients=0, CPUs=72, > heap=6.0GB] > [18:33:19] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, > heap=8.0GB] > > 2)[18:33:13] Security status [authentication=off, tls/ssl=off] > [18:33:19] To start Console Management & Monitoring run > ignitevisorcmd.{sh|bat} > [18:33:19] > [18:33:19] Ignite node started OK (id=5c8dfd50) > [18:33:19] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, > heap=8.0GB] > > 3)[18:33:12] Ignite node started OK (id=4e75d238) > [18:33:12] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, > heap=2.0GB] > [18:33:14] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, > heap=4.0GB] > [18:33:17] Topology snapshot [ver=3, servers=3, clients=0, CPUs=72, > heap=6.0GB] > [18:33:19] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, > heap=8.0GB] > > 4)[18:33:14] Ignite node started OK (id=250fcb93) > [18:33:14] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, > heap=4.0GB] > [18:33:17] Topology snapshot [ver=3, servers=3, clients=0, CPUs=72, > heap=6.0GB] > [18:33:19] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, > heap=8.0GB] > > faced issues========= > > 1、spark-shell test > ./spark-shell --jars > /u01/xxx/apache-ignite-hadoop-1.6.0-bin/libs/ignite-core-1. > 6.0.jar,/u01/xxx/apache-ignite-hadoop-1.6.0-bin/libs/ > ignite-spark/ignite-spark-1.6.0.jar,/u01/xxx/apache-ignite- > hadoop-1.6.0-bin/libs/cache-api-1.0.0.jar,/u01/xxx/apache- > ignite-hadoop-1.6.0-bin/libs/ignite-log4j/ignite-log4j-1.6. > 0.jar,/u01/xxx/apache-ignite-hadoop-1.6.0-bin/libs/ignite- > log4j/log4j-1.2.17.jar > --packages > org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0 > > 2、SQL context available as sqlContext. > > scala> import org.apache.ignite.spark._ > import org.apache.ignite.spark._ > > scala> import org.apache.ignite.configuration._ > import org.apache.ignite.configuration._ > > scala> val ic = new IgniteContext[Integer, Integer](sc, () => new > IgniteConfiguration()) > > > > > the spark-shell hang here........ > > Thanks again!!!! > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6961.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
