I still couldn't get it working on Yarn with spark-submit. Looking at the logs, the initial reading of default-config.xml succeeds and the ignite node is started OK, but is failing when it starts the Task. Please see below.
NB: I tried spark-submit from Spark Master node as well as the Spark client nodes but seeing the same error in both the cases. Is there a way to get this working on yarn with spark-submit? 17/11/06 06:06:26 main INFO XmlBeanDefinitionReader: Loading XML bean definitions from URL [file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml] 17/11/06 06:06:26 main INFO GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@77b27b57: startup date [Mon Nov 06 06:06:26 UTC 2017]; root of context hierarchy [06:06:26] __________ ________________ [06:06:26] / _/ ___/ |/ / _/_ __/ __/ [06:06:26] _/ // (7 7 // / / / / _/ [06:06:26] /___/\___/_/|_/___/ /_/ /___/ [06:06:26] [06:06:26] ver. 2.1.0#20170720-sha1:a6ca5c8a [06:06:26] 2017 Copyright(C) Apache Software Foundation [06:06:26] [06:06:26] Ignite documentation: http://ignite.apache.org [06:06:26] [06:06:26] Quiet mode. [06:06:26] ^-- Logging to file '/home/ec2-user/ apache-ignite-fabric-2.1.0-bin/work/log/ignite-bd37708a.0.log' [06:06:26] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [06:06:26] [06:06:26] OS: Linux 4.9.43-17.38.amzn1.x86_64 amd64 [06:06:26] VM information: Java(TM) SE Runtime Environment 1.8.0_60-b27 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.60-b23 [06:06:27] Configured plugins: [06:06:27] ^-- None [06:06:27] [06:06:27] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [06:06:27] Security status [authentication=off, tls/ssl=off] [06:06:27] REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property. [06:06:28] Performance suggestions for grid (fix if possible) [06:06:28] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [06:06:28] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [06:06:28] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [06:06:28] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [06:06:28] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [06:06:28] [06:06:28] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [06:06:28] *[06:06:28] Ignite node started OK (id=bd37708a)* *[06:06:28] Topology snapshot [ver=112, servers=4, clients=2, CPUs=40, heap=13.0GB]* *17/11/06 06:06:29 dispatcher-event-loop-4 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, ip-รง-162.us-west-2.compute.internal, executor 1, partition 1, PROCESS_LOCAL, 6495 bytes)* *17/11/06 06:06:29 task-result-getter-0 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, ip-10-xx-xx-162.us-west-2.compute.internal, executor 1): class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml, err=IOException parsing XML document from URL [file:/home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml]; nested exception is java.io.FileNotFoundException: /home/ec2-user/apache-ignite-fabric-2.1.0-bin/config/default-config.xml (Permission denied)]* On Fri, Nov 3, 2017 at 5:21 PM, roshan joe <[email protected]> wrote: > Thanks Alexey. I believe I have tried all the 4 methods you suggested > below without any success. > > 1. Specified the absolute path of the default-config.xml using the > file:/// as well as hdfs:/// > > 2. passed the config file as file parameter to spark-submit as > --files=file:///.../..../apache-ignite-fabric-2.1.0- > bin/config/default-config.xml > > 3. exported IGNITE_HOME=/..../..../apache-ignite-fabric-2.1.0-bin/ in all > the spark nodes and passed "config/default-config.xml" as relative path > > 4. created the config file under "src/resources/" but no luck. > > Please see the complete logs. > > Also, I have the default-config.xml file at the default location on all > spark nodes. Do I need to pass it in the code then? If I do not pass it, ""it > cannot find any IPs from multicast or IPFinder. " > > > Thank you. > > > > On Fri, Nov 3, 2017 at 6:52 AM, Alexey Kukushkin < > [email protected]> wrote: > >> Hi, >> >> The problem is Ignite cannot find your configuration file. >> >> Ignite tries these 3 steps to find configuration file: >> >> 1. Try to resolve it as a URI >> 2. If previous step fails, try to resolve it as $IGNITE_HOME + <the >> path you specified> >> 3. If previous step fails, try to resolve it in CLASSPATH. >> >> Thus, you have multiple options to specify config path: >> >> 1. Specify an absolute path >> 2. Pass the config path as a command line parameter >> 3. Create your path relative to $IGNITE_HOME >> 4. If you use maven or gradle java plugin, then you can create file >> under "resources/path-to-ignite-config.xml". On the "process >> resources" phase maven will copy it under >> target/classes/path-to-ignite-config.xml" >> and then it will go into JAR. Your app will find it by >> "path-to-ignite-config.xml" since it will be in CLASSPATH. >> >> >> > >
