Re: Ignite YARN deployment mode issues

[ilya.kasnacheev]

Hello Ray,

Yes, please try it and tell if it helps.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

[Ray]

Hi Ilya

Do you mean modify the original config from 
 
 
 
 
 

to 

 
 
 
 

if my understanding is correct?

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

[Ray]

Thanks for the reply, Ilya.

The IGNITE_HOME for yarn is
/yarn/nm/usercache/username/appcache/application_appid/container_containerID/ignite/

The log4j2.xml is under IGNITE_HOME/bin folder too.
If I launch ignite using ignist.sh, it's working fine and the log4j2.xml
under  IGNITE_HOME/bin folder  can be picked up by ignite.
But for YARN deployment, the issue still exists.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite YARN deployment mode issues

[Ilya Kasnacheev]

Hello Ray!

-- 
Ilya Kasnacheev

2017-09-19 10:51 GMT+03:00 Ray :

> I'm using ignite 2.1, my ignite config xml is as follows.
>
> I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not
> working.
>

I guess it should be relative to IGNITE_HOME, or absolute path (I expect
it's no-no with YARN).

What is IGNITE_HOME exactly in case of YARN, I will try to find out.

-- 
Ilya Kasnacheev

Re: Ignite YARN deployment mode issues

[Ray]

Figured the second question out by myself.
The node I'm running ./control.sh does not seem to have ignite grid started.
Here's the yarn log.



[08:09:30,866][SEVERE][main][IgniteKernal] Exception during start
processors, node will be stopped and close connections
class org.apache.ignite.IgniteException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_76/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.(GridCacheDatabaseSharedManager.java:2931)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:374)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:696)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:929)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: java.io.FileNotFoundException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_76/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.(GridCacheDatabaseSharedManager.java:2925)
... 16 more
[08:09:30,868][SEVERE][main][IgniteKernal] Got exception while starting
(will rollback startup routine).
class org.apache.ignite.IgniteException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_76/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.(GridCacheDatabaseSharedManager.java:2931)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:374)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:696)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:929)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: java.io.FileNotFoundException:

Ignite YARN deployment mode issues

[Ray]

I'm using ignite 2.1, my ignite config xml is as follows.

   












And in standalone mode, ignite can read the log4j2.xml in local directory
and works ok.
When deployed in YARN, ignite says it can find log4j2.xml with a bunch of
spring exceptions.
My cluster.properties is as follows 

IGNITE_NODE_COUNT=6
IGNITE_RUN_CPU_PER_NODE=6
IGNITE_MEMORY_PER_NODE=3
IGNITE_VERSION=2.1.0
IGNITE_PATH=/***/apache-ignite-fabric-2.1.0.zip
IGNITE_XML_CONFIG=/***/ignite-config/default-config.xml
IGNITE_USERS_LIBS=/***/ignite-libs/

I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not
working.

And another question is how to activate Ignite with persistent store enabled
in YARN mode.
I tried running ./control.sh --host ignitenode --port 11211 --activate
And the log says.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl 
WARNING: Failed to initialize topology on client start. Will retry in
background.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl 
INFO: Client started [id=5f00be2b-7679-46e7-9f8e-3435f7f1d759, protocol=TCP]
Something fail during activation, exception message: Latest topology update
failed.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl stop
INFO: Client stopped [id=5f00be2b-7679-46e7-9f8e-3435f7f1d759,
waitCompletion=true]
And there's no log in the server node side.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/