Hi, I am using IgniteRDD to query data from Ignite In memory cache cluster.I have setup the cluster on AWS. My Spark and Ignite Nodes are running on same AWS nodes. >From the logs I can see that it take around 3-5 seconds on every executor to >establish connection with Ignite Cluster.Is there is any way to make it faster. Please find the below logs indicating task started at 10:32:41 and till 10:32:45 it was trying to make connection with Ignite cluster.Is there is anyway to improve this ?
16/04/27 10:32:41 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/./riskAnalytics-0.0.1-SNAPSHOT.jar!/config/aws-cloud/risk-analytics-cache-client-aws.xml] 16/04/27 10:32:41 INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@6e7a6652: startup date [Wed Apr 27 10:32:41 UTC 2016]; root of context hierarchy 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 1.6.0-SNAPSHOT#19700101-sha1:DEV >>> 2016 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Config URL: n/a 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Daemon mode: off 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: OS: Linux 4.1.17-22.30.amzn1.x86_64 amd64 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: OS user: yarn 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Language runtime: Scala ver. 2.10.5 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM information: OpenJDK Runtime Environment 1.8.0_77-b03 Oracle Corporation OpenJDK 64-Bit Server VM 25.77-b03 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM total memory: 4.9GB 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Remote Management [restart: off, REST: on, JMX (remote: off)] 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: IGNITE_HOME=null 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: VM arguments: [-XX:OnOutOfMemoryError=kill %p, -Xms5120m, -Xmx5120m, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxHeapFreeRatio=70, -XX:+CMSClassUnloadingEnabled, -XX:OnOutOfMemoryError=kill -9 %p, -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/tmp, -Dspark.history.ui.port=18080, -Dspark.driver.port=50884, -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1461736583050_0013/container_1461736583050_0013_01_000002] 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache', 'default'] 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Non-loopback local IPs: 172.31.11.240, fe80:0:0:0:83f:42ff:fecd:2909%eth0 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Enabled local MACs: 0A3F42CD2909 16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor: Configured plugins: 16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor: ^-- None 16/04/27 10:32:42 INFO plugin.IgnitePluginProcessor: 16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: IPC shared memory server endpoint started [port=48102, tokDir=/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/tmp/ignite/work/ipc/shmem/98a94f2f-4f19-4430-bfe1-930312846191-87676] 16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: Successfully bound shared memory communication to TCP port [port=48102, locHost=0.0.0.0/0.0.0.0] 16/04/27 10:32:42 INFO tcp.TcpCommunicationSpi: Successfully bound to TCP port [port=47102, locHost=0.0.0.0/0.0.0.0] 16/04/27 10:32:42 WARN noop.NoopCheckpointSpi: Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) 16/04/27 10:32:42 WARN collision.GridCollisionManager: Collision resolution is disabled (all jobs will be activated upon arrival). 16/04/27 10:32:42 WARN noop.NoopSwapSpaceSpi: Swap space is disabled. To enable use FileSwapSpaceSpi. 16/04/27 10:32:42 INFO internal.IgniteKernal%riskAnalyticsGrid: Security status [authentication=off, tls/ssl=off] 16/04/27 10:32:43 INFO tcp.GridTcpRestProtocol: Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11212] 16/04/27 10:32:43 WARN s3.TcpDiscoveryS3IpFinder: Amazon client configuration is not set (will use default). 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode [id=623ea12b-cabb-4c1d-a13c-ac340c2f05f4, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.239], sockAddrs=[ip-172-31-11-239.us-west-2.compute.internal/172.31.11.239:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.31.11.239:0], discPort=0, order=72, intOrder=40, lastExchangeTime=1461753163906, loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true] 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=72, servers=3, clients=5, CPUs=56, heap=60.0GB] 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode [id=71337db2-7548-49e5-91e5-12544df08cc0, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.239], sockAddrs=[ip-172-31-11-239.us-west-2.compute.internal/172.31.11.239:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.31.11.239:0], discPort=0, order=73, intOrder=41, lastExchangeTime=1461753163906, loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true] 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=73, servers=3, clients=6, CPUs=56, heap=65.0GB] 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Added new node to topology: TcpDiscoveryNode [id=1894483f-1b3f-42d1-a336-9659ec81c339, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.31.11.241], sockAddrs=[ip-172-31-11-241.us-west-2.compute.internal/172.31.11.241:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.31.11.241:0], discPort=0, order=74, intOrder=42, lastExchangeTime=1461753163906, loc=false, ver=1.6.0#19700101-sha1:00000000, isClient=true] 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=74, servers=3, clients=7, CPUs=56, heap=70.0GB] 16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-sys-cache, mode=REPLICATED] 16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-atomics-sys-cache, mode=PARTITIONED] 16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=default, mode=PARTITIONED] 16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=ignite-marshaller-sys-cache, mode=REPLICATED] 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: Performance suggestions for grid 'riskAnalyticsGrid' (fix if possible) 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: ^-- Decrease number of backups (set 'keyBackups' to 0) 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: 16/04/27 10:32:44 INFO internal.IgniteKernal%riskAnalyticsGrid: >>> +------------------------------------------+ >>> Ignite ver. 1.6.0-SNAPSHOT#19700101-sha1:DEV >>> +------------------------------------------+ >>> OS name: Linux 4.1.17-22.30.amzn1.x86_64 amd64 >>> CPU(s): 16 >>> Heap: 5.0GB >>> VM name: 87676@ip-172-31-11-240 >>> Grid name: riskAnalyticsGrid >>> Local node [ID=98A94F2F-4F19-4430-BFE1-930312846191, order=71, >>> clientMode=true] >>> Local node addresses: >>> [ip-172-31-11-240.us-west-2.compute.internal/0:0:0:0:0:0:0:1%lo, >>> /127.0.0.1, /172.31.11.240] >>> Local ports: TCP:11212 TCP:47102 TCP:48102 16/04/27 10:32:44 INFO discovery.GridDiscoveryManager: Topology snapshot [ver=74, servers=3, clients=7, CPUs=56, heap=70.0GB] 16/04/27 10:32:44 INFO cache.GridCacheProcessor: Started cache [name=StockSimulationsCacheWriterPartitioned, mode=PARTITIONED] 16/04/27 10:32:45 INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1461736583050_0013/container_1461736583050_0013_01_000002/./riskAnalytics-0.0.1-SNAPSHOT.jar!/config/aws-cloud/risk-analytics-cache-client-aws.xml] 16/04/27 10:32:45 INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@7577a3f7: startup date [Wed Apr 27 10:32:45 UTC 2016]; root of context hierarchy 16/04/27 10:32:45 INFO executor.Executor: Finished task 5.0 in stage 0.0 (TID 5). 1224 bytes result sent to driver Regards,Vij
