Re: Ignite yarn resources keep on increasing

[ChandanS]

The problem is I am not able to kill my ignite yarn application even though
exception has been thrown while starting the ignite and the resources of the
ignite-yarn spark job keeps piling up. Please find attached image of spark
web-ui. From my "StartStandalone" spark application I use to submit the
ignite-yarn job "ignite-titan-resourceUP". You can see resources going up
for ignite-yarn job though it was submitted with same resources as parent
spark application. The only difference with negative scenario is I give the
ignite_path wrong.


 

 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite yarn resources keep on increasing

[Ilya Kasnacheev]

Hello!

I still have trouble understanding where does your problem lie, in Ignite
terms (as opposed to Yarn or ZooKeeper terms).

Why did your negative scenario fail? What is the difference in procedure
used?

Regards,
-- 
Ilya Kasnacheev


чт, 20 февр. 2020 г. в 08:37, ChandanS :

> Hi Ilya,
>
> Yes, I start the ignite in client mode and my other applications access
> this
> ignite cluster. I use to start the ignite cluster with 40-executors -and
> 2-cores. Usually, in success scenario my starting ignite and data loading
> finish within 5/6 minutes. The logs are for negative scenario, while my
> jobs
> are running for long time  and I have killed the long running job.
>
> Below config xml I have used for ignite:
>
> http://www.springframework.org/schema/beans;
>xmlns:util="http://www.springframework.org/schema/util;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> classpath:/org/springframework/beans/factory/xml/spring-beans-4.3.xsd">
>
>
>   
>   
>   
>
>   
>   class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> 
> 
>  
>   
>
>   
>   class="org.apache.ignite.spi.discovery.zk.ZookeeperDiscoverySpi">
>  value="x.int.westgroup.com:2181,x.int.westgroup.com:2181,
> x.int.westgroup.com:2181"/>
> 
>  value="/apacheIgnite-titan-resourceUP"/>
>  
>   
>
>   
>   class="org.apache.ignite.configuration.ClientConnectorConfiguration">
> 
> 
>  
>   
>
>   
>   class="org.apache.ignite.configuration.TransactionConfiguration">
>  value="6"/>
>  
>   
>
> 
>
>
> Thanks,
> Chandan
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Ilya,

Yes, I start the ignite in client mode and my other applications access this
ignite cluster. I use to start the ignite cluster with 40-executors -and
2-cores. Usually, in success scenario my starting ignite and data loading
finish within 5/6 minutes. The logs are for negative scenario, while my jobs
are running for long time  and I have killed the long running job.

Below config xml I have used for ignite:

http://www.springframework.org/schema/beans;
   xmlns:util="http://www.springframework.org/schema/util; 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
  
classpath:/org/springframework/beans/factory/xml/spring-beans-4.3.xsd">
   

  
  
  

  
 


 
  

  
 

  

 
  

  
 


 
  

  
 

 
  
   



Thanks,
Chandan
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Ilya,

Yes, I start the ignite in client mode and my other applications access this
ignite cluster. I use to start the ignite cluster with 40-executors -and
2-cores. Usually, in success scenario my starting ignite and data loading
finish within 5/6 minutes. The logs are for negative scenario, while my jobs
are running for long time  and I have killed the long running job.

Below config xml I have used for ignite:

http://www.springframework.org/schema/beans;
   xmlns:util="http://www.springframework.org/schema/util; 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
  
classpath:/org/springframework/beans/factory/xml/spring-beans-4.3.xsd">
   

  
  
  

  
 


 
  

  
 

  

 
  

  
 


 
  

  
 

 
  
   



Thanks,
Chandan
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite yarn resources keep on increasing

[Ilya Kasnacheev]

Helo!

I can see that clientMode is true, this means that it has to find some
server nodes ot join.

GridDiscoveryManager fails to start because you interrupt it. Ditto the Zk
connection that you quote next.

Regards,
-- 
Ilya Kasnacheev


ср, 19 февр. 2020 г. в 15:03, ChandanS :

> Hi Ilya,
>
> When it doesn't fine existing ignite cluster, I use to create a new one. If
> you see the logs, you will get
> >>> Starting Ignite Client with cluster name: titan-resourceUP
>
> You can also see ignite initial start logs as well:
>
> >>>__  
> >>>   /  _/ ___/ |/ /  _/_  __/ __/
> >>>  _/ // (7 7// /  / / / _/
> >>> /___/\___/_/|_/___/ /_/ /___/
> >>>
> >>> ver. 2.7.0#20181130-sha1:256ae401
> >>> 2018 Copyright(C) Apache Software Foundation
> >>>
> >>> Ignite documentation: http://ignite.apache.org
>
>
> The code snippet for starting ignite is a below:
>logInfo(s">>> Starting Ignite Client with cluster name:
> $clusterName")
>Ignition.setClientMode(true)
>val retIgnite = Ignition.start(configInputStream)
>logInfo(s">>> New Ignite cluster started")
>retIgnite
>
> Below exception is being thrown from Ignition.start method and caught in my
> parent caller method
> >>> ERROR while creating Ignite Cluster: class
> >>> org.apache.ignite.IgniteException: Failed to start manager:
> >>> GridManagerAdapter [enabled=true,
> >>>
> name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
>
> And then I try to kill the job which is failing as I described before with
> exception
>
> >>> ERROR while triggering Ignite Standalone job: java.io.IOException:
> >>> Failed on local exception:
> java.nio.channels.ClosedByInterruptException;
> >>> Host Details : local host is:
> >>> "c920qju.intqa.bigdata.int.thomsonreuters.com/10.204.100.65";
> >>> destination host is: "c411apy.int.westgroup.com":8032;
>
>
> My spark application is working fine if I don't give the ignite-path wrong:
> IGNITE_PATH=/project/wrong-folder-name/ignite/apache-ignite-2.7.0-bin.zip
>
>
> Thanks,
> Chandan
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Ilya,

When it doesn't fine existing ignite cluster, I use to create a new one. If
you see the logs, you will get 
>>> Starting Ignite Client with cluster name: titan-resourceUP

You can also see ignite initial start logs as well:

>>>__    
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7// /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.7.0#20181130-sha1:256ae401
>>> 2018 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org


The code snippet for starting ignite is a below:
   logInfo(s">>> Starting Ignite Client with cluster name:
$clusterName")
   Ignition.setClientMode(true)
   val retIgnite = Ignition.start(configInputStream)
   logInfo(s">>> New Ignite cluster started")
   retIgnite

Below exception is being thrown from Ignition.start method and caught in my
parent caller method 
>>> ERROR while creating Ignite Cluster: class
>>> org.apache.ignite.IgniteException: Failed to start manager:
>>> GridManagerAdapter [enabled=true,
>>> name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]

And then I try to kill the job which is failing as I described before with
exception 

>>> ERROR while triggering Ignite Standalone job: java.io.IOException:
>>> Failed on local exception: java.nio.channels.ClosedByInterruptException;
>>> Host Details : local host is:
>>> "c920qju.intqa.bigdata.int.thomsonreuters.com/10.204.100.65";
>>> destination host is: "c411apy.int.westgroup.com":8032; 


My spark application is working fine if I don't give the ignite-path wrong:
IGNITE_PATH=/project/wrong-folder-name/ignite/apache-ignite-2.7.0-bin.zip


Thanks,
Chandan





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Ilya,

>From my spark application, I am trying to do two things:
First, start Ignite yarn cluster using ignite-yarn.jar
Second, load data to ignite cluster.
In the first step, I have my own method to submit another spark job with
YarnConfiguration to start the ignite. In second step, I use to load my
parquet data from hdfs location to ignite-cluster. Both the steps working
fine for best case scenario. I can see from spark web UI, my spark
application finish successfully once data loading is over and Ignite job
keep on running for my other applications to consume data. But whenever I
use to provide some wrong path for ignite configuration, application throws
exception.
IGNITE_PATH=/project/wrong-folder-name/ignite/apache-ignite-2.7.0-bin.zip
I have the code to kill the ignite yarn spark job if it throws some
exception. Now while trying to kill the ignite it's throwing another
exception and both spark jobs are running infinitely without stopping and
ignite-yarn spark job resources keep in increasing. Below is the code
snippet that I am trying to kill ignite:
  logInfo(s">>> Triggering stop Ignite cluster interface 2")
  val yarnClient = IgniteClient.returnYarnClient()
  logInfo(s">>> Triggering stop Ignite cluster interface 3")
  if (yarnClient != null) {
logInfo(s">>> Triggering stop Ignite cluster interface 3.1")
yarnClient.killApplication(applicationID)
logInfo(s">>> Triggering stop Ignite cluster interface 3.2")
  }
Below exception I am getting ClosedByInterruptException from killApplication
method:
20/02/18 12:23:33 INFO api.StartStandalone: >>> Triggering stop Ignite
cluster interface 2
20/02/18 12:23:33 INFO api.StartStandalone: >>> Triggering stop Ignite
cluster interface 3
20/02/18 12:23:33 INFO api.StartStandalone: >>> Triggering stop Ignite
cluster interface 3.1
20/02/18 12:23:33 ERROR api.StartStandalone: >>> ERROR while triggering
Ignite Standalone job: java.io.IOException: Failed on local exception:
java.nio.channels.ClosedByInterruptException; Host Details : local host is:
"c920qju.intqa.bigdata.int.thomsonreuters.com/10.204.100.65"; destination
host is: "c411apy.int.westgroup.com":8032; 


More logs:
20/02/18 12:23:06 INFO ignite.IgniteDataLoader: >>> Starting Ignite Client
with cluster name: titan-resourceUP
20/02/18 12:23:06 INFO xml.XmlBeanDefinitionReader: Loading XML bean
definitions from InputStream resource [resource loaded through InputStream]
20/02/18 12:23:06 INFO support.GenericApplicationContext: Refreshing
org.springframework.context.support.GenericApplicationContext@10478488:
startup date [Tue Feb 18 12:23:06 CST 2020]; root of context hierarchy
20/02/18 12:23:07 WARN : Failed to resolve default logging config file:
config/java.util.logging.properties
Console logging handler is not configured.
20/02/18 12:23:07 INFO internal.IgniteKernal%titan-resourceUP: 

>>>__    
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7// /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.7.0#20181130-sha1:256ae401
>>> 2018 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

20/02/18 12:23:07 INFO internal.IgniteKernal%titan-resourceUP: Config URL:
n/a
20/02/18 12:23:07 INFO internal.IgniteKernal%titan-resourceUP:
IgniteConfiguration [igniteInstanceName=titan-resourceUP, pubPoolSize=40,
svcPoolSize=40, callbackPoolSize=40, stripedPoolSize=40, sysPoolSize=40,
mgmtPoolSize=4, igfsPoolSize=40, dataStreamerPoolSize=40,
utilityCachePoolSize=40, utilityCacheKeepAliveTime=6, p2pPoolSize=2,
qryPoolSize=40, igniteHome=null,
igniteWorkDir=/data/1/yarn/nm/usercache/bigdata-app-ecpdevbermuda-uncomparator/appcache/application_1564355610025_506973/container_e192_1564355610025_506973_01_01/tmp/ignite/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@5f8c4d8d,
nodeId=4333dd99-d826-4804-9884-9a17781d1eac, marsh=BinaryMarshaller [],
marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=1,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=ZookeeperDiscoverySpi [zkRootPath=/apacheIgnite-titan-resourceUP,
zkConnectionString=c995xyn.int.westgroup.com:2181,c981qbu.int.westgroup.com:2181,c411apy.int.westgroup.com:2181,
joinTimeout=0, sesTimeout=6, clientReconnectDisabled=false,
internalLsnr=null, stats=ZookeeperDiscoveryStatistics [joinedNodesCnt=0,
failedNodesCnt=0]], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=1,
commSpi=TcpCommunicationSpi [connectGate=null,
connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@35045795,
enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null,
locHost=null, locPort=45400, locPortRange=20, shmemPort=-1, directBuf=true,
directSndBuf=false, idleConnTimeout=60, connTimeout=5000,
maxConnTimeout=60, reconCnt=10, 

Re: Ignite yarn resources keep on increasing

[Ilya Kasnacheev]

Hello!

It's hard to say what's going on. Maybe you forget to stop the node, but
retry the same computation, and you have nodes piling up?

Can you provide logs?

Regards,
-- 
Ilya Kasnacheev


пт, 14 февр. 2020 г. в 22:29, ChandanS :

> Hi Ilya,
>
> My submitted Ignite-yarn application works fine for best case scenarios.
> The
> issue is with worst case scenarios. E.g., below is my correct ignite_path
> property and my application works fine with it:
> IGNITE_PATH=/project/ecpdevbermuda/ignite/apache-ignite-2.7.0-bin.zip
>
> But, if I give the path wrong my spark job runs indefinitely with
> increasing
> in resources, e.g.
> IGNITE_PATH=/project/wrong-folder-name/ignite/apache-ignite-2.7.0-bin.zip
>
> Though, I had originally submitted my spark application with 40 executors
> and 2 cores, every few minutes resources keep on increasing and application
> won't stop, keep on running indefinitely. I have checked the spark job
> alive
> till 15 hrs with resources increased up to 528 executors/1055 cores, then I
> killed the job. In best case scenario my job finished within 5 or 6
> minutes.
> Please suggest.
>
>
> Thanks,
> Chandan Sarma
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Ilya,

My submitted Ignite-yarn application works fine for best case scenarios. The
issue is with worst case scenarios. E.g., below is my correct ignite_path
property and my application works fine with it: 
IGNITE_PATH=/project/ecpdevbermuda/ignite/apache-ignite-2.7.0-bin.zip

But, if I give the path wrong my spark job runs indefinitely with increasing
in resources, e.g. 
IGNITE_PATH=/project/wrong-folder-name/ignite/apache-ignite-2.7.0-bin.zip

Though, I had originally submitted my spark application with 40 executors
and 2 cores, every few minutes resources keep on increasing and application
won't stop, keep on running indefinitely. I have checked the spark job alive
till 15 hrs with resources increased up to 528 executors/1055 cores, then I
killed the job. In best case scenario my job finished within 5 or 6 minutes.
Please suggest.


Thanks,
Chandan Sarma



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite yarn resources keep on increasing

[ChandanS]

Hi Andrei,

I am using below configurations:

IGNITE_NODE_COUNT=40
IGNITE_RUN_CPU_PER_NODE=5
IGNITE_MEMORY_PER_NODE=15000
IGNITE_PATH=/project/ecpdevbermuda/ignite/apache-ignite-2.7.0-bin.zip
IGNITE_VERSION=2.7.0
IGNITE_WORKING_DIR=/project/ecpdevbermuda/ignite/
IGNITE_XML_CONFIG=/project/ecpdevbermuda/ignite/ignite-config-unifiedignite-la1.xml
IGNITE_USERS_LIBS=/project/ecpdevbermuda/ignite/libs/
IGNITE_QUEUE=root.ecpdevbermuda
IGNITE_CLUSTER_NAME=titan-resourceUP

Part of the logs:

20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned shuffle 0
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 10
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 9
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 8
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 7
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 6
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 5
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 4
20/02/11 06:43:43 INFO spark.ContextCleaner: Cleaned accumulator 3
20/02/11 06:43:43 INFO ignite.IgniteDataLoader: >>> Starting Ignite cluster
class org.apache.ignite.IgniteIllegalStateException: Ignite instance with
provided name doesn't exist. Did you call Ignition.start(..) to start an
Ignite instance? [name=titan-resourceUP]
at org.apache.ignite.internal.IgnitionEx.grid(IgnitionEx.java:1390)
at org.apache.ignite.Ignition.ignite(Ignition.java:531)
at
un.api.dataloader.ignite.IgniteDataLoader.liftedTree1$1(IgniteDataLoader.scala:707)
at
un.api.dataloader.ignite.IgniteDataLoader.getIgnite(IgniteDataLoader.scala:706)
at
un.api.dataloader.ignite.IgniteDataLoader.createOfficialNameOACache(IgniteDataLoader.scala:737)
at
un.api.dataloader.ignite.IgniteDataLoader.loadOfficialNameCache(IgniteDataLoader.scala:308)
at
un.api.dataloader.ignite.IgniteServerDataLoader$.loadOAData(IgniteServerDataLoader.scala:35)
at
un.api.StartStandalone$.startIgniteAndDataloading(StartStandalone.scala:98)
at
un.api.StartStandalone$.triggerIgniteStandAlone(StartStandalone.scala:43)
at 
un.api.StartStandalone$delayedInit$body.apply(StartStandalone.scala:18)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at un.api.StartStandalone$.main(StartStandalone.scala:17)
at un.api.StartStandalone.main(StartStandalone.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:567)
20/02/11 06:43:43 INFO ignite.IgniteDataLoader: >>> Reading Ignite Client
config from: /project/ecpdevbermuda/ignite/ignite-config-unifiedignite.xml
20/02/11 06:43:43 INFO ignite.IgniteDataLoader: >>> Starting Ignite Client
with cluster name: titan-resourceUP
20/02/11 06:43:43 INFO xml.XmlBeanDefinitionReader: Loading XML bean
definitions from InputStream resource [resource loaded through InputStream]
20/02/11 06:43:44 INFO support.GenericApplicationContext: Refreshing
org.springframework.context.support.GenericApplicationContext@7b544245:
startup date [Tue Feb 11 06:43:44 UTC 2020]; root of context hierarchy
20/02/11 06:43:44 WARN : Failed to resolve default logging config file:
config/java.util.logging.properties
Console logging handler is not configured.
20/02/11 06:43:44 INFO internal.IgniteKernal%titan-resourceUP: 

>>>__    
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7// /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.7.0#20181130-sha1:256ae401
>>> 2018 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

20/02/11 06:43:44 INFO internal.IgniteKernal%titan-resourceUP: Config URL:
n/a
20/02/11 06:43:44 INFO internal.IgniteKernal%titan-resourceUP:
IgniteConfiguration [igniteInstanceName=titan-resourceUP, pubPoolSize=48,
svcPoolSize=48, callbackPoolSize=48, stripedPoolSize=48, sysPoolSize=48,
mgmtPoolSize=4, igfsPoolSize=48, dataStreamerPoolSize=48,
utilityCachePoolSize=48, utilityCacheKeepAliveTime=6, p2pPoolSize=2,
qryPoolSize=48, igniteHome=null,

Re: Ignite yarn resources keep on increasing

[Andrei Aleksandrov]

Hi,

Could you please provide more details:

1)Your configurations and environment variables (IGNITE_PATH?)
2)The logs of your Ignite nodes where you see the mentioned exception.

IGNITE_PATH - a path to unzipped Ignite distribution instead of the URL. 
Is it possible that you didn't unzip the binaries or forget to copy 
binaries to some node?


BR,
Andrei

2/13/2020 1:12 PM, ChandanS пишет:

I am using ignite 2.7 version for ignite yarn deployment. I have my own spark
application that start ignite yarn cluster and load data to ignite. It works
fine in positive scenarios, but whenever there is an exception from the
ignite-yarn.jar side like giving wrong path for some properties
(IGNITE_PATH), the resource uses keep on increasing with some time interval.
I have started my application with --num-executors 40 --executor-cores 2,
currently after keeping the application up for last 10 hrs number of
executors is 461 and cores 921 with increasing in memory as well. I am
getting the below exception from ignite-yarn application:

class org.apache.ignite.IgniteException: Failed to start manager:
GridManagerAdapter [enabled=true,
name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/