Re: ignite memory issues:Urgent in production

Can anyone help me to fix this issue as this issue happens in our production env? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-memory-issues-Urgent-in-production-tp7817p7842.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ignite memory issues:Urgent in production

*Another server logs and I found that several ignite server automaticlly shutdown. * Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=f59d7d01-b01d-46b2-b679-17b73313ae98, addrs=[y, 127.0.0.1], sockAddrs=[y/y:0,

ignite memory issues:Urgent in production

Hello, I have a urgent issues on our production env for ignite issues. I have deployed ignite cluster with standalone server for 7 server nodes, each ignite node memory is 40G. totally is 270G. [10:05:11] Topology snapshot [ver=2356, servers=7, clients=0, CPUs=1488, heap=270GB] we have

how to get all keys and values stored into ignite cache for ignite1.6?

Hello, I have request to get all the keys and values from igniteCache for ignite 1.6 version? Can anyone tell me how to do that? thanks!!! -- View this message in context:

Re: Yarn Ignite Container Automatically exit when other yarn application running

Thanks Nikolai Tikhonov-2 response. *Did you use kerberos authentication for YARN? *No *And could you clarify which version of hadoop are you using?* hadoop-2.6.0-cdh5.5.0 -- View this message in context:

Re: Spark stage was hang via ignite

We have two executors that running very slowly... 1 -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Spark-stage-was-hang-via-ignite-tp7485p7525.html Sent from the Apache Ignite Users

Can anyone give me a full demo for embedded ignite for spark?

-- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Can-anyone-give-me-a-full-demo-for-embedded-ignite-for-spark-tp7486.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Spark stage was hang via ignite

/u01/spark-1.6.0-hive/bin/spark-submit --driver-memory 4G --class com.ETLTransform --master yarn --executor-cores 4 --executor-memory 1000m --num-executors 20 --conf spark.rdd.compress=false --conf spark.shuffle.compress=false --conf spark.broadcast.compress=false /u01/spark_engine863.jar -quesize

Re: Ignite Cache Update(k,v)

IgniteCluster API provides startNodes() method that allows to do this programmatically on the remote machines (it uses SSH connection). Can you please give me full examples? or a link for this? thanks again -- View this message in context:

Re: Yarn Ignite Container Automatically exit when other yarn application running

but from yarn contains, i can't find any errors. sometimes the yarn am ignite was shutdown down(and sometimes restart a new AM, i don't know why)? -- View this message in context:

Re: Ignite Cache Update(k,v)

OK,thanks vkulichenko! If i use IgniteCache API directly on my scenarios, how to start ignite server on different worker machine? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-Update-k-v-tp7296p7362.html Sent from the Apache Ignite Users mailing

Yarn Ignite Container Automatically exit when other yarn application running

Hello, I faced important issues. I have deployed yarn ignite application successfully. everything is okay. But today, when others running spark job on yarn(this job can't contain ignite),and faced error message as below *16/08/26 16:23:52 ERROR YarnScheduler: Lost executor 1 on :

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

I have already fixed my issues, and add classpth://srping-bean.jar to load native jar NOT internet. Thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Yarn-deployment-for-static-TcpDiscoverySpi-issues-Urgent-In-Production-tp7205p7289.html Sent from

Re: Yarn deployment for memory capacity make a bigger than before: Urgent!!!

Thanks Nikolai Tikhonov-2, 2 questions 1)How to configure -Xmx and -Xms jvm options for client nodes? I have submitted this job via spark-sub, you mean change this memory. 2)You can't explain the reason? Can you please double check this? **why local port 127.0.0.1 join yarn ignite topology,

Yarn deployment for memory capacity make a bigger than before: Urgent!!!

Here is my detailed steps 1)root@sparkup1 config]# cat default-config.xml http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xmlns:util="http://www.springframework.org/schema/util;

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

Thanks Val, I have run successfully this with yarn on my local env. but when i run this on production env, I always get spring beans error message http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="

Re: Yarn deployment for static TcpDiscoverySpi issues:Urgent In Production

I have fixed the file issues "file:/usr/apache-ignite-fabric-1.6.0-bin/config/default-config.xml" on today. I verified again on our production env and found another issues as below 1) I have set IGNTE_HOME on spark driver machine and start to submit this jar to yarn cluster export

Re: Embedded mode ignite on spark

Thanks vkulichenko, I will use TcpDiscoveryVmIpFinder instead of mutinies for ignite, but i faced another issues. my integration mode is : embedded mode ignite on spark 1) Scala code for idea as below val spi = new TcpDiscoverySpi() val ipFinder = new TcpDiscoveryVmIpFinder() // Set initial IP

Re: Embedded mode ignite on spark

Anyone can help me to fix this issue? thanks -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Embedded-mode-ignite-on-spark-tp6942p7146.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Embedded mode ignite on spark

==5)=== SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/filecache/652/spark-assembly-1.6.0-hadoop2.5.0-cdh5.3.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in

Re: Embedded mode ignite on spark

4) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/filecache/785/spark-assembly-1.6.0-hadoop2.5.0-cdh5.3.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in

Re: Embedded mode ignite on spark

Thanks vkulichenko. 1)I have tried this code as your new method on our production env and submit to yarn cluster, final result is not correct, it seems that case data was lost. val sharedRDD = ic.fromCache(new CacheConfiguration[String, String]().setCacheMode(CacheMode.REPLICATED)); 2) this is

Re: Re:Re: Ignite for Spark on YARN Deployment

Can you pleas tell me what ports should be opened on driver or yarn cluster machine? this is our production env, I suspect that something need be configured on firewall ports. 1) I verified all these steps on my local cluster(3 spark machine[1 master and 2 workers], one yarn cluster machine),

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, YES, I can ping yarn cluster from driver machine. I try to telnet localhost 50075 on driver machine, but failed, is this root cause for this issues? thanks!!! -- View this message in context:

Re: Embedded mode ignite on spark

Thanks vkulichenko I will describe my issues detailed as below. I have the same code on two scenarios but firestone is correct and second one is not correct. I'm studying ignite recent days but can't get correct result on thisHopefully anyone can help me on this.

Re: Re:Re: Ignite for Spark on YARN Deployment

scala> [14:52:04] New version is available at ignite.apache.org: 1.7.0 16/08/12 14:52:46 ERROR TcpDiscoverySpi: Failed to reconnect to cluster (consider increasing 'networkTimeout' configuration property) [networkTimeout=5000] 16/08/12 15:37:54 ERROR GridClockSyncProcessor: Failed to send time

Re: Re:Re: Ignite for Spark on YARN Deployment

Get the following error message after a few minutes scala> val ic = new IgniteContext[Integer, Integer](sc,"/u01/yueyi/apache-ignite-fabric-1.6.0-bin/config/default-config.xml") ic: org.apache.ignite.spark.IgniteContext[Integer,Integer] = org.apache.ignite.spark.IgniteContext@26917c50 scala>

Re: Embedded mode ignite on spark

I verified this code as below val ic = new IgniteContext[Int, Int](sc, () => new IgniteConfiguration(),false) val sharedRDD = ic.fromCache("sharedBaselineCacheRDD") val initalRDD = sc.parallelize(1 to 10,10).map(i => (i, i)) println("initalRDD.couner=/. " + initalRDD.count()

Re: Embedded mode ignite on spark

thaks ,Alisher, Can you also give more ideas about discard datas for ignite cache? thanks! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Embedded-mode-ignite-on-spark-tp6942p6999.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, I'm a new ignite, can you please provide more detail steps for this? thanks again!!! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6986.html Sent from the Apache Ignite Users mailing list archive at

Re: Re:Re: Ignite for Spark on YARN Deployment

1) containers logs as below 1、[20:07:14] Ignite node started OK (id=a3a87e37) [20:07:14] Topology snapshot [ver=1, servers=1, clients=0, CPUs=24, heap=2.0GB] [20:07:14] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=4.0GB] [20:07:15] Topology snapshot [ver=3, servers=3, clients=0,

Re: Re:Re: Ignite for Spark on YARN Deployment

Hello, Nikolai, 1、 Just updated configuration default-config.xm on /u01/XXX/apache-ignite-fabric-1.6.0-bin/config/default-config.xm 2）./hdfs dfs -put /u01/XXX/apache-ignite-fabric-1.6.0-bin/config/default-config.xm /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/ 3) scala> import

Re: Re:Re: Ignite for Spark on YARN Deployment

Thanks Nikolai very much. As your request, and changed configuration /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml but this file is hfs file 1) $./hdfs dfs -text /ignite/release16/apache-ignite-fabric-1.6.0-bin/config/default-config.xml SLF4J: Class path contains

Re: Ignite for Spark on YARN Deployment

$cat default-config.xml http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd;>

Re: Re:Re: Ignite for Spark on YARN Deployment

1、Adjusted cluster.properites as below $cat cluster16.properties # The number of nodes in the cluster. IGNITE_NODE_COUNT=4 # The number of CPU Cores for each Apache Ignite node. IGNITE_RUN_CPU_PER_NODE=1 # The number of Megabytes of RAM for each Apache Ignite node. IGNITE_MEMORY_PER_NODE=2048 #

Re:Re: Ignite for Spark on YARN Deployment

GNITE_USERS_LIBS not needed here. The property need to use when you want deploy to cluster your own libs. On Thu, Aug 11, 2016 at 7:54 AM, percent620 <[hidden email]> wrote: Thanks vkulichenko's quick response. Here is my detailed steps about how to deploy and integration spark with ignite

Embedded mode ignite on spark

Hello, I want to integrate ignite with spark on Embedded mode , and here is my detailed steps for this. 1、added maven dependencies for pom.xml org.apache.ignite ignite-core 1.6.0 org.apache.ignite

Re: Ignite for Spark on YARN Deployment

Thanks vkulichenko's quick response. Here is my detailed steps about how to deploy and integration spark with ignite as below. 1、Followed this guidelines about how to deploy ignite-yarn application. it's successfully