The documentation recommends against using embedded mode for what’s likely to be a related reason.
Embedded mode implies starting Ignite server nodes within Spark executors which can cause unexpected rebalancing or even data loss. Therefore this mode is currently deprecated and will be eventually discontinued. Consider starting a separate Ignite cluster and using standalone mode to avoid data consistency and performance issues. https://apacheignite-fs.readme.io/docs/installation-deployment#embedded-deployment Regards, Stephen > On 13 Jun 2019, at 12:57, vinod.jv <[email protected]> wrote: > > Hi, > > We are using Apache Ignite in embedded mode to store data in key value pairs > and query the data. > Sometimes, the spark jobs run for really long time than expected and in > those scenarios we have noticed that Ignite nodes are not responding in the > heart beat interval time and hence the re-balancing of data is happening and > followed by query cancellation. > > When the job is running in the expected time we don't see any exceptions in > the log. > > Here are the exceptions we get. > > org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 > seconds]. This timeout is controlled by spark.executor.heartbeatInterval > > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [10 seconds] > > 19/06/13 02:14:08 ERROR twostep.GridMapQueryExecutor: Failed to execute > local query. > class org.apache.ignite.cache.query.QueryCancelledException: The query was > cancelled while executing. > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:558) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:449) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:203) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:178) > at > org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1915) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:710) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:102) > at > org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:673) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 19/06/13 02:14:08 ERROR twostep.GridMapQueryExecutor: Failed to execute > local query. > class org.apache.ignite.cache.query.QueryCancelledException: The query was > cancelled while executing. > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:595) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:449) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:203) > at > org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:178) > at > org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:1915) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:710) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:102) > at > org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:673) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
