[
https://issues.apache.org/jira/browse/IGNITE-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Ozerov updated IGNITE-5181:
------------------------------------
Priority: Major (was: Critical)
> Connection problem in ignite map reduce tasks
> ---------------------------------------------
>
> Key: IGNITE-5181
> URL: https://issues.apache.org/jira/browse/IGNITE-5181
> Project: Ignite
> Issue Type: Bug
> Components: compute
> Affects Versions: 1.9
> Environment: ignite server:
> centos6.5 with kernel 2.6.32-431.el6.x86_64
> ignite version 1.9
> hadoop version 2.6.2
> 3 server nodes with each having '-Xms16g -Xmx16g -server -XX:+AggressiveOpts
> -XX:MaxMetaspaceSize=256m' set when started
> Reporter: Allen Xu
>
> I run a map reduce test job with ignite map reduce. The job is simply getting
> the average number for each people. The data is like:
> Jack 0.35
> Tom 0.78
> Lily 0.92
> Jack 0.28
> Tom 0.18
> ...
> At first, I generated a data set of 100M lines. It's about 2.53GB. The job
> finished correctly in about 30s. Then I generated a data set of 1 Billion
> lines, about 25.3GB. The job always failed with exceptions. I tried several
> times but the same result.
> ==========================================================
> The ignite server node threw exception below:
> [15:06:56,804][ERROR][sys-#2740%null%][GridTcpRestProtocol] Failed to process
> client request [ses=GridSelectorNioSessionImpl
> [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=549
> lim=549 cap=8192], super=AbstractNioClientWorker
> [selector=sun.nio.ch.EPollSelectorImpl@1cba0431, idx=3, bytesRcvd=0,
> bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-3, gridName=null, finished=false,
> isCancelled=false, hashCode=906881587, interrupted=false,
> runner=grid-nio-worker-tcp-rest-3-#50%null%]]], writeBuf=null, readBuf=null,
> inRecovery=null, outRecovery=null, super=GridNioSessionImpl
> [locAddr=/172.31.68.204:11211, rmtAddr=/172.31.68.202:39473,
> createTime=1493967985751, closeTime=1493968009502, bytesSent=2715,
> bytesRcvd=2641, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1493968016794,
> lastSndTime=1493967998303, lastRcvTime=1493968009502, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser
> [jdkMarshaller=JdkMarshaller [], routerClient=false], directMode=false]],
> accepted=true]], msg=GridClientTaskRequest
> [taskName=o.a.i.i.processors.hadoop.proto.HadoopProtocolJobStatusTask,
> arg=HadoopProtocolTaskArguments []]]
> class org.apache.ignite.IgniteCheckedException: Failed to send message
> (connection was closed): GridSelectorNioSessionImpl
> [worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=549
> lim=549 cap=8192], super=AbstractNioClientWorker
> [selector=sun.nio.ch.EPollSelectorImpl@1cba0431, idx=3, bytesRcvd=0,
> bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-3, gridName=null, finished=false,
> isCancelled=false, hashCode=906881587, interrupted=false,
> runner=grid-nio-worker-tcp-rest-3-#50%null%]]], writeBuf=null, readBuf=null,
> inRecovery=null, outRecovery=null, super=GridNioSessionImpl
> [locAddr=/172.31.68.204:11211, rmtAddr=/172.31.68.202:39473,
> createTime=1493967985751, closeTime=1493968009502, bytesSent=2715,
> bytesRcvd=2641, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1493968016794,
> lastSndTime=1493967998303, lastRcvTime=1493968009502, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser
> [jdkMarshaller=JdkMarshaller [], routerClient=false], directMode=false]],
> accepted=true]]
> at
> org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7239)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:264)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1$1.apply(GridTcpRestNioListener.java:261)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:261)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:229)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:332)
> at
> org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:158)
> at
> org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:155)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:332)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:332)
> at
> org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:294)
> at
> org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:257)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1579)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1547)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.reduce(GridTaskWorker.java:1157)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:942)
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:996)
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1221)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to send message (connection was
> closed): GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker
> [readBuf=java.nio.HeapByteBuffer[pos=549 lim=549 cap=8192],
> super=AbstractNioClientWorker
> [selector=sun.nio.ch.EPollSelectorImpl@1cba0431, idx=3, bytesRcvd=0,
> bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
> [name=grid-nio-worker-tcp-rest-3, gridName=null, finished=false,
> isCancelled=false, hashCode=906881587, interrupted=false,
> runner=grid-nio-worker-tcp-rest-3-#50%null%]]], writeBuf=null, readBuf=null,
> inRecovery=null, outRecovery=null, super=GridNioSessionImpl
> [locAddr=/172.31.68.204:11211, rmtAddr=/172.31.68.202:39473,
> createTime=1493967985751, closeTime=1493968009502, bytesSent=2715,
> bytesRcvd=2641, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1493968016794,
> lastSndTime=1493967998303, lastRcvTime=1493968009502, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser
> [jdkMarshaller=JdkMarshaller [], routerClient=false], directMode=false]],
> accepted=true]]
> at
> org.apache.ignite.internal.util.nio.GridNioServer.send0(GridNioServer.java:554)
> at
> org.apache.ignite.internal.util.nio.GridNioServer.send(GridNioServer.java:494)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionWrite(GridNioServer.java:3036)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:118)
> at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:94)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:118)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:264)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:189)
> at
> org.apache.ignite.internal.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:108)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestNioListener$1.apply(GridTcpRestNioListener.java:258)
> ... 40 more
> ==========================================================
> The job client threw exception below:
> java.io.IOException: Failed to get job status:
> job_1fbf9083-9a44-4be9-9199-695a97652dc2_0002
> at
> org.apache.ignite.internal.processors.hadoop.impl.proto.HadoopClientProtocol.getJobStatus(HadoopClientProtocol.java:197)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:326)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:323)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:611)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1357)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1318)
> at com.tscloud.sdk.test.ignite.MRTest.run(MRTest.java:81)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at com.tscloud.sdk.test.ignite.MRTest.main(MRTest.java:53)
> Caused by: class
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException:
> Failed to perform request (connection failed): /172.31.68.204:11211
> at
> org.apache.ignite.internal.client.impl.connection.GridClientConnection.getCloseReasonAsException(GridClientConnection.java:491)
> at
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:339)
> at
> org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.close(GridClientNioTcpConnection.java:299)
> at
> org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onDisconnected(GridClientConnectionManagerAdapter.java:630)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClosed(GridNioFilterChain.java:253)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
> at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClosed(GridNioCodecFilter.java:70)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClosed(GridNioFilterAdapter.java:93)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onSessionClosed(GridNioServer.java:3005)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClosed(GridNioFilterChain.java:147)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.close(GridNioServer.java:2306)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:929)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2026)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1568)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> ==========================================================
> I had job configuration below:
> Configuration configuration = new Configuration();
> configuration.set(MRConfig.FRAMEWORK_NAME,
> IgniteHadoopClientProtocolProvider.FRAMEWORK_NAME);
> configuration.set(MRConfig.MASTER_ADDRESS, "172.31.68.202:11211");
> configuration.set("fs.igfs.impl",
> "org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem");
> configuration.set("fs.default.name", "igfs://igfs@172.31.68.202/");
> Any help is appreciated. Thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
