Hi All, During running pig application doing geometric data expansion, "Java.nio.channels.ClosedChannelException" error happened. Could you please help to see this issue and give your suggestions? Thanks
Pig statement: A = LOAD '$tmpfiledir1' AS (f1:chararray); B = LOAD '$tmpfiledir1' AS (f1:chararray); C = UNION A, B; STORE C INTO '$tmpfiledir2'; The output file $tmpfiledir2 will be the input again as $tmpfiledir1. And the original size of $tmpfiledir1 is 1GB. Then the output will be: 2GB -> 4GB -> 8GB -> 16GB -> 32GB -> 64GB -> 128GB -> 256GB -> 512GB -> 1TB -> 2TB -> 4TB -> 8TB -> 16TB Error from tasktracker: 07-14 17:50:08,261 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001616_0, with JvmId: jvm_201307141733_0015_m_-1094010055 at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435) at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(AccessController.java:310) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) 2013-07-14 17:50:08,262 WARN org.apache.hadoop.ipc.Server: IPC Server Responder, call statusUpdate(attempt_201307141733_0015_m_001616_0, org.apache.hadoop.mapred.MapTaskStatus@33fb33fb, org.apache.hadoop.mapred.JvmContext@40354035) from 127.0.0.1:15007: output error 2013-07-14 17:50:08,262 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54168 caught: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:147) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:338) at org.apache.hadoop.ipc.Server.channelWrite(Server.java:1705) at org.apache.hadoop.ipc.Server.access$2000(Server.java:95) at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:741) at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:805) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1421) 2013-07-14 17:50:08,265 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /hadoop/sdc/mapred/local/ttprivate/taskTracker/biadmin/jobcache/job_201307141733_0015/attempt_201307141733_0015_m_002271_0/taskjvm.sh 2013-07-14 17:50:08,266 WARN org.apache.hadoop.mapred.TaskTracker: Failed validating JVM java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201307141733_0015_m_001641_0, with JvmId: jvm_201307141733_0015_m_22934798 at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:3435) at org.apache.hadoop.mapred.TaskTracker.statusUpdate(TaskTracker.java:3505) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(AccessController.java:310) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) 2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exit code from task is : 66 2013-07-14 17:50:08,297 WARN org.apache.hadoop.mapred.LinuxTaskController: Exception thrown while launching task JVM : org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) at org.apache.hadoop.util.Shell.run(Shell.java:182) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) at org.apache.hadoop.mapred.LinuxTaskController.launchTask(LinuxTaskController.java:238) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:497) at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:471)
