I have included the output from lsof, Unfortunately though I only restarted the server a few mins ago and so it doesn't really illustrate the problem. I'll try to post back with the same command tomorrow so that we can diff between the two.
Thanks, Andy. java 16908 hadoop cwd DIR 0,17 240 7037865 /var/run/hadoop-0.20 java 16908 hadoop rtd DIR 8,1 4096 2 / java 16908 hadoop txt REG 8,1 50810 3555462 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/bin/java java 16908 hadoop mem REG 8,1 127296 3443156 /usr/lib/liblzo2.so.2.0.0 java 16908 hadoop mem REG 8,1 69472 3735429 /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/libgplcompression.so.0.0.0 java 16908 hadoop mem REG 8,1 44257 3555570 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnio.so java 16908 hadoop mem REG 8,1 80760 1164352 /lib/libresolv-2.7.so java 16908 hadoop mem REG 8,1 22856 1164345 /lib/libnss_dns-2.7.so java 16908 hadoop mem REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar java 16908 hadoop mem REG 8,1 38105 3555614 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libmanagement.so java 16908 hadoop mem REG 8,1 111901 3555569 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libnet.so java 16908 hadoop mem REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar java 16908 hadoop mem REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar java 16908 hadoop mem REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar java 16908 hadoop mem REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar java 16908 hadoop mem REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar java 16908 hadoop mem REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar java 16908 hadoop mem REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar java 16908 hadoop mem REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar java 16908 hadoop mem REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar java 16908 hadoop mem REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar java 16908 hadoop mem REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar java 16908 hadoop mem REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar java 16908 hadoop mem REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar java 16908 hadoop mem REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar java 16908 hadoop mem REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar java 16908 hadoop mem REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar java 16908 hadoop mem REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar java 16908 hadoop mem REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar java 16908 hadoop mem REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar java 16908 hadoop mem REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar java 16908 hadoop mem REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar java 16908 hadoop mem REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar java 16908 hadoop mem REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar java 16908 hadoop mem REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar java 16908 hadoop mem REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar java 16908 hadoop mem REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar java 16908 hadoop mem REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar java 16908 hadoop mem REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar java 16908 hadoop mem REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar java 16908 hadoop mem REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar java 16908 hadoop mem REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar java 16908 hadoop mem REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar java 16908 hadoop mem REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar java 16908 hadoop mem REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar java 16908 hadoop mem REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar java 16908 hadoop mem REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar java 16908 hadoop mem REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar java 16908 hadoop mem REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar java 16908 hadoop mem REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar java 16908 hadoop mem REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar java 16908 hadoop mem REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar java 16908 hadoop mem REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar java 16908 hadoop mem REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar java 16908 hadoop mem REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar java 16908 hadoop mem REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar java 16908 hadoop mem REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar java 16908 hadoop mem REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar java 16908 hadoop mem REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar java 16908 hadoop mem REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar java 16908 hadoop mem REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar java 16908 hadoop mem REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar java 16908 hadoop mem REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar java 16908 hadoop mem REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar java 16908 hadoop mem REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar java 16908 hadoop mem REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar java 16908 hadoop mem REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar java 16908 hadoop mem REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar java 16908 hadoop mem REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar java 16908 hadoop mem REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar java 16908 hadoop mem REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar java 16908 hadoop mem REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar java 16908 hadoop mem REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar java 16908 hadoop mem REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar java 16908 hadoop mem REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar java 16908 hadoop mem REG 8,1 25700 3441590 /usr/lib/gconv/gconv-modules.cache java 16908 hadoop mem REG 8,1 254076 3465218 /usr/lib/locale/en_GB.utf8/LC_CTYPE java 16908 hadoop mem REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar java 16908 hadoop mem REG 8,1 92117 3555579 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libzip.so java 16908 hadoop mem REG 8,1 47528 1164346 /lib/libnss_files-2.7.so java 16908 hadoop mem REG 8,1 43480 1164348 /lib/libnss_nis-2.7.so java 16908 hadoop mem REG 8,1 35632 1164344 /lib/libnss_compat-2.7.so java 16908 hadoop mem REG 8,1 55850 3555581 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/native_threads/libhpi.so java 16908 hadoop mem REG 8,1 93080 1164343 /lib/libnsl-2.7.so java 16908 hadoop mem REG 8,1 234713 3555592 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libjava.so java 16908 hadoop mem REG 8,1 66580 3555574 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/libverify.so java 16908 hadoop mem REG 8,1 35784 1164353 /lib/librt-2.7.so java 16908 hadoop mem REG 8,1 526560 1164341 /lib/libm-2.7.so java 16908 hadoop mem REG 8,1 10987257 3555599 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/server/libjvm.so java 16908 hadoop mem REG 8,1 1436976 1164337 /lib/libc-2.7.so java 16908 hadoop mem REG 8,1 14624 1164340 /lib/libdl-2.7.so java 16908 hadoop mem REG 8,1 130224 1164351 /lib/libpthread-2.7.so java 16908 hadoop mem REG 8,1 127480 1164334 /lib/ld-2.7.so java 16908 hadoop mem REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar java 16908 hadoop mem REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar java 16908 hadoop mem REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar java 16908 hadoop mem REG 8,1 32768 3989505 /tmp/hsperfdata_hadoop/16908 java 16908 hadoop mem REG 8,1 48335 3555565 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/amd64/jli/libjli.so java 16908 hadoop mem REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar java 16908 hadoop 0u CHR 136,2 4 /dev/pts/2 (deleted) java 16908 hadoop 1u CHR 136,2 4 /dev/pts/2 (deleted) java 16908 hadoop 2u CHR 136,2 4 /dev/pts/2 (deleted) java 16908 hadoop 3r REG 8,1 51669471 3548774 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/rt.jar java 16908 hadoop 4r REG 8,1 12588713 3556085 /usr/lib/jvm/java-6-sun-1.6.0.14/lib/tools.jar java 16908 hadoop 5r REG 8,1 2771746 3631241 /usr/lib/hadoop-0.20/hadoop-0.20.1+133-core.jar java 16908 hadoop 6r REG 8,1 41123 3721331 /usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar java 16908 hadoop 7r REG 8,1 46725 3721329 /usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar java 16908 hadoop 8r REG 8,1 112341 3721350 /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar java 16908 hadoop 9r REG 8,1 279781 3721330 /usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar java 16908 hadoop 10r REG 8,1 38015 3721336 /usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar java 16908 hadoop 11r REG 8,1 26202 3721366 /usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar java 16908 hadoop 12r REG 8,1 180792 3721357 /usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar java 16908 hadoop 13r REG 8,1 3566844 3721335 /usr/lib/hadoop-0.20/lib/core-3.1.1.jar java 16908 hadoop 14r REG 8,1 37157 3721359 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-fairscheduler.jar java 16908 hadoop 15r REG 8,1 18684 3721361 /usr/lib/hadoop-0.20/lib/hadoop-0.20.1+133-scribe-log4j.jar java 16908 hadoop 16r REG 8,1 37210 6038622 /usr/lib/hadoop-0.20/lib/hadoop-gpl-compression-0.1.0-dev.jar java 16908 hadoop 17r REG 8,1 706710 3721337 /usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar java 16908 hadoop 18r REG 8,1 706710 3721348 /usr/lib/hadoop-0.20/lib/hsqldb.jar java 16908 hadoop 19r REG 8,1 405086 3721360 /usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar java 16908 hadoop 20r REG 8,1 76698 3721363 /usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar java 16908 hadoop 21r REG 8,1 321806 3721333 /usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar java 16908 hadoop 22r REG 8,1 516429 3721334 /usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar java 16908 hadoop 23r REG 8,1 163121 3721356 /usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar java 16908 hadoop 24r REG 8,1 121070 3721354 /usr/lib/hadoop-0.20/lib/junit-3.8.1.jar java 16908 hadoop 25r REG 8,1 198945 3721353 /usr/lib/hadoop-0.20/lib/junit-4.5.jar java 16908 hadoop 26r REG 8,1 11428 3721352 /usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar java 16908 hadoop 27r REG 8,1 114342 3721365 /usr/lib/hadoop-0.20/lib/libfb303.jar java 16908 hadoop 28r REG 8,1 73374 3721367 /usr/lib/hadoop-0.20/lib/libthrift.jar java 16908 hadoop 29r REG 8,1 391834 3721364 /usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar java 16908 hadoop 30r REG 8,1 540852 3721351 /usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar java 16908 hadoop 31r REG 8,1 65261 3721362 /usr/lib/hadoop-0.20/lib/oro-2.0.8.jar java 16908 hadoop 32r REG 8,1 132368 3721358 /usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar java 16908 hadoop 33r REG 8,1 15345 3721332 /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar java 16908 hadoop 34r REG 8,1 8601 3721349 /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar java 16908 hadoop 35r REG 8,1 15010 3721355 /usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar java 16908 hadoop 36r REG 8,1 1024681 3727362 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar java 16908 hadoop 37r REG 8,1 134910 3727363 /usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar java 16908 hadoop 38r REG 8,1 105209 3532075 /usr/lib/hive/lib/antlr-runtime-3.0.1.jar java 16908 hadoop 39r REG 8,1 43035 3532057 /usr/lib/hive/lib/asm-3.1.jar java 16908 hadoop 40r REG 8,1 258337 3532056 /usr/lib/hive/lib/commons-cli-2.0-SNAPSHOT.jar java 16908 hadoop 41r REG 8,1 46725 3532054 /usr/lib/hive/lib/commons-codec-1.3.jar java 16908 hadoop 42r REG 8,1 575389 3532080 /usr/lib/hive/lib/commons-collections-3.2.1.jar java 16908 hadoop 43r REG 8,1 261809 3532059 /usr/lib/hive/lib/commons-lang-2.4.jar java 16908 hadoop 44r REG 8,1 38015 3532060 /usr/lib/hive/lib/commons-logging-1.0.4.jar java 16908 hadoop 45r REG 8,1 26202 3532079 /usr/lib/hive/lib/commons-logging-api-1.0.4.jar java 16908 hadoop 46r REG 8,1 1923209 3532061 /usr/lib/hive/lib/datanucleus-core-1.1.2.jar java 16908 hadoop 47r REG 8,1 176174 3532053 /usr/lib/hive/lib/datanucleus-enhancer-1.1.2.jar java 16908 hadoop 48r REG 8,1 1154170 3532082 /usr/lib/hive/lib/datanucleus-rdbms-1.1.2.jar java 16908 hadoop 49r REG 8,1 2446767 3532072 /usr/lib/hive/lib/derby.jar java 16908 hadoop 50r REG 8,1 7092 3532067 /usr/lib/hive/lib/hive_anttasks.jar java 16908 hadoop 51r REG 8,1 12641 3532064 /usr/lib/hive/lib/hive_cli.jar java 16908 hadoop 52r REG 8,1 14099 3532074 /usr/lib/hive/lib/hive_common.jar java 16908 hadoop 53r REG 8,1 11412 3532091 /usr/lib/hive/lib/hive_contrib.jar java 16908 hadoop 54r REG 8,1 1798010 3532071 /usr/lib/hive/lib/hive_exec.jar java 16908 hadoop 55r REG 8,1 19349 3532081 /usr/lib/hive/lib/hive_hwi.jar java 16908 hadoop 56r REG 8,1 31931 3532070 /usr/lib/hive/lib/hive_jdbc.jar java 16908 hadoop 57r REG 8,1 500917 3532063 /usr/lib/hive/lib/hive_metastore.jar java 16908 hadoop 58r REG 8,1 390114 3532076 /usr/lib/hive/lib/hive_serde.jar java 16908 hadoop 59r REG 8,1 95488 3532068 /usr/lib/hive/lib/hive_service.jar java 16908 hadoop 60r REG 8,1 20202 3532073 /usr/lib/hive/lib/hive_shims.jar java 16908 hadoop 61r REG 8,1 179508 3532062 /usr/lib/hive/lib/jdo2-api-2.3-SNAPSHOT.jar java 16908 hadoop 62r REG 8,1 87325 3532030 /usr/lib/hive/lib/jline-0.9.94.jar java 16908 hadoop 63r REG 8,1 45946 3532058 /usr/lib/hive/lib/json.jar java 16908 hadoop 64r REG 8,1 121070 3532069 /usr/lib/hive/lib/junit-3.8.1.jar java 16908 hadoop 65r REG 8,1 114342 3532078 /usr/lib/hive/lib/libfb303.jar java 16908 hadoop 66r REG 8,1 169666 3532090 /usr/lib/hive/lib/libthrift.jar java 16908 hadoop 67r REG 8,1 391834 3532077 /usr/lib/hive/lib/log4j-1.2.15.jar java 16908 hadoop 68r REG 8,1 724225 3532093 /usr/lib/hive/lib/mysql-connector-java-5.1.10-bin.jar java 16908 hadoop 69r REG 8,1 229928 3532055 /usr/lib/hive/lib/stringtemplate-3.1b1.jar java 16908 hadoop 70r REG 8,1 392124 3532065 /usr/lib/hive/lib/velocity-1.5.jar java 16908 hadoop 71r CHR 1,8 1818 /dev/random java 16908 hadoop 72r CHR 1,9 1824 /dev/urandom java 16908 hadoop 73u IPv6 153142741 TCP *:webmin (LISTEN) java 16908 hadoop 74u sock 0,4 153142739 can't identify protocol java 16908 hadoop 76w REG 8,1 6867 5043336 /tmp/hadoop/hive_job_log_hadoop_201002152257_1407395533.txt java 16908 hadoop 77u IPv6 154300915 TCP hadoopmaster.cluster.trafficbroker.co.uk:41800->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 78r REG 8,1 1145980 3548785 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/resources.jar java 16908 hadoop 79u IPv6 153477399 TCP hadoopmaster.cluster.trafficbroker.co.uk:33453->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 80u IPv6 153594480 TCP hadoopmaster.cluster.trafficbroker.co.uk:55995->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 81w REG 8,1 15491 5042961 /tmp/hadoop/hive_job_log_hadoop_201002151135_1344776163.txt java 16908 hadoop 82u IPv6 153177590 TCP hadoopmaster.cluster.trafficbroker.co.uk:37032->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 83r REG 8,1 618950 3548767 /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/jsse.jar java 16908 hadoop 84u IPv6 153177591 TCP hadoopmaster.cluster.trafficbroker.co.uk:45753->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 85u unix 0xffff8101b5090000 153143422 socket java 16908 hadoop 86w REG 8,1 62510 5043340 /tmp/hadoop/hive_job_log_hadoop_201002152307_1020703855.txt java 16908 hadoop 87w REG 8,1 960 5042962 /tmp/hadoop/hive_job_log_hadoop_201002151135_2129614227.txt java 16908 hadoop 88u IPv6 153219191 TCP hadoopmaster.cluster.trafficbroker.co.uk:55630->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 89u IPv6 153143432 TCP hadoopmaster.cluster.trafficbroker.co.uk:39643->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 90u IPv6 153143434 TCP hadoopmaster.cluster.trafficbroker.co.uk:39645->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 91u IPv6 153219192 TCP hadoopmaster.cluster.trafficbroker.co.uk:55631->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 92u IPv6 153477400 TCP hadoopmaster.cluster.trafficbroker.co.uk:47916->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 93u IPv6 153445420 TCP hadoopmaster.cluster.trafficbroker.co.uk:41432->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 94u IPv6 153385926 TCP hadoopmaster.cluster.trafficbroker.co.uk:59893->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 95u IPv6 153322551 TCP hadoopmaster.cluster.trafficbroker.co.uk:33148->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 96u IPv6 153220498 TCP hadoopmaster.cluster.trafficbroker.co.uk:60801->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 97u IPv6 153220499 TCP hadoopmaster.cluster.trafficbroker.co.uk:52976->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 98u IPv6 153273816 TCP hadoopmaster.cluster.trafficbroker.co.uk:46298->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 99u IPv6 153273819 TCP hadoopmaster.cluster.trafficbroker.co.uk:46299->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 100u IPv6 153293338 TCP hadoopmaster.cluster.trafficbroker.co.uk:55614->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 102u IPv6 153708229 TCP hadoopmaster.cluster.trafficbroker.co.uk:48799->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 103u IPv6 153276086 TCP hadoopmaster.cluster.trafficbroker.co.uk:59132->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 104u IPv6 153276087 TCP hadoopmaster.cluster.trafficbroker.co.uk:37820->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 105u IPv6 153291015 TCP hadoopmaster.cluster.trafficbroker.co.uk:37060->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 106u IPv6 153291016 TCP hadoopmaster.cluster.trafficbroker.co.uk:37061->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 107u IPv6 153293339 TCP hadoopmaster.cluster.trafficbroker.co.uk:45869->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 108u IPv6 153291444 TCP hadoopmaster.cluster.trafficbroker.co.uk:37071->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 109u IPv6 153291445 TCP hadoopmaster.cluster.trafficbroker.co.uk:37072->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 110u IPv6 153294552 TCP hadoopmaster.cluster.trafficbroker.co.uk:37855->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 111u IPv6 153294209 TCP hadoopmaster.cluster.trafficbroker.co.uk:41043->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 112u IPv6 153294210 TCP hadoopmaster.cluster.trafficbroker.co.uk:41044->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 113w REG 8,1 11562 5043337 /tmp/hadoop/hive_job_log_hadoop_201002152258_253764886.txt java 16908 hadoop 114u IPv6 153294232 TCP hadoopmaster.cluster.trafficbroker.co.uk:36967->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 115u IPv6 153294233 TCP hadoopmaster.cluster.trafficbroker.co.uk:36968->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 116u IPv6 153294553 TCP hadoopmaster.cluster.trafficbroker.co.uk:37856->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 117u IPv6 153308598 TCP hadoopmaster.cluster.trafficbroker.co.uk:60192->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 118u IPv6 153299396 TCP hadoopmaster.cluster.trafficbroker.co.uk:37922->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 119u IPv6 153299397 TCP hadoopmaster.cluster.trafficbroker.co.uk:41128->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 120u IPv6 153305498 TCP hadoopmaster.cluster.trafficbroker.co.uk:45027->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 121u IPv6 153305501 TCP hadoopmaster.cluster.trafficbroker.co.uk:45028->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 122u IPv6 153322053 TCP hadoopmaster.cluster.trafficbroker.co.uk:47336->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 123u IPv6 153306847 TCP hadoopmaster.cluster.trafficbroker.co.uk:35011->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 124u IPv6 153306848 TCP hadoopmaster.cluster.trafficbroker.co.uk:55583->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 125u IPv6 153308599 TCP hadoopmaster.cluster.trafficbroker.co.uk:58929->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 126u IPv6 153308072 TCP hadoopmaster.cluster.trafficbroker.co.uk:60180->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 127u IPv6 153308073 TCP hadoopmaster.cluster.trafficbroker.co.uk:60181->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 128u IPv6 153322552 TCP hadoopmaster.cluster.trafficbroker.co.uk:46618->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 129u IPv6 153318287 TCP hadoopmaster.cluster.trafficbroker.co.uk:35532->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 130u IPv6 153318288 TCP hadoopmaster.cluster.trafficbroker.co.uk:49915->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 131u IPv6 153322054 TCP hadoopmaster.cluster.trafficbroker.co.uk:47515->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 132u IPv6 153321874 TCP hadoopmaster.cluster.trafficbroker.co.uk:50015->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 133u IPv6 153321875 TCP hadoopmaster.cluster.trafficbroker.co.uk:50016->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 134u IPv6 153708231 TCP hadoopmaster.cluster.trafficbroker.co.uk:54749->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 135u IPv6 153322490 TCP hadoopmaster.cluster.trafficbroker.co.uk:37912->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 136u IPv6 153322491 TCP hadoopmaster.cluster.trafficbroker.co.uk:37913->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 137u IPv6 153330756 TCP hadoopmaster.cluster.trafficbroker.co.uk:52062->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 138u IPv6 153322687 TCP hadoopmaster.cluster.trafficbroker.co.uk:43574->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 139u IPv6 153322688 TCP hadoopmaster.cluster.trafficbroker.co.uk:59376->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 140u IPv6 153325417 TCP hadoopmaster.cluster.trafficbroker.co.uk:40817->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 141u IPv6 153322873 TCP hadoopmaster.cluster.trafficbroker.co.uk:50386->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 142u IPv6 153322874 TCP hadoopmaster.cluster.trafficbroker.co.uk:36612->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 143u IPv6 153325418 TCP hadoopmaster.cluster.trafficbroker.co.uk:46662->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 144u IPv6 153355133 TCP hadoopmaster.cluster.trafficbroker.co.uk:56408->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 145u IPv6 153325537 TCP hadoopmaster.cluster.trafficbroker.co.uk:60663->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 146u IPv6 153325538 TCP hadoopmaster.cluster.trafficbroker.co.uk:45424->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 147u IPv6 153330757 TCP hadoopmaster.cluster.trafficbroker.co.uk:55620->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 148u IPv6 153330673 TCP hadoopmaster.cluster.trafficbroker.co.uk:57503->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 149u IPv6 153330674 TCP hadoopmaster.cluster.trafficbroker.co.uk:57504->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 150u IPv6 153342009 TCP hadoopmaster.cluster.trafficbroker.co.uk:41027->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 151u IPv6 153342010 TCP hadoopmaster.cluster.trafficbroker.co.uk:36849->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 152u IPv6 153343873 TCP hadoopmaster.cluster.trafficbroker.co.uk:59210->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 153u IPv6 153342462 TCP hadoopmaster.cluster.trafficbroker.co.uk:34719->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 154u IPv6 153342463 TCP hadoopmaster.cluster.trafficbroker.co.uk:34720->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 155u IPv6 153343874 TCP hadoopmaster.cluster.trafficbroker.co.uk:59211->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 156u IPv6 153348641 TCP hadoopmaster.cluster.trafficbroker.co.uk:57131->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 157u IPv6 153344781 TCP hadoopmaster.cluster.trafficbroker.co.uk:59463->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 158u IPv6 153344782 TCP hadoopmaster.cluster.trafficbroker.co.uk:45281->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 159u IPv6 153348642 TCP hadoopmaster.cluster.trafficbroker.co.uk:57132->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 160u IPv6 153354839 TCP hadoopmaster.cluster.trafficbroker.co.uk:34546->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 161u IPv6 153353725 TCP hadoopmaster.cluster.trafficbroker.co.uk:59303->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 162u IPv6 153353726 TCP hadoopmaster.cluster.trafficbroker.co.uk:35686->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 163u IPv6 153354840 TCP hadoopmaster.cluster.trafficbroker.co.uk:56196->hadoopslave17.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 164u IPv6 153363372 TCP hadoopmaster.cluster.trafficbroker.co.uk:40547->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 165u IPv6 153354979 TCP hadoopmaster.cluster.trafficbroker.co.uk:39825->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 166u IPv6 153354980 TCP hadoopmaster.cluster.trafficbroker.co.uk:39826->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 167u IPv6 153355134 TCP hadoopmaster.cluster.trafficbroker.co.uk:56409->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 168u IPv6 153385927 TCP hadoopmaster.cluster.trafficbroker.co.uk:47979->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 169u IPv6 153445421 TCP hadoopmaster.cluster.trafficbroker.co.uk:41433->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 170u IPv6 153405211 TCP hadoopmaster.cluster.trafficbroker.co.uk:60176->hadoopslave15.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 171u IPv6 153363371 TCP hadoopmaster.cluster.trafficbroker.co.uk:36383->hadoopslave5.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 172u IPv6 153405212 TCP hadoopmaster.cluster.trafficbroker.co.uk:48012->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 173u IPv6 153373260 TCP hadoopmaster.cluster.trafficbroker.co.uk:33359->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 174u IPv6 153460225 TCP hadoopmaster.cluster.trafficbroker.co.uk:50021->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 175u IPv6 153449798 TCP hadoopmaster.cluster.trafficbroker.co.uk:43609->hadoopslave8.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 176u IPv6 153360445 TCP hadoopmaster.cluster.trafficbroker.co.uk:45813->hadoopslave14.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 177u IPv6 153360459 TCP hadoopmaster.cluster.trafficbroker.co.uk:38494->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 178u IPv6 153449799 TCP hadoopmaster.cluster.trafficbroker.co.uk:51056->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 179u IPv6 153373262 TCP hadoopmaster.cluster.trafficbroker.co.uk:58887->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 180u IPv6 153509124 TCP hadoopmaster.cluster.trafficbroker.co.uk:52742->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 181u IPv6 153411087 TCP hadoopmaster.cluster.trafficbroker.co.uk:52252->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 182u IPv6 153411089 TCP hadoopmaster.cluster.trafficbroker.co.uk:52253->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 183u IPv6 153460228 TCP hadoopmaster.cluster.trafficbroker.co.uk:50022->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 184u IPv6 153428120 TCP hadoopmaster.cluster.trafficbroker.co.uk:59224->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 185u IPv6 153428121 TCP hadoopmaster.cluster.trafficbroker.co.uk:33391->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 187u IPv6 153600041 TCP hadoopmaster.cluster.trafficbroker.co.uk:55643->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 188u IPv6 153557843 TCP hadoopmaster.cluster.trafficbroker.co.uk:44136->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 189u IPv6 153462763 TCP hadoopmaster.cluster.trafficbroker.co.uk:51620->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 190u IPv6 153460417 TCP hadoopmaster.cluster.trafficbroker.co.uk:35784->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 191u IPv6 153460418 TCP hadoopmaster.cluster.trafficbroker.co.uk:48084->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 192u IPv6 153462764 TCP hadoopmaster.cluster.trafficbroker.co.uk:50062->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 193u IPv6 153471908 TCP hadoopmaster.cluster.trafficbroker.co.uk:46818->hadoopslave9.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 194u IPv6 153468521 TCP hadoopmaster.cluster.trafficbroker.co.uk:48176->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 195u IPv6 153468522 TCP hadoopmaster.cluster.trafficbroker.co.uk:54246->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 196u IPv6 153471980 TCP hadoopmaster.cluster.trafficbroker.co.uk:52708->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 197u IPv6 153557844 TCP hadoopmaster.cluster.trafficbroker.co.uk:44137->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 198u IPv6 153476603 TCP hadoopmaster.cluster.trafficbroker.co.uk:47901->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 199u IPv6 153476604 TCP hadoopmaster.cluster.trafficbroker.co.uk:36379->hadoopslave11.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 200u IPv6 153508848 TCP hadoopmaster.cluster.trafficbroker.co.uk:60022->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 201u IPv6 153507533 TCP hadoopmaster.cluster.trafficbroker.co.uk:37573->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 202u IPv6 153507534 TCP hadoopmaster.cluster.trafficbroker.co.uk:56770->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 203u IPv6 153508419 TCP hadoopmaster.cluster.trafficbroker.co.uk:49127->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 204u IPv6 153508234 TCP hadoopmaster.cluster.trafficbroker.co.uk:43983->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 205u IPv6 153508235 TCP hadoopmaster.cluster.trafficbroker.co.uk:49488->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 206u IPv6 153508422 TCP hadoopmaster.cluster.trafficbroker.co.uk:49128->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 207u IPv6 153509125 TCP hadoopmaster.cluster.trafficbroker.co.uk:42097->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 209u IPv6 153508845 TCP hadoopmaster.cluster.trafficbroker.co.uk:49517->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 210u IPv6 153508846 TCP hadoopmaster.cluster.trafficbroker.co.uk:59266->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 211u IPv6 153572667 TCP hadoopmaster.cluster.trafficbroker.co.uk:50125->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 212u IPv6 153508849 TCP hadoopmaster.cluster.trafficbroker.co.uk:60023->hadoopslave16.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 213u IPv6 153516692 TCP hadoopmaster.cluster.trafficbroker.co.uk:38832->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 214u IPv6 153516695 TCP hadoopmaster.cluster.trafficbroker.co.uk:38833->hadoopslave7.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 216u IPv6 153610112 TCP hadoopmaster.cluster.trafficbroker.co.uk:52494->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 217u IPv6 153610114 TCP hadoopmaster.cluster.trafficbroker.co.uk:52495->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 218u IPv6 153557944 TCP hadoopmaster.cluster.trafficbroker.co.uk:34058->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 219u IPv6 153557945 TCP hadoopmaster.cluster.trafficbroker.co.uk:50784->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 220u IPv6 153572668 TCP hadoopmaster.cluster.trafficbroker.co.uk:50126->hadoopmaster.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 221u IPv6 153585054 TCP hadoopmaster.cluster.trafficbroker.co.uk:43430->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 222u IPv6 153582445 TCP hadoopmaster.cluster.trafficbroker.co.uk:57895->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 223u IPv6 153582447 TCP hadoopmaster.cluster.trafficbroker.co.uk:40966->hadoopslave6.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 224u IPv6 153585055 TCP hadoopmaster.cluster.trafficbroker.co.uk:43431->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 225u IPv6 153708561 TCP hadoopmaster.cluster.trafficbroker.co.uk:36958->hadoopslave8.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 226u IPv6 153594476 TCP hadoopmaster.cluster.trafficbroker.co.uk:43585->hadoopslave20.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 227u IPv6 153638916 TCP hadoopmaster.cluster.trafficbroker.co.uk:49413->hadoopslave18.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 229u IPv6 153708562 TCP hadoopmaster.cluster.trafficbroker.co.uk:56463->hadoopslave12.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 230u IPv6 153600134 TCP hadoopmaster.cluster.trafficbroker.co.uk:55644->hadoopslave1.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 231u IPv6 153638922 TCP hadoopmaster.cluster.trafficbroker.co.uk:54177->hadoopslave12.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 232u IPv6 153635793 TCP hadoopmaster.cluster.trafficbroker.co.uk:53403->hadoopslave2.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 233u IPv6 153708619 TCP hadoopmaster.cluster.trafficbroker.co.uk:33396->hadoopslave7.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 234u IPv6 153703803 TCP hadoopmaster.cluster.trafficbroker.co.uk:59998->hadoopslave13.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 235u IPv6 153703804 TCP hadoopmaster.cluster.trafficbroker.co.uk:40745->hadoopslave3.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 236u IPv6 153604398 TCP hadoopmaster.cluster.trafficbroker.co.uk:41618->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 237u IPv6 153604506 TCP hadoopmaster.cluster.trafficbroker.co.uk:41619->hadoopslave10.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 238u IPv6 153708578 TCP hadoopmaster.cluster.trafficbroker.co.uk:46417->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 241u IPv6 153635795 TCP hadoopmaster.cluster.trafficbroker.co.uk:59353->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 244r FIFO 0,5 154318314 pipe java 16908 hadoop 245u IPv6 153706355 TCP hadoopmaster.cluster.trafficbroker.co.uk:53521->hadoopslave19.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 246u IPv6 153706356 TCP hadoopmaster.cluster.trafficbroker.co.uk:41546->hadoopslave4.cluster.trafficbroker.co.uk:50010 (ESTABLISHED) java 16908 hadoop 247w FIFO 0,5 154318314 pipe java 16908 hadoop 248u IPv6 153708579 TCP hadoopmaster.cluster.trafficbroker.co.uk:46418->hadoopslave6.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 249w REG 8,1 15230 5043335 /tmp/hadoop/hive_job_log_hadoop_201002152251_963848561.txt java 16908 hadoop 250u IPv6 153708593 TCP hadoopmaster.cluster.trafficbroker.co.uk:57193->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 251u IPv6 153708594 TCP hadoopmaster.cluster.trafficbroker.co.uk:57194->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 252u IPv6 153708620 TCP hadoopmaster.cluster.trafficbroker.co.uk:52501->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 253u 0000 0,6 0 493 /anon_inode:[eventpoll] java 16908 hadoop 255u IPv6 153742520 TCP hadoopmaster.cluster.trafficbroker.co.uk:35222->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 256u IPv6 153742521 TCP hadoopmaster.cluster.trafficbroker.co.uk:35223->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 259u IPv6 153742533 TCP hadoopmaster.cluster.trafficbroker.co.uk:60662->hadoopslave9.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 260u IPv6 153742534 TCP hadoopmaster.cluster.trafficbroker.co.uk:60641->hadoopslave4.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 262u IPv6 153742547 TCP hadoopmaster.cluster.trafficbroker.co.uk:57104->hadoopslave19.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 263u IPv6 153742548 TCP hadoopmaster.cluster.trafficbroker.co.uk:35235->hadoopslave14.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 265u IPv6 153742562 TCP hadoopmaster.cluster.trafficbroker.co.uk:39561->hadoopslave20.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) java 16908 hadoop 266u IPv6 153742563 TCP hadoopmaster.cluster.trafficbroker.co.uk:51464->hadoopslave5.cluster.trafficbroker.co.uk:50010 (CLOSE_WAIT) On 15 Feb 2010, at 22:42, Zheng Shao wrote: > Can you go to that box, sudo as root, and do "lsof | grep 12345" where > 12345 is the process id of the hive server? > We should be able to see the names of the files that are open. > > Zheng > > On Mon, Feb 15, 2010 at 7:42 AM, Andy Kent <[email protected]> wrote: >> Nope, no luck so far. >> >> We have upped the number of file descriptors and are having to restart hive >> every week or so :( >> >> Any other suggestions would be greatly appreciated. >> >> On 15 Feb 2010, at 14:09, Bennie Schut wrote: >> >>> Did this help? I'm running into a similar problem. slowly leaking >>> connections to 50010 and after a hive restart all is ok again. >>> >>> Andy Kent wrote: >>>> I can give try and give it a go. I'm not convinced though as we are >>>> working with CSV files and don't touch sequence files at all at the moment. >>>> >>>> We are using the Clodera Ubuntu Packages for Hadoop 0.20.1+133 and Hive >>>> 0.40 >>>> >>>> >>>> On 25 Jan 2010, at 15:30, Jay Booth wrote: >>>> >>>> >>>>> Actually, we had an issue with this, it was a bug in SequenceFile where >>>>> if there were problems opening a file, it would leave a filehandle open >>>>> and never close it. >>>>> >>>>> Here's the patch -- It's already fixed in 0.21/trunk, if I get some time >>>>> this week I'll submit it against 0.20.2 -- could you apply this to hadoop >>>>> and let me know if it fixes things for you? >>>>> >>>>> On Mon, Jan 25, 2010 at 10:11 AM, Jay Booth >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> Yeah, I'd guess that this is a Hive issue, although it could be a >>>>> combination.. maybe if you're doing queries and then closing your thrift >>>>> connection before reading all results, Hive doesn't know what to do and >>>>> leaves the connection open? Once the west coast folks wake up, they >>>>> might have a better answer for you than I do. >>>>> >>>>> >>>>> On Mon, Jan 25, 2010 at 9:06 AM, Andy Kent >>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>> On 25 Jan 2010, at 13:59, Jay Booth wrote: >>>>> >>>>> >>>>>> That's the datanode port.. if I had to guess, Hive's connecting to DFS >>>>>> directly for some reason (maybe for "select *" queries?) and not >>>>>> finishing their reads or closing the connections after. >>>>>> >>>>> Thanks for the response. >>>>> >>>>> That's what I was suspecting. I have triple checked and our Ruby code and >>>>> it is defiantly closing it's thrift connections properly. >>>>> >>>>> I'll try running some different queries and see if I can suss out some >>>>> examples of which ones are leaky. Is this something that I should post to >>>>> Jira or is it a known issue? I can't believe other people haven't noticed >>>>> this? >>>>> >>>>> >>>>> <SequenceFile.patch> >>>>> >>>> >>>> >>> >>> >> >> > > > > -- > Yours, > Zheng >
