First, here is my environment:
Hadoop 1.2.1
Accumulo 1.4.4
Zookeeper 3.4.5
Gora 0.3
Solr 4.5.1
I have been trying to get a 4 node Hadoop cluster to start a distributed
crawl but have been running into some issue with just injecting the seeds.
I have successfully been able to get the runtime/local nutch crawl to work
(obviously including the inject step) but when I switched over to try the
runtime/deploy nutch job, the inject step would just timeout.
I have turned up all logging levels I could and tried to figure out what is
causing the issue, but I'm running out of ideas. One thing I did notice is
that I see several Counters getting run (SPILLED_RECORDS, SPLIT_RAW_BYTES,
MAP_INPUT_RECORDS, BYTES_READ, MAP_OUTPUT_RECORDS, BYTES_WRITTEN), but in
the job page, I only ever see "Spilled Records 0".
I can also confirm that my seeds file exists in hadoop and there is no
blank line at the end of the seeds.txt file (which caused an inject issue
in the runtime/local process).
$ hadoop dfs -ls /user/search/urls/
Found 1 items
-rw-r--r-- 3 search supergroup 1139 2013-11-12 16:38
/user/search/urls/seeds.txt
Here is the syslog for the job I create:
2013-11-14 15:35:43,107 DEBUG org.apache.hadoop.mapred.Child: Child starting
2013-11-14 15:35:43,443 DEBUG org.apache.hadoop.security.Groups: Creating
new Groups object
2013-11-14 15:35:43,448 DEBUG org.apache.hadoop.security.Groups: Group
mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
2013-11-14 15:35:43,477 DEBUG
org.apache.hadoop.security.UserGroupInformation: hadoop login
2013-11-14 15:35:43,477 DEBUG
org.apache.hadoop.security.UserGroupInformation: hadoop login commit
2013-11-14 15:35:43,481 DEBUG
org.apache.hadoop.security.UserGroupInformation: using local
user:UnixPrincipal: search
2013-11-14 15:35:43,483 DEBUG
org.apache.hadoop.security.UserGroupInformation: UGI loginUser:search
2013-11-14 15:35:43,487 DEBUG org.apache.hadoop.fs.FileSystem: Creating
filesystem for
file:///tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jobToken
2013-11-14 15:35:43,499 DEBUG
org.apache.hadoop.mapreduce.security.TokenCache: Task: Loaded jobTokenFile
from:
/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jobToken;
num of sec keys = 0 Number of tokens 1
2013-11-14 15:35:43,499 DEBUG org.apache.hadoop.mapred.Child: loading
token. # keys =0; from
file=/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jobToken
2013-11-14 15:35:43,500 DEBUG
org.apache.hadoop.security.UserGroupInformation: PriviledgedAction
as:job_201311141534_0001
from:org.apache.hadoop.mapred.Child.main(Child.java:109)
2013-11-14 15:35:43,523 DEBUG org.apache.hadoop.ipc.Client: The ping
interval is60000ms.
2013-11-14 15:35:43,524 DEBUG org.apache.hadoop.ipc.Client: Use SIMPLE
authentication for protocol TaskUmbilicalProtocol
2013-11-14 15:35:43,525 DEBUG org.apache.hadoop.ipc.Client: Connecting to /
127.0.0.1:37341
2013-11-14 15:35:43,543 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #0
2013-11-14 15:35:43,547 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001: starting, having
connections 1
2013-11-14 15:35:43,549 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #0
2013-11-14 15:35:43,555 DEBUG org.apache.hadoop.ipc.RPC: Call:
getProtocolVersion 36
2013-11-14 15:35:43,556 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #1
2013-11-14 15:35:43,558 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #1
2013-11-14 15:35:43,573 DEBUG org.apache.hadoop.mapred.SortedRanges:
currentIndex 0 0:0
2013-11-14 15:35:43,614 DEBUG org.apache.hadoop.mapred.Counters: Creating
group org.apache.hadoop.mapred.Task$Counter with bundle
2013-11-14 15:35:43,614 DEBUG org.apache.hadoop.mapred.Counters: Adding
SPILLED_RECORDS
2013-11-14 15:35:43,616 DEBUG org.apache.hadoop.mapred.Counters: Creating
group org.apache.hadoop.mapred.Task$Counter with bundle
2013-11-14 15:35:43,616 DEBUG org.apache.hadoop.mapred.SortedRanges:
currentIndex 0 0:0
2013-11-14 15:35:43,616 DEBUG org.apache.hadoop.mapred.SortedRanges:
currentIndex 1 0:0
2013-11-14 15:35:43,616 DEBUG org.apache.hadoop.ipc.RPC: Call: getTask 60
2013-11-14 15:35:43,676 DEBUG org.apache.hadoop.mapred.TaskRunner:
mapred.local.dir for child :
/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0
2013-11-14 15:35:43,685 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Trying to load the custom-built native-hadoop library...
2013-11-14 15:35:43,685 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-11-14 15:35:43,756 DEBUG org.apache.hadoop.mapred.TaskRunner: Fully
deleting contents of
/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0/work
2013-11-14 15:35:43,949 DEBUG org.apache.hadoop.mapred.Child: Creating
remote user to execute task: search
2013-11-14 15:35:43,967 DEBUG
org.apache.hadoop.security.UserGroupInformation: PriviledgedAction
as:search from:org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-11-14 15:35:43,970 DEBUG org.apache.hadoop.fs.FileSystem: Creating
filesystem for hdfs://dev1o:9000
2013-11-14 15:35:44,054 DEBUG org.apache.hadoop.io.retry.RetryUtils:
multipleLinearRandomRetry = null
2013-11-14 15:35:44,060 DEBUG org.apache.hadoop.ipc.Client: The ping
interval is60000ms.
2013-11-14 15:35:44,060 DEBUG org.apache.hadoop.ipc.Client: Use SIMPLE
authentication for protocol ClientProtocol
2013-11-14 15:35:44,060 DEBUG org.apache.hadoop.ipc.Client: Connecting to
dev1o/10.2.156.115:9000
2013-11-14 15:35:44,064 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search sending #2
2013-11-14 15:35:44,065 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search: starting, having
connections 2
2013-11-14 15:35:44,066 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search got value #2
2013-11-14 15:35:44,067 DEBUG org.apache.hadoop.ipc.RPC: Call:
getProtocolVersion 7
2013-11-14 15:35:44,067 DEBUG org.apache.hadoop.hdfs.DFSClient: Short
circuit read is false
2013-11-14 15:35:44,067 DEBUG org.apache.hadoop.hdfs.DFSClient: Connect to
datanode via hostname is false
2013-11-14 15:35:44,070 DEBUG org.apache.hadoop.mapred.Task: using new api
for output committer
2013-11-14 15:35:44,082 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2013-11-14 15:35:44,086 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@27982
2013-11-14 15:35:44,152 DEBUG
org.apache.hadoop.util.ProcfsBasedProcessTree: [ 26877 ]
2013-11-14 15:35:44,160 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search sending #3
2013-11-14 15:35:44,162 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search got value #3
2013-11-14 15:35:44,165 DEBUG org.apache.hadoop.ipc.RPC: Call:
getBlockLocations 5
2013-11-14 15:35:44,166 DEBUG org.apache.hadoop.hdfs.DFSClient: Connecting
to /10.2.156.136:50010
2013-11-14 15:35:44,169 DEBUG org.apache.hadoop.fs.FSInputChecker:
DFSClient readChunk got seqno 0 offsetInBlock 0 lastPacketInBlock true
packetLen 151
2013-11-14 15:35:44,174 DEBUG org.apache.hadoop.mapred.Counters: Adding
SPLIT_RAW_BYTES
2013-11-14 15:35:44,175 INFO org.apache.hadoop.mapred.MapTask: Processing
split: hdfs://dev1o:9000/user/search/urls/seeds.txt:0+1139
2013-11-14 15:35:44,176 DEBUG org.apache.hadoop.mapred.Counters: Adding
MAP_INPUT_RECORDS
2013-11-14 15:35:44,179 DEBUG org.apache.hadoop.mapred.Counters: Creating
group org.apache.hadoop.mapreduce.lib.input.FileInputFormat$Counter with
bundle
2013-11-14 15:35:44,179 DEBUG org.apache.hadoop.mapred.Counters: Adding
BYTES_READ
2013-11-14 15:35:44,182 DEBUG org.apache.hadoop.mapred.Counters: Adding
MAP_OUTPUT_RECORDS
2013-11-14 15:35:44,185 DEBUG org.apache.hadoop.mapred.Counters: Creating
group org.apache.hadoop.mapreduce.lib.output.FileOutputFormat$Counter with
bundle
2013-11-14 15:35:44,185 DEBUG org.apache.hadoop.mapred.Counters: Adding
BYTES_WRITTEN
2013-11-14 15:35:44,479 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14 GMT
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=dev6o
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_30
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/data/java/jdk1.6.0_30/jre
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/data/search/hadoop/hadoop-1.2.1/libexec/../conf:/webroot/java/default_jdk/lib/tools.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/classes:/data/search/hadoop/hadoop-1.2.1/libexec/../build:/data/search/hadoop/hadoop-1.2.1/libexec/../build/test/classes:/data/search/hadoop/hadoop-1.2.1/libexec/../build/tools:/data/search/hadoop/hadoop-1.2.1/libexec/..:/data/search/hadoop/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/slf4j-api-1.4.3.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/slf4j-log4j12-1.4.3.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/asm-3.2.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/aspectjrt-1.6.11.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/aspectjtools-1.6.11.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-beanutils-1.7.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-beanutils-core-1.8.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-cli-1.2.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-codec-1.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-collections-3.2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-configuration-1.6.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-daemon-1.0.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-digester-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-el-1.0.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-httpclient-3.0.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-io-2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-lang-2.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-logging-1.1.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-logging-api-1.0.4.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-math-2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/commons-net-3.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/core-3.1.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jackson-core-asl-1.8.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jackson-mapper-asl-1.8.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jasper-compiler-5.5.12.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jasper-runtime-5.5.12.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jdeb-0.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jersey-core-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jersey-json-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jersey-server-1.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jets3t-0.6.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jetty-6.1.26.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jetty-util-6.1.26.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/jsch-0.1.42.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/junit-4.5.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/log4j-1.2.15.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/mockito-all-1.8.5.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/oro-2.0.8.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/servlet-api-2.5-20081211.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/slf4j-api-1.4.3.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/slf4j-log4j12-1.4.3.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../build/ivy/lib/Hadoop/common/xmlenc-0.52.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/data/search/hadoop/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/asm-3.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/xmlenc-0.52.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-lang-2.6.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jsr305-1.3.9.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jackson-core-asl-1.8.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/gora-accumulo-0.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/geronimo-stax-api_1.0_spec-1.0.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/wstx-asl-3.2.7.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-cli-1.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/org.restlet-2.0.5.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-beanutils-core-1.8.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jetty-util5-6.1.26.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/ftplet-api-1.0.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/neethi-3.0.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-digester-1.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/guava-11.0.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/xmlschema-core-2.0.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/zookeeper-3.3.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/juniversalchardet-1.0.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-rt-transports-common-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-aop-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/servlet-api-2.4.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/xmlParserAPIs-2.6.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-beanutils-1.7.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/gora-core-0.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/servlet-api-2.5-20081211.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-api-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/log4j-1.2.16.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/lucene-memory-3.6.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/mina-core-2.0.0-M5.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jets3t-0.6.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/ftpserver-core-1.0.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jersey-core-1.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-rt-core-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/paranamer-2.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/wsdl4j-1.6.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/tika-core-1.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jdom-1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-logging-1.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-web-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/httpcore-4.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-core-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/org.restlet.ext.jackson-2.0.5.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/activation-1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-rt-bindings-xml-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jaxb-impl-2.2.3-1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/xercesImpl-2.9.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/libthrift-0.6.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jersey-json-1.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/paranamer-generator-2.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-rt-transports-http-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/lucene-queries-3.6.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/lucene-core-3.6.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jackson-xc-1.7.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/geronimo-javamail_1.4_spec-1.7.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/serializer-2.7.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/hamcrest-core-1.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/org.osgi.core-4.0.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/oro-2.0.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/accumulo-start-1.4.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jetty-sslengine-6.1.26.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/paranamer-ant-2.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-logging-api-1.0.4.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/junit-4.11.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jackson-mapper-asl-1.8.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jersey-server-1.8.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/solr-solrj-3.4.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-codec-1.4.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jackson-jaxrs-1.7.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/elasticsearch-0.19.4.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/lucene-analyzers-3.6.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cloudtrace-1.4.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/httpclient-4.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-asm-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/avro-1.3.3.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-rt-frontend-jaxrs-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/core-3.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jetty-util-6.1.26.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-jci-core-1.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/icu4j-4.0.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jaxb-api-2.2.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/accumulo-core-1.4.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-collections-3.2.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-expression-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-context-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/servlet-api-2.5-6.1.14.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/slf4j-log4j12-1.6.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jline-0.9.94.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/cxf-common-utilities-2.5.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jsr311-api-1.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-net-1.4.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/slf4j-api-1.6.6.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-configuration-1.6.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/spring-beans-3.0.6.RELEASE.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/aopalliance-1.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-math-2.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jettison-1.3.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/stax-api-1.0.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/woodstox-core-asl-4.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jetty-6.1.26.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/stax2-api-3.1.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/jetty-client-6.1.26.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-jci-fam-1.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/stax-api-1.0-2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/lucene-highlighter-3.6.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-el-1.0.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/xml-apis-1.3.04.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/qdox-1.10.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-io-2.4.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/ftpserver-deprecated-1.0.0-M2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/commons-httpclient-3.1.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/lib/crawler-commons-0.2.jar:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars/classes:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/jars:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0/work
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/data/search/hadoop/hadoop-1.2.1/libexec/../lib/native/Linux-i386-32:/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0/work
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0/work/tmp
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=i386
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-358.23.2.el6.x86_64
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=search
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/data/home/search
2013-11-14 15:35:44,480 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/tmp/hadoop-search/mapred/local/taskTracker/search/jobcache/job_201311141534_0001/attempt_201311141534_0001_m_000000_0/work
2013-11-14 15:35:44,481 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=dev1o,dev5o,dev6o sessionTimeout=30000
watcher=org.apache.accumulo.core.zookeeper.ZooSession$AccumuloWatcher@df2d38
2013-11-14 15:35:44,483 DEBUG org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2013-11-14 15:35:44,493 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server dev1o/10.2.156.115:2181
2013-11-14 15:35:44,494 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to dev1o/10.2.156.115:2181, initiating session
2013-11-14 15:35:44,496 DEBUG org.apache.zookeeper.ClientCnxn: Session
establishment request sent on dev1o/10.2.156.115:2181
2013-11-14 15:35:44,503 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server dev1o/10.2.156.115:2181, sessionid =
0x142573f8b440007, negotiated timeout = 30000
2013-11-14 15:35:44,601 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 1,3 replyHeader:: 1,81604378745,0 request::
'/accumulo/instances/dev1o,T response::
s{25769803868,25769803868,1384364228759,1384364228759,0,0,0,0,36,0,25769803868}
2013-11-14 15:35:44,604 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 2,4 replyHeader:: 2,81604378745,0 request::
'/accumulo/instances/dev1o,T response::
#30393062626266622d323561622d343766352d393637622d646330386161306366343731,s{25769803868,25769803868,1384364228759,1384364228759,0,0,0,0,36,0,25769803868}
2013-11-14 15:35:44,609 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 3,3 replyHeader:: 3,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471,T response::
s{25769803869,25769803869,1384364228762,1384364228762,0,16,0,0,0,16,25769803957}
2013-11-14 15:35:44,610 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 4,4 replyHeader:: 4,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471,T response::
,s{25769803869,25769803869,1384364228762,1384364228762,0,16,0,0,0,16,25769803957}
2013-11-14 15:35:44,631 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 5,8 replyHeader:: 5,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/tservers,T response:: v{'
10.2.156.115:9997}
2013-11-14 15:35:44,633 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 6,8 replyHeader:: 6,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/tservers/10.2.156.115:9997,T
response:: v{'zlock-0000000000}
2013-11-14 15:35:44,634 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 7,3 replyHeader:: 7,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/tservers/
10.2.156.115:9997/zlock-0000000000,T response::
s{81604378640,81604378640,1384443283392,1384443283392,0,0,0,90730872931876864,30,0,81604378640}
2013-11-14 15:35:44,635 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 8,4 replyHeader:: 8,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/tservers/
10.2.156.115:9997/zlock-0000000000,T response::
#54534552565f434c49454e543d31302e322e3135362e3131353a39393937,s{81604378640,81604378640,1384443283392,1384443283392,0,0,0,90730872931876864,30,0,81604378640}
2013-11-14 15:35:44,693 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 9,8 replyHeader:: 9,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/masters/lock,T response::
v{'zlock-0000000023}
2013-11-14 15:35:44,695 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 10,3 replyHeader:: 10,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/masters/lock/zlock-0000000023,T
response::
s{81604378645,81604378645,1384443287585,1384443287585,0,0,0,234846061027328001,17,0,81604378645}
2013-11-14 15:35:44,696 DEBUG org.apache.zookeeper.ClientCnxn: Reading
reply sessionid:0x142573f8b440007, packet:: clientPath:null serverPath:null
finished:false header:: 11,4 replyHeader:: 11,81604378745,0 request::
'/accumulo/090bbbfb-25ab-47f5-967b-dc08aa0cf471/masters/lock/zlock-0000000023,T
response::
#31302e322e3135362e3131353a39393939,s{81604378645,81604378645,1384443287585,1384443287585,0,0,0,234846061027328001,17,0,81604378645}
2013-11-14 15:35:47,077 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #4
2013-11-14 15:35:47,078 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #4
2013-11-14 15:35:47,078 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 2
2013-11-14 15:35:50,078 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #5
2013-11-14 15:35:50,079 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #5
2013-11-14 15:35:50,079 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:35:53,079 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #6
2013-11-14 15:35:53,080 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #6
2013-11-14 15:35:53,081 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 2
2013-11-14 15:35:54,160 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search: closed
2013-11-14 15:35:54,160 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to dev1o/10.2.156.115:9000 from search: stopped, remaining
connections 1
2013-11-14 15:35:54,701 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid: 0x142573f8b440007 after 1ms
2013-11-14 15:35:56,081 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #7
2013-11-14 15:35:56,082 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #7
2013-11-14 15:35:56,082 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:35:59,082 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #8
2013-11-14 15:35:59,083 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #8
2013-11-14 15:35:59,083 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:36:02,084 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #9
2013-11-14 15:36:02,085 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #9
2013-11-14 15:36:02,085 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 2
2013-11-14 15:36:04,709 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid: 0x142573f8b440007 after 0ms
2013-11-14 15:36:05,085 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #10
2013-11-14 15:36:05,086 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #10
2013-11-14 15:36:05,086 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:36:08,086 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #11
2013-11-14 15:36:08,087 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #11
2013-11-14 15:36:08,087 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:36:11,088 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #12
2013-11-14 15:36:11,088 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #12
2013-11-14 15:36:11,089 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 2
2013-11-14 15:36:14,089 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #13
2013-11-14 15:36:14,090 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #13
2013-11-14 15:36:14,090 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:36:14,718 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
response for sessionid: 0x142573f8b440007 after 0ms
2013-11-14 15:36:17,091 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #14
2013-11-14 15:36:17,092 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #14
2013-11-14 15:36:17,092 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:36:20,092 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #15
2013-11-14 15:36:20,093 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #15
2013-11-14 15:36:20,094 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 2
2013-11-14 15:45:41,347 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 sending #202
2013-11-14 15:45:41,348 DEBUG org.apache.hadoop.ipc.Client: IPC Client (47)
connection to /127.0.0.1:37341 from job_201311141534_0001 got value #202
2013-11-14 15:45:41,348 DEBUG org.apache.hadoop.ipc.RPC: Call: ping 1
2013-11-14 15:45:43,050 DEBUG org.apache.hadoop.fs.FileSystem: Starting
clear of FileSystem cache with 2 elements.
2013-11-14 15:45:43,052 DEBUG org.apache.hadoop.fs.FileSystem: Removing
filesystem for hdfs://dev1o:9000
2013-11-14 15:45:43,052 DEBUG org.apache.hadoop.fs.FileSystem: Removing
filesystem for file:///
2013-11-14 15:45:43,053 DEBUG org.apache.hadoop.fs.FileSystem: Removing
filesystem for file:///
2013-11-14 15:45:43,053 DEBUG org.apache.hadoop.fs.FileSystem: Done
clearing cache
--
Jon Uhal
