One thing i noticed is 3 of my regions servers shut down:
giving the following errors:
2009-04-05 00:08:45,727 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
7674788444062751362 lease expired
2009-04-05 00:08:45,731 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [...@132a2bc closing
scanner
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.close(StoreFileScanner.java:356)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.closeScanner(HStoreScanner.java:289)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.doClose(HStoreScanner.java:309)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.close(HStoreScanner.java:303)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.closeScanner(HRegion.java:2119)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.close(HRegion.java:2139)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:1759)
at org.apache.hadoop.hbase.Leases.run(Leases.java:95)
2009-04-05 00:08:45,732 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [...@159f8be closing
scanner
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.close(StoreFileScanner.java:356)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.closeScanner(HStoreScanner.java:289)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.doClose(HStoreScanner.java:309)
at
org.apache.hadoop.hbase.regionserver.HStoreScanner.close(HStoreScanner.java:303)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.closeScanner(HRegion.java:2119)
at
org.apache.hadoop.hbase.regionserver.HRegion$HScanner.close(HRegion.java:2139)
at
org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:1759)
at org.apache.hadoop.hbase.Leases.run(Leases.java:95)
2009-04-05 00:08:45,732 ERROR
org.apache.hadoop.hbase.regionserver.StoreFileScanner: [...@1d9ba08 closing
scanner
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:1573)
at java.io.FilterInputStream.close(FilterInputStream.java:155)
at
org.apache.hadoop.hbase.io.SequenceFile$Reader.close(SequenceFile.java:1598)
at org.apache.hadoop.hbase.io.MapFile$Reader.close(MapFile.java:586)
......
what could be the reason why this is happening?
is it a problem because i am performing a split on the table inside my map??
Thanks
Raakhi.
>
>
> On Sun, Apr 5, 2009 at 12:18 PM, Rakhi Khatwani
> <[email protected]>wrote:
>
>> Hi,
>> I am executing a job on ec2 (set up on cluster with 18 nodes... my
>> job has 7 Map tasks). however my tasks gets killed without reporting an
>> error.
>>
>> i even tried going through the logs, which happens to be fine.
>>
>> On the UI the tasks fail and the status shows as KILLED (error column
>> being empty).
>> and eventually the job fails.
>>
>> what could go wrong?? I am finding it weird since this is the 1st time i
>> am comming across this kinda failure.
>>
>> Thanks,
>> Raakhi
>>
>
>
