Lauren,
Can you post your pig script,
I don't know If this would help but I have seen a similar error when I was
creating
FileSystem through Java map reduce, example
String uri = "localhost"
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
this would create a error FileSystem Closed, where as when
my
String uri = "hdfs://localhost" it would work fine.
--
Richipal Singh
On Sun, Oct 21, 2012 at 9:34 AM, Lauren Blau <
[email protected]> wrote:
> I have a pig job that keeps failing at near completion. After 3 runs (long
> ones), I've finally found something out of the ordinary in a log:
> Anyone have any ideas what could be causing this?
>
> Thanks
>
> $Proxy1.complete(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
> at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1133)
> at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:243)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:349)
> at
> org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1616)
> at
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:1586)
> 2012-10-21 12:08:33,158 INFO
> org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
> truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2012-10-21 12:08:33,159 WARN org.apache.hadoop.mapred.Child: Error running
> child
> java.io.IOException: java.io.IOException: Filesystem closed
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:470)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:232)
> at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:70)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:3236)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:132)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:121)
> at
> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:112)
> at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
> at java.io.DataOutputStream.write(Unknown Source)
> at java.io.DataOutputStream.writeUTF(Unknown Source)
> at java.io.DataOutputStream.writeUTF(Unknown Source)
> at
> org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66)
> at
> org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:543)
> at
> org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> at
> org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> at
> org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> at
> org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> at
> org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> at
> org.apache.pig.impl.io.InterRecordWriter.write(InterRecordWriter.java:73)
> at
> org.apache.pig.impl.io.InterStorage.putNext(InterStorage.java:87)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:514)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:468)
> ... 11 more
>
> ------------------------------
>
