I don't know why I can't see my emails immediately sent to the group ...
anyways,
I'm sorting a sequenceFile using it's sorter on my local filesystem. The
inputFile size is 1937690478 bytes.
but after 14 minutes of sorting.. I get :
TEST SORTING ..
java.io.FileNotFoundException: File does not exist:
/usr/mark/tmp/mapred/local/SortedOutput.0
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1353)
at
org.apache.hadoop.io.SequenceFile$Sorter.cloneFileAttributes(SequenceFile.java:2663)
at
org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:2712)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2285)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2324)
at
CrossPartitionSimilarity.TestSorter(CrossPartitionSimilarity.java:164)
at CrossPartitionSimilarity.main(CrossPartitionSimilarity.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Yet, the file is still there: wc -c SortedOutput.0 ---> 1918661230
../tmp/mapred/local/SortedOutput.0
and if it is because of space, I checked and it can hold up to 209 GB. So,
my question are there restrictions on some JVM configurations that I should
take care of ?
Thank you,
Maha
