Hello,
I want to append to a HDFS file and I use the following code:
val fs: FileSystem = super.testFileSystem
val path: Path = new Path("/a/b.txt")
val os: FSDataOutputStream = fs.create(path)
os.writeBytes("asd")
os.close()
println("file length after first writing: " +
fs.getFileStatus(path).getLen)
val os2: FSDataOutputStream = fs.append(path)
os2.writeBytes("qwe")
os2.close()
println("file length after second writing: " +
fs.getFileStatus(path).getLen)
I am using a one-datanode test HDFS on localhost. Could somebody show me
what's the proper way to append to a file?
Sorry for crossposting, I just realized this is actually more hdfs-user
than hdfs-dev...
Thanks in advance, the exception follows:
0 [main] INFO com.ubeeko.client.model.hdfs.HdfsTreeTest - will
connect to hdfs://localhost:8020 as x
2012-02-17 18:34:16.788 java[20012:1903] Unable to load realm info from
SCDynamicStore
2012-02-17 18:34:16.790 java[20012:1903] Unable to load realm info from
SCDynamicStore
830 [main] INFO com.ubeeko.client.model.hdfs.HdfsTreeTest - will
connect to hdfs://localhost:8020 as x
Failed to close file /a/b.txt. Lease recovery is in progress. Try again
later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1359)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1187)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1112)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:328)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1493)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1487)
org.apache.hadoop.hdfs.protocol.RecoveryInProgressException: Failed to
close file /a/b.txt. Lease recovery is in progress. Try again later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1359)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1187)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1112)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:328)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1493)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1487)
at org.apache.hadoop.ipc.Client.call(Client.java:1086)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
at $Proxy6.create(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
at $Proxy6.create(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1245)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:878)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:836)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:217)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:774)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:755)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:654)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:643)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest$$anonfun$5.apply$mcV$sp(HdfsTreeTest.scala:76)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest$$anonfun$5.apply(HdfsTreeTest.scala:73)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest$$anonfun$5.apply(HdfsTreeTest.scala:73)
at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:1265)
at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest.withFixture(HdfsTreeTest.scala:10)
at
org.scalatest.FunSuite$class.invokeWithFixture$1(FunSuite.scala:1262)
at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
at org.scalatest.FunSuite$$anonfun$runTest$1.apply(FunSuite.scala:1271)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
at org.scalatest.FunSuite$class.runTest(FunSuite.scala:1271)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest.runTest(HdfsTreeTest.scala:10)
at
org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
at
org.scalatest.FunSuite$$anonfun$runTests$1.apply(FunSuite.scala:1304)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:281)
at org.scalatest.FunSuite$class.runTests(FunSuite.scala:1304)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest.runTests(HdfsTreeTest.scala:10)
at org.scalatest.Suite$class.run(Suite.scala:2286)
at
com.ubeeko.client.model.hdfs.HdfsTreeTest.org$scalatest$FunSuite$$super$run(HdfsTreeTest.scala:10)
at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
at org.scalatest.FunSuite$$anonfun$run$1.apply(FunSuite.scala:1310)
at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
at org.scalatest.FunSuite$class.run(FunSuite.scala:1310)
at com.ubeeko.client.model.hdfs.HdfsTreeTest.run(HdfsTreeTest.scala:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runSingleTest(ScalaTestRunner.java:97)
at
org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
