When Sqoop tries to import data into Hive, it creates a data file under.i.e. /export/home/myusername, before publishing the data to the hive table.
I'm wondering if the $HOME should be part of the HDFS...? .I'm assuming HDFS is not able to replicate the data when using direct mode for high volume table. Thx Taco!! On Fri, Jun 15, 2012 at 10:03 AM, cmdr taco <[email protected]> wrote: > Experts!! > > I use sqoop to import data from RDBMS (postgres) to HIVE Table > and Occasionally the Job fails with below exception..any idea, why this > happens..What I noticed is the failure only happens with Larger Volume > tables. > > Log: > > > > 12/06/15 03:46:35 INFO hdfs.DFSClient: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not > replicated yet:/user/etl_user/fact_web_events/data-00000 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1401) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:588) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313) > > at org.apache.hadoop.ipc.Client.call(Client.java:1054) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) > at $Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3166) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3036) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClient.java:2288) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2483) > > 12/06/15 03:46:35 WARN hdfs.DFSClient: NotReplicatedYetException sleeping > /user/etl_user/fact_web_events/data-00000 retries left 4 > >
