Hi, This seems to be related to HDFS configuration and not necessarily a Sqoop issue. Please check your NN/DN logs to see if there are other exceptions that could give an idea of what is going on with your cluster.
Regards, Arvind Prabhakar On Mon, Jun 18, 2012 at 8:00 AM, cmdr taco <[email protected]> wrote: > When Sqoop tries to import data into Hive, it creates a data file > under.i.e. /export/home/myusername, before publishing the data to the hive > table. > > I'm wondering if the $HOME should be part of the HDFS...? .I'm assuming > HDFS is not able to replicate the data when using direct mode for high > volume table. > > Thx > Taco!! > > > > On Fri, Jun 15, 2012 at 10:03 AM, cmdr taco <[email protected]>wrote: > >> Experts!! >> >> I use sqoop to import data from RDBMS (postgres) to HIVE Table >> and Occasionally the Job fails with below exception..any idea, why this >> happens..What I noticed is the failure only happens with Larger Volume >> tables. >> >> Log: >> >> >> >> 12/06/15 03:46:35 INFO hdfs.DFSClient: >> org.apache.hadoop.ipc.RemoteException: >> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not >> replicated yet:/user/etl_user/fact_web_events/data-00000 >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1401) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:588) >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1054) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) >> at $Proxy0.addBlock(Unknown Source) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >> at $Proxy0.addBlock(Unknown Source) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3166) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3036) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClient.java:2288) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2483) >> >> 12/06/15 03:46:35 WARN hdfs.DFSClient: NotReplicatedYetException >> sleeping /user/etl_user/fact_web_events/data-00000 retries left 4 >> >> >
