Hi, Cheolsoo and Jarcec, Thanks for your reply, it helps a lot.
I checked my network topology again and I found some nodes in the cluster can not connect to the database, I think that's why I got the error. 2012/5/17 Jarek Jarcec Cecho <[email protected]> > Hi Jason, > database and hadoop don't have to be on the same network. However sqoop is > generating map reduce job, so each your node have to be able to create > connection to database. Based on the exception "java.net.SocketException: > Network is unreachable" I would guess that some nodes where are you running > TaskTrackers are not allowed to reach your database server. > > You can verify whether particular node is allowed to access the database > using following command (execute from that particular node): > > mysql --host=<IP Address> --database=test --user=<username> > --password=<password> > > Jarcec > > On May 17, 2012, at 11:15 AM, jason Yang wrote: > > > Hello, > > > > Recently, I'm trying to import MySQL data into Hive by using sqoop. but > I have encountered the "Communications link failure" problem (the output is > attached at the end of this mail) > > > > To solve this problem, I have checked the user guide of sqoop and do the > solution mentioned in the troubleshooting section. however, It doesn't > work. After that, I read some posts about the basic process of sqoop > import, and I found that during the process of import, the sqoop would > submit a map-only Hadoop job to the cluster, which actually do the data > transfer works. so I was wondering that should the source database be in > the same network with the hadoop cluster? > > > > Any suggestion would be appreciated. > > > > > > --output-- > > [root@node10 ~]# sqoop import --connect jdbc:mysql:// > 172.18.11.54:3306/hive --username root -P --table test --hive-import -m1 > > Enter password: > > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific > delimiters for output. You can override > > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with > --fields-terminated-by, etc. > > 12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation > > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: > SELECT t.* FROM `test` AS t LIMIT 1 > > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: > SELECT t.* FROM `test` AS t LIMIT 1 > > 12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is > /srv/hadoop-0.20.2/bin/.. > > 12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at: > /srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar > > 12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename > /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java to > /root/./test.java > > 12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar > > 12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are > importing from mysql. > > 12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be > faster! Use the --direct > > 12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a > MySQL-specific fast path. > > 12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME > behavior to convertToNull (mysql) > > 12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test > > 12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement: > SELECT t.* FROM `test` AS t LIMIT 1 > > 12/05/17 15:33:13 INFO mapred.JobClient: Running job: > job_201205151001_0019 > > 12/05/17 15:33:14 INFO mapred.JobClient: map 0% reduce 0% > > 12/05/17 15:33:23 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_0, Status : FAILED > > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > > ... 5 more > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > > at > com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > > at > com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > > at java.sql.DriverManager.getConnection(DriverManager.java:582) > > at java.sql.DriverManager.getConnection(DriverManager.java:185) > > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > > ... 6 more > > Caused by: java.net.SocketException: Network is unreachable > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > at java.net.Socket.connect(Socket.java:529) > > at java.net.Socket.connect(Socket.java:478) > > at java.net.Socket.<init>(Socket.java:375) > > at java.net.Socket.<init>(Socket.java:218) > > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > > ... 22 more > > > > 12/05/17 15:33:29 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_1, Status : FAILED > > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > > ... 5 more > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > > at > com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > > at > com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > > at java.sql.DriverManager.getConnection(DriverManager.java:582) > > at java.sql.DriverManager.getConnection(DriverManager.java:185) > > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > > ... 6 more > > Caused by: java.net.SocketException: Network is unreachable > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > at java.net.Socket.connect(Socket.java:529) > > at java.net.Socket.connect(Socket.java:478) > > at java.net.Socket.<init>(Socket.java:375) > > at java.net.Socket.<init>(Socket.java:218) > > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > > ... 22 more > > > > 12/05/17 15:33:35 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_2, Status : FAILED > > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications > link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > > ... 5 more > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > > > The last packet sent successfully to the server was 0 milliseconds ago. > The driver has not received any packets from the server. > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > > at > com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > > at > com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at > com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > > at java.sql.DriverManager.getConnection(DriverManager.java:582) > > at java.sql.DriverManager.getConnection(DriverManager.java:185) > > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > > ... 6 more > > Caused by: java.net.SocketException: Network is unreachable > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > at java.net.Socket.connect(Socket.java:529) > > at java.net.Socket.connect(Socket.java:478) > > at java.net.Socket.<init>(Socket.java:375) > > at java.net.Socket.<init>(Socket.java:218) > > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > > ... 22 more > > > > 12/05/17 15:33:44 INFO mapred.JobClient: Job complete: > job_201205151001_0019 > > 12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2 > > 12/05/17 15:33:44 INFO mapred.JobClient: Job Counters > > 12/05/17 15:33:44 INFO mapred.JobClient: Launched map tasks=4 > > 12/05/17 15:33:44 INFO mapred.JobClient: Failed map tasks=1 > > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in > 32.2704 seconds (0 bytes/sec) > > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records. > > 12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job > failed! > > [root@node10 ~]# > > ------ > > > > -- > > YANG, Lin > > > > > -- YANG, Lin
