Hi Jason, database and hadoop don't have to be on the same network. However sqoop is generating map reduce job, so each your node have to be able to create connection to database. Based on the exception "java.net.SocketException: Network is unreachable" I would guess that some nodes where are you running TaskTrackers are not allowed to reach your database server.
You can verify whether particular node is allowed to access the database using following command (execute from that particular node): mysql --host=<IP Address> --database=test --user=<username> --password=<password> Jarcec On May 17, 2012, at 11:15 AM, jason Yang wrote: > Hello, > > Recently, I'm trying to import MySQL data into Hive by using sqoop. but I > have encountered the "Communications link failure" problem (the output is > attached at the end of this mail) > > To solve this problem, I have checked the user guide of sqoop and do the > solution mentioned in the troubleshooting section. however, It doesn't work. > After that, I read some posts about the basic process of sqoop import, and I > found that during the process of import, the sqoop would submit a map-only > Hadoop job to the cluster, which actually do the data transfer works. so I > was wondering that should the source database be in the same network with the > hadoop cluster? > > Any suggestion would be appreciated. > > > --output-- > [root@node10 ~]# sqoop import --connect jdbc:mysql://172.18.11.54:3306/hive > --username root -P --table test --hive-import -m1 > Enter password: > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for > output. You can override > 12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with > --fields-terminated-by, etc. > 12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT > t.* FROM `test` AS t LIMIT 1 > 12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT > t.* FROM `test` AS t LIMIT 1 > 12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is > /srv/hadoop-0.20.2/bin/.. > 12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at: > /srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar > 12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename > /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java to > /root/./test.java > 12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar > 12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are importing > from mysql. > 12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be faster! Use > the --direct > 12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a > MySQL-specific fast path. > 12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME behavior > to convertToNull (mysql) > 12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test > 12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement: SELECT > t.* FROM `test` AS t LIMIT 1 > 12/05/17 15:33:13 INFO mapred.JobClient: Running job: job_201205151001_0019 > 12/05/17 15:33:14 INFO mapred.JobClient: map 0% reduce 0% > 12/05/17 15:33:23 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_0, Status : FAILED > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > ... 5 more > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:185) > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > ... 6 more > Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:529) > at java.net.Socket.connect(Socket.java:478) > at java.net.Socket.<init>(Socket.java:375) > at java.net.Socket.<init>(Socket.java:218) > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > ... 22 more > > 12/05/17 15:33:29 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_1, Status : FAILED > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > ... 5 more > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:185) > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > ... 6 more > Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:529) > at java.net.Socket.connect(Socket.java:478) > at java.net.Socket.<init>(Socket.java:375) > at java.net.Socket.<init>(Socket.java:218) > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > ... 22 more > > 12/05/17 15:33:35 INFO mapred.JobClient: Task Id : > attempt_201205151001_0019_m_000000_2, Status : FAILED > java.lang.RuntimeException: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > Caused by: java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link > failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) > ... 5 more > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet sent successfully to the server was 0 milliseconds ago. The > driver has not received any packets from the server. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344) > at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332) > at > com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369) > at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153) > at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792) > at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381) > at > com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:185) > at > com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) > at > com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) > ... 6 more > Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:529) > at java.net.Socket.connect(Socket.java:478) > at java.net.Socket.<init>(Socket.java:375) > at java.net.Socket.<init>(Socket.java:218) > at > com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) > at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294) > ... 22 more > > 12/05/17 15:33:44 INFO mapred.JobClient: Job complete: job_201205151001_0019 > 12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2 > 12/05/17 15:33:44 INFO mapred.JobClient: Job Counters > 12/05/17 15:33:44 INFO mapred.JobClient: Launched map tasks=4 > 12/05/17 15:33:44 INFO mapred.JobClient: Failed map tasks=1 > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in > 32.2704 seconds (0 bytes/sec) > 12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records. > 12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job > failed! > [root@node10 ~]# > ------ > > -- > YANG, Lin >
signature.asc
Description: Message signed with OpenPGP using GPGMail
