Hello, Recently, I'm trying to import MySQL data into Hive by using sqoop. but I have encountered the "Communications link failure" problem (the output is attached at the end of this mail)
To solve this problem, I have checked the user guide of sqoop and do the solution mentioned in the troubleshooting section. however, It doesn't work. After that, I read some posts about the basic process of sqoop import, and I found that during the process of import, the sqoop would submit a map-only Hadoop job to the cluster, which actually do the data transfer works. so I was wondering that should the source database be in the same network with the hadoop cluster? Any suggestion would be appreciated. *--output--* *[root@node10 ~]# sqoop import --connect jdbc:mysql://172.18.11.54:3306/hive--username root -P --table test --hive-import -m1 * *Enter password: * *12/05/17 15:33:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override* *12/05/17 15:33:10 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.* *12/05/17 15:33:10 INFO tool.CodeGenTool: Beginning code generation* *12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `test` AS t LIMIT 1* *12/05/17 15:33:10 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `test` AS t LIMIT 1* *12/05/17 15:33:11 INFO orm.CompilationManager: HADOOP_HOME is /srv/hadoop-0.20.2/bin/..* *12/05/17 15:33:11 INFO orm.CompilationManager: Found hadoop core jar at: /srv/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar* *12/05/17 15:33:12 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.java to /root/./test.java* *12/05/17 15:33:12 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/6571e4a050817f31b3846917ae805e49/test.jar* *12/05/17 15:33:12 WARN manager.MySQLManager: It looks like you are importing from mysql.* *12/05/17 15:33:12 WARN manager.MySQLManager: This transfer can be faster! Use the --direct* *12/05/17 15:33:12 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.* *12/05/17 15:33:12 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)* *12/05/17 15:33:12 INFO mapreduce.ImportJobBase: Beginning import of test* *12/05/17 15:33:12 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `test` AS t LIMIT 1* *12/05/17 15:33:13 INFO mapred.JobClient: Running job: job_201205151001_0019 * *12/05/17 15:33:14 INFO mapred.JobClient: map 0% reduce 0%* *12/05/17 15:33:23 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_0, Status : FAILED* *java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) * * at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)* * at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) * * at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)* * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)* * at org.apache.hadoop.mapred.Child.main(Child.java:170)* *Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) * * ... 5 more* *Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)* * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)* * at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)* * at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)* * at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)* * at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)* * at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)* * at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)* * at java.sql.DriverManager.getConnection(DriverManager.java:582)* * at java.sql.DriverManager.getConnection(DriverManager.java:185)* * at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) * * ... 6 more* *Caused by: java.net.SocketException: Network is unreachable* * at java.net.PlainSocketImpl.socketConnect(Native Method)* * at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)* * at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)* * at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)* * at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)* * at java.net.Socket.connect(Socket.java:529)* * at java.net.Socket.connect(Socket.java:478)* * at java.net.Socket.<init>(Socket.java:375)* * at java.net.Socket.<init>(Socket.java:218)* * at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) * * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)* * ... 22 more* * * * 12/05/17 15:33:29 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_1, Status : FAILED* *java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) * * at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)* * at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) * * at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)* * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)* * at org.apache.hadoop.mapred.Child.main(Child.java:170)* *Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) * * ... 5 more* *Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)* * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)* * at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)* * at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)* * at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)* * at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)* * at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)* * at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)* * at java.sql.DriverManager.getConnection(DriverManager.java:582)* * at java.sql.DriverManager.getConnection(DriverManager.java:185)* * at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) * * ... 6 more* *Caused by: java.net.SocketException: Network is unreachable* * at java.net.PlainSocketImpl.socketConnect(Native Method)* * at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)* * at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)* * at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)* * at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)* * at java.net.Socket.connect(Socket.java:529)* * at java.net.Socket.connect(Socket.java:478)* * at java.net.Socket.<init>(Socket.java:375)* * at java.net.Socket.<init>(Socket.java:218)* * at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) * * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)* * ... 22 more* * * * 12/05/17 15:33:35 INFO mapred.JobClient: Task Id : attempt_201205151001_0019_m_000000_2, Status : FAILED* *java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:164) * * at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)* * at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) * * at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)* * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)* * at org.apache.hadoop.mapred.Child.main(Child.java:170)* *Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:190) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:159) * * ... 5 more* *Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure* * * *The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)* * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)* * at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)* * at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)* * at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)* * at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)* * at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* * at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) * * at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) * * at java.lang.reflect.Constructor.newInstance(Constructor.java:513)* * at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)* * at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)* * at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)* * at java.sql.DriverManager.getConnection(DriverManager.java:582)* * at java.sql.DriverManager.getConnection(DriverManager.java:185)* * at com.cloudera.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:152) * * at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:184) * * ... 6 more* *Caused by: java.net.SocketException: Network is unreachable* * at java.net.PlainSocketImpl.socketConnect(Native Method)* * at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)* * at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)* * at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)* * at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)* * at java.net.Socket.connect(Socket.java:529)* * at java.net.Socket.connect(Socket.java:478)* * at java.net.Socket.<init>(Socket.java:375)* * at java.net.Socket.<init>(Socket.java:218)* * at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257) * * at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)* * ... 22 more* * * * 12/05/17 15:33:44 INFO mapred.JobClient: Job complete: job_201205151001_0019* *12/05/17 15:33:44 INFO mapred.JobClient: Counters: 2* *12/05/17 15:33:44 INFO mapred.JobClient: Job Counters * *12/05/17 15:33:44 INFO mapred.JobClient: Launched map tasks=4* *12/05/17 15:33:44 INFO mapred.JobClient: Failed map tasks=1* *12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 32.2704 seconds (0 bytes/sec)* *12/05/17 15:33:44 INFO mapreduce.ImportJobBase: Retrieved 0 records.* *12/05/17 15:33:44 ERROR tool.ImportTool: Error during import: Import job failed!* *[root@node10 ~]#* *------* -- YANG, Lin
