Hi, - check your mysql max connections - verify if mysqld is started without the --skip-networking option
cheers - Alex -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF On May 29, 2012, at 1:22 AM, Srinivas Surasani wrote: > Hello Jerec/Kathleen, > > Thanks for your replies.. > > Below is sqoop log with verbose enabled. I am able to run only 5 > number of jobs which is equal to my datanodes count. Also maxnumber of > connections I can make to MySql database is 1500. I have run more > than 5 jobs with only one mapper for each job and still I get the same > error. Please adivise.. > > below is the sqoop log. > > > 12/05/26 13:07:29 DEBUG tool.BaseSqoopTool: Enabled debug logging. > 12/05/26 13:07:29 WARN tool.BaseSqoopTool: Setting your password on > the command-line is insecure. Consider using -P instead. > 12/05/26 13:07:29 DEBUG sqoop.ConnFactory: Loaded manager factory: > com.cloudera.sqoop.manager.DefaultManagerFactory > 12/05/26 13:07:29 DEBUG sqoop.ConnFactory: Trying ManagerFactory: > com.cloudera.sqoop.manager.DefaultManagerFactory > 12/05/26 13:07:29 DEBUG manager.DefaultManagerFactory: Trying with > scheme: jdbc:mysql: > 12/05/26 13:07:29 INFO manager.MySQLManager: Preparing to use a MySQL > streaming resultset. > 12/05/26 13:07:29 DEBUG sqoop.ConnFactory: Instantiated ConnManager > com.cloudera.sqoop.manager.DirectMySQLManager@1cb8deef > 12/05/26 13:07:29 INFO tool.CodeGenTool: Beginning code generation > 12/05/26 13:07:29 DEBUG manager.SqlManager: No connection paramenters > specified. Using regular API for making connection. > 12/05/26 13:07:36 DEBUG manager.SqlManager: Using fetchSize for next > query: -2147483648 > 12/05/26 13:07:36 INFO manager.SqlManager: Executing SQL statement: > SELECT t.* FROM `db` AS t LIMIT 1 > 12/05/26 13:07:36 ERROR manager.SqlManager: Error executing statement: > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet successfully received from the server was 4,270 > milliseconds ago. The last packet sent successfully to the server was > 12 milliseconds ago. > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet successfully received from the server was 4,270 > milliseconds ago. The last packet sent successfully to the server was > 12 milliseconds ago. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3591) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3480) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4021) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2671) > at > com.mysql.jdbc.StatementImpl.executeSimpleNonQuery(StatementImpl.java:1644) > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2232) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:487) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:496) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:195) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:179) > at > com.cloudera.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:116) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1226) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1051) > at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:84) > at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:370) > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:456) > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:146) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230) > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239) > Caused by: java.io.EOFException: Can not read response from server. > Expected to read 4 bytes, read 0 bytes before connection was > unexpectedly lost. > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3041) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3491) > ... 23 more > 12/05/26 13:07:36 ERROR manager.CatalogQueryManager: Failed to > rollback transaction > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > No operations allowed after connection closed. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.Util.getInstance(Util.java:386) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) > at > com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1245) > at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1237) > at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4717) > at > com.cloudera.sqoop.manager.CatalogQueryManager.getColumnNames(CatalogQueryManager.java:157) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1198) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1053) > at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:84) > at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:370) > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:456) > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:146) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230) > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239) > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet successfully received from the server was 4,270 > milliseconds ago. The last packet sent successfully to the server was > 12 milliseconds ago. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3591) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3480) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4021) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2671) > at > com.mysql.jdbc.StatementImpl.executeSimpleNonQuery(StatementImpl.java:1644) > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2232) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:487) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:496) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:195) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:179) > at > com.cloudera.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:116) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1226) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1051) > ... 9 more > Caused by: java.io.EOFException: Can not read response from server. > Expected to read 4 bytes, read 0 bytes before connection was > unexpectedly lost. > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3041) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3491) > ... 23 more > 12/05/26 13:07:36 ERROR manager.CatalogQueryManager: Failed to list columns > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > No operations allowed after connection closed. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.Util.getInstance(Util.java:386) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) > at > com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1245) > at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1237) > at > com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2536) > at > com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2518) > at > com.cloudera.sqoop.manager.CatalogQueryManager.getColumnNames(CatalogQueryManager.java:148) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1198) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1053) > at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:84) > at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:370) > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:456) > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:146) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230) > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239) > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet successfully received from the server was 4,270 > milliseconds ago. The last packet sent successfully to the server was > 12 milliseconds ago. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3591) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3480) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4021) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2671) > at > com.mysql.jdbc.StatementImpl.executeSimpleNonQuery(StatementImpl.java:1644) > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2232) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:487) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:496) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:195) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:179) > at > com.cloudera.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:116) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1226) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1051) > ... 9 more > Caused by: java.io.EOFException: Can not read response from server. > Expected to read 4 bytes, read 0 bytes before connection was > unexpectedly lost. > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3041) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3491) > ... 23 more > 12/05/26 13:07:36 ERROR sqoop.Sqoop: Got exception running Sqoop: > java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > No operations allowed after connection closed. > java.lang.RuntimeException: > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > No operations allowed after connection closed. > at > com.cloudera.sqoop.manager.CatalogQueryManager.getColumnNames(CatalogQueryManager.java:163) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnNames(ClassWriter.java:1198) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1053) > at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:84) > at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:370) > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:456) > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:146) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:182) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:221) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:230) > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:239) > Caused by: > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > No operations allowed after connection closed. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.Util.getInstance(Util.java:386) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1013) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:982) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927) > at > com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1245) > at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1237) > at > com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2536) > at > com.mysql.jdbc.ConnectionImpl.createStatement(ConnectionImpl.java:2518) > at > com.cloudera.sqoop.manager.CatalogQueryManager.getColumnNames(CatalogQueryManager.java:148) > ... 11 more > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: > Communications link failure > > The last packet successfully received from the server was 4,270 > milliseconds ago. The last packet sent successfully to the server was > 12 milliseconds ago. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3591) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3480) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4021) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2490) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2651) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2671) > at > com.mysql.jdbc.StatementImpl.executeSimpleNonQuery(StatementImpl.java:1644) > at > com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2232) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:487) > at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:496) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:195) > at > com.cloudera.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:179) > at > com.cloudera.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:116) > at > com.cloudera.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1226) > at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1051) > ... 9 more > Caused by: java.io.EOFException: Can not read response from server. > Expected to read 4 bytes, read 0 bytes before connection was > unexpectedly lost. > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3041) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3491) > ... 23 more > > Regards, > Srinivas > > > On Mon, May 28, 2012 at 3:39 AM, Kathleen Ting <[email protected]> wrote: >> Hi Srinivas, >> >> In addition to Jarcec's excellent advice, perhaps you might find >> http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_mysql_connection_failure >> helpful? >> >> Regards, >> Kathleen >> >> On Sat, May 26, 2012 at 6:54 AM, Jarek Jarcec Cecho <[email protected]> >> wrote: >>> >>> Hi Srinivas, >>> exception "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: >>> Communications link failure" is very generic and basically means that there >>> was something wrong with connecting to the MySQL. Could you share your >>> entire execution log that was generated with --verbose flag? >>> >>> Blindly shooting - it's suspicious that your issue appears only when >>> certain amount of jobs is generated. MySQL is allowing limiting maximal >>> number of concurrent connections. I would recommend checking defined >>> connection limit on MySQL side (by SQL query show variables like >>> "max_connections";) and compare it to number of created map tasks. In case >>> that configured maximal value for connections is lower, then you need to >>> either reconfigure mysql or sqoop (for example limit number of used mappers >>> with --num-mappers argument or do not import all tables at once). >>> >>> Jarcec >>> >>> On Sat, May 26, 2012 at 09:40:47AM -0400, Srinivas wrote: >>>> >>>>> I have couple of problems that I am running into: >>>>> >>>>> 1. I was able to initiate only 5 sqoop jobs parallel which is = >>>>> number of datanode's. >>>>> 2. If i try to invoke more jobs parallel then only 5 jobs are >>>>> picked by and the rest of them throwing an error saying >>>>> "com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications >>>>> link failure" >>>>> >>>>> Do you have any idea why this is happening. >>>>> >>>>> Also, I have made sure that the mysql allows parallel connections from >>>>> datanode's. >>>>> >>>>> >>>>> >> >> > > > > -- > -- Srinivas > [email protected]
