Hello all,
I'm having trouble importing data from a mysql database (version 5.26) with
sqoop (version 1.4.6).
I have a number of imports that are happening in AWS EMR and all of them
are working that go against tables. I have one import that goes against a
view that is not working. It works on our dev database but for some reason
not on our prod server.
Here's the command I'm running:
sqoop import --connect jdbc:mysql://$databaseHost/inventory
--connection-manager org.apache.sqoop.manager.MySQLManager --username
$user_name
--password-file=s3://$s3Bucket/dibs/settings/sqoop_db_credentials.properties
--as-avrodatafile --target-dir /user/hadoop/inventory/category_view
--delete-target-dir --table category_view --split-by id --verbose
I've tried with and without specifying the jdbc driver and without
specifying the connection-manager. I've also tried overriding the default
SQL it is using but nothing seems to help. I've also seen a number of other
users with this issue that all point to the table_definition_cache but
playing with those values has not fixed the issue.
>From the below stacktrace you can see that it is using a
MariaDbServerPreparedStatement. This seems weird to me and would expect it
to be using the Mysql specific PreparedStatement class. Is this the normal
behavior?
Here's the stacktrace:
2016-08-04 19:49:40,589 INFO org.apache.sqoop.manager.SqlManager
(main): Executing SQL statement: SELECT t.* FROM `category_view` AS t
LIMIT 1
2016-08-04 19:49:40,601 ERROR org.apache.sqoop.manager.SqlManager
(main): Error executing statement: java.sql.BatchUpdateException:
Prepared statement needs to be re-prepared
java.sql.BatchUpdateException: Prepared statement needs to be re-prepared
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:374)
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQuery(MariaDbServerPreparedStatement.java:341)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:758)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
at
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
at
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at
org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at
org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at
org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.sql.SQLException: Prepared statement needs to be re-prepared
at
org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:136)
at
org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQueryEpilog(MariaDbServerPreparedStatement.java:336)
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:291)
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:369)
... 18 more
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Prepared
statement needs to be re-prepared
at
org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:479)
at
org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:592)
at
org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:279)
... 19 more
2016-08-04 19:49:40,606 ERROR org.apache.sqoop.tool.ImportTool (main):
Encountered IOException running import job: java.io.IOException: No
columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Any thoughts/recommendations would be greatly appreciated.
Thanks,
--
*Geren White | Senior Java Developer*
*(e)* [email protected]
