[
https://issues.apache.org/jira/browse/SQOOP-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716246#comment-13716246
]
wenzhi wang commented on SQOOP-1054:
------------------------------------
Hi,
I tried your suggestion, but it doesn't work, could you have a check? thanks.
bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>./sqoop
--options-file ./import_3col_imsi.txt
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: $HADOOP_HOME is deprecated.
13/07/23 16:57:57 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
13/07/23 16:57:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming
resultset.
13/07/23 16:57:58 INFO tool.CodeGenTool: Beginning code generation
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:57:59 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:57:59 ERROR orm.ClassWriter: No parser available for Java type
org.apache.hadoop.io.BytesWritable
13/07/23 16:57:59 ERROR orm.ClassWriter: No parser available for Java type
org.apache.hadoop.io.BytesWritable
13/07/23 16:57:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/local/wenzhiw/hadoop-1.0.4
Note:
/tmp/sqoop-wenzhiw/compile/279635f8e9ce0376053e3f5c30674f2e/QueryResult.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/07/23 16:58:05 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-wenzhiw/compile/279635f8e9ce0376053e3f5c30674f2e/QueryResult.jar
13/07/23 16:58:06 INFO mapreduce.ImportJobBase: Beginning query import.
13/07/23 16:58:10 INFO mapred.JobClient: Cleaning up the staging area
hdfs://135.252.181.253:50080/local/wenzhiw/hadoop-1.0.4/tmp/mapred/staging/wenzhiw/.staging/job_201307221026_0002
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class
org.apache.hadoop.mapreduce.JobContext, but interface was expected
at
org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:53)
at
com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
at
org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:121)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at
org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
at
org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:590)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:405)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>./sqoop
--options-file ./import_3col_imsi.txt
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: $HADOOP_HOME is deprecated.
13/07/23 16:58:15 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
13/07/23 16:58:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming
resultset.
13/07/23 16:58:16 INFO tool.CodeGenTool: Beginning code generation
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:58:17 INFO manager.SqlManager: Executing SQL statement: select
convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where (1 = 0)
13/07/23 16:58:17 ERROR orm.ClassWriter: No parser available for Java type
org.apache.hadoop.io.BytesWritable
13/07/23 16:58:17 ERROR orm.ClassWriter: No parser available for Java type
org.apache.hadoop.io.BytesWritable
13/07/23 16:58:17 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/local/wenzhiw/hadoop-1.0.4
Note:
/tmp/sqoop-wenzhiw/compile/fd98a50ab15fb67c0b5b3322a7dd50d9/QueryResult.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/07/23 16:58:23 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-wenzhiw/compile/fd98a50ab15fb67c0b5b3322a7dd50d9/QueryResult.jar
13/07/23 16:58:23 INFO mapreduce.ImportJobBase: Beginning query import.
13/07/23 16:58:29 INFO mapred.JobClient: Cleaning up the staging area
hdfs://135.252.181.253:50080/local/wenzhiw/hadoop-1.0.4/tmp/mapred/staging/wenzhiw/.staging/job_201307221026_0003
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class
org.apache.hadoop.mapreduce.JobContext, but interface was expected
at
org.apache.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:53)
at
com.cloudera.sqoop.config.ConfigurationHelper.getJobNumMaps(ConfigurationHelper.java:36)
at
org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:121)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at
org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
at
org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:590)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:405)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
bl1253{wenzhiw}/local/wenzhiw/sqoop2/sqoop-1.4.3.bin__hadoop-0.23/bin>cat
import_3col_imsi.txt
import
--connect
jdbc:mysql://135.252.158.250/hlr
--username
root
--password
admin
--query
'select convert(mSubIdentificationNumberId using utf8),
mobileNetworkCodeId,mobileCountryCodeId from IMSI where $CONDITIONS'
--split-by
mSubIdentificationNumberId
--target-dir
/local/wenzhiw/3col_imsi
Best Regards,
Wenzhi
> New feature to support importing MySQL table via binary type splitter
> ----------------------------------------------------------------------
>
> Key: SQOOP-1054
> URL: https://issues.apache.org/jira/browse/SQOOP-1054
> Project: Sqoop
> Issue Type: New Feature
> Components: connectors/mysql, tools
> Affects Versions: 1.4.3
> Environment: $uname -a
> Linux bl1253.cn.alcatel-lucent.com 2.6.18-128.7.1.el5PAE #1 SMP Mon Aug 24
> 08:41:30 EDT 2009 i686 athlon i386 GNU/Linux
> Reporter: wenzhi wang
>
> I'm using (Sqoop 1.4.3 + Hadoop 1.0.4) handle data of our product, which
> employs MySQL 5.1.6 and most tables are defined with primary key on binary
> columns. Sqoop does not support binary type column as splitter now, and I
> have to import tables by restricting mapper number to 1 which lowers the
> performance. So, I open this feature to request Sqoop support binary type
> splitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
