Hi Markus, Had not tried that yet, but just tested it and no luck, get the same "not a specific class" error for char.
-Mark On Thu, Aug 4, 2016 at 5:02 PM, Markus Kemper <[email protected]> wrote: > Hello Mark, > > I will try to test this but, am curious if you tried: > > sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --query "select * > from ci_84adea33-9194-4753-925f-529a87656048 where \$CONDITIONS" > --as-parquetfile --class-name > mydata1 --username USERNAME -P --map-column-java <CHAR_COL>=String > > > Markus Kemper > Customer Operations Engineer > [image: www.cloudera.com] <http://www.cloudera.com> > > > On Thu, Aug 4, 2016 at 3:48 PM, Mark Wagoner <[email protected]> > wrote: > >> Hi Markus, >> >> Thanks for the feedback. >> >> I was able to use a slight variation of your suggestion to get past the >> table name issue, but then encountered a new error related apparently to >> the class of the column type I am importing (varchar), which is still >> unresolved as of today (https://issues.apache.org/jira/browse/SQOOP-2408) >> so I'll likely be importing in another format and then converting until >> this bug is fixed. >> >> [hadoop@ip-172-31-25-10 ~]$ sqoop import --connect >> jdbc:mysql://DATABASE_ENDPOINT --query 'select * from >> `ci_84adea33-9194-4753-925f-529a87656048` t1 where $CONDITIONS' >> --class-name mydata0 --as-parquetfile --target-dir /home/hadoop/mydata0 >> --split-by t1.rsid --username USERNAME -P >> Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will >> fail. >> Please set $ACCUMULO_HOME to the root of your Accumulo installation. >> 16/08/04 19:14:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 >> Enter password: >> 16/08/04 19:14:08 INFO manager.MySQLManager: Preparing to use a MySQL >> streaming resultset. >> 16/08/04 19:14:08 INFO tool.CodeGenTool: Beginning code generation >> 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: >> select * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = >> 0) >> 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: >> select * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = >> 0) >> 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: >> select * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = >> 0) >> 16/08/04 19:14:08 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is >> /usr/lib/hadoop-mapreduce >> Note: /tmp/sqoop-hadoop/compile/d06024b7489332878d1144702c6a7923/mydata0.java >> uses or overrides a deprecated API. >> Note: Recompile with -Xlint:deprecation for details. >> 16/08/04 19:14:12 INFO orm.CompilationManager: Writing jar file: >> /tmp/sqoop-hadoop/compile/d06024b7489332878d1144702c6a7923/mydata0.jar >> 16/08/04 19:14:12 INFO mapreduce.ImportJobBase: Beginning query import. >> SLF4J: Class path contains multiple SLF4J bindings. >> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/ >> slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/lo >> g4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. >> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> 16/08/04 19:14:13 INFO Configuration.deprecation: mapred.jar is >> deprecated. Instead, use mapreduce.job.jar >> 16/08/04 19:14:13 INFO manager.SqlManager: Executing SQL statement: >> select * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = >> 0) >> 16/08/04 19:14:13 INFO manager.SqlManager: Executing SQL statement: >> select * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = >> 0) >> 16/08/04 19:14:15 INFO Configuration.deprecation: mapred.map.tasks is >> deprecated. Instead, use mapreduce.job.maps >> 16/08/04 19:14:16 INFO impl.TimelineClientImpl: Timeline service address: >> http://ip-172-31-25-105.us-west-2.compute.internal:8188/ws/v1/timeline/ >> 16/08/04 19:14:16 INFO client.RMProxy: Connecting to ResourceManager at >> ip-172-31-25-105.us-west-2.compute.internal/172.31.25.105:8032 >> 16/08/04 19:14:17 INFO db.DBInputFormat: Using read commited transaction >> isolation >> 16/08/04 19:14:17 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: >> SELECT MIN(t1.rsid), MAX(t1.rsid) FROM (select * from >> `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 1) ) AS t1 >> 16/08/04 19:14:18 WARN db.TextSplitter: Generating splits for a textual >> index column. >> 16/08/04 19:14:18 WARN db.TextSplitter: If your database sorts in a >> case-insensitive order, this may result in a partial import or duplicate >> records. >> 16/08/04 19:14:18 WARN db.TextSplitter: You are strongly encouraged to >> choose an integral split column. >> 16/08/04 19:14:18 INFO mapreduce.JobSubmitter: number of splits:5 >> 16/08/04 19:14:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: >> job_1470337090744_0001 >> 16/08/04 19:14:19 INFO impl.YarnClientImpl: Submitted application >> application_1470337090744_0001 >> 16/08/04 19:14:19 INFO mapreduce.Job: The url to track the job: >> http://ip-172-31-25-105.us-west-2.compute.internal:20888/pro >> xy/application_1470337090744_0001/ >> 16/08/04 19:14:19 INFO mapreduce.Job: Running job: job_1470337090744_0001 >> 16/08/04 19:14:30 INFO mapreduce.Job: Job job_1470337090744_0001 running >> in uber mode : false >> 16/08/04 19:14:30 INFO mapreduce.Job: map 0% reduce 0% >> 16/08/04 19:14:43 INFO mapreduce.Job: map 20% reduce 0% >> 16/08/04 19:14:43 INFO mapreduce.Job: Task Id : >> attempt_1470337090744_0001_m_000000_0, Status : FAILED >> Error: org.apache.avro.AvroRuntimeException: Not a Specific class: char >> at org.apache.avro.specific.SpecificData.createSchema(SpecificD >> ata.java:213) >> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData >> .java:303) >> at org.apache.avro.reflect.ReflectData.createFieldSchema(Reflec >> tData.java:430) >> at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSch >> ema(DataModelUtil.java:55) >> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData >> .java:354) >> at org.apache.avro.reflect.ReflectData.createFieldSchema(Reflec >> tData.java:430) >> at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSch >> ema(DataModelUtil.java:55) >> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData >> .java:354) >> at org.apache.avro.reflect.ReflectData.createFieldSchema(Reflec >> tData.java:430) >> at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSch >> ema(DataModelUtil.java:55) >> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData >> .java:354) >> at org.apache.avro.specific.SpecificData.getSchema(SpecificData >> .java:154) >> at org.kitesdk.data.spi.DataModelUtil.getReaderSchema(DataModel >> Util.java:171) >> at org.kitesdk.data.spi.DataModelUtil.resolveType(DataModelUtil >> .java:148) >> at org.kitesdk.data.spi.AbstractDataset.<init>(AbstractDataset. >> java:44) >> at org.kitesdk.data.spi.filesystem.FileSystemDataset.<init>( >> FileSystemDataset.java:85) >> at org.kitesdk.data.spi.filesystem.FileSystemDataset.<init>( >> FileSystemDataset.java:115) >> at org.kitesdk.data.spi.filesystem.FileSystemDataset$Builder. >> build(FileSystemDataset.java:541) >> at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository. >> load(FileSystemDatasetRepository.java:194) >> at org.kitesdk.data.spi.AbstractDatasetRepository.load(Abstract >> DatasetRepository.java:40) >> at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.loadJobDat >> aset(DatasetKeyOutputFormat.java:544) >> at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.loadOrCrea >> teTaskAttemptDataset(DatasetKeyOutputFormat.java:555) >> at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.loadOrCrea >> teTaskAttemptView(DatasetKeyOutputFormat.java:568) >> at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getRecordW >> riter(DatasetKeyOutputFormat.java:426) >> at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<i >> nit>(MapTask.java:656) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:776) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >> upInformation.java:1657) >> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >> >> Best, >> Mark >> >> On Wed, Aug 3, 2016 at 4:10 PM, Markus Kemper <[email protected]> >> wrote: >> >>> Hello Mark, >>> >>> Have you tried the following: >>> >>> sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --query "select * >>> from ci_84adea33-9194-4753-925f-529a87656048 where \$CONDITIONS" >>> --as-parquetfile --class-name >>> mydata1 --username USERNAME -P >>> >>> >>> Markus Kemper >>> Customer Operations Engineer >>> [image: www.cloudera.com] <http://www.cloudera.com> >>> >>> >>> On Tue, Aug 2, 2016 at 4:59 PM, Mark Wagoner <[email protected]> >>> wrote: >>> >>>> hi, >>>> >>>> I need to be able to import table names with hypens but keep getting the >>>> following error. Is there any way to specify a table alias to rename a >>>> table replacing hypens with underscores? >>>> >>>> sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --table >>>> ci_84adea33-9194-4753-925f-529a87656048 --as-parquetfile --class-name >>>> mydata1 --username USERNAME -P >>>> >>>> >>>> 16/08/02 20:11:22 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM `ci_84adea33-9194-4753-925f-529a87656048` AS t LIMIT 1 >>>> 16/08/02 20:11:23 ERROR sqoop.Sqoop: Got exception running Sqoop: >>>> org.apache.avro.SchemaParseException: Illegal character in: >>>> ci_84adea33-9194-4753-925f-529a87656048 >>>> org.apache.avro.SchemaParseException: Illegal character in: >>>> ci_84adea33-9194-4753-925f-529a87656048 >>>> at org.apache.avro.Schema.validateName(Schema.java:1042) >>>> at org.apache.avro.Schema.access$200(Schema.java:78) >>>> at org.apache.avro.Schema$Name.<init>(Schema.java:431) >>>> at org.apache.avro.Schema.createRecord(Schema.java:144) >>>> at >>>> org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchema >>>> Generator.java:83) >>>> at >>>> org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroS >>>> chema(DataDrivenImportJob.java:133) >>>> at >>>> org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapp >>>> er(DataDrivenImportJob.java:106) >>>> at >>>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJob >>>> Base.java:260) >>>> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.j >>>> ava:673) >>>> at >>>> org.apache.sqoop.manager.MySQLManager.importTable(MySQLManag >>>> er.java:118) >>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java >>>> :497) >>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>> >>>> Thanks, >>>> Mark >>>> >>> >>> >> >
