[jira] [Updated] (SQOOP-3040) Lost time millisecond precision for Time data type when importing
[
https://issues.apache.org/jira/browse/SQOOP-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Lin updated SQOOP-3040:
Assignee: Eric Lin
> Lost time millisecond precision for Time data type when importing
> -
>
> Key: SQOOP-3040
> URL: https://issues.apache.org/jira/browse/SQOOP-3040
> Project: Sqoop
> Issue Type: Bug
> Components: tools
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>
> To re-produce, create a MySQL database with time(6) data type:
> {code}
> CREATE TABLE `test` (
> `a` time(6) DEFAULT NULL
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> INSERT INTO test VALUES ('16:56:53.09');
> {code}
> Import the data into Sqoop and data will become "16:56:53", the millisecond
> precision will be lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SQOOP-1904) support for DB2 XML data type when importing to hdfs
[ https://issues.apache.org/jira/browse/SQOOP-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Cao updated SQOOP-1904: Attachment: SQOOP-1904.3.patch Update patch base on Bogi comments > support for DB2 XML data type when importing to hdfs > > > Key: SQOOP-1904 > URL: https://issues.apache.org/jira/browse/SQOOP-1904 > Project: Sqoop > Issue Type: Improvement > Components: connectors >Affects Versions: 1.4.5 > Environment: RedHat6.4 + Sqoop1.4.5 + Hadoop 2.4.1 + Hive 0.14.0 + > DB2 10.5 >Reporter: xieshiju >Assignee: xieshiju > Labels: features > Fix For: 1.4.7 > > Attachments: SQOOP-1904.2.patch, SQOOP-1904.3.patch, SQOOP-1904.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > The DB2 connector should add XML data type support otherwise you will get an > error that Hive does not support the SQL type. > Example with column DESCRIPTION data type XML: > $SQOOP_HOME/bin/sqoop create-hive-table --connect > 'jdbc:db2://xxx.svl.ibm.com:6/SAMPLE' --username db2inst1 --password > db2inst1 --hive-table product_ct --table PRODUCT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-2894) Hive import with Parquet failed in Kerberos enabled cluster
[ https://issues.apache.org/jira/browse/SQOOP-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823391#comment-15823391 ] Ying Cao commented on SQOOP-2894: - Hi ping, This issue is fixed by SQOOP-2406:Add support for secure mode when importing Parquet files into Hive > Hive import with Parquet failed in Kerberos enabled cluster > --- > > Key: SQOOP-2894 > URL: https://issues.apache.org/jira/browse/SQOOP-2894 > Project: Sqoop > Issue Type: Bug > Components: hive-integration, tools >Affects Versions: 1.4.6 > Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1 >Reporter: Ping Wang > Labels: security > Fix For: 1.4.7 > > > Importing data from external database to hive with Parquet option failed in > the kerberos environment. (It can success without kerberos). > The sqoop command I used: > sqoop import --connect jdbc:db2://xxx:5/testdb --username xxx --password > xxx --table users --hive-import -hive-table users3 --as-parquetfile -m 1 > The import job failed: > .. > 2016-02-26 04:20:07,020 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. > 2016-02-26 04:20:08,088 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config > null > 2016-02-26 04:20:08,918 INFO [main] hive.metastore: Trying to connect to > metastore with URI thrift://xxx:9083 > 2016-02-26 04:30:09,207 WARN [main] hive.metastore: set_ugi() not successful, > Likely cause: new client talking to old server. Continuing without it. > org.apache.thrift.transport.TTransportException: > java.net.SocketTimeoutException: Read timed out > at > org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688) > at > org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:448) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:237) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:182) > at org.kitesdk.data.spi.hive.MetaStoreUtil.(MetaStoreUtil.java:82) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.getMetaStoreUtil(HiveAbstractMetadataProvider.java:63) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:270) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:255) > at > org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:102) > at > org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192) > at org.kitesdk.data.Datasets.load(Datasets.java:108) > at org.kitesdk.data.Datasets.load(Datasets.java:165) > at > org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510) > at > org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SQOOP-3123) Import from oracle using oraoop with map-column-java to avro fails if special characters encounter in table name or column name
[
https://issues.apache.org/jira/browse/SQOOP-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Zagorulkin updated SQOOP-3123:
-
Component/s: (was: connectors/oracle)
> Import from oracle using oraoop with map-column-java to avro fails if special
> characters encounter in table name or column name
>
>
> Key: SQOOP-3123
> URL: https://issues.apache.org/jira/browse/SQOOP-3123
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6, 1.4.7
>Reporter: Dmitry Zagorulkin
>
> I'm trying to import data from oracle to avro using oraoop.
> My table:
> {code}
> CREATE TABLE "IBS"."BRITISH#CATS"
> ("ID" NUMBER,
> "C_CODE" VARCHAR2(10),
> "C_USE_START#DATE" DATE,
> "C_USE_USE#NEXT_DAY" VARCHAR2(1),
> "C_LIM_MIN#DAT" DATE,
> "C_LIM_MIN#TIME" TIMESTAMP,
> "C_LIM_MIN#SUM" NUMBER,
> "C_OWNCODE" VARCHAR2(1),
> "C_LIMIT#SUM_LIMIT" NUMBER(17,2),
> "C_L@M" NUMBER(17,2),
> "C_1_THROW" NUMBER NOT NULL ENABLE,
> "C_#_LIMITS" NUMBER NOT NULL ENABLE
> ) SEGMENT CREATION IMMEDIATE
> PCTFREE 70 PCTUSED 40 INITRANS 2 MAXTRANS 255
> NOCOMPRESS LOGGING
> STORAGE(INITIAL 2097152 NEXT 524288 MINEXTENTS 1 MAXEXTENTS 2147483645
> PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
> BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
> TABLESPACE "WORK" ;
> {code}
> My first script is:
> {code}
> ./sqoop import \
>
> -Doraoop.timestamp.string=false \
> --direct \
> --connect jdbc:oracle:thin:@localhost:49161:XE \
> --username system \
> --password oracle \
> --table IBS.BRITISH#CATS \
> --target-dir /Users/Dmitry/Developer/Java/sqoop/bin/imported \
> --as-avrodatafile \
> --map-column-java
> ID=String,C_CODE=String,C_USE_START#DATE=String,C_USE_USE#NEXT_DAY=String,C_LIM_MIN#DAT=String,C_LIM_MIN#TIME=String,C_LIM_MIN#SUM=String,C_OWNCODE=String,C_LIMIT#SUM_LIMIT=String,C_L_M=String,C_1_THROW=String,C_#_LIMITS=String
> {code}
> fails with
> {code}
> 2017-01-13 16:11:21,348 ERROR [main] tool.ImportTool
> (ImportTool.java:run(625)) - Import failed: No column by the name
> C_LIMIT#SUM_LIMITfound while importing data; expecting one of
> [C_LIMIT_SUM_LIMIT, C_OWNCODE, C_L_M, C___LIMITS, C_LIM_MIN_DAT, C_1_THROW,
> C_CODE, C_USE_START_DATE, C_LIM_MIN_SUM, ID, C_LIM_MIN_TIME,
> C_USE_USE_NEXT_DAY]
> {code}
> After i've found that sqoop has replaced all special characters with
> underscore. My second script is:
> {code}
> ./sqoop import \
>
> -D oraoop.timestamp.string=false \
> --direct \
> --connect jdbc:oracle:thin:@localhost:49161:XE \
> --username system \
> --password oracle \
> --table IBS.BRITISH#CATS \
> --target-dir /Users/Dmitry/Developer/Java/sqoop/bin/imported \
> --as-avrodatafile \
> --map-column-java
> ID=String,C_CODE=String,C_USE_START_DATE=String,C_USE_USE_NEXT_DAY=String,C_LIM_MIN_DAT=String,C_LIM_MIN_TIME=String,C_LIM_MIN_SUM=String,C_OWNCODE=String,C_LIMIT_SUM_LIMIT=String,C_L_M=String,C_1_THROW=String,C___LIMITS=String
> \
> --verbose
> {code}
> Fails with: Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","long"]: 2017-01-13 11:22:53.0
> {code}
> 2017-01-13 16:14:54,687 WARN [Thread-26] mapred.LocalJobRunner
> (LocalJobRunner.java:run(560)) - job_local1372531461_0001
> java.lang.Exception:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]:
> 2017-01-13 11:22:53.0
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","long"]:
> 2017-01-13 11:22:53.0
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.sqoop.mapreduce.AvroOutputFormat$1.write(AvroOutputFormat.java:112)
> at
> org.apache.sqoop.mapreduce.AvroOutputFormat$1.write(AvroOutputFormat.java:108)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:655)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.sqoop.mapreduce.AvroImportMapper.map(AvroImportMapper.java:73)
> at
>
