Markus Kemper created SQOOP-3095:
------------------------------------
Summary: Add (import + --as-parquetfile) with Oracle BINARY_DOUBLE
Key: SQOOP-3095
URL: https://issues.apache.org/jira/browse/SQOOP-3095
Project: Sqoop
Issue Type: Improvement
Reporter: Markus Kemper
Some users are not able to easily implement the Sqoop option
(--map-column-java, Example 1 below).
The ask here is to find a way to natively support the Oracle BINARY_DOUBLE type
in Sqoop natively without using --map-column-java (see Example 2 below)
*Example 1*
{noformat}
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1
--target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1
--as-parquetfile --map-column-java C2=String
hdfs dfs -ls /user/user1/t1/*.parquet
parquet-tools cat --json
'hdfs://namenode.cloudera.com/user/user1/t1/9c4742a1-68e2-4fd7-8264-6e84110ac409.parquet'
Output:
16/12/27 10:34:05 INFO mapreduce.ImportJobBase: Transferred 1.5537 KB in
42.1369 seconds (37.7579 bytes/sec)
16/12/27 10:34:05 INFO mapreduce.ImportJobBase: Retrieved 1 records.
---
-rw-r--r-- 3 user1 user1 565 2016-12-27 10:34
/user/user1/t1/9c4742a1-68e2-4fd7-8264-6e84110ac409.parquet
---
{"C1":"1","C2":"1.1"}
{noformat}
*Example 2*
{noformat}
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1
--target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 1
--as-parquetfile
Output:
16/12/27 10:05:43 INFO manager.SqlManager: Executing SQL statement: SELECT t.*
FROM T1 t WHERE 1=0
16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C1 of type [2, 38, 0]
16/12/27 10:05:43 DEBUG manager.SqlManager: Found column C2 of type [101, 0, 0]
16/12/27 10:05:43 DEBUG util.ClassLoaderStack: Restoring classloader:
java.net.FactoryURLClassLoader@55465b1f
16/12/27 10:05:43 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL
type 101
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1
--target-dir /user/user1/t1 --delete-target-dir --verbose --num-mappers 2
--as-parquetfile --direct
Output:
16/12/27 10:08:50 DEBUG oracle.OraOopUtilities: The Oracle table context has
been derived from:
oracleConnectionUserName = sqoop
tableStr = T1
as:
owner : SQOOP
table : T1
16/12/27 10:08:50 INFO oracle.OraOopManagerFactory:
**************************************************
*** Using Data Connector for Oracle and Hadoop ***
**************************************************
<SNIP>
16/12/27 10:08:53 INFO manager.SqlManager: Executing SQL statement: SELECT
C1,C2 FROM T1 WHERE 1=0
16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C1
16/12/27 10:08:53 DEBUG manager.SqlManager: Found column C2
16/12/27 10:08:53 DEBUG util.ClassLoaderStack: Restoring classloader:
java.net.FactoryURLClassLoader@7e087bf5
16/12/27 10:08:53 ERROR tool.ImportTool: Imported Failed: Cannot convert SQL
type 101
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
