[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wanghao updated KYLIN-3934:
---------------------------
Description:
when column value from jdbc is null, sqoop will write it into hive table with
blank string.
eg
jdbc:
A | B
1 | 1
2 | 2
a | null
hive table:
A | B
1 | 1
2 | 2
a |
because of this, when I use count(distinct B), it return 3, not 2, and it can
lead to other problems
{code:java}
String cmd = String.format(Locale.ROOT,
"%s/bin/sqoop import" + generateSqoopConfigArgString()
+ "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
\\$CONDITIONS\" "
+ "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
+ "--fields-terminated-by '%s' --num-mappers %d",
sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
jobWorkingDir, hiveTable,
splitColumn, bquery, filedDelimiter, mapperNum);
{code}
the param '–null=string' should be '
\\N' instead of blank string ''.
I resolved this problem by replace the param. But it needs be configured in
kylin.properties
was:
when column value from jdbc is null, sqoop will write it into hive table with
blank string.
eg
jdbc:
A | B
1 | 1
2 | 2
a | null
hive table:
A | B
1 | 1
2 | 2
a |
because of this, when I use count(distinct B), it return 3, not 2, and it can
lead to other problems
{code:java}
String cmd = String.format(Locale.ROOT,
"%s/bin/sqoop import" + generateSqoopConfigArgString()
+ "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
\\$CONDITIONS\" "
+ "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
+ "--fields-terminated-by '%s' --num-mappers %d",
sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
jobWorkingDir, hiveTable,
splitColumn, bquery, filedDelimiter, mapperNum);
{code}
the param '–null=string' should be '\\N' instead of blank string ''.
I resolved this problem by replace the param. But it needs be configured in
kylin.properties
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -----------------------------------------------------------------------------------------
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
> Affects Versions: v2.6.0
> Reporter: wanghao
> Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
