GitHub user huaxingao opened a pull request:
https://github.com/apache/spark/pull/10262
[SPARK-12270][SQL]remove empty space after getString from database
{code}
conn.prepareStatement(
"create table people (name char(32)").executeUpdate()
conn.prepareStatement("insert into people values
('fred')").executeUpdate()
sql(
s"""
|CREATE TEMPORARY TABLE foobar
|USING org.apache.spark.sql.jdbc
|OPTIONS (url '$url', dbtable 'PEOPLE', user 'testuser', password
'testpassword')
""".stripMargin.replaceAll("\n", " "))
val df = sqlContext.sql("SELECT * FROM foobar WHERE NAME = 'fred'")
{code}
I am expecting to see one row with content 'fred' in df. However, there is
no row returned. If I changed the data type to varchar (32) in the create table
ddl , then I can get the row back correctly. The cause of the problem is that
for data type char (num), DB2 defines it as fixed-length character strings, so
if I have char (32), when doing "SELECT * FROM foobar WHERE NAME = 'fred'", DB2
returns 'fred' padded with 28 empty space. Spark treats "fred' padded with
empty space not the same as 'fred' so df doesn't have any row. If I have
varchar (32), DB2 just returns 'fred' for the select statement and df has the
right row. In order to make DB2 char (num) works for spark, I suggest to change
spark code to trim the empty space after get the data from database.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huaxingao/spark spark-12270
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10262.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10262
----
commit 63a3d835119ace22a359af03c267451aa68cc570
Author: Huaxin Gao <huax...@us.ibm.com>
Date: 2015-12-08T07:41:25Z
[SPARK-12270][SQL]remove empty space after getString from database
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
