[
https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mykhailo Osypov updated SPARK-18859:
------------------------------------
Description:
When joining two tables via LEFT JOIN, columns in right table may be NULLs,
however catalyst codegen cannot recognize it.
Example:
{code:title=schema.sql|borderStyle=solid}
create table masterdata.testtable(
id int not null,
age int
);
create table masterdata.jointable(
id int not null,
name text *not null*
);
{code}
{code:title=query_to_select.sql|borderStyle=solid}
(select t.id, t.age, j.name from masterdata.testtable t left join
masterdata.jointable j on t.id = j.id) as testtable;
{code}
{code:title=master code|borderStyle=solid}
val df = sqlContext
.read
.format("jdbc")
.option("dbTable", "query to select")
....
.load
//df generated schema
/*
root
|-- id: integer (nullable = false)
|-- age: integer (nullable = true)
|-- name: string (nullable = false)
*/
{code}
{code:title=Codegen|borderStyle=solid}
/* 038 */ scan_rowWriter.write(0, scan_value);
/* 039 */
/* 040 */ if (scan_isNull1) {
/* 041 */ scan_rowWriter.setNullAt(1);
/* 042 */ } else {
/* 043 */ scan_rowWriter.write(1, scan_value1);
/* 044 */ }
/* 045 */
/* 046 */ scan_rowWriter.write(2, scan_value2);
{code}
Since *j.name* is from right table of *left join* query, it may be null.
However generated schema doesn't think so (probably because it defined as *name
text not null*)
was:
When joining two tables via LEFT JOIN, columns in right table may be NULLs,
however catalyst codegen cannot recognize it.
Example:
{code:title=schema.sql|borderStyle=solid}
create table masterdata.testtable(
id int not null,
age int
);
create table masterdata.jointable(
id int not null,
name text not null
);
(select t.id, t.age, j.name from masterdata.testtable t left join
masterdata.jointable j on t.id = j.id) as testtable;
{code}
{code:title=master code|borderStyle=solid}
val df = sqlContext
.read
.format("jdbc")
..option("dbTable", "query to select)
....
.load
//df generated schema
/*
root
|-- id: integer (nullable = false)
|-- age: integer (nullable = true)
|-- name: string (nullable = false)
*/
{code}
{code:title=Codegen|borderStyle=solid}
/* 038 */ scan_rowWriter.write(0, scan_value);
/* 039 */
/* 040 */ if (scan_isNull1) {
/* 041 */ scan_rowWriter.setNullAt(1);
/* 042 */ } else {
/* 043 */ scan_rowWriter.write(1, scan_value1);
/* 044 */ }
/* 045 */
/* 046 */ scan_rowWriter.write(2, scan_value2);
{code}
Since j.name is from right table of left join query, it may be null. However
generated schema doesn't think so (probably because it defined as name text not
null)
> Catalyst codegen does not mark column as nullable when it should. Causes NPE
> ----------------------------------------------------------------------------
>
> Key: SPARK-18859
> URL: https://issues.apache.org/jira/browse/SPARK-18859
> Project: Spark
> Issue Type: Bug
> Components: Optimizer, SQL
> Affects Versions: 2.0.2
> Reporter: Mykhailo Osypov
> Priority: Critical
>
> When joining two tables via LEFT JOIN, columns in right table may be NULLs,
> however catalyst codegen cannot recognize it.
> Example:
> {code:title=schema.sql|borderStyle=solid}
> create table masterdata.testtable(
> id int not null,
> age int
> );
> create table masterdata.jointable(
> id int not null,
> name text *not null*
> );
> {code}
> {code:title=query_to_select.sql|borderStyle=solid}
> (select t.id, t.age, j.name from masterdata.testtable t left join
> masterdata.jointable j on t.id = j.id) as testtable;
> {code}
> {code:title=master code|borderStyle=solid}
> val df = sqlContext
> .read
> .format("jdbc")
> .option("dbTable", "query to select")
> ....
> .load
> //df generated schema
> /*
> root
> |-- id: integer (nullable = false)
> |-- age: integer (nullable = true)
> |-- name: string (nullable = false)
> */
> {code}
> {code:title=Codegen|borderStyle=solid}
> /* 038 */ scan_rowWriter.write(0, scan_value);
> /* 039 */
> /* 040 */ if (scan_isNull1) {
> /* 041 */ scan_rowWriter.setNullAt(1);
> /* 042 */ } else {
> /* 043 */ scan_rowWriter.write(1, scan_value1);
> /* 044 */ }
> /* 045 */
> /* 046 */ scan_rowWriter.write(2, scan_value2);
> {code}
> Since *j.name* is from right table of *left join* query, it may be null.
> However generated schema doesn't think so (probably because it defined as
> *name text not null*)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
