[
https://issues.apache.org/jira/browse/CALCITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882309#comment-17882309
]
david radley commented on CALCITE-6579:
---------------------------------------
[~mbudiu] that sounds like a good suggestion.
I see in your Jira you ended with the comment "Since no other SQL dialects
accept table declarations where the nullability of elements of an array can be
specified, it is unlikely for this to be the right solution."
I think we are saying that the Flink SQL dialect does need this. I am concerned
that this could change the behaviour for the other SQL dialects. I think it
would be safer for the existing behaviour to continue unaffected - but find a
way to introduce honouring non nullable children of nullables so Flink could
use this. Can we have a Calcite configuration flag to enable this behaviour or
a SQL hint?
WDYT?
> Unexpected nullable checks for nullable types with non-nullable children
> -------------------------------------------------------------------------
>
> Key: CALCITE-6579
> URL: https://issues.apache.org/jira/browse/CALCITE-6579
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.37.0
> Reporter: david radley
> Priority: Major
>
> This Jira is raised for the issue discussed in
> [https://lists.apache.org/thread/x857njnd42gcvrldbcd7y85to6fb58dn]
>
> I am using flink and have a table definition like this.
> _CREATE TABLE source_1 ( `order_id` STRING NOT NULL, `order_time` STRING NOT
> NULL, `buyer` ROW< `first_name` STRING, `last_name` STRING NOT NULL, `title`
> STRING NOT NULL_
> _) WITH ( 'connector' = 'kafka', 'topic' = 'mytopic',
> 'properties.bootstrap.servers' = 'localhost:9092', 'value.format' = 'avro',
> 'value.fields-include' = 'ALL', 'scan.startup.mode' = 'earliest-offset' );_
>
> This errors as code in Calcite does not create the correct schema.
> It sets`last_name` STRING NOT NULL, `title` STRING NOT NULL to both be
> nullable.
> The cause of this is in the table planner when we convert SqlDataTypeSpec to
> the RelDataType. We push the nullable of buyer onto all of its fields, losing
> the children’s nullable status . In the debugger I see that it is taking the
> nullable true (from buyer) and putting this on all the children in
> [fixupNullability|http://example.com]https://github.com/davidradl/calcite/blob/ad2e843c5d9b3bec001d22e680ebe6b5de4e2078/core/src/main/java/org/apache/calcite/sql/SqlDataTypeSpec.java#L238.
> I can see that the SqlDataTypeSpec has this information but it is not used.
>
> Flink maps this directly to an Avro schema that incorreclty polices the Avro
> payload.
>
> In terms of fixing this, I wonder if the type could be created with the
> correct nullable so it doesn’t need to be fixed up or whether we should pass
> down the SqlDataTypeSpec so it has the right information to set the nullable
> status.
>
> This subject has previously been raised on the dev list
> [https://lists.apache.org/thread/s4nd9rk0fzckoctokl7kjdbtfbvxncy7] by
> [~dwysakowicz]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
