[
https://issues.apache.org/jira/browse/FLINK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230745#comment-17230745
]
Benoît Paris commented on FLINK-20105:
--------------------------------------
Thanks a lot, [~jark]!
Works like a charm with the deduplication operation!
Your kind explanations helped me understand more the PRIMARY KEY streaming
semantics. Let me rephrase to check that I understood correctly:
A PRIMARY KEY is a field which can only be repeated in a streaming context as
part of an UPDATE/UPSERT semantic pivoting around that key.
It is required for Temporal Table Joins because it is the only way a Temporal
Table Join has a use for; Therefore it is good to enforce it.
Any appending table may declare UPSERT semantics with a deduplication
operation. Retracting tables may not because they already have a PRIMARY KEY
defined.
I'll follow kafka-upsert as well!
>From my part, this JIRA can be requalified as a very minor convenience issue;
>I did feel a little lost by the error message. An error message with what you
>said in your comment would have worked wonders.
On another note, and this might not be the right semantics, but the DESCRIBE
statement could convey that the VIEW field has PK powers. It currently displays
nothing on a deduplicating view. I don't know if the SQL standard allows for
stating that on a view, though.
Anyway, thanks for the hard work on Flink!
Ben
> Temporary Table does not work when Kafka is used as the versioned side
> (planner PK problem)
> -------------------------------------------------------------------------------------------
>
> Key: FLINK-20105
> URL: https://issues.apache.org/jira/browse/FLINK-20105
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Planner
> Affects Versions: 1.12.0
> Reporter: Benoît Paris
> Priority: Major
> Fix For: 1.12.0
>
> Attachments: flink-test-temporal-table.zip
>
>
>
> This is probably an early bug report, that I'm making before 0.12 is out.
> In 1.12-SNAPSHOT, doing this:
>
> {code:java}
> INSERT INTO output_table
> SELECT
> o.f_sequence * r.f_int,
> o.f_random_str
> FROM datagen_1 AS o
> LEFT JOIN input_2 FOR SYSTEM_TIME AS OF o.ts r
> ON o.f_random = r.f_int_pk{code}
>
> works when input_2 is build with datagen, but fails when data comes from
> kafka; yielding the following error that comes from planner code:
>
> {code:java}
> Type INT NOT NULL of table field 'f_int_pk' does not match with the physical
> type INT of the 'f_int_pk' field of the TableSource return type.{code}
>
> Included is code for a complete reproduction, with:
> * docker-compose file for ZooKeeper and Kafka (latest)
> * pom.xml
> * OK_TempTableSQLTestDatagen.java: it works with the datagen
> * KO_TempTableSQLTestKafka.java: fails with Kafka
> * KO_TempTableSQLTestKafkaNoPK.java: I tried to have no PK, it fails
> * KO_TempTableSQLTestKafkaNull.java: I tried with the PK being nullable, it
> fails
> * KO_TempTableSQLTestKafkaNullif.java: I tried with using the PK in a
> NULLIF(pk, '') as [advertised
> here|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Inserting-nullable-data-into-NOT-NULL-columns-td34198.html],
> but if fails (loses PK powers)
> I just can't think of a workaround. I even tried to GROUP BY on the PK.
> From memory, the Temporal Table Function suffers from a similar problem; My
> usual workaround being to do a
> COALESCE(problematic_temp_table_function_primary_key, null), but it fails
> here as well (interestingly, it does not fail because of losing PK powers,
> but because of the NOT NULL planner difference).
> It seems like the same problem of having transformations of the same field
> being NULL and NOT NULL between planner transformations.
> ----
>
> This isssue is probably related to the ongoing FLIP-132 Temporal Table DDL
> and Temporal Table Join developments.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
