Mate Czagany created FLINK-41255:
------------------------------------
Summary: Value-format metadata columns are NULL with key.fields
and value.fields-include=ALL
Key: FLINK-41255
URL: https://issues.apache.org/jira/browse/FLINK-41255
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Mate Czagany
A {{kafka}} table with {{{}key.fields{}}}, the default {{'value.fields-include'
= 'ALL'}} and a metadata column from the value format (such as
{{value.source.timestamp}} from {{{}debezium-json{}}}) gets NULL in that column.
{code:sql}
CREATE TABLE t (
id INT,
src_ts TIMESTAMP_LTZ(3) METADATA FROM 'value.source.timestamp' VIRTUAL
) WITH (
'connector' = 'kafka',
'topic' = 't',
'properties.bootstrap.servers' = 'localhost:9092',
'scan.startup.mode' = 'earliest-offset',
'key.format' = 'json',
'key.fields' = 'id',
'value.format' = 'debezium-json'
);
{code}
{noformat}
key: {"id":1}
value:
{"before":null,"after":{"id":1},"source":{"ts_ms":1700000000000},"op":"c"}
{noformat}
{noformat}
expected: +I(1, 2023-11-14 22:13:20.000)
actual: +I(1, NULL)
{noformat}
With {{{}'value.fields-include' = 'EXCEPT_KEY'{}}}, or without
{{{}key.format{}}}/{{{}key.fields{}}}, the row is correct.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
