[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Description:
*Summary:*
Given a schema registered to a topic with name and namespace
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz row it tries to produce is not compatible with the schema
registered
*Root cause:*
The upsert-kafka connector auto generates a schema with the +*name as `record`
and no namespace*+. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
Schema generated by the upsert-kafka connector which is using AvroRowSerializer
interanally
!image-2021-10-12-12-21-46-177.png|width=813,height=23!
{color:#cc7832}Schema Registered to the subject: {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
Table SQL with upsert-kafka connector
!image-2021-10-12-12-18-53-016.png|width=351,height=176!
The name "record" hardcoded
!image-2021-10-12-12-21-02-008.png|width=464,height=138!
was:
*Summary:*
Given a schema registered to a topic with name and namespave
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz row it tries to produce is not compatible with the schema
registered
*Root cause:*
The upsert-kafka connector auto generates a schema with the +*name as `record`
and no namespace*+. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
Schema generated by the upsert-kafka connector which is using AvroRowSerializer
interanally
!image-2021-10-12-12-21-46-177.png|width=813,height=23!
{color:#cc7832}Schema Registered to a subject:
{color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
Table SQL with upsert-kafka connector
!image-2021-10-12-12-18-53-016.png|width=351,height=176!
The name "record" hardcoded
!image-2021-10-12-12-21-02-008.png|width=464,height=138!
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png,
> image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
> *Summary:*
> Given a schema registered to a topic with name and namespace
> when the
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Affects Version/s: 1.14.0
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.14.0, 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png,
> image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
> *Summary:*
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatible with the schema
> registered
>
> *Root cause:*
> The upsert-kafka connector auto generates a schema with the +*name as
> `record` and no namespace*+. The below schema is generated by the connector.
> I'm expecting the connector should pull the schema from the subject and use
> ConfluentAvroRowSerialization to[which is not there today i believe]
> serialize using the schema from the subject.
> Schema generated by the upsert-kafka connector which is using
> AvroRowSerializer interanally
> !image-2021-10-12-12-21-46-177.png|width=813,height=23!
> {color:#cc7832}Schema Registered to a subject:
> {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> Table SQL with upsert-kafka connector
> !image-2021-10-12-12-18-53-016.png|width=351,height=176!
>
> The name "record" hardcoded
> !image-2021-10-12-12-21-02-008.png|width=464,height=138!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Description:
*Summary:*
Given a schema registered to a topic with name and namespave
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz row it tries to produce is not compatible with the schema
registered
*Root cause:*
The upsert-kafka connector auto generates a schema with the +*name as `record`
and no namespace*+. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
Schema generated by the upsert-kafka connector which is using AvroRowSerializer
interanally
!image-2021-10-12-12-21-46-177.png|width=813,height=23!
{color:#cc7832}Schema Registered to a subject:
{color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
Table SQL with upsert-kafka connector
!image-2021-10-12-12-18-53-016.png|width=351,height=176!
The name "record" hardcoded
!image-2021-10-12-12-21-02-008.png|width=464,height=138!
was:
Given a schema registered to a topic with name and namespave
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz row it tries to produce is not compatibale with the schema
registered
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png,
> image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
> *Summary:*
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatible with the schema
> registered
>
> *Root cause:*
> The upsert-kafka connector auto generates a schema with the +*name as
> `record` and no namespace*+. The below schema is generated by the connector.
> I'm expecting the connector should pull the schema from the subject and use
> ConfluentAvroRowSerialization to[which is not there today i believe]
> serialize using the schema from the subject.
> Schema generated by the upsert-kafka connector which is using
> AvroRowSerializer interanally
> !image-2021-10-12-12-21-46-177.png|width=813,height=23!
> {color:#cc7832}Schema Registered to a subject:
> {color}
> {
>
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: image-2021-10-12-12-21-46-177.png
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png,
> image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
>
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatibale with the schema
> registered
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: image-2021-10-12-12-21-02-008.png
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png,
> image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
>
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatibale with the schema
> registered
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: image-2021-10-12-12-19-37-227.png
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png
>
>
>
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatibale with the schema
> registered
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: image-2021-10-12-12-18-53-016.png
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png,
> image-2021-10-12-12-18-53-016.png
>
>
>
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatibale with the schema
> registered
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Description:
Given a schema registered to a topic with name and namespave
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz row it tries to produce is not compatibale with the schema
registered
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
was:
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with name and namespave
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatibale with the schema
> registered
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Description:
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
was:
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
The full code is here [^KafkaTableTumbling.java]
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default"
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Summary: Avro Confluent Registry SQL kafka connector fails to write to the
topic with schema (was: Avro Confluent Registry SQL not fails to write to the
topic with schema with default value for int)
> Avro Confluent Registry SQL kafka connector fails to write to the topic with
> schema
> -
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: (was: KafkaTableTumbling.java)
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: (was: image-2021-10-12-12-09-24-259.png)
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: (was: image-2021-10-12-12-06-16-262.png)
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png,
> image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427871#comment-17427871
]
Mahendran Ponnusamy edited comment on FLINK-24494 at 10/12/21, 7:11 PM:
Thank you [~MartijnVisser], I figured the issue is not with the default, it is
with the schema name that is generated by the connector.
The subjects has the below schema registered
!image-2021-10-12-12-11-04-664.png|width=1135,height=17!
Where as the connector generates the schema with the name as `record` hardcoded
and no namespace. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
!image-2021-10-12-12-09-30-374.png|width=586,height=14!
and it throws "Failed to serialize row"
!Screen Shot 2021-10-12 at 10.38.55 AM.png|width=501,height=142!
I believe I should close this bug and create a new one, If its a bug
was (Author: mahen):
Thank you [~MartijnVisser], I figured the issue is not with the default, it is
with the schema name that is generated by the connector.
The subjects has the below schema registered
{quote}{"type":"record","name":"SimpleCustomer","namespace":"com....example.model","fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int","default":0}]}{quote}
Where as the connector generates the schema with the name as `record` hardcoded
and no namespace. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
{quote}{"type":"record",{color:#ff}"name":"record"{color},"fields":[\\{"name":"customerId","type":"string"},\\{"name":"age","type":"int"}]}
{quote}
and it throws "Failed to serialize row"
!Screen Shot 2021-10-12 at 10.38.55 AM.png|width=501,height=142!
I believe I should close this bug and create a new one, If its a bug
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: KafkaTableTumbling.java, Screen Shot 2021-10-12 at
> 10.38.55 AM.png, image-2021-10-12-12-06-16-262.png,
> image-2021-10-12-12-09-24-259.png, image-2021-10-12-12-09-30-374.png,
> image-2021-10-12-12-11-04-664.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427871#comment-17427871
]
Mahendran Ponnusamy edited comment on FLINK-24494 at 10/12/21, 7:04 PM:
Thank you [~MartijnVisser], I figured the issue is not with the default, it is
with the schema name that is generated by the connector.
The subjects has the below schema registered
{quote}{"type":"record","name":"SimpleCustomer","namespace":"com....example.model","fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int","default":0}]}{quote}
Where as the connector generates the schema with the name as `record` hardcoded
and no namespace. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
{quote}{"type":"record",{color:#ff}"name":"record"{color},"fields":[\\{"name":"customerId","type":"string"},\\{"name":"age","type":"int"}]}
{quote}
and it throws "Failed to serialize row"
!Screen Shot 2021-10-12 at 10.38.55 AM.png|width=501,height=142!
I believe I should close this bug and create a new one, If its a bug
was (Author: mahen):
Thank you [~MartijnVisser], I figured the issue is not with the default, it is
with the schema name that is generated by the connector.
The subjects has the below schema registered
{quote}{"type":"record","name":"SimpleCustomer","namespace":"com.nordstrom.nap.onehop.example.model","fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int","default":0}]}
{quote}
Where as the connector generates the schema with the name as `record` hardcoded
and no namespace. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
{quote}{"type":"record",{color:#FF}"name":"record"{color},"fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int"}]}
{quote}
and it throws "Failed to serialize row"
!Screen Shot 2021-10-12 at 10.38.55 AM.png|width=501,height=142!
I believe I should close this bug and create a new one, If its a bug
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: KafkaTableTumbling.java, Screen Shot 2021-10-12 at
> 10.38.55 AM.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427871#comment-17427871
]
Mahendran Ponnusamy commented on FLINK-24494:
-
Thank you [~MartijnVisser], I figured the issue is not with the default, it is
with the schema name that is generated by the connector.
The subjects has the below schema registered
{quote}{"type":"record","name":"SimpleCustomer","namespace":"com.nordstrom.nap.onehop.example.model","fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int","default":0}]}
{quote}
Where as the connector generates the schema with the name as `record` hardcoded
and no namespace. The below schema is generated by the connector. I'm
expecting the connector should pull the schema from the subject and use
ConfluentAvroRowSerialization to[which is not there today i believe] serialize
using the schema from the subject.
{quote}{"type":"record",{color:#FF}"name":"record"{color},"fields":[\{"name":"customerId","type":"string"},\{"name":"age","type":"int"}]}
{quote}
and it throws "Failed to serialize row"
!Screen Shot 2021-10-12 at 10.38.55 AM.png|width=501,height=142!
I believe I should close this bug and create a new one, If its a bug
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: KafkaTableTumbling.java, Screen Shot 2021-10-12 at
> 10.38.55 AM.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro Confluent Registry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Attachment: Screen Shot 2021-10-12 at 10.38.55 AM.png
> Avro Confluent Registry SQL not fails to write to the topic with schema with
> default value for int
> --
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: KafkaTableTumbling.java, Screen Shot 2021-10-12 at
> 10.38.55 AM.png
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The full code is here [^KafkaTableTumbling.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24494) Avro conluent Resigry SQL not fails to write to the topic with schema with default value for int
[
https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahendran Ponnusamy updated FLINK-24494:
Description:
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
{color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
{color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type"
{color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
{color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color}
}]
}
The full code is here [^KafkaTableTumbling.java]
was:
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},
{color} {color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},
{color} {color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},
{color} {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},
{color} {color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"
{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},
{color} {color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},
{color} {color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0
{color} } ][^KafkaTableTumbling.java]
}
> Avro conluent Resigry SQL not fails to write to the topic with schema with
> default value for int
>
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.13.2
>Reporter: Mahendran Ponnusamy
>Priority: Critical
> Attachments: KafkaTableTumbling.java
>
>
>
> Given a schema registered to a topic with default value as '0' for int,
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz the schema registered is not compatible with the data [schema]
> it is producing where there is not default value [{color:#9876aa}"default"
> {color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
> {color:#cc7832} {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"record"{color}{color:#cc7832},{color}
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color}
> {color:#9876aa}"namespace" {color}{color:#cc7832}:
> {color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},{color}
> {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}:
> {color}{color:#6a8759}"age"{color}{color:#cc7832},{color}
> {color:#9876aa}"type" {color}{color:#cc7832}:
> {color}{color:#6a8759}"int"{color}{color:#cc7832},{color}
> {color:#9876aa}"default" {color}{color:#cc7832}:
> {color}{color:#6897bb}0{color} }]
> }
>
> The
[jira] [Created] (FLINK-24494) Avro conluent Resigry SQL not fails to write to the topic with schema with default value for int
Mahendran Ponnusamy created FLINK-24494:
---
Summary: Avro conluent Resigry SQL not fails to write to the topic
with schema with default value for int
Key: FLINK-24494
URL: https://issues.apache.org/jira/browse/FLINK-24494
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.13.2
Reporter: Mahendran Ponnusamy
Attachments: KafkaTableTumbling.java
Given a schema registered to a topic with default value as '0' for int,
when the flink sql with upsert-kafka connector writes to the topic,
it fails coz the schema registered is not compatible with the data [schema] it
is producing where there is not default value [{color:#9876aa}"default"
{color}{color:#cc7832}: {color}{color:#6897bb}0{color}] for the int.
{color:#cc7832} {color}
{
{color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"record"{color}{color:#cc7832},
{color} {color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},
{color} {color:#9876aa}"namespace" {color}{color:#cc7832}:
{color}{color:#6a8759}"com.nordstrom.nap.onehop.example.model"{color}{color:#cc7832},
{color} {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"customerId"{color}{color:#cc7832},
{color} {color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"string"
{color} }{color:#cc7832}, {color}{
{color:#9876aa}"name" {color}{color:#cc7832}:
{color}{color:#6a8759}"age"{color}{color:#cc7832},
{color} {color:#9876aa}"type" {color}{color:#cc7832}:
{color}{color:#6a8759}"int"{color}{color:#cc7832},
{color} {color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0
{color} } ][^KafkaTableTumbling.java]
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
