Re: Problem with Flink job and Kafka.

2021-10-20 Thread Marco Villalobos
Hi Qingsheng,

Thank you.

I am running Flink in EMR 6.3.0 which uses Flink version 1.12.1.  We AWS
MSK Kafka, and we are currently using Kafka 2.2.1.

The stack trace seems empty, it only states this error:

org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to
write a non-default producerId at version 1

I notice that this configuration uses Flink 2.5.0, which is ahead of Kafka
2.2.1.  I take your word that this is a version issue.

I am going to try and upgrade our Kafka clusters and see if that makes a
difference.

Thank you. Any further insight will be highly appreciated.

-Marco


On Mon, Oct 18, 2021 at 8:03 PM Qingsheng Ren  wrote:

> Hi Marco,
>
> Sorry I forgot to cc the user mailing list just now.
>
> From the exception message it looks like a versioning issue. Could you
> provide some additional information, such as Flink & Kafka connector
> version, Kafka broker version, and full exception stack? Also it will be
> helpful to paste part of your code (on DataStream API) or SQL (on Table &
> SQL API).
>
> --
> Best Regards,
>
> Qingsheng Ren
> Email: renqs...@gmail.com
> On Oct 19, 2021, 9:28 AM +0800, Marco Villalobos <
> mvillalo...@kineteque.com>, wrote:
>
> I have the simplest Flink job that simply deques off of a kafka topic and
> writes to another kafka topic, but with headers, and manually copying the
> event time into the kafka sink.
>
> It works as intended, but sometimes I am getting this error:
>
> org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to
> write a non-default producerId at version 1.
>
> Does anybody know what this means and how to fix this?
>
> Thank you.
>
>


Re: Problem with Flink job and Kafka.

2021-10-18 Thread Qingsheng Ren
Hi Marco,

Sorry I forgot to cc the user mailing list just now.

From the exception message it looks like a versioning issue. Could you provide 
some additional information, such as Flink & Kafka connector version, Kafka 
broker version, and full exception stack? Also it will be helpful to paste part 
of your code (on DataStream API) or SQL (on Table & SQL API).

--
Best Regards,

Qingsheng Ren
Email: renqs...@gmail.com
On Oct 19, 2021, 9:28 AM +0800, Marco Villalobos , 
wrote:
> I have the simplest Flink job that simply deques off of a kafka topic and 
> writes to another kafka topic, but with headers, and manually copying the 
> event time into the kafka sink.
>
> It works as intended, but sometimes I am getting this error:
>
> org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to 
> write a non-default producerId at version 1.
>
> Does anybody know what this means and how to fix this?
>
> Thank you.
>


Problem with Flink job and Kafka.

2021-10-18 Thread Marco Villalobos
I have the simplest Flink job that simply deques off of a kafka topic and
writes to another kafka topic, but with headers, and manually copying the
event time into the kafka sink.

It works as intended, but sometimes I am getting this error:

org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to
write a non-default producerId at version 1.

Does anybody know what this means and how to fix this?

Thank you.