Re: [VOTE] Reverting sink metric name changes made in 1.15

[Xintong Song]

+1 (binding)

Best,

Xintong



On Fri, Oct 14, 2022 at 11:19 AM Jark Wu  wrote:

> Thanks a lot for driving this, Qingsheng!
>
> +1 (binding)
>
>
> Best,
> Jark
>
> On Fri, 14 Oct 2022 at 10:16, Hang Ruan  wrote:
>
> > +1 from me (non-binding).
> >
> > The documents about these Kafka sink metrics( Flink metrics
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope
> > >
> > and Kafka sink monitoring
> > <
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1
> > >)
> > do not contain the usage of the metrics like numRecordsSend by now. Only
> > Kinesis Sink contains these metrics.
> > If the document about the metrics could be found in other places, please
> > correct me. ;)
> >
> > Best,
> > Hang
> >
> > Chesnay Schepler  于2022年10月14日周五 01:56写道：
> >
> > > Do we have a PoC that achieves this without re-introducing the bug
> where
> > > the numRecordsOut was simply wrong because it counted both records
> > > written to the external system and the downstream committer?
> > > It's gonna be quite the dirty hack I assume.
> > >
> > > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > > Hi devs,
> > > >
> > > > I'd like to start a vote about reverting sink metric name changes
> made
> > in
> > > > 1.15 considering compatibility issues. These metrics include:
> > > >
> > > > - numRecordsSend -> numRecordsOut
> > > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > > - numBytesSend -> numBytesOut
> > > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > > - numRecordsSendError -> numRecordsOutError
> > > >
> > > > which reflect the output of the sink to the external system. "send"
> > > metric
> > > > series will be kept with the same value as "out" metric series. This
> > > change
> > > > will be applied to 1.15 and 1.16. More details could be found in the
> > > > discussion thread [1].
> > > >
> > > > The vote will open for at least 72 hours.
> > > >
> > > > Looking forward to your feedback!
> > > >
> > > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > >
> > >
> >
>

RE: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Han Yin]

Hi Hangxiang,

Thanks for the proposal. It seems more reasonable to let the new serializer 
claim the compatibility in the cases you mentioned.

I have but one question here. What happens in the case of 
“compatibleAfterMigration” after we completely reverse the direction (in step 
3)?  To be specific, migration from an old schema calls for the previous 
serializer to read bytes into state objects. How should a new serializer decide 
whether the migration is possible?

Best,
Han

On 2022/10/12 12:41:07 Hangxiang Yu wrote:
> Dear Flink developers,
> 
> I would like to start a discussion thread on FLIP-263[1] proposing to
> improve the usability of resolving schema compatibility.
> 
> Currently, the place for compatibility checks is
> TypeSerializerSnapshot#resolveSchemaCompatibility
> which belongs to the old serializer, There are no ways for users to specify 
> the
> compatibility with the old serializer in the new customized serializer.
> 
> The FLIP hopes to reverse the direction of resolving schema compatibility
> to improve the usability of resolving schema compatibility.
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
> 

Re: [Discuss]- Donate Iceberg Flink Connector

[Jark Wu]

Thank Abid for the discussion,

I'm also fine with maintaining it under the Flink project.
But I'm also interested in the response to Martijn's question.

Besides, once the code is moved to the Flink project, are there any initial
maintainers for the connector we can find?
In addition, do we still maintain documentation under Iceberg
https://iceberg.apache.org/docs/latest/flink/ ?

Best,
Jark


On Thu, 13 Oct 2022 at 17:52, yuxia  wrote:

> +1. Thanks for driving it. Hope I can find some chances to take part in
> the future development of Iceberg Flink Connector.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Zheng Yu Chen" 
> 收件人: "dev" 
> 发送时间: 星期四, 2022年 10 月 13日 上午 11:26:29
> 主题: Re: [Discuss]- Donate Iceberg Flink Connector
>
> +1, thanks to drive it
>
> Abid Mohammed  于2022年10月10日周一 09:22写道：
>
> > Hi,
> >
> > I would like to start a discussion about contributing Iceberg Flink
> > Connector to Flink.
> >
> > I created a doc <
> >
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing
> >
> > with all the details following the Flink Connector template as I don’t
> have
> > permissions to create a FLIP yet.
> > High level details are captured below:
> >
> > Motivation:
> >
> > This FLIP aims to contribute the existing Apache Iceberg Flink Connector
> > to Flink.
> >
> > Apache Iceberg is an open table format for huge analytic datasets.
> Iceberg
> > adds tables to compute engines including Spark, Trino, PrestoDB, Flink,
> > Hive and Impala using a high-performance table format that works just
> like
> > a SQL table.
> > Iceberg avoids unpleasant surprises. Schema evolution works and won’t
> > inadvertently un-delete data. Users don’t need to know about partitioning
> > to get fast queries. Iceberg was designed to solve correctness problems
> in
> > eventually-consistent cloud object stores.
> >
> > Iceberg supports both Flink’s DataStream API and Table API. Based on the
> > guideline of the Flink community, only the latest 2 minor versions are
> > actively maintained. See the Multi-Engine Support#apache-flink for
> further
> > details.
> >
> >
> > Iceberg connector supports:
> >
> > • Source: detailed Source design <
> >
> https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit#
> >,
> > based on FLIP-27
> > • Sink: detailed Sink design and interfaces used <
> >
> https://docs.google.com/document/d/1O-dPaFct59wUWQECXEEYIkl9_MOoG3zTbC2V-fZRwrg/edit#
> > >
> > • Usable in both DataStream and Table API/SQL
> > • DataStream read/append/overwrite
> > • SQL create/alter/drop table, select, insert into, insert
> > overwrite
> > • Streaming or batch read in Java API
> > • Support for Flink’s Python API
> >
> > See Iceberg Flink   >for
> > detailed usage instructions.
> >
> > Looking forward to the discussion!
> >
> > Thanks
> > Abid
>

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

[Jark Wu]

Hi Xintong,

In terms of code, I think it's not complicated. It's all about we need a
public discussion for the new metric name.
And we don't want to block the release for the rarely used metric.

Best,
Jark

On Fri, 14 Oct 2022 at 10:07, Xintong Song  wrote:

> @Qingsheng,
>
> I'm overall +1 to your proposal, with only one question: How complicated
> is it to come up with a metric for the internal traffic?
>
> I'm asking because, as the new feature is already out for 1.15 & 1.16, it
> would be nice if the corresponding new metrics can also be available in
> these versions.
>
> I'm not saying this should block the 1.16 release. Just trying to
> understand the efforts needed. If adding such metrics is not something that
> can be done shortly, I'd also be fine with releasing 1.16 without them.
>
> Best,
>
> Xintong
>
>
>
> On Fri, Oct 14, 2022 at 1:31 AM Qingsheng Ren  wrote:
>
>> Hi devs and users,
>>
>> It looks like we are getting an initial consensus in the discussion so I
>> started a voting thread [1] just now. Looking forward to your feedback!
>>
>> [1] https://lists.apache.org/thread/ozlf82mkm6ndx2n1vdgq532h156p4lt6
>>
>> Best,
>> Qingsheng
>>
>>
>> On Thu, Oct 13, 2022 at 10:41 PM Jing Ge  wrote:
>>
>>> Hi Qingsheng,
>>>
>>> Thanks for the clarification. +1, I like the idea. Pointing both
>>> numXXXOut and numXXXSend to the same external data transfer metric does not
>>> really break the new SinkV2 design, since there was no requirement to
>>> monitor the internal traffic. So, I think both developer and user can live
>>> with it. It might not be the perfect solution but is indeed the currently
>>> best trade-off solution after considering the backward compatibility.  I
>>> would suggest firing a follow-up ticket after the PR to take care of the
>>> new metric for the internal traffic in the future.
>>>
>>> Best regards,
>>> Jing
>>>
>>>
>>> On Thu, Oct 13, 2022 at 3:08 PM Qingsheng Ren  wrote:
>>>
 Hi Jing,

 Thanks for the reply!

 Let me rephrase my proposal: we’d like to use numXXXOut registered on
 SinkWriterOperator to reflect the traffic to the external system for
 compatibility with old versions before 1.15, and make numXXXSend have the
 same value as numXXXOut for compatibility within 1.15. That means both
 numXXXOut and numXXXSend are used for external data transfers, which end
 users care more about. As for the internal traffic within the sink, we
 could name a new metric for it because this is a _new_ feature in the _new_
 sink, and end users usually don’t pay attention to internal implementation.
 The name of the new metric could be discussed later after 1.16 release.

 > but it might end up with monitoring unexpected metrics, which is
 even worse for users, i.e. I didn't change anything, but something has been
 broken since the last update.

 Yeah this is exactly what we are trying to fix with this proposal. I
 believe users are more concerned with the output to the external system
 than the internal data delivery in the sink, so I think we’ll have more
 cases reporting like “I set up a panel on numRecordsOut in sink to monitor
 the output of the job, but after upgrading to 1.15 this value is extremely
 low and I didn’t change anything” if we stick to the current situation. I
 think only a few end users care about the number of committables sending to
 downstream as most of them don’t care how the sink works.

 We do need a re-design to fully distinguish the internal and external
 traffic on metrics, not only in sink but in all operators as it’s quite
 common for operators to make IO. This needs time to design, discuss, adjust
 and vote, but considering this is blocking 1.16, maybe it’s better to
 rescue the compatibility for now, and leave the huge reconstruction to
 future versions (maybe 2.0).

 Best,
 Qingsheng

 On Wed, Oct 12, 2022 at 7:21 PM Jing Ge  wrote:

> Hi Qingsheng,
>
> Just want to make sure we are on the same page. Are you suggesting
> switching the naming between "numXXXSend" and "numXXXOut" or reverting all
> the changes we did with FLINK-26126 and FLINK-26492?
>
> For the naming switch, please pay attention that the behaviour has
> been changed since we introduced SinkV2[1]. So, please be aware of
> different numbers(behaviour change) even with the same metrics name.
> Sticking with the old name with the new behaviour (very bad idea, IMHO)
> might seem like saving the effort in the first place, but it might end up
> with monitoring unexpected metrics, which is even worse for users, i.e. I
> didn't change anything, but something has been broken since
> the last update.
>
> For reverting, I am not sure how to fix the issue mentioned in
> FLINK-26126 after reverting all changes. Like Chesnay has already pointed
> out, with SinkV2 we have two 

Re: [VOTE] Reverting sink metric name changes made in 1.15

[Jark Wu]

Thanks a lot for driving this, Qingsheng!

+1 (binding)


Best,
Jark

On Fri, 14 Oct 2022 at 10:16, Hang Ruan  wrote:

> +1 from me (non-binding).
>
> The documents about these Kafka sink metrics( Flink metrics
> <
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/metrics//#scope
> >
> and Kafka sink monitoring
> <
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka//#monitoring-1
> >)
> do not contain the usage of the metrics like numRecordsSend by now. Only
> Kinesis Sink contains these metrics.
> If the document about the metrics could be found in other places, please
> correct me. ;)
>
> Best,
> Hang
>
> Chesnay Schepler  于2022年10月14日周五 01:56写道：
>
> > Do we have a PoC that achieves this without re-introducing the bug where
> > the numRecordsOut was simply wrong because it counted both records
> > written to the external system and the downstream committer?
> > It's gonna be quite the dirty hack I assume.
> >
> > On 13/10/2022 19:24, Qingsheng Ren wrote:
> > > Hi devs,
> > >
> > > I'd like to start a vote about reverting sink metric name changes made
> in
> > > 1.15 considering compatibility issues. These metrics include:
> > >
> > > - numRecordsSend -> numRecordsOut
> > > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > > - numBytesSend -> numBytesOut
> > > - numBytesSendPerSecond -> numBytesOutPerSecond
> > > - numRecordsSendError -> numRecordsOutError
> > >
> > > which reflect the output of the sink to the external system. "send"
> > metric
> > > series will be kept with the same value as "out" metric series. This
> > change
> > > will be applied to 1.15 and 1.16. More details could be found in the
> > > discussion thread [1].
> > >
> > > The vote will open for at least 72 hours.
> > >
> > > Looking forward to your feedback!
> > >
> > > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> > >
> > > Best,
> > > Qingsheng
> > >
> >
> >
>

Re: [DISCUSS] FLIP-263: Improve resolving schema compatibility

[Hangxiang Yu]

Hi, Zakelly.
Thanks for your suggestion and I think it makes sense.

I think both two options should be adopted. The option 2 is just what I
mentioned in step 2. At the same time, we still need option 1 because we
should make sure the existing user code could run normally without
modifying any codes before we remove the deprecated method (
TypeSerializerSnapshot#resolveSchemaCompatibility).


Hi, Yanfei.
Thanks for your feedback.

Will Typeserializer#resolveSchemaCompatibility become an abstract method
that must be implemented here?

Considering the compatibility before removing the deprecated method (
TypeSerializerSnapshot#resolveSchemaCompatibility), we may still need to
have a default implementation for
Typeserializer#resolveSchemaCompatibility. When we remove the deprecated
method (In the step 3), I think it deserves a discussion whether we need to
make it abstract or not (it will break changes twice).

Does it support the migration from built-in serializer -> custom serializer
-> built-in serializer ?

I think after reversing the direction of resolving schema compatibility,
users could just consider the compatibility result in the newest
serializer. If the built-in serializer cannot work well, users could just
specify their new serializer. It could make users always have ways to
restore from the snapshot if they want to do so.

On Thu, Oct 13, 2022 at 11:37 AM yanfei lei  wrote:

> Hi Hangxiang,
>
>
> Thanks for raising the discussion, +1 for reversing the direction of
> resolving schema compatibility.
>
> As you described, in 'Step 1', Typeserializer#resolveSchemaCompatibility
> will return TYPE.INCOMPATIBLE default,
> Typeserializer#resolveSchemaCompatibility is a default method; in 'Step
> 2&3', you proposed deprecating
> TypeSerializerSnapshot#resolveSchemaCompatibility for the long run, will
> Typeserializer#resolveSchemaCompatibility become an abstract method that
> must be implemented here?
>
>
> After this FILP， the new serializer claims its compatibility with the old
> serializer， does it support the migration from
>
> built-in serializer -> custom serializer -> built-in serializer ?
>
> as you mentioned, some built-in serializers are final class, we can't
> change the #resolveSchemaCompatibility() method of them.
>
>
> Best,
>
> Yanfei
>
> Zakelly Lan  于2022年10月13日周四 00:56写道：
>
> > Hi Hangxiang,
> >
> > Thanks for driving this. It is reasonable to let the new serializer
> > claim its compatibility with the old serializer. However, I think
> > there is a little confusion as you described in your proposed change
> > 'Step 1'. You mean that we let the new serializer check the
> > compatibility first, and if it gives the 'INCOMPATIBLE' then we let
> > the old serializer do the check as before. The 'INCOMPATIBLE' from the
> > new serializer means 'do not know' or 'unresolved' here, but sometimes
> > the new serializer should express a decisive meaning that it is really
> > not compatible with the older one, which is the 'INCOMPATIBLE' should
> > mean. There is a semantic ambiguity or missing. I think there are two
> > options to resolve this:
> > 1. Provide 'UNRESOLVED' in type of the compatibility checking result,
> > and let the proposed new interface return it by default. OR
> > 2. Replace any use of the old interface with the new interface. In the
> > default implementation of the new interface, call the old interface to
> > leverage the old result. This approach provides the ability to totally
> > replace original checking logic (by implementing the new interface in
> > new serializer) while maintaining good backward compatibility.
> >
> > What do you think?
> >
> > Best,
> > Zakelly.
> >
> >
> > On Wed, Oct 12, 2022 at 8:41 PM Hangxiang Yu 
> wrote:
> > >
> > > Dear Flink developers,
> > >
> > > I would like to start a discussion thread on FLIP-263[1] proposing to
> > > improve the usability of resolving schema compatibility.
> > >
> > > Currently, the place for compatibility checks is
> > > TypeSerializerSnapshot#resolveSchemaCompatibility
> > > which belongs to the old serializer, There are no ways for users to
> > specify the
> > > compatibility with the old serializer in the new customized serializer.
> > >
> > > The FLIP hopes to reverse the direction of resolving schema
> compatibility
> > > to improve the usability of resolving schema compatibility.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-263%3A+Improve+resolving+schema+compatibility
> >
>


-- 
Best,
Hangxiang.

[jira] [Created] (FLINK-29637) Improve to reuse threads in TaskManager for different tasks between jobs

[Shammon (Jira)]

Shammon created FLINK-29637:
---

 Summary: Improve to reuse threads in TaskManager for different 
tasks between jobs
 Key: FLINK-29637
 URL: https://issues.apache.org/jira/browse/FLINK-29637
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Affects Versions: 1.17.0
Reporter: Shammon


Currently `TaskManager` will create multiple threads for each task, such as 
execution thread, cancel thread, source thread, flush thread and etc. These 
threads will be destroyed when the task is finished. In olap cluster, the 
`TaskManager` frequently creates and destroys threads, which causes a great 
waste of resources. We should improve the `TaskManager` to reuse these thread 
after the task is finished.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29636) Add micro benchmark module in flink table store

[Shammon (Jira)]

Shammon created FLINK-29636:
---

 Summary: Add micro benchmark module in flink table store
 Key: FLINK-29636
 URL: https://issues.apache.org/jira/browse/FLINK-29636
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.2.2
Reporter: Shammon


Currently there's a `flink-table-store-benchmark` to run a query in flink 
cluster and collect metrics to measure the performance of `flink-table-store`. 
There're some key operation steps such as read/write/compaction in 
`flink-table-store`, we should add a `flink-table-store-micro-benchmark` for 
them



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Reverting sink metric name changes made in 1.15

[Hang Ruan]

+1 from me (non-binding).

The documents about these Kafka sink metrics( Flink metrics

and Kafka sink monitoring
)
do not contain the usage of the metrics like numRecordsSend by now. Only
Kinesis Sink contains these metrics.
If the document about the metrics could be found in other places, please
correct me. ;)

Best,
Hang

Chesnay Schepler  于2022年10月14日周五 01:56写道：

> Do we have a PoC that achieves this without re-introducing the bug where
> the numRecordsOut was simply wrong because it counted both records
> written to the external system and the downstream committer?
> It's gonna be quite the dirty hack I assume.
>
> On 13/10/2022 19:24, Qingsheng Ren wrote:
> > Hi devs,
> >
> > I'd like to start a vote about reverting sink metric name changes made in
> > 1.15 considering compatibility issues. These metrics include:
> >
> > - numRecordsSend -> numRecordsOut
> > - numRecordsSendPerSecond -> numRecordsOutPerSecond
> > - numBytesSend -> numBytesOut
> > - numBytesSendPerSecond -> numBytesOutPerSecond
> > - numRecordsSendError -> numRecordsOutError
> >
> > which reflect the output of the sink to the external system. "send"
> metric
> > series will be kept with the same value as "out" metric series. This
> change
> > will be applied to 1.15 and 1.16. More details could be found in the
> > discussion thread [1].
> >
> > The vote will open for at least 72 hours.
> >
> > Looking forward to your feedback!
> >
> > [1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv
> >
> > Best,
> > Qingsheng
> >
>
>

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

[Xintong Song]

@Qingsheng,

I'm overall +1 to your proposal, with only one question: How complicated is
it to come up with a metric for the internal traffic?

I'm asking because, as the new feature is already out for 1.15 & 1.16, it
would be nice if the corresponding new metrics can also be available in
these versions.

I'm not saying this should block the 1.16 release. Just trying to
understand the efforts needed. If adding such metrics is not something that
can be done shortly, I'd also be fine with releasing 1.16 without them.

Best,

Xintong



On Fri, Oct 14, 2022 at 1:31 AM Qingsheng Ren  wrote:

> Hi devs and users,
>
> It looks like we are getting an initial consensus in the discussion so I
> started a voting thread [1] just now. Looking forward to your feedback!
>
> [1] https://lists.apache.org/thread/ozlf82mkm6ndx2n1vdgq532h156p4lt6
>
> Best,
> Qingsheng
>
>
> On Thu, Oct 13, 2022 at 10:41 PM Jing Ge  wrote:
>
>> Hi Qingsheng,
>>
>> Thanks for the clarification. +1, I like the idea. Pointing both
>> numXXXOut and numXXXSend to the same external data transfer metric does not
>> really break the new SinkV2 design, since there was no requirement to
>> monitor the internal traffic. So, I think both developer and user can live
>> with it. It might not be the perfect solution but is indeed the currently
>> best trade-off solution after considering the backward compatibility.  I
>> would suggest firing a follow-up ticket after the PR to take care of the
>> new metric for the internal traffic in the future.
>>
>> Best regards,
>> Jing
>>
>>
>> On Thu, Oct 13, 2022 at 3:08 PM Qingsheng Ren  wrote:
>>
>>> Hi Jing,
>>>
>>> Thanks for the reply!
>>>
>>> Let me rephrase my proposal: we’d like to use numXXXOut registered on
>>> SinkWriterOperator to reflect the traffic to the external system for
>>> compatibility with old versions before 1.15, and make numXXXSend have the
>>> same value as numXXXOut for compatibility within 1.15. That means both
>>> numXXXOut and numXXXSend are used for external data transfers, which end
>>> users care more about. As for the internal traffic within the sink, we
>>> could name a new metric for it because this is a _new_ feature in the _new_
>>> sink, and end users usually don’t pay attention to internal implementation.
>>> The name of the new metric could be discussed later after 1.16 release.
>>>
>>> > but it might end up with monitoring unexpected metrics, which is even
>>> worse for users, i.e. I didn't change anything, but something has been
>>> broken since the last update.
>>>
>>> Yeah this is exactly what we are trying to fix with this proposal. I
>>> believe users are more concerned with the output to the external system
>>> than the internal data delivery in the sink, so I think we’ll have more
>>> cases reporting like “I set up a panel on numRecordsOut in sink to monitor
>>> the output of the job, but after upgrading to 1.15 this value is extremely
>>> low and I didn’t change anything” if we stick to the current situation. I
>>> think only a few end users care about the number of committables sending to
>>> downstream as most of them don’t care how the sink works.
>>>
>>> We do need a re-design to fully distinguish the internal and external
>>> traffic on metrics, not only in sink but in all operators as it’s quite
>>> common for operators to make IO. This needs time to design, discuss, adjust
>>> and vote, but considering this is blocking 1.16, maybe it’s better to
>>> rescue the compatibility for now, and leave the huge reconstruction to
>>> future versions (maybe 2.0).
>>>
>>> Best,
>>> Qingsheng
>>>
>>> On Wed, Oct 12, 2022 at 7:21 PM Jing Ge  wrote:
>>>
 Hi Qingsheng,

 Just want to make sure we are on the same page. Are you suggesting
 switching the naming between "numXXXSend" and "numXXXOut" or reverting all
 the changes we did with FLINK-26126 and FLINK-26492?

 For the naming switch, please pay attention that the behaviour has been
 changed since we introduced SinkV2[1]. So, please be aware of different
 numbers(behaviour change) even with the same metrics name. Sticking with
 the old name with the new behaviour (very bad idea, IMHO) might seem like
 saving the effort in the first place, but it might end up with monitoring
 unexpected metrics, which is even worse for users, i.e. I didn't change
 anything, but something has been broken since the last update.

 For reverting, I am not sure how to fix the issue mentioned in
 FLINK-26126 after reverting all changes. Like Chesnay has already pointed
 out, with SinkV2 we have two different output lines - one with the external
 system and the other with the downstream operator. In this case,
 "numXXXSend" is rather a new metric than a replacement of "numXXXOut". The
 "numXXXOut" metric can still be used, depending on what the user wants to
 monitor.


 Best regards,
 Jing

 [1]
 

Re: Utilizing Kafka headers in Flink Kafka connector

[Shengkai Fang]

hi.

You can use SQL API to parse or write the header in the Kafka record[1] if
you are using Flink SQL.

Best,
Shengkai

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#available-metadata

Yaroslav Tkachenko  于2022年10月13日周四 02:21写道：

> Hi,
>
> You can implement a custom KafkaRecordDeserializationSchema (example
> https://docs.immerok.cloud/docs/cookbook/reading-apache-kafka-headers-with-apache-flink/#the-custom-deserializer)
> and just avoid emitting the record if the header value matches what you
> need.
>
> On Wed, Oct 12, 2022 at 11:04 AM Great Info  wrote:
>
>> I have some flink applications that read streams from Kafka, now
>> the producer side code has introduced some additional information in Kafka
>> headers while producing records.
>> Now I need to change my consumer-side logic to process the records if the
>> header contains a specific value, if the header value is different than the
>> one I am looking I just need to move forward with the next steam.
>>
>> I got some sample reference code
>> but this logic needs
>> to deserialize and verify the header. Is there any simple way to ignore the
>> record before deserializing?
>>
>

[jira] [Created] (FLINK-29635) Hive sink should supports merge small files in batch mode

[luoyuxia (Jira)]

luoyuxia created FLINK-29635:


 Summary: Hive sink should supports merge small files in batch mode
 Key: FLINK-29635
 URL: https://issues.apache.org/jira/browse/FLINK-29635
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Reporter: luoyuxia
 Fix For: 1.17.0


When using Flink write Hive table in batch mode, there may produce small files. 
We should provide a mechanism to merge these small files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re:[VOTE] Remove HCatalog connector

[Roc Marshal]


+1 (non-binding)
Thanks for driving.

Best regards,
Roc Marshal.

At 2022-10-13 03:45:57, "Martijn Visser"  wrote:
>Hi everyone,
>
>Since no comments were made, I'm opening a vote to remove the HCatalog
>connector [1]
>
>The voting period will be open at least 72hrs.
>
>Best regards,
>
>Martijn
>
>[1]
>https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84
>-- 
>Martijn
>https://twitter.com/MartijnVisser82
>https://github.com/MartijnVisser

[jira] [Created] (FLINK-29634) Support periodic checkpoint triggering

[Thomas Weise (Jira)]

Thomas Weise created FLINK-29634:


 Summary: Support periodic checkpoint triggering
 Key: FLINK-29634
 URL: https://issues.apache.org/jira/browse/FLINK-29634
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator
Reporter: Thomas Weise


Similar to the support for periodic savepoints, the operator should support 
triggering periodic checkpoints to break the incremental checkpoint chain.

Support for external triggering will come with 1.17: 
https://issues.apache.org/jira/browse/FLINK-27101 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Externalized connector release details​

[Chesnay Schepler]

I will write this all down in the wiki once the vote is over, but here 
are some example.



Let's start out with a happy-case scenario with one connector supporting 
the last 2 Flink versions.
This will commonly be the scenario for connectors when they have been 
externalized:


v1: 1.14-1.15


Now we create a v2 that only support 1.15:

v1: 1.14-1.15 (patch support)
v2: 1.15 (feature support)

4.1) kicks in, both versions getting support, but only the latest 
getting new features.



Now 1.16 is released, which v2 also supports.

v1: 1.14-1.15 (patch support)
v2: 1.15-1.16 (feature support)

Nothing changes.


Now 1.17 is released:

v1: 1.14-1.15 (no support)
v2: 1.15-1.17

Here 4.1.a kicks in; v1 supports no supported Flink version and lost 
support.



Now we create v3 targeting 1.17, and shortly thereafter v4, also 
targeting 1.17, because we messed something up or are just that excited 
about finally having major releases.


v2: 1.15-1.17 (patch support)
v3: 1.17 (patch support)
v4: 1.17 (feature support)

Here 4.1.b kicks in, ensuring that v2 is still supported since we need 
to support all Flink versions.



Now 1.18 is released, with v3 and v4 supporting it.

v2: 1.15-1.17 (no support)
v3: 1.17 (patch support)
v4: 1.17 (feature support)

General 4.1) rule kicks in, with only the last 2 major versions being 
supported.


On 13/10/2022 16:25, Jing Ge wrote:

+1 and I would suggest giving a concrete example to explain 4) support. It
is still not quite easy to understand the text. Not many (future) connector
developers could join this discussion. It is better to make it as clear as
possible at the beginning than spend more time explaining multiple times.
Just my two cents.

Best regards,
Jing

On Thu, Oct 13, 2022 at 2:02 PM Ryan Skraba 
wrote:


+1 non-binding!  I've been following (and generally agreeing) with the
thread -- it's a perfectly reasonable way to start, and I'm sure we can
adjust the process if it turns out to be unsuitable or unexpected as the
connectors evolve in their external repositories.

On Thu, Oct 13, 2022 at 12:37 PM Thomas Weise  wrote:


+1 (binding) for the vote and thanks for the explanation

On Thu, Oct 13, 2022 at 5:58 AM Chesnay Schepler 
wrote:


@Thomas:
Version-specific modules that either contain a connector or shims to
support that Flink version.
Alternatively, since the addition of such code (usually) goes beyond a
patch release you'd create a new minor version and could have that only
support the later version.

On 13/10/2022 02:05, Thomas Weise wrote:

"Branches are not specific to a Flink version. (i.e., no v3.2-1.15)"

Sorry for the late question. I could not find in the discussion

thread

how

a connector can make use of features of the latest Flink version that

were

not present in the previous Flink version, when branches cannot be

Flink

version specific?

Thanks,
Thomas

On Wed, Oct 12, 2022 at 4:09 PM Ferenc Csaky


wrote:


+1 from my side (non-binding)

Best,
F


--- Original Message ---
On Wednesday, October 12th, 2022 at 15:47, Martijn Visser <
martijnvis...@apache.org> wrote:



+1 (binding), I am indeed assuming that Chesnay meant the last two

minor

versions as supported.

Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer

dannycran...@apache.org

Thanks for the concise summary Chesnay.

+1 from me (binding)

Just one clarification, for "3.1) The Flink versions supported by

the

project (last 2 major Flink versions) must be supported.". Do we

actually

mean major here, as in Flink 1.x.x and 2.x.x? Right now we would

only

support Flink 1.15.x and not 1.14.x? I would be inclined to

support

the

latest 2 minor Flink versions (major.minor.patch) given that we

only

have 1

active major Flink version.

Danny

On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler

ches...@apache.org

wrote:


Since the discussion
(

https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)

has

stalled a bit but we need a conclusion to move forward I'm

opening

a

vote.

Proposal summary:

1) Branch model
1.1) The default branch is called "main" and used for the next

major

iteration.
1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
1.3) Branches are not specific to a Flink version. (i.e., no

v3.2-1.15)

2) Versioning
2.1) Source releases: major.minor.patch
2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
(This may imply releasing the exact same connector jar multiple

times

under different versions)

3) Flink compatibility
3.1) The Flink versions supported by the project (last 2 major

Flink

versions) must be supported.
3.2) How this is achived is left to the connector, as long as it
conforms to the rest of the proposal.

4) Support
4.1) The last 2 major connector releases are supported with only

the

latter receiving additional features, with the following

exceptions:

4.1.a) If the older major connector version does not support any
currently supported Flink version, then it is no longer

supported.

4.1.b) If 

Re: [VOTE] Reverting sink metric name changes made in 1.15

[Chesnay Schepler]

Do we have a PoC that achieves this without re-introducing the bug where 
the numRecordsOut was simply wrong because it counted both records 
written to the external system and the downstream committer?

It's gonna be quite the dirty hack I assume.

On 13/10/2022 19:24, Qingsheng Ren wrote:

Hi devs,

I'd like to start a vote about reverting sink metric name changes made in
1.15 considering compatibility issues. These metrics include:

- numRecordsSend -> numRecordsOut
- numRecordsSendPerSecond -> numRecordsOutPerSecond
- numBytesSend -> numBytesOut
- numBytesSendPerSecond -> numBytesOutPerSecond
- numRecordsSendError -> numRecordsOutError

which reflect the output of the sink to the external system. "send" metric
series will be kept with the same value as "out" metric series. This change
will be applied to 1.15 and 1.16. More details could be found in the
discussion thread [1].

The vote will open for at least 72 hours.

Looking forward to your feedback!

[1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv

Best,
Qingsheng

[jira] [Created] (FLINK-29633) Operator doesn't pass initialSavepointPath as fromSavepoint argument

[Yaroslav Tkachenko (Jira)]

Yaroslav Tkachenko created FLINK-29633:
--

 Summary: Operator doesn't pass initialSavepointPath as 
fromSavepoint argument
 Key: FLINK-29633
 URL: https://issues.apache.org/jira/browse/FLINK-29633
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.2.0
Reporter: Yaroslav Tkachenko


The Kubernetes Operator doesn't pass *initialSavepointPath* from the JobSpec as 
a *--fromSavepoint* argument to the JobManager. The operator does update the 
configuration, but in the standalone mode, Flink actually [overrides 
that|https://github.com/apache/flink/blob/012dc6a9b800bae0cfa5250d38de992ccbabc015/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterEntryPoint.java#L57-L63]
 based on the command-line arguments. 

*CmdStandaloneJobManagerDecorator* should be updated to include *fromSavepoint.*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

[Qingsheng Ren]

Hi devs and users,

It looks like we are getting an initial consensus in the discussion so I
started a voting thread [1] just now. Looking forward to your feedback!

[1] https://lists.apache.org/thread/ozlf82mkm6ndx2n1vdgq532h156p4lt6

Best,
Qingsheng


On Thu, Oct 13, 2022 at 10:41 PM Jing Ge  wrote:

> Hi Qingsheng,
>
> Thanks for the clarification. +1, I like the idea. Pointing both numXXXOut
> and numXXXSend to the same external data transfer metric does not really
> break the new SinkV2 design, since there was no requirement to monitor the
> internal traffic. So, I think both developer and user can live with it. It
> might not be the perfect solution but is indeed the currently best
> trade-off solution after considering the backward compatibility.  I would
> suggest firing a follow-up ticket after the PR to take care of the new
> metric for the internal traffic in the future.
>
> Best regards,
> Jing
>
>
> On Thu, Oct 13, 2022 at 3:08 PM Qingsheng Ren  wrote:
>
>> Hi Jing,
>>
>> Thanks for the reply!
>>
>> Let me rephrase my proposal: we’d like to use numXXXOut registered on
>> SinkWriterOperator to reflect the traffic to the external system for
>> compatibility with old versions before 1.15, and make numXXXSend have the
>> same value as numXXXOut for compatibility within 1.15. That means both
>> numXXXOut and numXXXSend are used for external data transfers, which end
>> users care more about. As for the internal traffic within the sink, we
>> could name a new metric for it because this is a _new_ feature in the _new_
>> sink, and end users usually don’t pay attention to internal implementation.
>> The name of the new metric could be discussed later after 1.16 release.
>>
>> > but it might end up with monitoring unexpected metrics, which is even
>> worse for users, i.e. I didn't change anything, but something has been
>> broken since the last update.
>>
>> Yeah this is exactly what we are trying to fix with this proposal. I
>> believe users are more concerned with the output to the external system
>> than the internal data delivery in the sink, so I think we’ll have more
>> cases reporting like “I set up a panel on numRecordsOut in sink to monitor
>> the output of the job, but after upgrading to 1.15 this value is extremely
>> low and I didn’t change anything” if we stick to the current situation. I
>> think only a few end users care about the number of committables sending to
>> downstream as most of them don’t care how the sink works.
>>
>> We do need a re-design to fully distinguish the internal and external
>> traffic on metrics, not only in sink but in all operators as it’s quite
>> common for operators to make IO. This needs time to design, discuss, adjust
>> and vote, but considering this is blocking 1.16, maybe it’s better to
>> rescue the compatibility for now, and leave the huge reconstruction to
>> future versions (maybe 2.0).
>>
>> Best,
>> Qingsheng
>>
>> On Wed, Oct 12, 2022 at 7:21 PM Jing Ge  wrote:
>>
>>> Hi Qingsheng,
>>>
>>> Just want to make sure we are on the same page. Are you suggesting
>>> switching the naming between "numXXXSend" and "numXXXOut" or reverting all
>>> the changes we did with FLINK-26126 and FLINK-26492?
>>>
>>> For the naming switch, please pay attention that the behaviour has been
>>> changed since we introduced SinkV2[1]. So, please be aware of different
>>> numbers(behaviour change) even with the same metrics name. Sticking with
>>> the old name with the new behaviour (very bad idea, IMHO) might seem like
>>> saving the effort in the first place, but it might end up with monitoring
>>> unexpected metrics, which is even worse for users, i.e. I didn't change
>>> anything, but something has been broken since the last update.
>>>
>>> For reverting, I am not sure how to fix the issue mentioned in
>>> FLINK-26126 after reverting all changes. Like Chesnay has already pointed
>>> out, with SinkV2 we have two different output lines - one with the external
>>> system and the other with the downstream operator. In this case,
>>> "numXXXSend" is rather a new metric than a replacement of "numXXXOut". The
>>> "numXXXOut" metric can still be used, depending on what the user wants to
>>> monitor.
>>>
>>>
>>> Best regards,
>>> Jing
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/51fc20db30d001a95de95b3b9993eeb06f558f6c/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SinkWriterMetricGroup.java#L48
>>>
>>>
>>> On Wed, Oct 12, 2022 at 12:48 PM Qingsheng Ren  wrote:
>>>
 As a supplement, considering it could be a big reconstruction
 redefining internal and external traffic and touching metric names in
 almost all operators, this requires a lot of discussions and we might
 do it finally in Flink 2.0. I think compatibility is a bigger blocker
 in front of us, as the output of sink is a metric that users care a
 lot about.

 Thanks,
 Qingsheng

 On Wed, Oct 12, 2022 at 6:20 PM Qingsheng Ren  wrote:

[VOTE] Reverting sink metric name changes made in 1.15

[Qingsheng Ren]

Hi devs,

I'd like to start a vote about reverting sink metric name changes made in
1.15 considering compatibility issues. These metrics include:

- numRecordsSend -> numRecordsOut
- numRecordsSendPerSecond -> numRecordsOutPerSecond
- numBytesSend -> numBytesOut
- numBytesSendPerSecond -> numBytesOutPerSecond
- numRecordsSendError -> numRecordsOutError

which reflect the output of the sink to the external system. "send" metric
series will be kept with the same value as "out" metric series. This change
will be applied to 1.15 and 1.16. More details could be found in the
discussion thread [1].

The vote will open for at least 72 hours.

Looking forward to your feedback!

[1] https://lists.apache.org/thread/vxhty3q97s7pw2zn0jhkyd6sxwwodzbv

Best,
Qingsheng

[jira] [Created] (FLINK-29632) Support nodeSelector in helm template for flink operator deployment

[Jeesmon Jacob (Jira)]

Jeesmon Jacob created FLINK-29632:
-

 Summary: Support nodeSelector in helm template for flink operator 
deployment
 Key: FLINK-29632
 URL: https://issues.apache.org/jira/browse/FLINK-29632
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Jeesmon Jacob


Currently helm chart of flink-kubernetes-operator doesn't allow adding 
nodeSelector to operator deployment. There are cases where we want to schedule 
operator to specific nodes and it will be great if helm chart can support it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29631) [CVE-2022-42003] flink-shaded-jackson

[Sergio Sainz (Jira)]

Sergio Sainz created FLINK-29631:


 Summary: [CVE-2022-42003] flink-shaded-jackson
 Key: FLINK-29631
 URL: https://issues.apache.org/jira/browse/FLINK-29631
 Project: Flink
  Issue Type: Bug
  Components: BuildSystem / Shaded
Affects Versions: shaded-16.0
Reporter: Sergio Sainz


flink-shaded-jackson vulnerable to 7.5 (high) 
[https://nvd.nist.gov/vuln/detail/CVE-2022-42003]

 

Ref:

[https://nvd.nist.gov/vuln/detail/CVE-2022-42003]

[https://repo1.maven.org/maven2/org/apache/flink/flink-shaded/16.0/flink-shaded-16.0.pom]

[https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-jackson-parent/2.13.4-16.0]

[https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-jackson/2.13.4-16.0/flink-shaded-jackson-2.13.4-16.0.pom]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

[Jing Ge]

Hi Qingsheng,

Thanks for the clarification. +1, I like the idea. Pointing both numXXXOut
and numXXXSend to the same external data transfer metric does not really
break the new SinkV2 design, since there was no requirement to monitor the
internal traffic. So, I think both developer and user can live with it. It
might not be the perfect solution but is indeed the currently best
trade-off solution after considering the backward compatibility.  I would
suggest firing a follow-up ticket after the PR to take care of the new
metric for the internal traffic in the future.

Best regards,
Jing


On Thu, Oct 13, 2022 at 3:08 PM Qingsheng Ren  wrote:

> Hi Jing,
>
> Thanks for the reply!
>
> Let me rephrase my proposal: we’d like to use numXXXOut registered on
> SinkWriterOperator to reflect the traffic to the external system for
> compatibility with old versions before 1.15, and make numXXXSend have the
> same value as numXXXOut for compatibility within 1.15. That means both
> numXXXOut and numXXXSend are used for external data transfers, which end
> users care more about. As for the internal traffic within the sink, we
> could name a new metric for it because this is a _new_ feature in the _new_
> sink, and end users usually don’t pay attention to internal implementation.
> The name of the new metric could be discussed later after 1.16 release.
>
> > but it might end up with monitoring unexpected metrics, which is even
> worse for users, i.e. I didn't change anything, but something has been
> broken since the last update.
>
> Yeah this is exactly what we are trying to fix with this proposal. I
> believe users are more concerned with the output to the external system
> than the internal data delivery in the sink, so I think we’ll have more
> cases reporting like “I set up a panel on numRecordsOut in sink to monitor
> the output of the job, but after upgrading to 1.15 this value is extremely
> low and I didn’t change anything” if we stick to the current situation. I
> think only a few end users care about the number of committables sending to
> downstream as most of them don’t care how the sink works.
>
> We do need a re-design to fully distinguish the internal and external
> traffic on metrics, not only in sink but in all operators as it’s quite
> common for operators to make IO. This needs time to design, discuss, adjust
> and vote, but considering this is blocking 1.16, maybe it’s better to
> rescue the compatibility for now, and leave the huge reconstruction to
> future versions (maybe 2.0).
>
> Best,
> Qingsheng
>
> On Wed, Oct 12, 2022 at 7:21 PM Jing Ge  wrote:
>
>> Hi Qingsheng,
>>
>> Just want to make sure we are on the same page. Are you suggesting
>> switching the naming between "numXXXSend" and "numXXXOut" or reverting all
>> the changes we did with FLINK-26126 and FLINK-26492?
>>
>> For the naming switch, please pay attention that the behaviour has been
>> changed since we introduced SinkV2[1]. So, please be aware of different
>> numbers(behaviour change) even with the same metrics name. Sticking with
>> the old name with the new behaviour (very bad idea, IMHO) might seem like
>> saving the effort in the first place, but it might end up with monitoring
>> unexpected metrics, which is even worse for users, i.e. I didn't change
>> anything, but something has been broken since the last update.
>>
>> For reverting, I am not sure how to fix the issue mentioned in
>> FLINK-26126 after reverting all changes. Like Chesnay has already pointed
>> out, with SinkV2 we have two different output lines - one with the external
>> system and the other with the downstream operator. In this case,
>> "numXXXSend" is rather a new metric than a replacement of "numXXXOut". The
>> "numXXXOut" metric can still be used, depending on what the user wants to
>> monitor.
>>
>>
>> Best regards,
>> Jing
>>
>> [1]
>> https://github.com/apache/flink/blob/51fc20db30d001a95de95b3b9993eeb06f558f6c/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SinkWriterMetricGroup.java#L48
>>
>>
>> On Wed, Oct 12, 2022 at 12:48 PM Qingsheng Ren  wrote:
>>
>>> As a supplement, considering it could be a big reconstruction
>>> redefining internal and external traffic and touching metric names in
>>> almost all operators, this requires a lot of discussions and we might
>>> do it finally in Flink 2.0. I think compatibility is a bigger blocker
>>> in front of us, as the output of sink is a metric that users care a
>>> lot about.
>>>
>>> Thanks,
>>> Qingsheng
>>>
>>> On Wed, Oct 12, 2022 at 6:20 PM Qingsheng Ren  wrote:
>>> >
>>> > Thanks Chesnay for the reply. +1 for making a unified and clearer
>>> > metric definition distinguishing internal and external data transfers.
>>> > As you described, having IO in operators is quite common such as
>>> > dimension tables in Table/SQL API. This definitely deserves a FLIP and
>>> > an overall design.
>>> >
>>> > However I think it's necessary to change the metric back to
>>> > 

Re: [VOTE] Externalized connector release details​

[Jing Ge]

+1 and I would suggest giving a concrete example to explain 4) support. It
is still not quite easy to understand the text. Not many (future) connector
developers could join this discussion. It is better to make it as clear as
possible at the beginning than spend more time explaining multiple times.
Just my two cents.

Best regards,
Jing

On Thu, Oct 13, 2022 at 2:02 PM Ryan Skraba 
wrote:

> +1 non-binding!  I've been following (and generally agreeing) with the
> thread -- it's a perfectly reasonable way to start, and I'm sure we can
> adjust the process if it turns out to be unsuitable or unexpected as the
> connectors evolve in their external repositories.
>
> On Thu, Oct 13, 2022 at 12:37 PM Thomas Weise  wrote:
>
> > +1 (binding) for the vote and thanks for the explanation
> >
> > On Thu, Oct 13, 2022 at 5:58 AM Chesnay Schepler 
> > wrote:
> >
> > > @Thomas:
> > > Version-specific modules that either contain a connector or shims to
> > > support that Flink version.
> > > Alternatively, since the addition of such code (usually) goes beyond a
> > > patch release you'd create a new minor version and could have that only
> > > support the later version.
> > >
> > > On 13/10/2022 02:05, Thomas Weise wrote:
> > > > "Branches are not specific to a Flink version. (i.e., no v3.2-1.15)"
> > > >
> > > > Sorry for the late question. I could not find in the discussion
> thread
> > > how
> > > > a connector can make use of features of the latest Flink version that
> > > were
> > > > not present in the previous Flink version, when branches cannot be
> > Flink
> > > > version specific?
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > > On Wed, Oct 12, 2022 at 4:09 PM Ferenc Csaky
> >  > > >
> > > > wrote:
> > > >
> > > >> +1 from my side (non-binding)
> > > >>
> > > >> Best,
> > > >> F
> > > >>
> > > >>
> > > >> --- Original Message ---
> > > >> On Wednesday, October 12th, 2022 at 15:47, Martijn Visser <
> > > >> martijnvis...@apache.org> wrote:
> > > >>
> > > >>
> > > >>>
> > > >>> +1 (binding), I am indeed assuming that Chesnay meant the last two
> > > minor
> > > >>> versions as supported.
> > > >>>
> > > >>> Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer
> > > >> dannycran...@apache.org
> > >  Thanks for the concise summary Chesnay.
> > > 
> > >  +1 from me (binding)
> > > 
> > >  Just one clarification, for "3.1) The Flink versions supported by
> > the
> > >  project (last 2 major Flink versions) must be supported.". Do we
> > > >> actually
> > >  mean major here, as in Flink 1.x.x and 2.x.x? Right now we would
> > only
> > >  support Flink 1.15.x and not 1.14.x? I would be inclined to
> support
> > > the
> > >  latest 2 minor Flink versions (major.minor.patch) given that we
> only
> > > >> have 1
> > >  active major Flink version.
> > > 
> > >  Danny
> > > 
> > >  On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler
> ches...@apache.org
> > >  wrote:
> > > 
> > > > Since the discussion
> > > > (
> https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)
> > > >> has
> > > > stalled a bit but we need a conclusion to move forward I'm
> opening
> > a
> > > > vote.
> > > >
> > > > Proposal summary:
> > > >
> > > > 1) Branch model
> > > > 1.1) The default branch is called "main" and used for the next
> > major
> > > > iteration.
> > > > 1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
> > > > 1.3) Branches are not specific to a Flink version. (i.e., no
> > > >> v3.2-1.15)
> > > > 2) Versioning
> > > > 2.1) Source releases: major.minor.patch
> > > > 2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
> > > > (This may imply releasing the exact same connector jar multiple
> > times
> > > > under different versions)
> > > >
> > > > 3) Flink compatibility
> > > > 3.1) The Flink versions supported by the project (last 2 major
> > Flink
> > > > versions) must be supported.
> > > > 3.2) How this is achived is left to the connector, as long as it
> > > > conforms to the rest of the proposal.
> > > >
> > > > 4) Support
> > > > 4.1) The last 2 major connector releases are supported with only
> > the
> > > > latter receiving additional features, with the following
> > exceptions:
> > > > 4.1.a) If the older major connector version does not support any
> > > > currently supported Flink version, then it is no longer
> supported.
> > > > 4.1.b) If the last 2 major versions do not cover all supported
> > Flink
> > > > versions, then the latest connector version that supports the
> older
> > > > Flink version /additionally /gets patch support.
> > > > 4.2) For a given major connector version only the latest minor
> > > >> version
> > > > is supported.
> > > > (This means if 1.1.x is released there will be no more 1.0.x
> > release)
> > > >
> > > > I'd like to clarify that these 

Re: State of the Rescale API

[Maximilian Michels]

Thank you everyone for the clarification. It looks like the Rescale API
will require some more work before it can be fully taken advantage of. I'm
wondering whether in addition to the Rescale API we want to provide a means
to do a rescale without an explicit API call but as part of the job
submission.

Clearly, the Rescale API + the Adaptive Scheduler will provide a better
"rescale experience" in case of a running job. But some users have
implemented autoscaling in a way that will update the existing job
deployment in k8s to trigger a redeploy of the job with an updated
configuration and job graph parallelisms. The problem is that we only allow
setting the default parallelism so far.

Would something like along the lines of
https://issues.apache.org/jira/browse/FLINK-29501 be conceivable?

-Max

On Wed, Oct 12, 2022 at 5:58 AM Jiangang Liu 
wrote:

> Thanks for the attention to the rescale api. Dynamic resource adjust is
> useful for streaming jobs since the throughput can change in different
> time. The rescale api is a lightweight way to change the job's parallelism.
> This is importance for some jobs, for example, the jobs are in activities
> or related to money which can not be delayed.
> In our production scenario，we have supported a simple rescale api which
> may be not perfect. By this chance, I suggest to support the rescale api in
> adaptive scheduler for auto-scaling.
>
> Chesnay Schepler  于2022年10月11日周二 20:36写道：
>
>> The AdaptiveScheduler is not limited to reactive mode. There are no
>> deployment limitations for the scheduler itself.
>> In a nutshell, all that reactive mode does is crank the target
>> parallelism to infinity, when usually it is the parallelism the user has
>> set in the job/configuration.
>>
>> I think it would be fine if a new/revised rescale API were only
>> available in the Adaptive Scheduler (without reactive mode!) for starters.
>> We'd require way more stuff to make this useful for batch workloads.
>>
>> On 10/10/2022 16:47, Maximilian Michels wrote:
>> > Hey Gyula,
>> >
>> > Is the Adaptive Scheduler limited to the Reactive mode? I agree that if
>> we
>> > move forward with the Adaptive Scheduler solution it should support all
>> > deployment scenarios.
>> >
>> > Thanks,
>> > -Max
>> >
>> > On Sun, Oct 9, 2022 at 6:10 AM Gyula Fóra  wrote:
>> >
>> >> Hi!
>> >>
>> >> I think we have to make sure that the Rescale API will work also
>> without
>> >> the adaptive scheduler (for instance when we are running Flink with the
>> >> Kubernetes Native Integration or in other cases where the adaptive
>> >> scheduler is not supported).
>> >>
>> >> What do you think?
>> >>
>> >> Cheers
>> >> Gyula
>> >>
>> >>
>> >>
>> >> On Fri, Oct 7, 2022 at 3:50 PM Maximilian Michels 
>> wrote:
>> >>
>> >>> We've been looking into ways to support programmatic rescaling of job
>> >>> vertices. This feature is typically required for any type of Flink
>> >>> autoscaler which does not merely set the default parallelism but
>> adjusts
>> >>> the parallelisms on a JobVertex level.
>> >>>
>> >>> We've had an initial discussion here:
>> >>> https://issues.apache.org/jira/browse/FLINK-29501 where Chesnay
>> suggested
>> >>> to use the infamous "rescaling" API:
>> >>>
>> >>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-rescaling
>> >>> This API is disabled as of
>> >>> https://issues.apache.org/jira/browse/FLINK-12312
>> >>> .
>> >>>
>> >>> Since there is the Adaptive Scheduler in Flink now, it may be
>> feasible to
>> >>> re-enable the API (at least for streaming jobs) and allow overriding
>> the
>> >>> parallelism of job vertices in addition to the default parallelism.
>> >>>
>> >>> Any thoughts?
>> >>>
>> >>> -Max
>> >>>
>>
>>

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

[Matthias Pohl]

+1 Agree, good idea!

On Thu, Oct 13, 2022 at 2:52 PM Jing Ge  wrote:

> +1 good idea!
>
> On Thu, Oct 13, 2022 at 2:42 PM Yang Wang  wrote:
>
> > +1 for increasing the visibility of flink-kubernetes-operator.
> >
> > Best,
> > Yang
> >
> > Thomas Weise  于2022年10月13日周四 07:49写道：
> >
> > > +1
> > >
> > >
> > > On Wed, Oct 12, 2022 at 5:03 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > +1 from my end to include the operator in the related Kubernetes
> > sections
> > > > of the Flink docs
> > > >
> > > > On Wed, Oct 12, 2022 at 5:31 PM Chesnay Schepler  >
> > > > wrote:
> > > >
> > > > > I don't see a reason for why we shouldn't at least mention the
> > operator
> > > > > in the kubernetes docs.
> > > > >
> > > > > On 12/10/2022 16:25, Gyula Fóra wrote:
> > > > > > Hi Devs!
> > > > > >
> > > > > > I would like to start a discussion about referencing the Flink
> > > > Kubernetes
> > > > > > Operator directly from the Flink Kubernetes deployment
> > documentation.
> > > > > >
> > > > > > Currently the Flink deployment/resource provider docs provide
> some
> > > > > > information for the Standalone and Native Kubernetes integration
> > > > without
> > > > > > any reference to the operator.
> > > > > >
> > > > > > I think we reached a point with the operator where we should
> > provide
> > > a
> > > > > bit
> > > > > > more visibility and value to the users by directly proposing to
> use
> > > the
> > > > > > operator when considering Flink on Kubernetes. We should
> definitely
> > > > keep
> > > > > > the current docs but make the point that for most users the
> easiest
> > > way
> > > > > to
> > > > > > use Flink on Kubernetes is probably through the operator (where
> > they
> > > > can
> > > > > > now benefit from both standalone and native integration under the
> > > > hood).
> > > > > > This should help us avoid cases where a new user completely
> misses
> > > the
> > > > > > existence of the operator when starting out based on the Flink
> > docs.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Gyula
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Drop Gelly

[Maximilian Michels]

+1

On Thu, Oct 13, 2022 at 12:00 PM Konstantin Knauf  wrote:

> +1
>
> Am Do., 13. Okt. 2022 um 10:56 Uhr schrieb Niels Basjes :
>
> > +1
> >
> > On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser <
> martijnvis...@apache.org>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to open a vote for dropping Gelly, which was discussed a
> > long
> > > time ago but never put to a vote [1].
> > >
> > > Voting will be open for at least 72 hours.
> > >
> > > Best regards,
> > >
> > > Martijn
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
> > >
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Maximilian Michels]

Congratulations Danny! Well deserved :)

-Max

On Thu, Oct 13, 2022 at 2:40 PM Yang Wang  wrote:

> Congratulations Danny!
>
> Best,
> Yang
>
> Hang Ruan  于2022年10月13日周四 10:58写道：
>
> > Congratulations Danny!
> >
> > Best,
> > Hang
> >
> > Yun Gao  于2022年10月13日周四 10:56写道：
> >
> > > Congratulations Danny!
> > > Best,
> > > Yun Gao
> > > --
> > > From:yuxia 
> > > Send Time:2022 Oct. 12 (Wed.) 09:49
> > > To:dev 
> > > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > Congratulations Danny!
> > > Best regards,
> > > Yuxia
> > > - 原始邮件 -
> > > 发件人: "Xingbo Huang" 
> > > 收件人: "dev" 
> > > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > > Congratulations Danny!
> > > Best,
> > > Xingbo
> > > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > > Congratulations, Danny
> > > >
> > > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee 
> > wrote:
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Congxian Qiu  于2022年10月11日周二 19:42写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > >
> > > > > > Best,
> > > > > > Congxian
> > > > > >
> > > > > >
> > > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > > >
> > > > > > > Congratulations Danny!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Leonard
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

[Qingsheng Ren]

Hi Jing,

Thanks for the reply!

Let me rephrase my proposal: we’d like to use numXXXOut registered on
SinkWriterOperator to reflect the traffic to the external system for
compatibility with old versions before 1.15, and make numXXXSend have the
same value as numXXXOut for compatibility within 1.15. That means both
numXXXOut and numXXXSend are used for external data transfers, which end
users care more about. As for the internal traffic within the sink, we
could name a new metric for it because this is a _new_ feature in the _new_
sink, and end users usually don’t pay attention to internal implementation.
The name of the new metric could be discussed later after 1.16 release.

> but it might end up with monitoring unexpected metrics, which is even
worse for users, i.e. I didn't change anything, but something has been
broken since the last update.

Yeah this is exactly what we are trying to fix with this proposal. I
believe users are more concerned with the output to the external system
than the internal data delivery in the sink, so I think we’ll have more
cases reporting like “I set up a panel on numRecordsOut in sink to monitor
the output of the job, but after upgrading to 1.15 this value is extremely
low and I didn’t change anything” if we stick to the current situation. I
think only a few end users care about the number of committables sending to
downstream as most of them don’t care how the sink works.

We do need a re-design to fully distinguish the internal and external
traffic on metrics, not only in sink but in all operators as it’s quite
common for operators to make IO. This needs time to design, discuss, adjust
and vote, but considering this is blocking 1.16, maybe it’s better to
rescue the compatibility for now, and leave the huge reconstruction to
future versions (maybe 2.0).

Best,
Qingsheng

On Wed, Oct 12, 2022 at 7:21 PM Jing Ge  wrote:

> Hi Qingsheng,
>
> Just want to make sure we are on the same page. Are you suggesting
> switching the naming between "numXXXSend" and "numXXXOut" or reverting all
> the changes we did with FLINK-26126 and FLINK-26492?
>
> For the naming switch, please pay attention that the behaviour has been
> changed since we introduced SinkV2[1]. So, please be aware of different
> numbers(behaviour change) even with the same metrics name. Sticking with
> the old name with the new behaviour (very bad idea, IMHO) might seem like
> saving the effort in the first place, but it might end up with monitoring
> unexpected metrics, which is even worse for users, i.e. I didn't change
> anything, but something has been broken since the last update.
>
> For reverting, I am not sure how to fix the issue mentioned in FLINK-26126
> after reverting all changes. Like Chesnay has already pointed out, with
> SinkV2 we have two different output lines - one with the external system
> and the other with the downstream operator. In this case, "numXXXSend" is
> rather a new metric than a replacement of "numXXXOut". The "numXXXOut"
> metric can still be used, depending on what the user wants to monitor.
>
>
> Best regards,
> Jing
>
> [1]
> https://github.com/apache/flink/blob/51fc20db30d001a95de95b3b9993eeb06f558f6c/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SinkWriterMetricGroup.java#L48
>
>
> On Wed, Oct 12, 2022 at 12:48 PM Qingsheng Ren  wrote:
>
>> As a supplement, considering it could be a big reconstruction
>> redefining internal and external traffic and touching metric names in
>> almost all operators, this requires a lot of discussions and we might
>> do it finally in Flink 2.0. I think compatibility is a bigger blocker
>> in front of us, as the output of sink is a metric that users care a
>> lot about.
>>
>> Thanks,
>> Qingsheng
>>
>> On Wed, Oct 12, 2022 at 6:20 PM Qingsheng Ren  wrote:
>> >
>> > Thanks Chesnay for the reply. +1 for making a unified and clearer
>> > metric definition distinguishing internal and external data transfers.
>> > As you described, having IO in operators is quite common such as
>> > dimension tables in Table/SQL API. This definitely deserves a FLIP and
>> > an overall design.
>> >
>> > However I think it's necessary to change the metric back to
>> > numRecordsOut instead of sticking with numRecordsSend in 1.15 and
>> > 1.16. The most important argument is for compatibility as I mentioned
>> > in my previous email, otherwise all users have to modify their configs
>> > of metric systems after upgrading to Flink 1.15+, and all custom
>> > connectors have to change their implementations to migrate to the new
>> > metric name. I believe other ones participating and approving this
>> > proposal share the same concern about compatibility too. Also
>> > considering this issue is blocking the release of 1.16, maybe we could
>> > fix this asap, and as for defining a new metric for internal data
>> > transfers we can have an in-depth discussion later. WDYT?
>> >
>> > Best,
>> > Qingsheng
>> >
>> > On Tue, Oct 11, 2022 at 6:06 PM 

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

[Jing Ge]

+1 good idea!

On Thu, Oct 13, 2022 at 2:42 PM Yang Wang  wrote:

> +1 for increasing the visibility of flink-kubernetes-operator.
>
> Best,
> Yang
>
> Thomas Weise  于2022年10月13日周四 07:49写道：
>
> > +1
> >
> >
> > On Wed, Oct 12, 2022 at 5:03 PM Martijn Visser  >
> > wrote:
> >
> > > +1 from my end to include the operator in the related Kubernetes
> sections
> > > of the Flink docs
> > >
> > > On Wed, Oct 12, 2022 at 5:31 PM Chesnay Schepler 
> > > wrote:
> > >
> > > > I don't see a reason for why we shouldn't at least mention the
> operator
> > > > in the kubernetes docs.
> > > >
> > > > On 12/10/2022 16:25, Gyula Fóra wrote:
> > > > > Hi Devs!
> > > > >
> > > > > I would like to start a discussion about referencing the Flink
> > > Kubernetes
> > > > > Operator directly from the Flink Kubernetes deployment
> documentation.
> > > > >
> > > > > Currently the Flink deployment/resource provider docs provide some
> > > > > information for the Standalone and Native Kubernetes integration
> > > without
> > > > > any reference to the operator.
> > > > >
> > > > > I think we reached a point with the operator where we should
> provide
> > a
> > > > bit
> > > > > more visibility and value to the users by directly proposing to use
> > the
> > > > > operator when considering Flink on Kubernetes. We should definitely
> > > keep
> > > > > the current docs but make the point that for most users the easiest
> > way
> > > > to
> > > > > use Flink on Kubernetes is probably through the operator (where
> they
> > > can
> > > > > now benefit from both standalone and native integration under the
> > > hood).
> > > > > This should help us avoid cases where a new user completely misses
> > the
> > > > > existence of the operator when starting out based on the Flink
> docs.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Gyula
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Reference operator from Flink Kubernetes deployment docs

[Yang Wang]

+1 for increasing the visibility of flink-kubernetes-operator.

Best,
Yang

Thomas Weise  于2022年10月13日周四 07:49写道：

> +1
>
>
> On Wed, Oct 12, 2022 at 5:03 PM Martijn Visser 
> wrote:
>
> > +1 from my end to include the operator in the related Kubernetes sections
> > of the Flink docs
> >
> > On Wed, Oct 12, 2022 at 5:31 PM Chesnay Schepler 
> > wrote:
> >
> > > I don't see a reason for why we shouldn't at least mention the operator
> > > in the kubernetes docs.
> > >
> > > On 12/10/2022 16:25, Gyula Fóra wrote:
> > > > Hi Devs!
> > > >
> > > > I would like to start a discussion about referencing the Flink
> > Kubernetes
> > > > Operator directly from the Flink Kubernetes deployment documentation.
> > > >
> > > > Currently the Flink deployment/resource provider docs provide some
> > > > information for the Standalone and Native Kubernetes integration
> > without
> > > > any reference to the operator.
> > > >
> > > > I think we reached a point with the operator where we should provide
> a
> > > bit
> > > > more visibility and value to the users by directly proposing to use
> the
> > > > operator when considering Flink on Kubernetes. We should definitely
> > keep
> > > > the current docs but make the point that for most users the easiest
> way
> > > to
> > > > use Flink on Kubernetes is probably through the operator (where they
> > can
> > > > now benefit from both standalone and native integration under the
> > hood).
> > > > This should help us avoid cases where a new user completely misses
> the
> > > > existence of the operator when starting out based on the Flink docs.
> > > >
> > > > What do you think?
> > > >
> > > > Gyula
> > > >
> > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Yang Wang]

Congratulations Danny!

Best,
Yang

Hang Ruan  于2022年10月13日周四 10:58写道：

> Congratulations Danny!
>
> Best,
> Hang
>
> Yun Gao  于2022年10月13日周四 10:56写道：
>
> > Congratulations Danny!
> > Best,
> > Yun Gao
> > --
> > From:yuxia 
> > Send Time:2022 Oct. 12 (Wed.) 09:49
> > To:dev 
> > Subject:Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > Congratulations Danny!
> > Best regards,
> > Yuxia
> > - 原始邮件 -
> > 发件人: "Xingbo Huang" 
> > 收件人: "dev" 
> > 发送时间: 星期三, 2022年 10 月 12日 上午 9:44:22
> > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer
> > Congratulations Danny!
> > Best,
> > Xingbo
> > Sergey Nuyanzin  于2022年10月12日周三 01:26写道：
> > > Congratulations, Danny
> > >
> > > On Tue, Oct 11, 2022, 15:18 Lincoln Lee 
> wrote:
> > >
> > > > Congratulations Danny!
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Congxian Qiu  于2022年10月11日周二 19:42写道：
> > > >
> > > > > Congratulations Danny!
> > > > >
> > > > > Best,
> > > > > Congxian
> > > > >
> > > > >
> > > > > Leonard Xu  于2022年10月11日周二 18:03写道：
> > > > >
> > > > > > Congratulations Danny!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Leonard
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Remove HCatalog connector

[Samrat Deb]

+1 (non-binding)

Bests,
Samrat


On Thu, 13 Oct 2022 at 8:23 AM, yuxia  wrote:

> +1 (non-binding)
> Thanks for driving.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Hang Ruan" 
> 收件人: "dev" 
> 发送时间: 星期四, 2022年 10 月 13日 上午 10:16:46
> 主题: Re: [VOTE] Remove HCatalog connector
>
> +1,
>
> Best,
> Hang
>
> Jingsong Li  于2022年10月13日周四 10:09写道：
>
> > +1
> >
> > Thanks for driving.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Oct 13, 2022 at 3:46 AM Martijn Visser  >
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > Since no comments were made, I'm opening a vote to remove the HCatalog
> > > connector [1]
> > >
> > > The voting period will be open at least 72hrs.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > [1]
> > > https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84
> > > --
> > > Martijn
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> >
>

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

[Márton Balassi]

Hi Martjin,

After some more careful consideration I am in favor of dropping the Scala
API support in with Flink 2.0 given that we add Java 17 support earlier or
latest at the same time.

Best,
Marton

On Thu, Oct 13, 2022 at 12:01 PM Chesnay Schepler 
wrote:

> Support for records has not been investigated yet. We're still at the
> stage of getting things to run at all on Java 17.
>
> It _may_ be possible, it _may_ not be.
>
> On 13/10/2022 07:39, Salva Alcántara wrote:
>
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written "manually"
> or using Lombok) to Java records without breaking serialization? (assuming
> that those POJOs are used as immutable values, e.g., setters are never
> used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser 
> wrote:
>
>> Hi everyone,
>>
>> Thanks again for all your feedback. It's very much appreciated.
>>
>> My overall feeling is that people are not opposed to the FLIP. There is
>> demand for adding Java 17 support before dropping the Scala APIs. Given
>> that the proposal for actually dropping the Scala APIs would only happen
>> with a Flink 2.0 and Java 17 support would either happen in a new minor
>> version or a new major version (I haven't seen a FLIP or discussion being
>> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
>> would either be there earlier (in a new minor version) or at the same time
>> (with Flink 2.0) when the Scala APIs would be dropped.
>>
>> If there are no more discussion topics, I would move this FLIP to a vote
>> at the beginning of next week.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists 
>> wrote:
>>
>>> Hi Martijn
>>>
>>> I do not maintain a large production application based on Flink, so it
>>> would not be a problem for me to convert existing implementations to
>>> whatever API.
>>>
>>> I am working in the area of cultural heritage, which is mainly about the
>>> processing of structured (meta)-data (scientific libraries, archives and
>>> museums)
>>> My impression: People without much background/experience with Java
>>> implementations find it easier to get into the functional mindset as
>>> supported in Scala. That's why I think it would be very unfortunate if the
>>> use of Scala in Flink becomes more and more limited or neglected.
>>>
>>> I think using the Java API in Scala is a possible way also in my
>>> environment.
>>>
>>> In the last weeks I tried to port the examples from the "Flink Course"
>>> of Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>>> Scala courses), which are exclusively based on the native Scala API, to the
>>> Java API. This has worked without any problems as far as I can see. So far
>>> I haven't tried any examples based on the Table API or streaming workflows
>>> in batch mode (which would be important for our environment).
>>>
>>> My main trouble: So far I don't know enough about the limitations of
>>> using the Java API in a Scala implementation and what that means. My
>>> current understanding: the limitation is mainly in deriving the type
>>> information in generic APIs with Scala types. For me it would be very
>>> significant and helpful if there would be more information, descriptions
>>> and examples about this topic.
>>>
>>> So far unfortunately I had too little time to deal with a wrapper like
>>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>>> current alternative is probably going to be deprecated in the future (
>>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>>
>>> Günter
>>>
>>>
>>> On 04.10.22 13:58, Martijn Visser wrote:
>>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>>> support in a future version of Flink. Your feedback on this topic is very
>>> much appreciated.
>>>
>>> Regarding the large Scala codebase for Flink, a potential alternative
>>> could be to have a wrapper for all Java APIs that makes them available as
>>> Scala APIs. However, this still requires Scala maintainers and I don't
>>> think that we currently have those in our community. The easiest solution
>>> for them would be to use the Java APIs directly. Yes it would involve work,
>>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>>> there's still time for that :)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi 
>>> wrote:
>>>
 Hi 

[jira] [Created] (FLINK-29630) Junit 5.8.1 run unit test with temporary directory will occur Failed to delete temp directory.

[Aiden Gong (Jira)]

Aiden Gong created FLINK-29630:
--

 Summary: Junit 5.8.1 run unit test with temporary directory will 
occur Failed to delete temp directory.
 Key: FLINK-29630
 URL: https://issues.apache.org/jira/browse/FLINK-29630
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.3.0
Reporter: Aiden Gong
 Fix For: table-store-0.3.0


Junit 5.8.1 run unit test with temporary directory will occur Failed to delete 
temp directory.

My local :

windows 10

jdk1.8

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29629) FlameGraph is empty for Legacy Source Threads

[Peter Vary (Jira)]

Peter Vary created FLINK-29629:
--

 Summary: FlameGraph is empty for Legacy Source Threads
 Key: FLINK-29629
 URL: https://issues.apache.org/jira/browse/FLINK-29629
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Reporter: Peter Vary


Thread dump gets the stack trace for the {{Custom Source}} thread, but this 
thread is always in {{TIMED_WAITING}}:
{code}
"Source: Custom Source -> A random source (1/2)#0" ...
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
- parking to wait for  <0xea775750> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos()
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
at 
org.apache.flink.streaming.runtime.tasks.mailbox.TaskMailboxImpl.take()
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:335)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
[..]
{code}

The actual code is run in the {{Legacy Source Thread}}:
{code}
"Legacy Source Thread - Source: Custom Source -> A random source (1/2)#0" ...
   java.lang.Thread.State: RUNNABLE
{code}

This causes the WebUI FlameGraph to be empty of any useful data.

This is an example code to reproduce:
{code}
DataStream inputStream = env.addSource(new 
RandomRecordSource(recordSize));
inputStream = inputStream.map(new CounterMapper());
FlinkSink.forRowData(inputStream).tableLoader(loader).append();
{code}





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Externalized connector release details​

[Ryan Skraba]

+1 non-binding!  I've been following (and generally agreeing) with the
thread -- it's a perfectly reasonable way to start, and I'm sure we can
adjust the process if it turns out to be unsuitable or unexpected as the
connectors evolve in their external repositories.

On Thu, Oct 13, 2022 at 12:37 PM Thomas Weise  wrote:

> +1 (binding) for the vote and thanks for the explanation
>
> On Thu, Oct 13, 2022 at 5:58 AM Chesnay Schepler 
> wrote:
>
> > @Thomas:
> > Version-specific modules that either contain a connector or shims to
> > support that Flink version.
> > Alternatively, since the addition of such code (usually) goes beyond a
> > patch release you'd create a new minor version and could have that only
> > support the later version.
> >
> > On 13/10/2022 02:05, Thomas Weise wrote:
> > > "Branches are not specific to a Flink version. (i.e., no v3.2-1.15)"
> > >
> > > Sorry for the late question. I could not find in the discussion thread
> > how
> > > a connector can make use of features of the latest Flink version that
> > were
> > > not present in the previous Flink version, when branches cannot be
> Flink
> > > version specific?
> > >
> > > Thanks,
> > > Thomas
> > >
> > > On Wed, Oct 12, 2022 at 4:09 PM Ferenc Csaky
>  > >
> > > wrote:
> > >
> > >> +1 from my side (non-binding)
> > >>
> > >> Best,
> > >> F
> > >>
> > >>
> > >> --- Original Message ---
> > >> On Wednesday, October 12th, 2022 at 15:47, Martijn Visser <
> > >> martijnvis...@apache.org> wrote:
> > >>
> > >>
> > >>>
> > >>> +1 (binding), I am indeed assuming that Chesnay meant the last two
> > minor
> > >>> versions as supported.
> > >>>
> > >>> Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer
> > >> dannycran...@apache.org
> >  Thanks for the concise summary Chesnay.
> > 
> >  +1 from me (binding)
> > 
> >  Just one clarification, for "3.1) The Flink versions supported by
> the
> >  project (last 2 major Flink versions) must be supported.". Do we
> > >> actually
> >  mean major here, as in Flink 1.x.x and 2.x.x? Right now we would
> only
> >  support Flink 1.15.x and not 1.14.x? I would be inclined to support
> > the
> >  latest 2 minor Flink versions (major.minor.patch) given that we only
> > >> have 1
> >  active major Flink version.
> > 
> >  Danny
> > 
> >  On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler ches...@apache.org
> >  wrote:
> > 
> > > Since the discussion
> > > (https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)
> > >> has
> > > stalled a bit but we need a conclusion to move forward I'm opening
> a
> > > vote.
> > >
> > > Proposal summary:
> > >
> > > 1) Branch model
> > > 1.1) The default branch is called "main" and used for the next
> major
> > > iteration.
> > > 1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
> > > 1.3) Branches are not specific to a Flink version. (i.e., no
> > >> v3.2-1.15)
> > > 2) Versioning
> > > 2.1) Source releases: major.minor.patch
> > > 2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
> > > (This may imply releasing the exact same connector jar multiple
> times
> > > under different versions)
> > >
> > > 3) Flink compatibility
> > > 3.1) The Flink versions supported by the project (last 2 major
> Flink
> > > versions) must be supported.
> > > 3.2) How this is achived is left to the connector, as long as it
> > > conforms to the rest of the proposal.
> > >
> > > 4) Support
> > > 4.1) The last 2 major connector releases are supported with only
> the
> > > latter receiving additional features, with the following
> exceptions:
> > > 4.1.a) If the older major connector version does not support any
> > > currently supported Flink version, then it is no longer supported.
> > > 4.1.b) If the last 2 major versions do not cover all supported
> Flink
> > > versions, then the latest connector version that supports the older
> > > Flink version /additionally /gets patch support.
> > > 4.2) For a given major connector version only the latest minor
> > >> version
> > > is supported.
> > > (This means if 1.1.x is released there will be no more 1.0.x
> release)
> > >
> > > I'd like to clarify that these won't be set in stone for eternity.
> > > We should re-evaluate how well this model works over time and
> adjust
> > >> it
> > > accordingly, consistently across all connectors.
> > > I do believe that as is this strikes a good balance between
> > > maintainability for us and clarity to users.
> > >
> > > Voting schema:
> > >
> > > Consensus, committers have binding votes, open for at least 72
> hours.
> > >>> --
> > >>> Martijn
> > >>> https://twitter.com/MartijnVisser82
> > >>> https://github.com/MartijnVisser
> >
> >
> >
>

[jira] [Created] (FLINK-29628) Bump aws-java-sdk-s3 to 1.12.319

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29628:
--

 Summary: Bump aws-java-sdk-s3 to 1.12.319
 Key: FLINK-29628
 URL: https://issues.apache.org/jira/browse/FLINK-29628
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / FileSystem, FileSystems
Reporter: Martijn Visser
Assignee: Martijn Visser


As reported by Dependabot in https://github.com/apache/flink/pull/20285



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Externalized connector release details​

[Thomas Weise]

+1 (binding) for the vote and thanks for the explanation

On Thu, Oct 13, 2022 at 5:58 AM Chesnay Schepler  wrote:

> @Thomas:
> Version-specific modules that either contain a connector or shims to
> support that Flink version.
> Alternatively, since the addition of such code (usually) goes beyond a
> patch release you'd create a new minor version and could have that only
> support the later version.
>
> On 13/10/2022 02:05, Thomas Weise wrote:
> > "Branches are not specific to a Flink version. (i.e., no v3.2-1.15)"
> >
> > Sorry for the late question. I could not find in the discussion thread
> how
> > a connector can make use of features of the latest Flink version that
> were
> > not present in the previous Flink version, when branches cannot be Flink
> > version specific?
> >
> > Thanks,
> > Thomas
> >
> > On Wed, Oct 12, 2022 at 4:09 PM Ferenc Csaky  >
> > wrote:
> >
> >> +1 from my side (non-binding)
> >>
> >> Best,
> >> F
> >>
> >>
> >> --- Original Message ---
> >> On Wednesday, October 12th, 2022 at 15:47, Martijn Visser <
> >> martijnvis...@apache.org> wrote:
> >>
> >>
> >>>
> >>> +1 (binding), I am indeed assuming that Chesnay meant the last two
> minor
> >>> versions as supported.
> >>>
> >>> Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer
> >> dannycran...@apache.org
>  Thanks for the concise summary Chesnay.
> 
>  +1 from me (binding)
> 
>  Just one clarification, for "3.1) The Flink versions supported by the
>  project (last 2 major Flink versions) must be supported.". Do we
> >> actually
>  mean major here, as in Flink 1.x.x and 2.x.x? Right now we would only
>  support Flink 1.15.x and not 1.14.x? I would be inclined to support
> the
>  latest 2 minor Flink versions (major.minor.patch) given that we only
> >> have 1
>  active major Flink version.
> 
>  Danny
> 
>  On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler ches...@apache.org
>  wrote:
> 
> > Since the discussion
> > (https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)
> >> has
> > stalled a bit but we need a conclusion to move forward I'm opening a
> > vote.
> >
> > Proposal summary:
> >
> > 1) Branch model
> > 1.1) The default branch is called "main" and used for the next major
> > iteration.
> > 1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
> > 1.3) Branches are not specific to a Flink version. (i.e., no
> >> v3.2-1.15)
> > 2) Versioning
> > 2.1) Source releases: major.minor.patch
> > 2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
> > (This may imply releasing the exact same connector jar multiple times
> > under different versions)
> >
> > 3) Flink compatibility
> > 3.1) The Flink versions supported by the project (last 2 major Flink
> > versions) must be supported.
> > 3.2) How this is achived is left to the connector, as long as it
> > conforms to the rest of the proposal.
> >
> > 4) Support
> > 4.1) The last 2 major connector releases are supported with only the
> > latter receiving additional features, with the following exceptions:
> > 4.1.a) If the older major connector version does not support any
> > currently supported Flink version, then it is no longer supported.
> > 4.1.b) If the last 2 major versions do not cover all supported Flink
> > versions, then the latest connector version that supports the older
> > Flink version /additionally /gets patch support.
> > 4.2) For a given major connector version only the latest minor
> >> version
> > is supported.
> > (This means if 1.1.x is released there will be no more 1.0.x release)
> >
> > I'd like to clarify that these won't be set in stone for eternity.
> > We should re-evaluate how well this model works over time and adjust
> >> it
> > accordingly, consistently across all connectors.
> > I do believe that as is this strikes a good balance between
> > maintainability for us and clarity to users.
> >
> > Voting schema:
> >
> > Consensus, committers have binding votes, open for at least 72 hours.
> >>> --
> >>> Martijn
> >>> https://twitter.com/MartijnVisser82
> >>> https://github.com/MartijnVisser
>
>
>

[jira] [Created] (FLINK-29627) Sink - Duplicate key exception during recover more than 1 committable.

[Krzysztof Chmielewski (Jira)]

Krzysztof Chmielewski created FLINK-29627:
-

 Summary: Sink - Duplicate key exception during recover more than 1 
committable.
 Key: FLINK-29627
 URL: https://issues.apache.org/jira/browse/FLINK-29627
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.15.2, 1.16.0, 1.17.0, 1.16.1
Reporter: Krzysztof Chmielewski


Recovery more then one Committable  causes `IllegalStateException` and prevents 
cluster to start.

When we recover the `CheckpointCommittableManager` we deserialize 
SubtaskCommittableManager instances from recovery state and we put them into 
`Map>`. The key of this map is 
subtaskId of the recovered manager. However this will fail if we have to 
recover more than one committable. 

What w should do is to call `SubtaskCommittableManager::merge` if we already 
deserialzie manager for this subtaskId.


Stack Trace:
{code:java}
28603 [flink-akka.actor.default-dispatcher-8] INFO  
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Global 
Committer (1/1) 
(485dc57aca56235b9d1ab803c8c966ad_47d89856a1cf553f16e7063d953b7d42_0_1) 
switched from INITIALIZING to FAILED on 2ed5c848-d360-48ae-9a92-730b022c8a39 @ 
kubernetes.docker.internal (dataPort=-1).
java.lang.IllegalStateException: Duplicate key 0 (attempted merging values 
org.apache.flink.streaming.runtime.operators.sink.committables.SubtaskCommittableManager@631940ac
 and 
org.apache.flink.streaming.runtime.operators.sink.committables.SubtaskCommittableManager@7ff3bd7)
at 
java.util.stream.Collectors.duplicateKeyException(Collectors.java:133) ~[?:?]
at 
java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
 ~[?:?]
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) 
~[?:?]
at 
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) 
~[?:?]
at 
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) 
~[?:?]
at 
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) 
~[?:?]
at 
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at 
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at 
org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer$CheckpointSimpleVersionedSerializer.deserialize(CommittableCollectorSerializer.java:153)
 ~[classes/:?]
at 
org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer$CheckpointSimpleVersionedSerializer.deserialize(CommittableCollectorSerializer.java:124)
 ~[classes/:?]
at 
org.apache.flink.core.io.SimpleVersionedSerialization.readVersionAndDeserializeList(SimpleVersionedSerialization.java:148)
 ~[classes/:?]
at 
org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer.deserializeV2(CommittableCollectorSerializer.java:105)
 ~[classes/:?]
at 
org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer.deserialize(CommittableCollectorSerializer.java:82)
 ~[classes/:?]
at 
org.apache.flink.streaming.runtime.operators.sink.committables.CommittableCollectorSerializer.deserialize(CommittableCollectorSerializer.java:41)
 ~[classes/:?]
at 
org.apache.flink.core.io.SimpleVersionedSerialization.readVersionAndDeSerialize(SimpleVersionedSerialization.java:121)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.connector.sink2.GlobalCommitterSerializer.deserializeV2(GlobalCommitterSerializer.java:128)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.connector.sink2.GlobalCommitterSerializer.deserialize(GlobalCommitterSerializer.java:99)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.connector.sink2.GlobalCommitterSerializer.deserialize(GlobalCommitterSerializer.java:42)
 ~[classes/:?]
at 
org.apache.flink.core.io.SimpleVersionedSerialization.readVersionAndDeSerialize(SimpleVersionedSerialization.java:227)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.operators.util.SimpleVersionedListState$DeserializingIterator.next(SimpleVersionedListState.java:138)
 ~[classes/:?]
at java.lang.Iterable.forEach(Iterable.java:74) ~[?:?]
at 
org.apache.flink.streaming.api.connector.sink2.GlobalCommitterOperator.initializeState(GlobalCommitterOperator.java:133)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
 ~[classes/:?]
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)
 ~[classes/:?]
at 

Re: [VOTE] Remove HCatalog connector

[Chesnay Schepler]

+1

On 12/10/2022 21:45, Martijn Visser wrote:

Hi everyone,

Since no comments were made, I'm opening a vote to remove the HCatalog
connector [1]

The voting period will be open at least 72hrs.

Best regards,

Martijn

[1]
https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

[Chesnay Schepler]

Support for records has not been investigated yet. We're still at the 
stage of getting things to run at all on Java 17.


It _may_ be possible, it _may_ not be.

On 13/10/2022 07:39, Salva Alcántara wrote:

Hi Martijn,

Maybe a bit of an off-topic, but regarding Java 17 support, will it be 
possible to replace POJOs with Java records in existing applications?


In a project I maintain we use Lombok a lot, but with Java records we 
would probably stop using it (or significantly reduce its usage).


Will there be a way to promote existing POJOs (either written 
"manually" or using Lombok) to Java records without breaking 
serialization? (assuming that those POJOs are used as immutable 
values, e.g., setters are never used).


Regards,

Salva

On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser 
 wrote:


Hi everyone,

Thanks again for all your feedback. It's very much appreciated.

My overall feeling is that people are not opposed to the FLIP.
There is demand for adding Java 17 support before dropping the
Scala APIs. Given that the proposal for actually dropping the
Scala APIs would only happen with a Flink 2.0 and Java 17 support
would either happen in a new minor version or a new major version
(I haven't seen a FLIP or discussion being opened adding Java 17
support, only on deprecating Java 8), Java 17 support would either
be there earlier (in a new minor version) or at the same time
(with Flink 2.0) when the Scala APIs would be dropped.

If there are no more discussion topics, I would move this FLIP to
a vote at the beginning of next week.

Best regards,

Martijn

On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists
 wrote:

Hi Martijn

I do not maintain a large production application based on
Flink, so it would not be a problem for me to convert existing
implementations to whatever API.

I am working in the area of cultural heritage, which is mainly
about the processing of structured (meta)-data (scientific
libraries, archives and museums)
My impression: People without much background/experience with
Java implementations find it easier to get into the functional
mindset as supported in Scala. That's why I think it would be
very unfortunate if the use of Scala in Flink becomes more and
more limited or neglected.

I think using the Java API in Scala is a possible way also in
my environment.

In the last weeks I tried to port the examples from the "Flink
Course" of Daniel Ciorcilan (https://rockthejvm.com/p/flink -
he mainly offers Scala courses), which are exclusively based
on the native Scala API, to the Java API. This has worked
without any problems as far as I can see. So far I haven't
tried any examples based on the Table API or streaming
workflows in batch mode (which would be important for our
environment).

My main trouble: So far I don't know enough about the
limitations of using the Java API in a Scala implementation
and what that means. My current understanding: the limitation
is mainly in deriving the type information in generic APIs
with Scala types. For me it would be very significant and
helpful if there would be more information, descriptions and
examples about this topic.

So far unfortunately I had too little time to deal with a
wrapper like flink-scala-api
(https://github.com/findify/flink-scala-api ) and the current
alternative is probably going to be deprecated in the future
(https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808
)

Günter


On 04.10.22 13:58, Martijn Visser wrote:

Hi Marton,

You're making a good point, I originally wanted to include
already the User mailing list to get their feedback but
forgot to do so. I'll do some more outreach via other
channels as well.

@Users of Flink, I've made a proposal to deprecate and remove
Scala API support in a future version of Flink. Your feedback
on this topic is very much appreciated.

Regarding the large Scala codebase for Flink, a potential
alternative could be to have a wrapper for all Java APIs that
makes them available as Scala APIs. However, this still
requires Scala maintainers and I don't think that we
currently have those in our community. The easiest solution
for them would be to use the Java APIs directly. Yes it would
involve work, but we won't actually be able to remove the
Scala APIs until Flink 2.0 so there's still time for that :)

Best regards,

Martijn

On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi
 wrote:

Hi Martjin,

Thanks for compiling the FLIP. I agree with the 

Re: [VOTE] Drop Gelly

[Konstantin Knauf]

+1

Am Do., 13. Okt. 2022 um 10:56 Uhr schrieb Niels Basjes :

> +1
>
> On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser 
> wrote:
>
> > Hi everyone,
> >
> > I would like to open a vote for dropping Gelly, which was discussed a
> long
> > time ago but never put to a vote [1].
> >
> > Voting will be open for at least 72 hours.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
> >
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [VOTE] Externalized connector release details​

[Chesnay Schepler]

@Thomas:
Version-specific modules that either contain a connector or shims to 
support that Flink version.
Alternatively, since the addition of such code (usually) goes beyond a 
patch release you'd create a new minor version and could have that only 
support the later version.


On 13/10/2022 02:05, Thomas Weise wrote:

"Branches are not specific to a Flink version. (i.e., no v3.2-1.15)"

Sorry for the late question. I could not find in the discussion thread how
a connector can make use of features of the latest Flink version that were
not present in the previous Flink version, when branches cannot be Flink
version specific?

Thanks,
Thomas

On Wed, Oct 12, 2022 at 4:09 PM Ferenc Csaky 
wrote:


+1 from my side (non-binding)

Best,
F


--- Original Message ---
On Wednesday, October 12th, 2022 at 15:47, Martijn Visser <
martijnvis...@apache.org> wrote:




+1 (binding), I am indeed assuming that Chesnay meant the last two minor
versions as supported.

Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer

dannycran...@apache.org

Thanks for the concise summary Chesnay.

+1 from me (binding)

Just one clarification, for "3.1) The Flink versions supported by the
project (last 2 major Flink versions) must be supported.". Do we

actually

mean major here, as in Flink 1.x.x and 2.x.x? Right now we would only
support Flink 1.15.x and not 1.14.x? I would be inclined to support the
latest 2 minor Flink versions (major.minor.patch) given that we only

have 1

active major Flink version.

Danny

On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler ches...@apache.org
wrote:


Since the discussion
(https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo)

has

stalled a bit but we need a conclusion to move forward I'm opening a
vote.

Proposal summary:

1) Branch model
1.1) The default branch is called "main" and used for the next major
iteration.
1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
1.3) Branches are not specific to a Flink version. (i.e., no

v3.2-1.15)

2) Versioning
2.1) Source releases: major.minor.patch
2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
(This may imply releasing the exact same connector jar multiple times
under different versions)

3) Flink compatibility
3.1) The Flink versions supported by the project (last 2 major Flink
versions) must be supported.
3.2) How this is achived is left to the connector, as long as it
conforms to the rest of the proposal.

4) Support
4.1) The last 2 major connector releases are supported with only the
latter receiving additional features, with the following exceptions:
4.1.a) If the older major connector version does not support any
currently supported Flink version, then it is no longer supported.
4.1.b) If the last 2 major versions do not cover all supported Flink
versions, then the latest connector version that supports the older
Flink version /additionally /gets patch support.
4.2) For a given major connector version only the latest minor

version

is supported.
(This means if 1.1.x is released there will be no more 1.0.x release)

I'd like to clarify that these won't be set in stone for eternity.
We should re-evaluate how well this model works over time and adjust

it

accordingly, consistently across all connectors.
I do believe that as is this strikes a good balance between
maintainability for us and clarity to users.

Voting schema:

Consensus, committers have binding votes, open for at least 72 hours.

--
Martijn
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

Re: [VOTE] Externalized connector release details​

[Chesnay Schepler]

I mean minor. I always get confused on the Flink side because we use 
"major" instead of "minor" releases in many places.


On 12/10/2022 20:18, Danny Cranmer wrote:

Thanks for the concise summary Chesnay.

+1 from me (binding)

Just one clarification, for "3.1) The Flink versions supported by the
project (last 2 major Flink versions) must be supported.". Do we actually
mean major here, as in Flink 1.x.x and 2.x.x? Right now we would only
support Flink 1.15.x and not 1.14.x? I would be inclined to support the
latest 2 minor Flink versions (major.minor.patch) given that we only have 1
active major Flink version.

Danny

On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler  wrote:


Since the discussion
(https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo) has
stalled a bit but we need a conclusion to move forward I'm opening a vote.

Proposal summary:

1) Branch model
1.1) The default branch is called "main" and used for the next major
iteration.
1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2)
1.3) Branches are not specific to a Flink version. (i.e., no v3.2-1.15)

2) Versioning
2.1) Source releases: major.minor.patch
2.2) Jar artifacts: major.minor.match-flink-major.flink-minor
(This may imply releasing the exact same connector jar multiple times
under different versions)

3) Flink compatibility
3.1) The Flink versions supported by the project (last 2 major Flink
versions) must be supported.
3.2) How this is achived is left to the connector, as long as it
conforms to the rest of the proposal.

4) Support
4.1) The last 2 major connector releases are supported with only the
latter receiving additional features, with the following exceptions:
4.1.a) If the older major connector version does not support any
currently supported Flink version, then it is no longer supported.
4.1.b) If the last 2 major versions do not cover all supported Flink
versions, then the latest connector version that supports the older
Flink version /additionally /gets patch support.
4.2) For a given major connector version only the latest minor version
is supported.
(This means if 1.1.x is released there will be no more 1.0.x release)


I'd like to clarify that these won't be set in stone for eternity.
We should re-evaluate how well this model works over time and adjust it
accordingly, consistently across all connectors.
I do believe that as is this strikes a good balance between
maintainability for us and clarity to users.


Voting schema:

Consensus, committers have binding votes, open for at least 72 hours.

Re: [Discuss]- Donate Iceberg Flink Connector

[yuxia]

+1. Thanks for driving it. Hope I can find some chances to take part in the 
future development of Iceberg Flink Connector.

Best regards,
Yuxia

- 原始邮件 -
发件人: "Zheng Yu Chen" 
收件人: "dev" 
发送时间: 星期四, 2022年 10 月 13日 上午 11:26:29
主题: Re: [Discuss]- Donate Iceberg Flink Connector

+1, thanks to drive it

Abid Mohammed  于2022年10月10日周一 09:22写道：

> Hi,
>
> I would like to start a discussion about contributing Iceberg Flink
> Connector to Flink.
>
> I created a doc <
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing>
> with all the details following the Flink Connector template as I don’t have
> permissions to create a FLIP yet.
> High level details are captured below:
>
> Motivation:
>
> This FLIP aims to contribute the existing Apache Iceberg Flink Connector
> to Flink.
>
> Apache Iceberg is an open table format for huge analytic datasets. Iceberg
> adds tables to compute engines including Spark, Trino, PrestoDB, Flink,
> Hive and Impala using a high-performance table format that works just like
> a SQL table.
> Iceberg avoids unpleasant surprises. Schema evolution works and won’t
> inadvertently un-delete data. Users don’t need to know about partitioning
> to get fast queries. Iceberg was designed to solve correctness problems in
> eventually-consistent cloud object stores.
>
> Iceberg supports both Flink’s DataStream API and Table API. Based on the
> guideline of the Flink community, only the latest 2 minor versions are
> actively maintained. See the Multi-Engine Support#apache-flink for further
> details.
>
>
> Iceberg connector supports:
>
> • Source: detailed Source design <
> https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit#>,
> based on FLIP-27
> • Sink: detailed Sink design and interfaces used <
> https://docs.google.com/document/d/1O-dPaFct59wUWQECXEEYIkl9_MOoG3zTbC2V-fZRwrg/edit#
> >
> • Usable in both DataStream and Table API/SQL
> • DataStream read/append/overwrite
> • SQL create/alter/drop table, select, insert into, insert
> overwrite
> • Streaming or batch read in Java API
> • Support for Flink’s Python API
>
> See Iceberg Flink  for
> detailed usage instructions.
>
> Looking forward to the discussion!
>
> Thanks
> Abid

[jira] [Created] (FLINK-29626) Update Akka to 2.6.20

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29626:
--

 Summary: Update Akka to 2.6.20
 Key: FLINK-29626
 URL: https://issues.apache.org/jira/browse/FLINK-29626
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Coordination
Reporter: Martijn Visser
Assignee: Martijn Visser


Update Akka to the latest 2.6 version that's still under Apache license



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29625) Optimize changelog normalize

[Jiabao Sun (Jira)]

Jiabao Sun created FLINK-29625:
--

 Summary: Optimize changelog normalize
 Key: FLINK-29625
 URL: https://issues.apache.org/jira/browse/FLINK-29625
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.15.2
Reporter: Jiabao Sun


Currently, Flink will add an expensive operator _changelog normalize_ to the 
source of the upsert changelog mode to complete the _update_before_ value. 

Even inserting directly from upsert-kafka source to upsert-kafka sink will 
still add this operator, and there is an extra operator to clear 
_upsert_before_ messages, which is obviously redundant.

In CDC scenarios, some databases do not provide update before images, such as 
Cassandra、MongoDB、TiDB({_}Old Value{_} is not turned on) and Postgres 
({_}REPLICA IDENTITY{_} is not set to {_}FULL{_}). Using Flink SQL to process 
these changelog will have a lot of state overhead.

I don't know much about why this operator is needed, so I take the liberty to 
ask if we can get rid of changelog normalize completely or optimistic about it, 
adding it only if a normalized changelog is required by an after operator.

If this optimization is worthwhile, I'm happy to help with it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29624) Upgrade org.apache.commons:commons-lang3 from 3.3.2 to 3.12.0

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29624:
--

 Summary: Upgrade org.apache.commons:commons-lang3 from 3.3.2 to 
3.12.0
 Key: FLINK-29624
 URL: https://issues.apache.org/jira/browse/FLINK-29624
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Common, Connectors / FileSystem
Reporter: Martijn Visser
Assignee: Martijn Visser


Upgrade org.apache.commons:commons-lang3 from 3.3.2 to 3.12.0 to avoid being 
falsely flagged for CVEs CVE-2021-29425 and CVE-2020-15250



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29623) Bump Prometheus Java Client to 0.16.0

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29623:
--

 Summary: Bump Prometheus Java Client to 0.16.0
 Key: FLINK-29623
 URL: https://issues.apache.org/jira/browse/FLINK-29623
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Metrics
Reporter: Martijn Visser
Assignee: Martijn Visser


Flink uses an old and no longer supported version of 
io.prometheus:simpleclient. We should upgrade to the latest version



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Drop Gelly

[Niels Basjes]

+1

On Wed, Oct 12, 2022 at 11:00 PM Martijn Visser 
wrote:

> Hi everyone,
>
> I would like to open a vote for dropping Gelly, which was discussed a long
> time ago but never put to a vote [1].
>
> Voting will be open for at least 72 hours.
>
> Best regards,
>
> Martijn
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Created] (FLINK-29622) KerberosDelegationTokenManager fails to load DelegationTokenProvider due to NoClassDefFoundError in various tests

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-29622:
-

 Summary: KerberosDelegationTokenManager fails to load 
DelegationTokenProvider due to NoClassDefFoundError in various tests
 Key: FLINK-29622
 URL: https://issues.apache.org/jira/browse/FLINK-29622
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Tests
Affects Versions: 1.16.0, 1.17.0
Reporter: Matthias Pohl


There are multiple ITCases that print an error when trying to load the 
{{HadoopFSDelegationTokenProvider}} which is on the classpath through 
{{flink-runtime}} but the corresponding hadoop dependency seems to be missing:
{code}
186348 02:25:25,492 [main] INFO  
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - 
Loading delegation token providers
 186349 02:25:25,493 [main] ERROR 
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager [] - 
Failed to initialize delegation token provider hadoopfs
 186350 java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
 186351 at 
org.apache.flink.runtime.security.token.HadoopFSDelegationTokenProvider.init(HadoopFSDelegationTokenProvider.java:68)
 ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186352 at 
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.loadProviders(KerberosDelegationTokenManager.java:124)
 ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186353 at 
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.(KerberosDelegationTokenManager.java:109)
 ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186354 at 
org.apache.flink.runtime.security.token.KerberosDelegationTokenManager.(KerberosDelegationTokenManager.java:91)
 ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186355 at 
org.apache.flink.runtime.security.token.KerberosDelegationTokenManagerFactory.create(KerberosDelegationTokenManagerFactory.java:47)
 ~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186356 at 
org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:431) 
~[flink-runtime-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186357 at 
org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:234)
 ~[flink-runtime-1.16-SNAPSHOT-tests.jar:1.16-SNAPSHOT]
 186358 at 
org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:109)
 ~[flink-runtime-1.16-SNAPSHOT-tests.jar:1.16-SNAPSHOT]
 186359 at 
org.apache.flink.test.util.MiniClusterWithClientResource.before(MiniClusterWithClientResource.java:64)
 ~[flink-test-utils-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
 186360 at 
org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.setupTestCluster(EventTimeWindowCheckpointingITCase.java:253)
 ~[test-classes/:?]
 186361 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_292]
[...]
{code}

This error might be misleading/confusing to people investigating the logs. It 
looks like this error is actually expected since the tests not necessarily 
require Kerberos delegation tokens.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Drop Gelly

[Yun Gao]

Hi all,
For the successor of the Gelly library, currently in flink-ml we have already 
implemented a new iteration framework based on the DataStream API, 
which is necessary for a graph processing framework. It could also provide 
abilities like allowing for arbitrary states across rounds and asynchronous 
iteration that could help implementing more effective graph algorithms. 
Thus it looks to me that now we are ready to have a design of the new graph 
processing framework based on the unified DataStream API and 
replace the Gelly library built on the deprecated DataSet API. 
Best,
Yun Gao
[1] https://github.com/apache/flink-ml 
--
From:Martijn Visser 
Send Time:2022 Oct. 13 (Thu.) 15:52
To:dev 
Subject:Re: [VOTE] Drop Gelly
Hi Yu,
The vote is for option 1, remove all code and documentation of Gelly. For
the current users, they can stay on a previous version of Flink which still
contains Gelly. Since Gelly (and the DataSet API) haven't been touched in
multiple releases, there's most likely no need for them to upgrade anyway.
The source code is still available for previous versions, so moving it to
an external repository doesn't seem to add value.
I'm all for it if someone wants to drive a successor: that could be either
done under the ASF Flink umbrella (then I'm looking forwarding to a FLIP)
or of course as their own initiative under flink-extended.
Thanks,
Martijn
On Thu, Oct 13, 2022 at 6:31 AM Yu Li  wrote:
> Hi Martijn,
>
> From the last replies of the previous discussion, I could see there are
> still users using Gelly that expressed their wish to somehow keep this
> module [1] [2] [3]. Therefore, Before giving my vote, I'd like to
> confirm what "Drop Gelly" exactly means. Does it mean that 1) we remove the
> codes and documents of Gelly completely, or 2) just move it out of the main
> repository into a separate one and mark it as EOM (or maybe call for
> maintainers), or 3) something else? And if the #1 option, is there any
> replacement (or any plan)? Thanks.
>
> Best Regards,
> Yu
>
> [1] https://lists.apache.org/thread/4yxb7xnb2070h5lypcd3wxnsck9zwz8f 
> 
> [2] https://lists.apache.org/thread/x21p382vjt6nrjnj51fxxtcrp1dqtzyz 
> 
> [3] https://lists.apache.org/thread/w2f4yvb75tg5t7g3l7t9z6bvpwmd1t6y 
> 
>
>
> On Thu, 13 Oct 2022 at 10:56, Yun Gao 
> wrote:
>
> > +1
> > Best,
> > Yun Gao
> > --
> > From:Martijn Visser 
> > Send Time:2022 Oct. 13 (Thu.) 04:59
> > To:dev 
> > Subject:[VOTE] Drop Gelly
> > Hi everyone,
> > I would like to open a vote for dropping Gelly, which was discussed a
> long
> > time ago but never put to a vote [1].
> > Voting will be open for at least 72 hours.
> > Best regards,
> > Martijn
> > https://twitter.com/MartijnVisser82 
> > https://github.com/MartijnVisser 
> > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc 
> > 
> >
>

Re: [ANNOUNCE] Apache Flink Table Store 0.2.1 released

[Martijn Visser]

Congratulations and thanks to all those involved!

On Thu, Oct 13, 2022 at 4:47 AM Jingsong Lee 
wrote:

> The Apache Flink community is very happy to announce the release of
> Apache Flink Table Store 0.2.1.
>
> Apache Flink Table Store is a unified storage to build dynamic tables
> for both streaming and batch processing in Flink, supporting
> high-speed data ingestion and timely data query.
>
> Please check out the release blog post for an overview of the release:
> https://flink.apache.org/news/2022/10/13/release-table-store-0.2.1.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Table Store can be found at:
> https://search.maven.org/search?q=g:org.apache.flink%20table-store
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352257
>
> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
>
> Best,
> Jingsong Lee
>

Re: [VOTE] Remove HCatalog connector

[Sergey Nuyanzin]

+1 (non-binding)
Thanks Martijn for driving this

On Thu, Oct 13, 2022 at 8:47 AM Qingsheng Ren  wrote:

> +1
>
> Thanks for driving this Martijn!
>
> Best,
> Qingsheng
>
> > On Oct 13, 2022, at 03:45, Martijn Visser 
> wrote:
> >
> > Hi everyone,
> >
> > Since no comments were made, I'm opening a vote to remove the HCatalog
> > connector [1]
> >
> > The voting period will be open at least 72hrs.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1]
> > https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84
> > --
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
>
>

-- 
Best regards,
Sergey

[jira] [Created] (FLINK-29621) Append-only with eventual log.consistency can not work

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-29621:


 Summary: Append-only with eventual log.consistency can not work
 Key: FLINK-29621
 URL: https://issues.apache.org/jira/browse/FLINK-29621
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.3.0, table-store-0.2.2






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Drop Gelly

[Martijn Visser]

Hi Yu,

The vote is for option 1, remove all code and documentation of Gelly. For
the current users, they can stay on a previous version of Flink which still
contains Gelly. Since Gelly (and the DataSet API) haven't been touched in
multiple releases, there's most likely no need for them to upgrade anyway.
The source code is still available for previous versions, so moving it to
an external repository doesn't seem to add value.

I'm all for it if someone wants to drive a successor: that could be either
done under the ASF Flink umbrella (then I'm looking forwarding to a FLIP)
or of course as their own initiative under flink-extended.

Thanks,

Martijn

On Thu, Oct 13, 2022 at 6:31 AM Yu Li  wrote:

> Hi Martijn,
>
> From the last replies of the previous discussion, I could see there are
> still users using Gelly that expressed their wish to somehow keep this
> module [1] [2] [3]. Therefore, Before giving my vote, I'd like to
> confirm what "Drop Gelly" exactly means. Does it mean that 1) we remove the
> codes and documents of Gelly completely, or 2) just move it out of the main
> repository into a separate one and mark it as EOM (or maybe call for
> maintainers), or 3) something else? And if the #1 option, is there any
> replacement (or any plan)? Thanks.
>
> Best Regards,
> Yu
>
> [1] https://lists.apache.org/thread/4yxb7xnb2070h5lypcd3wxnsck9zwz8f
> [2] https://lists.apache.org/thread/x21p382vjt6nrjnj51fxxtcrp1dqtzyz
> [3] https://lists.apache.org/thread/w2f4yvb75tg5t7g3l7t9z6bvpwmd1t6y
>
>
> On Thu, 13 Oct 2022 at 10:56, Yun Gao 
> wrote:
>
> > +1
> > Best,
> > Yun Gao
> > --
> > From:Martijn Visser 
> > Send Time:2022 Oct. 13 (Thu.) 04:59
> > To:dev 
> > Subject:[VOTE] Drop Gelly
> > Hi everyone,
> > I would like to open a vote for dropping Gelly, which was discussed a
> long
> > time ago but never put to a vote [1].
> > Voting will be open for at least 72 hours.
> > Best regards,
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> > [1] https://lists.apache.org/thread/2m6wtgjvxcogbf9d5q7mqt4ofqjf2ojc
> >
>

[jira] [Created] (FLINK-29620) Flink deployment stuck in UPGRADING state when changing configuration

[liad shachoach (Jira)]

liad shachoach created FLINK-29620:
--

 Summary: Flink deployment stuck in UPGRADING state when changing 
configuration
 Key: FLINK-29620
 URL: https://issues.apache.org/jira/browse/FLINK-29620
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: 1.14.2
 Environment: AWS EKS v1.21

Operator version: 1.1.0
Reporter: liad shachoach


When I update the configuration of a flink deployment I observe one of two 
scenarios:

Success:

This happens when the job has not started - if I change the configuration quick 
enough:
{code:java}
2022-10-13 06:50:54,336 o.a.f.k.o.r.d.AbstractJobReconciler [INFO 
][load-streaming/validator-process-124] Upgrading/Restarting running job, 
suspending first...
2022-10-13 06:50:54,343 o.a.f.k.o.r.d.ApplicationReconciler [INFO 
][load-streaming/validator-process-124] Job is not running but HA metadata is 
available for last state restore, ready for upgrade
2022-10-13 06:50:54,353 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Deleting JobManager deployment while 
preserving HA metadata.
2022-10-13 06:50:58,415 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (5s)
2022-10-13 06:51:03,451 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (10s)
2022-10-13 06:51:06,469 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Cluster shutdown completed.
2022-10-13 06:51:06,470 o.a.f.k.o.c.FlinkDeploymentController [INFO 
][load-streaming/validator-process-124] End of reconciliation
2022-10-13 06:51:06,493 o.a.f.k.o.c.FlinkDeploymentController [INFO 
][load-streaming/validator-process-124] Starting reconciliation
2022-10-13 06:51:06,494 o.a.f.k.o.c.FlinkConfigManager [INFO 
][load-streaming/validator-process-124] Generating new config
 {code}
In this scenario I see that the job manager and task manager pods are 
terminated and then recreated.

 

 

Failure:

This happens when I let the job start (wait more than 30-60 seconds) and change 
the configuration:
{code:java}
2022-10-13 06:53:06,637 o.a.f.k.o.r.d.AbstractJobReconciler [INFO 
][load-streaming/validator-process-124] Upgrading/Restarting running job, 
suspending first...
2022-10-13 06:53:06,637 o.a.f.k.o.r.d.AbstractJobReconciler [INFO 
][load-streaming/validator-process-124] Job is in running state, ready for 
upgrade with SAVEPOINT
2022-10-13 06:53:06,659 o.a.f.k.o.s.FlinkService       [INFO 
][load-streaming/validator-process-124] Suspending job with savepoint.
2022-10-13 06:53:07,042 o.a.f.k.o.s.FlinkService       [INFO 
][load-streaming/validator-process-124] Job successfully suspended with 
savepoint 
s3://cu-flink-load-checkpoints-us-east-1/validator-process-124/savepoints/savepoint-00-947975b509b2.
2022-10-13 06:53:11,111 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (5s)
2022-10-13 06:53:16,176 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (10s)
2022-10-13 06:53:21,238 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (15s)
2022-10-13 06:53:26,293 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (20s)
2022-10-13 06:53:31,355 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (25s)
2022-10-13 06:53:36,412 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (30s)
2022-10-13 06:53:41,512 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (35s)
2022-10-13 06:53:46,568 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (40s)
2022-10-13 06:53:51,625 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (45s)
2022-10-13 06:53:56,740 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (50s)
2022-10-13 06:54:01,811 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (55s)
2022-10-13 06:54:06,866 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Waiting for cluster shutdown... (60s)
2022-10-13 06:54:07,866 o.a.f.k.o.u.FlinkUtils         [INFO 
][load-streaming/validator-process-124] Cluster shutdown completed.
2022-10-13 06:54:07,866 o.a.f.k.o.c.FlinkDeploymentController [INFO 
][load-streaming/validator-process-124] End of reconciliation
2022-10-13 06:54:07,894 o.a.f.k.o.c.FlinkDeploymentController [INFO 

[jira] [Created] (FLINK-29619) Remove redundant MeterView updater thread from KubernetesClientMetrics

[Matyas Orhidi (Jira)]

Matyas Orhidi created FLINK-29619:
-

 Summary: Remove redundant MeterView updater thread from 
KubernetesClientMetrics
 Key: FLINK-29619
 URL: https://issues.apache.org/jira/browse/FLINK-29619
 Project: Flink
  Issue Type: Bug
Reporter: Matyas Orhidi


The `MetricRegistryImpl` already has a solution to update `MeterView` objects 
periodically.

https://github.com/apache/flink/blob/7a509c46e45b9a91f2b7d01f13afcdef266b1faf/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java#L404



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29618) YARNSessionFIFOSecuredITCase.testDetachedMode timed out in Azure CI

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-29618:
-

 Summary: YARNSessionFIFOSecuredITCase.testDetachedMode timed out 
in Azure CI
 Key: FLINK-29618
 URL: https://issues.apache.org/jira/browse/FLINK-29618
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN, Tests
Affects Versions: 1.17.0
Reporter: Matthias Pohl


We experienced a [build 
failure|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41931=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=30284]
 that was caused (exclusively) by 
{{YARNSessionFIFOSecuredITCase.testDetachedMode}} running into a timeout.

The actual issue might be that the test thread failed due to an 
{{InterruptedException}} while waiting for the job to be submitted:
{code}
11:33:51,124 [ForkJoinPool-1-worker-25] WARN  
org.apache.flink.yarn.YarnTestBase   [] - Interruped
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method) ~[?:1.8.0_292]
at org.apache.flink.yarn.YarnTestBase.sleep(YarnTestBase.java:716) 
~[test-classes/:?]
at 
org.apache.flink.yarn.YarnTestBase.startWithArgs(YarnTestBase.java:906) 
~[test-classes/:?]
at 
org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:141)
 ~[test-classes/:?]
at 
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.lambda$testDetachedMode$2(YARNSessionFIFOSecuredITCase.java:173)
 ~[test-classes/:?]
at org.apache.flink.yarn.YarnTestBase.runTest(YarnTestBase.java:288) 
~[test-classes/:?]
at 
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode(YARNSessionFIFOSecuredITCase.java:160)
 ~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_292]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_292]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_292]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
[...]
{code}

The test specific logs which were extracted from the build's are attached to 
this Jira issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29617) Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster

[LI Mingkun (Jira)]

LI Mingkun created FLINK-29617:
--

 Summary: Cost too much time to start SourceCoordinator of 
hdfsFileSource when start JobMaster
 Key: FLINK-29617
 URL: https://issues.apache.org/jira/browse/FLINK-29617
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem, Runtime / Coordination
Affects Versions: 1.15.2
Reporter: LI Mingkun


h1. Scenario:
Our user use flink batch to compact small files in one day. Flink version : 1.15
He split pipeline into 24 for each hour. So there are 24 source
 
I find it  costs too much time to start SourceCoordinator of hdfsFileSource 
when start JobMaster
 
 as follow:
 
!https://mail.google.com/mail/u/0?ui=2=488d9ac3dd=0.1=msg-a:r-3013789195315215531=183cb292e567fd9f=fimg=ip=s0-l75-ft=ANGjdJ9SVAoAslMUGQdVQJ_ccmEf4LxhaONYKJvS_V8nvijvT3JXw_VlyRBAEE9EQhTtWdYPa4TLCO5rxjXGrTDK2_PGHX4RZDPTQTJ0LwKXAUr4BYlMhYZsjcrY9eo=emb=ii_l95bh7qy0|width=542,height=260!
 
h1. Root Cause:
I got the root cause after check: 
 # AbstractFileSource will enumerateSplits when createEnumerator
 # NotSplittingRecursiveEnumerator need to get fileblockLocation of every 
fileblock which is a heavy IO operation

!https://mail.google.com/mail/u/0?ui=2=488d9ac3dd=0.3=msg-a:r-3013789195315215531=183cb292e567fd9f=fimg=ip=s0-l75-ft=ANGjdJ8AoT071eCNMb_q3uJtcbrUmZnYbg3ucnDelMlRRPn7WLlXOBGj650srQk9vhqKyJEANvpOWoxHuH6jNHt7g6go8JkeRUZKc81yqT0yzzz7tbBciTe-YnRVQ7w=emb=ii_l95bp1832|width=542,height=456!
 
!https://mail.google.com/mail/u/0?ui=2=488d9ac3dd=0.2=msg-a:r-3013789195315215531=183cb292e567fd9f=fimg=ip=s0-l75-ft=ANGjdJ9phsX1nauTsx3xWje_YJM4uUaOLXKHcXKsm7WJquPQQGC7bQTni3OhQB5HtGYVOvrD-3Kbp9LURfUj6OiIUgsZU1AImSL0vj27cnDcf7HpVpLpaqdADtpoABU=emb=ii_l95bjh1g1|width=526,height=542!
 
h1. Suggestion
 # FileSource add option to disable location fetcher
 # Move location fetcher into IOExecutor



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29616) Polish Table Store Pom file to avoid warning.

[Aiden Gong (Jira)]

Aiden Gong created FLINK-29616:
--

 Summary: Polish Table Store Pom file to avoid warning.
 Key: FLINK-29616
 URL: https://issues.apache.org/jira/browse/FLINK-29616
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.3.0
Reporter: Aiden Gong
 Fix For: table-store-0.3.0
 Attachments: image-2022-10-13-14-49-39-582.png

!image-2022-10-13-14-49-39-582.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Remove HCatalog connector

[Qingsheng Ren]

+1

Thanks for driving this Martijn!

Best,
Qingsheng

> On Oct 13, 2022, at 03:45, Martijn Visser  wrote:
> 
> Hi everyone,
> 
> Since no comments were made, I'm opening a vote to remove the HCatalog
> connector [1]
> 
> The voting period will be open at least 72hrs.
> 
> Best regards,
> 
> Martijn
> 
> [1]
> https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84
> -- 
> Martijn
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser

[jira] [Created] (FLINK-29615) MetricStore does not remove metrics of nonexistent subtasks when adaptive scheduler lowers job parallelism

[Zhanghao Chen (Jira)]

Zhanghao Chen created FLINK-29615:
-

 Summary: MetricStore does not remove metrics of nonexistent 
subtasks when adaptive scheduler lowers job parallelism
 Key: FLINK-29615
 URL: https://issues.apache.org/jira/browse/FLINK-29615
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics, Runtime / REST
Affects Versions: 1.15.0, 1.16.0
Reporter: Zhanghao Chen
 Fix For: 1.17.0


*Problem*

MetricStore does not remove metrics of nonexistent subtasks when adaptive 
scheduler lowers job parallelism and users will see metrics of nonexistent 
subtasks on Web UI (e.g. the task backpressure page) or REST API response.

 

*Proposed Solution*

Thanks to [FLINK-29132] SubtaskMetricStore causes memory leak. - ASF JIRA 
(apache.org) & [FLINK-28588] Enhance REST API for Speculative Execution - ASF 
JIRA (apache.org),  Flink will now update current execution attempts when 
updating metrics. Since the active subtask info is included in the current 
execution attempt info, we are able to retain active subtasks using the current 
execution attempt info.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29614) Introduce Spark writer for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-29614:


 Summary: Introduce Spark writer for table store
 Key: FLINK-29614
 URL: https://issues.apache.org/jira/browse/FLINK-29614
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Jingsong Lee


The main difficulty is that the Spark SourceV2 interface currently does not 
support custom distribution, and the Table Store must have consistent 
distribution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

[Salva Alcántara]

Hi Martijn,

Maybe a bit of an off-topic, but regarding Java 17 support, will it be
possible to replace POJOs with Java records in existing applications?

In a project I maintain we use Lombok a lot, but with Java records we would
probably stop using it (or significantly reduce its usage).

Will there be a way to promote existing POJOs (either written "manually" or
using Lombok) to Java records without breaking serialization? (assuming
that those POJOs are used as immutable values, e.g., setters are never
used).

Regards,

Salva

On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser 
wrote:

> Hi everyone,
>
> Thanks again for all your feedback. It's very much appreciated.
>
> My overall feeling is that people are not opposed to the FLIP. There is
> demand for adding Java 17 support before dropping the Scala APIs. Given
> that the proposal for actually dropping the Scala APIs would only happen
> with a Flink 2.0 and Java 17 support would either happen in a new minor
> version or a new major version (I haven't seen a FLIP or discussion being
> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
> would either be there earlier (in a new minor version) or at the same time
> (with Flink 2.0) when the Scala APIs would be dropped.
>
> If there are no more discussion topics, I would move this FLIP to a vote
> at the beginning of next week.
>
> Best regards,
>
> Martijn
>
> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists 
> wrote:
>
>> Hi Martijn
>>
>> I do not maintain a large production application based on Flink, so it
>> would not be a problem for me to convert existing implementations to
>> whatever API.
>>
>> I am working in the area of cultural heritage, which is mainly about the
>> processing of structured (meta)-data (scientific libraries, archives and
>> museums)
>> My impression: People without much background/experience with Java
>> implementations find it easier to get into the functional mindset as
>> supported in Scala. That's why I think it would be very unfortunate if the
>> use of Scala in Flink becomes more and more limited or neglected.
>>
>> I think using the Java API in Scala is a possible way also in my
>> environment.
>>
>> In the last weeks I tried to port the examples from the "Flink Course" of
>> Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>> Scala courses), which are exclusively based on the native Scala API, to the
>> Java API. This has worked without any problems as far as I can see. So far
>> I haven't tried any examples based on the Table API or streaming workflows
>> in batch mode (which would be important for our environment).
>>
>> My main trouble: So far I don't know enough about the limitations of
>> using the Java API in a Scala implementation and what that means. My
>> current understanding: the limitation is mainly in deriving the type
>> information in generic APIs with Scala types. For me it would be very
>> significant and helpful if there would be more information, descriptions
>> and examples about this topic.
>>
>> So far unfortunately I had too little time to deal with a wrapper like
>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>> current alternative is probably going to be deprecated in the future (
>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>
>> Günter
>>
>>
>> On 04.10.22 13:58, Martijn Visser wrote:
>>
>> Hi Marton,
>>
>> You're making a good point, I originally wanted to include already the
>> User mailing list to get their feedback but forgot to do so. I'll do some
>> more outreach via other channels as well.
>>
>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
>> support in a future version of Flink. Your feedback on this topic is very
>> much appreciated.
>>
>> Regarding the large Scala codebase for Flink, a potential alternative
>> could be to have a wrapper for all Java APIs that makes them available as
>> Scala APIs. However, this still requires Scala maintainers and I don't
>> think that we currently have those in our community. The easiest solution
>> for them would be to use the Java APIs directly. Yes it would involve work,
>> but we won't actually be able to remove the Scala APIs until Flink 2.0 so
>> there's still time for that :)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Oct 4, 2022 at 1:26 AM Márton Balassi 
>> wrote:
>>
>>> Hi Martjin,
>>>
>>> Thanks for compiling the FLIP. I agree with the sentiment that Scala
>>> poses
>>> considerable maintenance overhead and key improvements (like 2.13 or
>>> 2.12.8
>>> supports) are hanging stale. With that said before we make this move we
>>> should attempt to understand the userbase affected.
>>> A quick Slack and user mailing list search does return quite a bit of
>>> results for scala (admittedly a cursory look at them suggest that many of
>>> them have to do with missing features in Scala that exist in Java or
>>> Scala
>>> versions). I would love to see some polls on this 

Re: [DISCUSS] Adding client.id.prefix to the KafkaSink

[Hang Ruan]

Hi Yaroslav,

+1 from me.  It is meaningful to keep the common Kafka client configuration
the same between Kafka source and sink.

Best,
Hang

Mason Chen  于2022年10月13日周四 13:51写道：

> Hi Yaroslav,
>
> +1 from my end too. I get some questions internally on the warning logs due
> to conflicting client-id and it would be nice to resolve them.
>
> Best,
> Mason
>
> On Wed, Oct 12, 2022 at 2:03 PM Martijn Visser 
> wrote:
>
> > Hi Yaroslav,
> >
> > +1 from my end. Thanks for bringing this up!
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Oct 12, 2022 at 6:34 PM Yaroslav Tkachenko
> >  wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to propose adding client.id.prefix to the KafkaSink to mirror
> > the
> > > functionality provided by the KafkaSource.
> > >
> > > Defining client.id is very important when running workloads with many
> > > different Kafka clients: they help with identification and enforcing
> > > quotas. Due to the specific implementation details, you can't simply
> set
> > > client.id for a KafkaSource or KafkaSink in Flink, you can find more
> > > context in https://issues.apache.org/jira/browse/FLINK-8093, which was
> > > opened in 2017 and is still unresolved.
> > >
> > > I don't see any reason to treat KafkaSink differently: if we can define
> > > a client.id.prefix for the KafkaSource we should be able to define it
> for
> > > the KafkaSink.
> > >
> > > I've created an issue
> https://issues.apache.org/jira/browse/FLINK-28842
> > > and
> > > a PR https://github.com/apache/flink/pull/20475 to resolve this.
> > >
> >
>

[jira] [Created] (FLINK-29613) An error occurred during the running of the Flink pulsar，it shows "We only support normal message id currently."

[qiaomengnan (Jira)]

qiaomengnan created FLINK-29613:
---

 Summary: An error occurred during the running of the Flink 
pulsar，it shows "We only support normal message id currently."
 Key: FLINK-29613
 URL: https://issues.apache.org/jira/browse/FLINK-29613
 Project: Flink
  Issue Type: Bug
Reporter: qiaomengnan


java.lang.RuntimeException: One or more fetchers have encountered exception
at 
nextMessageIdorg.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
at 
org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.pollNext(PulsarOrderedSourceReader.java:109)
at 
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
at 
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at 
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
at 
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: SplitFetcher thread 1 received 
unexpected exception while polling the records
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Suppressed: java.lang.RuntimeException: SplitFetcher thread 0 received 
unexpected exception while polling the records
... 7 more
Caused by: java.lang.IllegalArgumentException: We only support normal message 
id currently.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
at 
org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.unwrapMessageId(MessageIdUtils.java:61)
at 
org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.nextMessageId(MessageIdUtils.java:43)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader.beforeCreatingConsumer(PulsarOrderedPartitionSplitReader.java:94)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase.handleSplitsChanges(PulsarPartitionSplitReaderBase.java:160)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader.handleSplitsChanges(PulsarOrderedPartitionSplitReader.java:52)
at 
org.apache.flink.connector.base.source.reader.fetcher.AddSplitsTask.run(AddSplitsTask.java:51)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
... 6 more
Caused by: java.lang.IllegalArgumentException: We only support normal message 
id currently.
at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
at 
org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.unwrapMessageId(MessageIdUtils.java:61)
at 
org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.nextMessageId(MessageIdUtils.java:43)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader.beforeCreatingConsumer(PulsarOrderedPartitionSplitReader.java:94)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase.handleSplitsChanges(PulsarPartitionSplitReaderBase.java:160)
at 
org.apache.flink.connector.pulsar.source.reader.split.PulsarOrderedPartitionSplitReader.handleSplitsChanges(PulsarOrderedPartitionSplitReader.java:52)
at 
org.apache.flink.connector.base.source.reader.fetcher.AddSplitsTask.run(AddSplitsTask.java:51)
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
... 6 more



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Remove HCatalog connector

[Leonard Xu]

Thanks martijn for driving this work.

+1

Best,
Leonard


> 2022年10月13日 下午12:29，Gyula Fóra  写道：
> 
> +1
> 
> On Thu, 13 Oct 2022 at 04:53, yuxia  wrote:
> 
>> +1 (non-binding)
>> Thanks for driving.
>> 
>> Best regards,
>> Yuxia
>> 
>> - 原始邮件 -
>> 发件人: "Hang Ruan" 
>> 收件人: "dev" 
>> 发送时间: 星期四, 2022年 10 月 13日 上午 10:16:46
>> 主题: Re: [VOTE] Remove HCatalog connector
>> 
>> +1,
>> 
>> Best,
>> Hang
>> 
>> Jingsong Li  于2022年10月13日周四 10:09写道：
>> 
>>> +1
>>> 
>>> Thanks for driving.
>>> 
>>> Best,
>>> Jingsong
>>> 
>>> On Thu, Oct 13, 2022 at 3:46 AM Martijn Visser >> 
>>> wrote:
 
 Hi everyone,
 
 Since no comments were made, I'm opening a vote to remove the HCatalog
 connector [1]
 
 The voting period will be open at least 72hrs.
 
 Best regards,
 
 Martijn
 
 [1]
 https://lists.apache.org/thread/j8jc5zrhnqlv8y3lkmc3wdo9ysgmsr84
 --
 Martijn
 https://twitter.com/MartijnVisser82
 https://github.com/MartijnVisser
>>> 
>> 

Re: SQL Gateway and SQL Client

[yuxia]

> In what Flink’s release the connection from SQL Client to the Gateway is
expected to be added? 
Flink 1.17

> “Authentication module” (2) and “Persistent Gateway” (4) as
possible future work. Were there any recent discussions on these subjects?
No recent discussions on these subjects, but I think it'll come in Flink 1.17

> Another related topic: are there ideas around making SQL Gateway a 
> multi-tenant
component?
Yes.

Shengkaiis the maintainer of SQL Client and SQL gateway, maybe he can provide 
more information.



Best regards,
Yuxia

- 原始邮件 -
发件人: "Alexey Leonov-Vendrovskiy" 
收件人: "dev" 
发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08
主题: SQL Gateway and SQL Client

Hi all,

I’m Alexey from Confluent. This is my first email in this discussion list.
I’m rather new to Flink, and to local customs of communication. I want to
dive deeper and hopefully get more involved over time.

Currently I have a few questions around SQL Gateway and SQL Client.
Specifically I wanted to learn what is the vision around the nearest future
of these two components.

In what Flink’s release the connection from SQL Client to the Gateway is
expected to be added? I was looking at
https://issues.apache.org/jira/browse/FLINK-29486, and recently it got
renamed from “Enable SQL Client to Connect SQL Gateway in Remote Mode” to
“Introduce Client Parser to get statement type”.  I did some search, but
didn’t find a good place where the client's work in this direction is
discussed or tracked.

A couple questions about the SQL Gateway. The FLIP-91

mentions “Authentication module” (2) and “Persistent Gateway” (4) as
possible future work. Were there any recent discussions on these subjects?
Or maybe there are some ideas how to move these directions forward? Another
related topic: are there ideas around making SQL Gateway a multi-tenant
component?

Thank you,

Alexey

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

[Mason Chen]

Hi Ryan,

Thanks for the additional context! Yes, the offset initializer would need
to take a cluster as a parameter and the MultiClusterKafkaSourceSplit can
be exposed in an initializer.

Best,
Mason

On Thu, Oct 6, 2022 at 11:00 AM Ryan van Huuksloot <
ryan.vanhuuksl...@shopify.com> wrote:

> Hi Mason,
>
> Thanks for the clarification! In regards to the addition to the
> OffsetInitializer of this API - this would be an awesome addition and I
> think this entire FLIP would be a great addition to the Flink.
>
> To provide more context as to why we need particular offsets, we use
> Hybrid Source to currently backfill from buckets prior to reading from
> Kafka. We have a service that will tell us what offset has last been loaded
> into said bucket which we will use to initialize the KafkaSource
> OffsetsInitializer. We couldn't use a timestamp here and the offset would
> be different for each Cluster.
>
> In pseudocode, we'd want the ability to do something like this with
> HybridSources - if this is possible.
>
> ```scala
> val offsetsMetadata: Map[TopicPartition, Long] = // Get current offsets
> from OffsetReaderService
> val multiClusterArchiveSource: MultiBucketFileSource[T] = // Data is read
> from different buckets (multiple topics)
> val multiClusterKafkaSource: MultiClusterKafkaSource[T] =
> MultiClusterKafkaSource.builder()
>   .setKafkaMetadataService(new KafkaMetadataServiceImpl())
>   .setStreamIds(List.of("my-stream-1", "my-stream-2"))
>   .setGroupId("myConsumerGroup")
>
> .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
>   .setStartingOffsets(offsetsMetadata)
>   .setProperties(properties)
>   .build()
> val source =
> HybridSource.builder(multiClusterArchiveSource).addSource(multiClusterKafkaSource).build()
> ```
>
> Few notes:
> - TopicPartition won't work because the topic may be the same name as this
> is something that is supported IIRC
> - I chose to pass a map into starting offsets just for demonstrative
> purposes, I would be fine with whatever data structure would work best
>
> Ryan van Huuksloot
> Data Developer | Production Engineering | Streaming Capabilities
> [image: Shopify]
> 
>
>
> On Mon, Oct 3, 2022 at 11:29 PM Mason Chen  wrote:
>
>> Hi Ryan,
>>
>> Just copying your message over to the email chain.
>>
>> Hi Mason,
>>> First off, thanks for putting this FLIP together! Sorry for the delay.
>>> Full disclosure Mason and I chatted a little bit at Flink Forward 2022 but
>>> I have tried to capture the questions I had for him then.
>>> I'll start the conversation with a few questions:
>>> 1. The concept of streamIds is not clear to me in the proposal and could
>>> use some more information. If I understand correctly, they will be used in
>>> the MetadataService to link KafkaClusters to ones you want to use? If you
>>> assign stream ids using `setStreamIds`, how can you dynamically increase
>>> the number of clusters you consume if the list of StreamIds is static? I am
>>> basing this off of your example .setStreamIds(List.of("my-stream-1",
>>> "my-stream-2")) so I could be off base with my assumption. If you don't
>>> mind clearing up the intention, that would be great!
>>> 2. How would offsets work if you wanted to use this
>>> MultiClusterKafkaSource with a file based backfill? In the case I am
>>> thinking of, you have a bucket backed archive of Kafka data per cluster.
>>> and you want to pick up from the last offset in the archived system, how
>>> would you set OffsetInitializers "per cluster" potentially as a function or
>>> are you limited to setting an OffsetInitializer for the entire Source?
>>> 3. Just to make sure - because this system will layer on top of Flink-27
>>> and use KafkaSource for some aspects under the hood, the watermark
>>> alignment that was introduced in FLIP-182 / Flink 1.15 would be possible
>>> across multiple clusters if you assign them to the same alignment group?
>>> Thanks!
>>> Ryan
>>
>>
>> 1. The stream ids are static--however, what the physical clusters and
>> topics that they map to can mutate. Let's say my-stream-1 maps to cluster-1
>> and topic-1. The KafkaMetadataService can return a different mapping when
>> metadata is fetched the next time e.g. my-stream-1 mapping to cluster-1 and
>> topic-1, and cluster-2 and topic-2. Let me add more details on how the
>> KafkaMetadataService is used.
>> 2. The current design limits itself to a single configured
>> OffsetInitializer that is used for every underlying KafkaSource.
>> 3. Yes, it is in our plan to integrate this source with watermark
>> alignment in which the user can align watermarks from all clusters within
>> the single. It will leverage the Kafka Source implementation to achieve
>> this.
>>
>> With regards to 2, it's an interesting idea. I think we can extend the
>> design to support a map of offset initializers to clusters, which would
>> solve your file based backfill. If you