Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

[Leonard Xu]

Thanks Yuxia for driving this work.

+1(binding)

Best,
Leonard

> 2022年4月28日 下午8:29，Jing Zhang  写道：
> 
> +1 (binding)
> 
> Best,
> Jing Zhang
> 
> Martijn Visser  于2022年4月28日周四 20:11写道：
> 
>> +1 (Binding)
>> 
>> On Thu, 28 Apr 2022 at 13:40, ron  wrote:
>> 
>>> +1
>>> 
>>> Best,
>>> Ron
>>> 
>>> 
 -原始邮件-
 发件人: "Jark Wu" 
 发送时间: 2022-04-28 15:46:22 (星期四)
 收件人: dev 
 抄送:
 主题: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for
>>> migrating Hive dialect
 
 +1 (binding)
 
 Thank Yuxia, for driving this work.
 
 Best,
 Jark
 
 On Thu, 28 Apr 2022 at 11:58, Jingsong Li 
>>> wrote:
 
> +1 (Binding)
> 
> A very good step to move forward.
> 
> Best.
> Jingsong
> 
> On Wed, Apr 27, 2022 at 9:33 PM yuxia 
>>> wrote:
>> 
>> Hi, everyone
>> 
>> Thanks all for attention to FLIP-216: Introduce pluggable dialect
>> and
> plan for migrating Hive dialect [1] and participation in the
>>> discussion in
> the mail thread [2].
>> 
>> I'd like to start a vote for it. The vote will be open for at least
>>> 72
> hours unless there is an objection or not enough votes.
>> 
>> [1] [
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> |
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> ]
>> [2] [
>>> https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo
> | https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo ]
>> 
>> 
>> Best regards,
>> Yuxia
> 
>>> 
>>> 
>>> --
>>> Best,
>>> Ron
>>> 
>> 

Re: Add a init-file option into FLIP-91

[Shengkai Fang]

Hi. LuNing.

Thanks for your attention. I have responded in the discussion mails.

Best,
Shengkai

LuNing Wang  于2022年4月27日周三 17:29写道：

> Hi all,
>
> After I read FLIP-91[1], I want to add an init-file option. Its
> functionality is the same as option '-i' of Flink SQL Client.
>
> When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by this
> option after SQL Gateway starts every time.
>
> Shall we name this option `sql-gateway.session.init-file` and write it into
> the FLIP-91?
>
> Best regards,
>
> LuNing Wang
>
> [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91
>

Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint

[Shengkai Fang]

Hi, Jark and Martijn

Thanks for your feedback.

> Kyuubi provides three ways to configure Hive metastore [1]. Could we
provide similar abilities?

Yes. I have updated the FLIP about this and it takes some time to figure
out how the jdbc driver works. I added the section about how to use the
hive JDBC to configure the session-level catalog.

> I think we can improve the "HiveServer2 Compatibility" section.

Yes. I have updated the FLIP and added more details about the compatibility.

>  Prefer to first complete the discussion and vote on FLIP-91 then discuss
FLIP-223

Of course. We can wait until the discussion of the FLIP-91 finishes.

> Maintenance concerns about the hive

Actually we will only rely on the API in the Hive, which only contains the
thrift file and the generated code[1]. I think it will not influence us to
upgrade the java version.

[1] https://github.com/apache/hive/tree/master/service-rpc

Best,
Shengkai

Martijn Visser  于2022年4月26日周二 20:44写道：

> Hi all,
>
> I'm not too familiar with Hive and HiveServer2, but I do have a couple of
> questions/concerns:
>
> 1. What is the relationship between this FLIP and FLIP-91? My assumption
> would be that this FLIP (and therefore the HiveServer2) implementation
> would need to be integrated in the REST Gateway, is that correct? If so, I
> would prefer to first complete the discussion and vote on FLIP-91, else
> we'll have two moving FLIPs who have a direct relationship with each other.
>
> 2. While I understand that Hive is important (in the Chinese ecosystem, not
> so much in Europe and the US), I still have maintenance concerns on this
> topic. We know that the current Hive integration isn't exactly ideal and
> requires a lot of work to get in better shape. At the same time, Hive still
> doesn't support Java 11 while we need (and should, given the premier
> support has ended already) to move away from Java 8.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
>
> On Mon, 25 Apr 2022 at 12:13, Jark Wu  wrote:
>
> > Thank Shengkai for driving this effort,
> > I think this is an essential addition to Flink Batch.
> >
> > I have some small suggestions:
> > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could we
> > provide similar abilities?
> > Especially with the JDBC Connection URL, users can visit different Hive
> > metastore server instances.
> >
> > 2) I think we can improve the "HiveServer2 Compatibility" section.
> > We need to figure out two compatibility matrices. One is SQL Gateway with
> > different versions of Hive metastore,
> > and the other is different versions of Hive client (e.g., Hive JDBC) with
> > SQL Gateway. We need to clarify
> > what metastore and client versions we support and how users configure the
> > versions.
> >
> > Best,
> > Jark
> >
> >
> > [1]:
> >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> >
> > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang  wrote:
> >
> > > Hi, Jiang.
> > >
> > > Thanks for your feedback!
> > >
> > > > Integrating the Hive ecosystem should not require changing the
> service
> > > interface
> > >
> > > I move the API change to the FLIP-91. But I think it's possible we add
> > more
> > > interfaces to intergrate the new endpoints in the future because every
> > > endpoints's functionality is different. For example, the REST endpoint
> > > doen't support to fetch operation-level logs but the hiveserver2
> endpoint
> > > supports. In this case, we need to modify the shared GatewayService to
> > > support the functionality exposed by the new endpint.
> > >
> > > >  How to support different Hive versions?
> > >
> > > Do you means to support the different HiveServer2 version? The
> > HiveServer2
> > > uses the version to guarantee the compatibility. During the
> openSession,
> > > the client and server will determine the protocol
> version(minimun(client
> > > version, hiveendpoint version)). After that the client and the server
> > uses
> > > the determined version to communicate. In the HiveServer2 endpoint, it
> > > determines how the endpoint deserialize the results and the result
> > schema.
> > > I add a section about HiveServer2 compatiblity.
> > >
> > > > Could you please fully provide its definition including input
> > parameters
> > > and the corresponding return value schema?
> > >
> > > Because we implements the interface exposed by the Hive. So I add the
> > file
> > > link to the HiveServer2 interfaces[1], which contains all input
> > parameters
> > > and the results. Considering the file doesn't contain the output for
> the
> > > Operation, I add the output schema for all the supported Operation in
> the
> > > FLIP, which is not covered in the link. Hope these can address your
> > > question.
> > >
> > > Best,
> > > Shengkai
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > 

Re: [DISCUSS] Planning Flink 1.16

[Zhu Zhu]

+1 for a 5 months release cycle.
+1 target the feature freeze date 1.5 months before the release date. It
can better guard the release date.
Therefore, also +1 for mid August as the feature freeze date of 1.16

Thanks,
Zhu

Jark Wu  于2022年4月28日周四 22:24写道：

> I'm also fine with the end of July for the feature freeze.
>
> Best,
> Jark
>
> On Thu, 28 Apr 2022 at 21:00, Martijn Visser 
> wrote:
>
> > +1 for continuing to strive for a 5 months release cycle.
> >
> > And +1 to have the planned feature freeze mid August, which I would
> propose
> > to have happen on Monday the 15th of August 2022. I would also already
> > state that even though we know this is a holiday period, we should not
> > extend this deadline for that reason :)
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> >
> > On Thu, 28 Apr 2022 at 14:37, Jingsong Li 
> wrote:
> >
> > > Thanks for the check.
> > >
> > > > Continue to strive for a 5 months release cycle and 1.5 months before
> > to
> > > the desired release date
> > >
> > > Sounds good to me!
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, Apr 28, 2022 at 7:06 PM Konstantin Knauf 
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > thank you for the feedback so far. I've checked the length of the
> > > previous
> > > > last release cycles. Up until Flink 1.14, we've actually been
> > incredibly
> > > > good at maintaining a 5 months release cycle (see below).
> > Interestingly,
> > > > the community is officially targeting a 4 months release cycle [1].
> > > >
> > > > - 1.15.0 2022-05-01? (7 months, 2 days?)
> > > > - 1.14.0: 2021-09-29 (4 months, 26 days)
> > > > - 1.13.0: 2021-05-03 (4 months, 23 days)
> > > > - 1.12.0: 2020-12-10 (5 months, 3 days)
> > > > - 1.11.0: 2020-07-07 (4 months, 26 days)
> > > > - 1.10.0: 2020-02-11
> > > >
> > > > The 1.15 release cycle has took significantly longer. In my opinion
> we
> > > > should try to get back into the 5 months cadence with the next
> release.
> > > > Since empirically we always often end up moving the the feature
> freeze
> > > by a
> > > > week or two, and that we often need about a month for release
> testing &
> > > > stabilization and releasing, I don't think, we should move the
> planned
> > > > feature freeze to later than
> > > > *mid August. *
> > > > What do you think:
> > > > 1. Should we continue to strive for a 5 months release cycle (and
> > update
> > > > [1] accordingly)?
> > > > 2. Does it sound reasonable to target a feature freeze date, which is
> > 1.5
> > > > months before to the desired release date?
> > > >
> > > > Cheers,
> > > >
> > > > Konstantin
> > > >
> > > >  [1]
> > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
> > > >
> > > > Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
> > > > jingsongl...@gmail.com>:
> > > >
> > > > > Thanks Konstantin and Chesnay for starting the discussion. And
> thanks
> > > > > Konstantin, Chesnay, Godfrey, Martijn for volunteering.
> > > > >
> > > > > The 1.16 release at the end of July means that there are currently
> > > > > only 3 months to fully commit to 1.16 development, and I'm a little
> > > > > concerned that this is too short a time frame, which may result in
> > > > > some features only reaching a halfway decent state.
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > >
> > > > > On Wed, Apr 27, 2022 at 7:10 PM David Morávek 
> > wrote:
> > > > > >
> > > > > > Thanks Konstantin and Chesnay for starting the discussion and
> > > > > volunteering.
> > > > > > The timeline proposal sounds reasonable :+1:
> > > > > >
> > > > > > Best,
> > > > > > D.
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser <
> > > martijnvis...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks for starting this discussion. I would also volunteer to
> > help
> > > > > out as
> > > > > > > a release manager for the 1.16 release.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn Visser
> > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > https://github.com/MartijnVisser
> > > > > > >
> > > > > > >
> > > > > > > On Tue, 26 Apr 2022 at 13:19, godfrey he 
> > > wrote:
> > > > > > >
> > > > > > > > Hi Konstantin & Chesnay,
> > > > > > > >
> > > > > > > > Thanks for driving this discussion, I am willing to volunteer
> > as
> > > the
> > > > > > > > release manager for 1.16.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Godfrey
> > > > > > > >
> > > > > > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > With Flink 1.15 about to be released, the community has
> > started
> > > > > > > planning
> > > > > > > > &
> > > > > > > > > developing features for the next release, Flink 1.16. As
> > such,
> > > I
> > > > > would
> > > > > > > > like
> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang]

Hi Marijn and LuNing.

Thanks for your feedback!

> The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
which would be used by Flink's SQL Client (or other applications)?

Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename
the name to the "Support SQL Gateway". WDYT?

> From a user perspective, I would have expected that we start with the
REST endpoint before explaining how we would integrate this into Flink. Now
it's quite hard to first understand what we want to offer to users and if
that will be sufficient for a first version.

emmm. Considering that api is basically the operation of some concepts, is
it better to introduce the core concepts first? But I agree you are right
that we should start with the RESt endpoint. I reorganize the content to
introduce the REST first in the public interfaces.

> With Flink 1.15, we're introducing an OpenAPI specification. Can we
also do this straight away for the REST Gateway?

Yes. We will organize the related APIs into OpenAPI specification.

>Should we introduce the REST Gateway as part of Flink's main repository?
>Wouldn't we be better off to maintain this in a separate repository under
>ASF?

I think it's better to intergate the Gateway into the Flink code base. The
reason behind is

1. The Gateway relies on the Flink implementation,  I think we'd better to
maintain it inside the Flink. It really takes us much time to upgrade the
sql-gateway in ververica repo to the latest Flink version.

2. The Gateway is important to the Flink itself. Many users needs the
Gateway to manage the Flink SQL jobs. Actually Hive, Spark both have its
Gateway in its code base.

But I think it's fine to put other utils, e.g. JDBC under the ASF.

> Ideally you would like to be able to support multiple Flink versions
> with one version of the REST Gateway I think?

> Users can upgrade a large number of Flink jobs versions gradually in a
Gateway service.

Because the Gateway itself relies on the Flink inner implementation...I
think we can just use one Gateway per versions. Users can manage the
gateway with other utils.

>There's no mention of Batch or Streaming in this concept. If I recall
>correctly, the current Flink SQL Gateway can only support Batch. How will
>we support Streaming?

> I can imagine that if a user wants to use a REST Gateway, there's also a
> strong need to combine this with a Catalog.

Yes. I add a section about the Usage of the Gateway. Users can use the SQL
do everything in the Gateway, including
- configure the execution parameter, including exectuion mode
- manage the metadata with DDL, e.g. register catalog
- submit the job
...

>Will there be any requirement with JDBC, as there currently is?

In the FLIP-223, we implement the HiveServer2 endpint. Users can use the
hive jdbc to connect to the Flink SQL Gateway.

> Shall we name this option `sql-gateway.session.init-file` and write it
into
the FLIP-91?

Actually we already supports the -i parameters in the sql client. What's
more, Hive also supports the -i parameter in the client side[1].
I think it's fine to move this functionlity to the client rather than
gateway. WDYT?

[1]
https://github.com/apache/hive/blob/c3fa88a1b7d1475f44383fca913aecf9c664bab0/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L321

Best,
Shengkai





LuNing Wang  于2022年4月28日周四 10:04写道：

> > * Should we introduce the REST Gateway as part of Flink's main
> repository?
> Wouldn't we be better off to maintain this in a separate repository under
> ASF? Ideally you would like to be able to support multiple Flink versions
> with one version of the REST Gateway I think?
>
> We would be better off maintaining this in a separate repository. It is
> important to support multiple Flink versions. Users can upgrade a large
> number of Flink jobs versions gradually in a Gateway service.
>
> LuNing Wang  于2022年4月27日周三 17:54写道：
>
> > Hi ShengKai,
> >
> > After I read FLIP-91[1], I want to add an init-file option. Its
> > functionality is the same as option '-i' of Flink SQL Client.
> >
> > When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by
> > this option after SQL Gateway starts every time.
> >
> > Shall we name this option `sql-gateway.session.init-file` and write it
> > into the FLIP-91?
> >
> > Best regards,
> >
> > LuNing Wang
> >
> > [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91
> >
> > Martijn Visser  于2022年4月26日周二 20:32写道：
> >
> >> Hi Shengkai,
> >>
> >> Thanks for opening this discussion. I did a first brief pass over the
> FLIP
> >> and I have a couple of questions/remarks:
> >>
> >> * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
> >> which would be used by Flink's SQL Client (or other applications)?
> >>
> >> * From a user perspective, I would have expected that we start with the
> >> REST endpoint before explaining how we would integrate this into Flink.
> >> Now
> >> it's quite hard to first understand what we want to offer to users 

[VOTE] Apache Flink Table Store 0.1.0, release candidate #2

[Jingsong Li]

Hi everyone,

Please review and vote on the release candidate #2 for the version 0.1.0 of
Apache Flink Table Store, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

**Release Overview**

As an overview, the release consists of the following:
a) Table Store canonical source distribution, to be deployed to the
release repository at dist.apache.org
b) Maven artifacts to be deployed to the Maven Central Repository

**Staging Areas to Review**

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a) and b) can be found in the corresponding dev
repository at dist.apache.org [2]
* All artifacts for c) can be found at the Apache Nexus Repository [3]
* Pre Bundled Binaries Jar can work fine with quick start [4][5]

All artifacts are signed with the key
2C2B6A653B07086B65E4369F7C76245E0A318150 [6]

Other links for your review:
* JIRA release notes [7]
* source code tag "release-0.1.0-rc2" [8]
* PR to update the website Downloads page to include Table Store
links [9]

**Vote Duration**

The voting time will run for at least 72 hours.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Best,
Jingsong Lee

[1] 
https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release
[2] https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.1.0-rc2/
[3] https://repository.apache.org/content/repositories/orgapacheflink-1502/
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1502/org/apache/flink/flink-table-store-dist/0.1.0/flink-table-store-dist-0.1.0.jar
[5] 
https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/
[6] https://dist.apache.org/repos/dist/release/flink/KEYS
[7] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351234
[8] https://github.com/apache/flink-table-store/tree/release-0.1.0-rc2
[9] https://github.com/apache/flink-web/pull/531

Re: Failed Unit Test on Master Branch

[Guowei Ma]

Hi Haizhou

I ran the test and there is no problem.
And commit is "d940af688be90c92ce4f8b9ca883f6753c94aa0f"

Best,
Guowei


On Fri, Apr 29, 2022 at 5:39 AM Haizhou Zhao 
wrote:

> Hello Flink Community,
>
> I was encountering some unit test failure in the flink-avro sub-module when
> I tried to pull down the master branch and build.
>
> Here is the command I ran:
>
> mvn clean package -pl flink-formats/flink-avro
>
> Here is the test that fails:
>
> https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/test/java/org/apache/flink/formats/avro/AvroRowDeSerializationSchemaTest.java#L178
>
> Here is the exception that was thrown:
> [ERROR]
>
> org.apache.flink.formats.avro.AvroRowDeSerializationSchemaTest.testGenericDeserializeSeveralTimes
>  Time elapsed: 0.008 s  <<< ERROR!
> java.io.IOException: Failed to deserialize Avro record.
> ...
>
> Here is the latest commit of the HEAD I pulled:
> commit c5430e2e5d4eeb0aba14ce3ea8401747afe0182d (HEAD -> master,
> oss/master)
>
> Can someone confirm this is indeed a problem on the master branch? If yes,
> any suggestions on fixing it?
>
> Thank you,
> Haizhou Zhao
>

[jira] [Created] (FLINK-27449) The comment is lost when printing table schema

[Yuan Huang (Jira)]

Yuan Huang  created FLINK-27449:
---

 Summary: The comment is lost when printing table schema
 Key: FLINK-27449
 URL: https://issues.apache.org/jira/browse/FLINK-27449
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.14.4
Reporter: Yuan Huang 
 Attachments: test_result.png

User reported that the comment was lost when printing the table schema.

So this test will fail:
```

@Test
public void test() {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
DataStream dataStream = env.fromElements("Alice", "Bob", "John");
Schema.Builder builder = Schema.newBuilder();
builder.column("f0", DataTypes.of(String.class)).withComment("this is a 
comment");

Table table = tableEnv.fromDataStream(dataStream, 
builder.build()).as("user_name");
table.getResolvedSchema();
table.printSchema();
String expected = "(\n `user_name` STRING COMMENT 'this is a comment'\n)";
Assert.assertEquals(expected, table.getResolvedSchema().toString());
}

```

 

!test_result.png!

 

Is it a bug or just meets our expectations?

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Failed Unit Test on Master Branch

[Haizhou Zhao]

Hello Flink Community,

I was encountering some unit test failure in the flink-avro sub-module when
I tried to pull down the master branch and build.

Here is the command I ran:

mvn clean package -pl flink-formats/flink-avro

Here is the test that fails:
https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/test/java/org/apache/flink/formats/avro/AvroRowDeSerializationSchemaTest.java#L178

Here is the exception that was thrown:
[ERROR]
org.apache.flink.formats.avro.AvroRowDeSerializationSchemaTest.testGenericDeserializeSeveralTimes
 Time elapsed: 0.008 s  <<< ERROR!
java.io.IOException: Failed to deserialize Avro record.
...

Here is the latest commit of the HEAD I pulled:
commit c5430e2e5d4eeb0aba14ce3ea8401747afe0182d (HEAD -> master, oss/master)

Can someone confirm this is indeed a problem on the master branch? If yes,
any suggestions on fixing it?

Thank you,
Haizhou Zhao

Re: [DISCUSS] FLIP-227: Support overdraft buffer

[Anton Kalashnikov]

Hi,

You are right about LegacySource(is the same about BoundedStreamTask?). 
I am not really sure about hardcoded limit. We need to think about that.
Theoretically, when we create a task we know should it use overdraft or 
not but I am not really sure how we can use this knoweledge yet.


28.04.2022 06:39, rui fan пишет:

Hi Anton Kalashnikov,

Thanks for your very clear reply, I think you are totally right.

The 'maxBuffersNumber - buffersInUseNumber' can be used as the
overdraft buffer, it won't need the new buffer configuration.Flink users
can turn up the maxBuffersNumber to control the overdraft buffer size.

Also, I‘d like to add some information. For safety, we should limit the
maximum number of overdraft segments that each LocalBufferPool
can apply for.

Why do we limit it?
Some operators don't check the `recordWriter.isAvailable` during
processing records, such as LegacySource. I have mentioned it in
FLINK-26759 [1]. I'm not sure if there are other cases.

If don't add the limitation, the LegacySource will use up all remaining
memory in the NetworkBufferPool when the backpressure is severe.

How to limit it?
I prefer to hard code the `maxOverdraftBuffers=numberOfSubpartitions`
in the constructor of LocalBufferPool. The maxOverdraftBuffers is just
for safety, and it should be enough for most flink jobs. Or we can set
`maxOverdraftBuffers=Math.max(numberOfSubpartitions, 10)` to handle
some jobs of low parallelism.

Also if user don't enable the Unaligned Checkpoint, we can set
maxOverdraftBuffers=0 in the constructor of LocalBufferPool. Because
the overdraft isn't useful for the Aligned Checkpoint.

Please correct me if I'm wrong. Thanks a lot.

[1] https://issues.apache.org/jira/browse/FLINK-26759

Best wishes
fanrui

On Thu, Apr 28, 2022 at 12:29 AM Anton Kalashnikov 
wrote:


Hi fanrui,

Thanks for creating the FLIP.

In general, I think the overdraft is good idea and it should help in
described above cases. Here are my thoughts about configuration:

Please, correct me if I am wrong but as I understand right now we have
following calculation.

maxBuffersNumber(per TaskManager) = Network memory(calculated via
taskmanager.memory.network.fraction, taskmanager.memory.network.min,
taskmanager.memory.network.max and total memory size) /
taskmanager.memory.segment-size.

requiredBuffersNumber(per TaskManager) = (exclusive buffers *
parallelism + floating buffers) * subtasks number in TaskManager

buffersInUseNumber = real number of buffers which used at current
moment(always <= requiredBuffersNumber)

Ideally requiredBuffersNumber should be equal to maxBuffersNumber which
allows Flink work predictibly. But if requiredBuffersNumber >
maxBuffersNumber sometimes it is also fine(but not good) since not all
required buffers really mandatory(e.g. it is ok if Flink can not
allocate floating buffers)

But if maxBuffersNumber > requiredBuffersNumber, as I understand Flink
just never use these leftovers buffers(maxBuffersNumber -
requiredBuffersNumber). Which I propose to use. ( we can actualy use
even difference 'requiredBuffersNumber - buffersInUseNumber' since if
one TaskManager contains several operators including 'window' which can
temporally borrow buffers from the global pool).

My proposal, more specificaly(it relates only to requesting buffers
during processing single record while switching to unavalability between
records should be the same as we have it now):

* If one more buffer requested but maxBuffersPerChannel reached, then
just ignore this limitation and allocate this buffers from any
place(from LocalBufferPool if it has something yet otherwise from
NetworkBufferPool)

* If LocalBufferPool exceeds limit, then temporally allocate it from
NetworkBufferPool while it has something to allocate


Maybe I missed something and this solution won't work, but I like it
since on the one hand, it work from the scratch without any
configuration, on the other hand, it can be configuration by changing
proportion of maxBuffersNumber and requiredBuffersNumber.

The last thing that I want to say, I don't really want to implement new
configuration since even now it is not clear how to correctly configure
network buffers with existing configuration and I don't want to
complicate it, especially if it will be possible to resolve the problem
automatically(as described above).


So is my understanding about network memory/buffers correct?

--

Best regards,
Anton Kalashnikov

27.04.2022 07:46, rui fan пишет:

Hi everyone,

Unaligned Checkpoint (FLIP-76 [1]) is a major feature of Flink. It
effectively solves the problem of checkpoint timeout or slow
checkpoint when backpressure is severe.

We found that UC(Unaligned Checkpoint) does not work well when the
back pressure is severe and multiple output buffers are required to
process a single record. FLINK-14396 [2] also mentioned this issue
before. So we propose the overdraft buffer to solve it.

I created FLINK-26762[3] and FLIP-227[4] to detail the overdraft
buffer 

[jira] [Created] (FLINK-27447) Implement last-state with ZooKeeper HA

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27447:
-

 Summary: Implement last-state with ZooKeeper HA
 Key: FLINK-27447
 URL: https://issues.apache.org/jira/browse/FLINK-27447
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27448) Enable standalone mode for old Flink versions

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27448:
-

 Summary: Enable standalone mode for old Flink versions
 Key: FLINK-27448
 URL: https://issues.apache.org/jira/browse/FLINK-27448
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27445) Create Flink Standalone Service

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27445:
-

 Summary: Create Flink Standalone Service
 Key: FLINK-27445
 URL: https://issues.apache.org/jira/browse/FLINK-27445
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27446) Introduce Standalone Mode

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27446:
-

 Summary: Introduce Standalone Mode
 Key: FLINK-27446
 URL: https://issues.apache.org/jira/browse/FLINK-27446
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27444) Create standalone mode ClusterDescriptor

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27444:
-

 Summary: Create standalone mode ClusterDescriptor
 Key: FLINK-27444
 URL: https://issues.apache.org/jira/browse/FLINK-27444
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27443) Add standalone mode parameters and decorators

[Usamah Jassat (Jira)]

Usamah Jassat created FLINK-27443:
-

 Summary: Add standalone mode parameters and decorators 
 Key: FLINK-27443
 URL: https://issues.apache.org/jira/browse/FLINK-27443
 Project: Flink
  Issue Type: Sub-task
Reporter: Usamah Jassat


Add parameters and decorators  to create JM and TM deployments in standalone 
mode



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Planning Flink 1.16

[Jark Wu]

I'm also fine with the end of July for the feature freeze.

Best,
Jark

On Thu, 28 Apr 2022 at 21:00, Martijn Visser  wrote:

> +1 for continuing to strive for a 5 months release cycle.
>
> And +1 to have the planned feature freeze mid August, which I would propose
> to have happen on Monday the 15th of August 2022. I would also already
> state that even though we know this is a holiday period, we should not
> extend this deadline for that reason :)
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
>
> On Thu, 28 Apr 2022 at 14:37, Jingsong Li  wrote:
>
> > Thanks for the check.
> >
> > > Continue to strive for a 5 months release cycle and 1.5 months before
> to
> > the desired release date
> >
> > Sounds good to me!
> >
> > Best,
> > Jingsong
> >
> > On Thu, Apr 28, 2022 at 7:06 PM Konstantin Knauf 
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > thank you for the feedback so far. I've checked the length of the
> > previous
> > > last release cycles. Up until Flink 1.14, we've actually been
> incredibly
> > > good at maintaining a 5 months release cycle (see below).
> Interestingly,
> > > the community is officially targeting a 4 months release cycle [1].
> > >
> > > - 1.15.0 2022-05-01? (7 months, 2 days?)
> > > - 1.14.0: 2021-09-29 (4 months, 26 days)
> > > - 1.13.0: 2021-05-03 (4 months, 23 days)
> > > - 1.12.0: 2020-12-10 (5 months, 3 days)
> > > - 1.11.0: 2020-07-07 (4 months, 26 days)
> > > - 1.10.0: 2020-02-11
> > >
> > > The 1.15 release cycle has took significantly longer. In my opinion we
> > > should try to get back into the 5 months cadence with the next release.
> > > Since empirically we always often end up moving the the feature freeze
> > by a
> > > week or two, and that we often need about a month for release testing &
> > > stabilization and releasing, I don't think, we should move the planned
> > > feature freeze to later than
> > > *mid August. *
> > > What do you think:
> > > 1. Should we continue to strive for a 5 months release cycle (and
> update
> > > [1] accordingly)?
> > > 2. Does it sound reasonable to target a feature freeze date, which is
> 1.5
> > > months before to the desired release date?
> > >
> > > Cheers,
> > >
> > > Konstantin
> > >
> > >  [1]
> > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
> > >
> > > Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
> > > jingsongl...@gmail.com>:
> > >
> > > > Thanks Konstantin and Chesnay for starting the discussion. And thanks
> > > > Konstantin, Chesnay, Godfrey, Martijn for volunteering.
> > > >
> > > > The 1.16 release at the end of July means that there are currently
> > > > only 3 months to fully commit to 1.16 development, and I'm a little
> > > > concerned that this is too short a time frame, which may result in
> > > > some features only reaching a halfway decent state.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > >
> > > > On Wed, Apr 27, 2022 at 7:10 PM David Morávek 
> wrote:
> > > > >
> > > > > Thanks Konstantin and Chesnay for starting the discussion and
> > > > volunteering.
> > > > > The timeline proposal sounds reasonable :+1:
> > > > >
> > > > > Best,
> > > > > D.
> > > > >
> > > > > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser <
> > martijnvis...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks for starting this discussion. I would also volunteer to
> help
> > > > out as
> > > > > > a release manager for the 1.16 release.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn Visser
> > > > > > https://twitter.com/MartijnVisser82
> > > > > > https://github.com/MartijnVisser
> > > > > >
> > > > > >
> > > > > > On Tue, 26 Apr 2022 at 13:19, godfrey he 
> > wrote:
> > > > > >
> > > > > > > Hi Konstantin & Chesnay,
> > > > > > >
> > > > > > > Thanks for driving this discussion, I am willing to volunteer
> as
> > the
> > > > > > > release manager for 1.16.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Godfrey
> > > > > > >
> > > > > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > With Flink 1.15 about to be released, the community has
> started
> > > > > > planning
> > > > > > > &
> > > > > > > > developing features for the next release, Flink 1.16. As
> such,
> > I
> > > > would
> > > > > > > like
> > > > > > > > to start a discussion around managing this release.
> > > > > > > >
> > > > > > > > Specifically, Chesnay & myself would like to volunteer as
> > release
> > > > > > > managers.
> > > > > > > > Our focus as release managers would be
> > > > > > > > * to propose a release timeline
> > > > > > > > * to provide an overview of all ongoing development threads
> and
> > > > ideally
> > > > > > > > their current status to the community
> > > > > > > > * to keep an eye on build stability
> > > > > > > > * facilitate release testing
> > > > > > > > * to do 

[jira] [Created] (FLINK-27442) Module flink-sql-avro-confluent-registry does not configure Confluent repo

[Nicolaus Weidner (Jira)]

Nicolaus Weidner created FLINK-27442:


 Summary: Module flink-sql-avro-confluent-registry does not 
configure Confluent repo
 Key: FLINK-27442
 URL: https://issues.apache.org/jira/browse/FLINK-27442
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.15.0
Reporter: Nicolaus Weidner


flink-sql-avro-confluent-registry depends on org.apache.kafka:kafka-clients, 
which is not available in Maven Central, but only in the Confluent repo. 
However, it does not configure this repo. This causes the build to fail for me 
locally with the following exception:

{code:java}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) on project flink-sql-avro-confluent-registry: Error 
resolving project artifact: Could not transfer artifact 
org.apache.kafka:kafka-clients:pom:6.2.2-ccs from/to : Not 
authorized , ReasonPhrase: . for project 
org.apache.kafka:kafka-clients:jar:6.2.2-ccs -> [Help 1]
{code}

This may be build order dependent, but the module should probably configure the 
repo to be safe, like done elsewhere: 
https://github.com/apache/flink/blob/dd48d058c6b745f505870836048284a76a23f7cc/flink-end-to-end-tests/flink-confluent-schema-registry/pom.xml#L36-L41

Looks like this is the case since 1.12 at least.




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Keep the Community Member Information Up-to-Date

[tison]

Hi Xintong,

Thanks for starting this discussion.

+0 to replace the information with link to
https://projects.apache.org/committee.html?flink.
+1 to add such a link.

My opinion is that we the community doesn't have to keep the page up to
date since Apache has a member page[1]
that isn't up to date also.

We can add one line to redirect to the whole list so that those who are
"lazy" to add themselves on the page
don't have to do it. And keep the table so that those who are proud to
announce their membership or trying a commit
with their commit access can do.

Best,
tison.

[1] https://www.apache.org/foundation/members.html


Xintong Song  于2022年4月28日周四 21:26写道：

> >
> > Personally I'm tempted to just link to
> > https://projects.apache.org/committee.html?flink, if at all.
> >
>
> Despite its fashionless look, I'm thinking the same thing...
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Apr 28, 2022 at 8:41 PM Jingsong Li 
> wrote:
>
> > One value is that this page has avatars. :-)
> >
> > Best,
> > Jingsong
> >
> > On Thu, Apr 28, 2022 at 8:27 PM Chesnay Schepler 
> > wrote:
> > >
> > > Personally I'm tempted to just link to
> > > https://projects.apache.org/committee.html?flink, if at all.
> > >
> > > I'm not sure overall whether this listing really provides value in the
> > > first place.
> > >
> > > On 28/04/2022 13:58, Xintong Song wrote:
> > > > Hi Flink Committers & PMC members,
> > > >
> > > > I just noticed that the list of community members on our website [1]
> is
> > > > quite out-of-date. According to the ASF roster [2], this project
> > currently
> > > > has 87 committers, 39 of which are PMC members. However, there's only
> > 62
> > > > people listed on our website, and many (e.g., me) have outdated
> roles.
> > > >
> > > > I believe the list on the website is supposed to be updated by each
> new
> > > > committer / PMC member. I remember reading somewhere that suggested
> new
> > > > committers to add themselves to this list as the first trying-out for
> > > > merging changes. Unfortunately I couldn't find it anymore.
> > > >
> > > > Do you think we should keep the page manually updated, or shall we
> > > > investigate some way to keep it automatically synchronized?
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > > [1] https://flink.apache.org/community.html
> > > >
> > > > [2] https://whimsy.apache.org/roster/committee/flink
> > > >
> > >
> >
>

[jira] [Created] (FLINK-27441) Scrollbar is missing for particular UI elements (Accumulators, Backpressure, Watermarks)

[Ferenc Csaky (Jira)]

Ferenc Csaky created FLINK-27441:


 Summary: Scrollbar is missing for particular UI elements 
(Accumulators, Backpressure, Watermarks)
 Key: FLINK-27441
 URL: https://issues.apache.org/jira/browse/FLINK-27441
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Web Frontend
Affects Versions: 1.14.3, 1.15.0
Reporter: Ferenc Csaky


The angular version bump introduced a bug, where for {{nzScroll}} does not 
support percentage in CSS calc, so the scrollbar will be invisible. There is an 
easy workaround, the linked Angular discussion covers it.

Angular issue: https://github.com/NG-ZORRO/ng-zorro-antd/issues/3090



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] Keep the Community Member Information Up-to-Date

[Xintong Song]

>
> Personally I'm tempted to just link to
> https://projects.apache.org/committee.html?flink, if at all.
>

Despite its fashionless look, I'm thinking the same thing...


Thank you~

Xintong Song



On Thu, Apr 28, 2022 at 8:41 PM Jingsong Li  wrote:

> One value is that this page has avatars. :-)
>
> Best,
> Jingsong
>
> On Thu, Apr 28, 2022 at 8:27 PM Chesnay Schepler 
> wrote:
> >
> > Personally I'm tempted to just link to
> > https://projects.apache.org/committee.html?flink, if at all.
> >
> > I'm not sure overall whether this listing really provides value in the
> > first place.
> >
> > On 28/04/2022 13:58, Xintong Song wrote:
> > > Hi Flink Committers & PMC members,
> > >
> > > I just noticed that the list of community members on our website [1] is
> > > quite out-of-date. According to the ASF roster [2], this project
> currently
> > > has 87 committers, 39 of which are PMC members. However, there's only
> 62
> > > people listed on our website, and many (e.g., me) have outdated roles.
> > >
> > > I believe the list on the website is supposed to be updated by each new
> > > committer / PMC member. I remember reading somewhere that suggested new
> > > committers to add themselves to this list as the first trying-out for
> > > merging changes. Unfortunately I couldn't find it anymore.
> > >
> > > Do you think we should keep the page manually updated, or shall we
> > > investigate some way to keep it automatically synchronized?
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1] https://flink.apache.org/community.html
> > >
> > > [2] https://whimsy.apache.org/roster/committee/flink
> > >
> >
>

Re: [DISCUSS] Planning Flink 1.16

[Martijn Visser]

+1 for continuing to strive for a 5 months release cycle.

And +1 to have the planned feature freeze mid August, which I would propose
to have happen on Monday the 15th of August 2022. I would also already
state that even though we know this is a holiday period, we should not
extend this deadline for that reason :)

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser


On Thu, 28 Apr 2022 at 14:37, Jingsong Li  wrote:

> Thanks for the check.
>
> > Continue to strive for a 5 months release cycle and 1.5 months before to
> the desired release date
>
> Sounds good to me!
>
> Best,
> Jingsong
>
> On Thu, Apr 28, 2022 at 7:06 PM Konstantin Knauf 
> wrote:
> >
> > Hi everyone,
> >
> > thank you for the feedback so far. I've checked the length of the
> previous
> > last release cycles. Up until Flink 1.14, we've actually been incredibly
> > good at maintaining a 5 months release cycle (see below). Interestingly,
> > the community is officially targeting a 4 months release cycle [1].
> >
> > - 1.15.0 2022-05-01? (7 months, 2 days?)
> > - 1.14.0: 2021-09-29 (4 months, 26 days)
> > - 1.13.0: 2021-05-03 (4 months, 23 days)
> > - 1.12.0: 2020-12-10 (5 months, 3 days)
> > - 1.11.0: 2020-07-07 (4 months, 26 days)
> > - 1.10.0: 2020-02-11
> >
> > The 1.15 release cycle has took significantly longer. In my opinion we
> > should try to get back into the 5 months cadence with the next release.
> > Since empirically we always often end up moving the the feature freeze
> by a
> > week or two, and that we often need about a month for release testing &
> > stabilization and releasing, I don't think, we should move the planned
> > feature freeze to later than
> > *mid August. *
> > What do you think:
> > 1. Should we continue to strive for a 5 months release cycle (and update
> > [1] accordingly)?
> > 2. Does it sound reasonable to target a feature freeze date, which is 1.5
> > months before to the desired release date?
> >
> > Cheers,
> >
> > Konstantin
> >
> >  [1]
> https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
> >
> > Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
> > jingsongl...@gmail.com>:
> >
> > > Thanks Konstantin and Chesnay for starting the discussion. And thanks
> > > Konstantin, Chesnay, Godfrey, Martijn for volunteering.
> > >
> > > The 1.16 release at the end of July means that there are currently
> > > only 3 months to fully commit to 1.16 development, and I'm a little
> > > concerned that this is too short a time frame, which may result in
> > > some features only reaching a halfway decent state.
> > >
> > > Best,
> > > Jingsong
> > >
> > >
> > > On Wed, Apr 27, 2022 at 7:10 PM David Morávek  wrote:
> > > >
> > > > Thanks Konstantin and Chesnay for starting the discussion and
> > > volunteering.
> > > > The timeline proposal sounds reasonable :+1:
> > > >
> > > > Best,
> > > > D.
> > > >
> > > > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser <
> martijnvis...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for starting this discussion. I would also volunteer to help
> > > out as
> > > > > a release manager for the 1.16 release.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn Visser
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
> > > > >
> > > > >
> > > > > On Tue, 26 Apr 2022 at 13:19, godfrey he 
> wrote:
> > > > >
> > > > > > Hi Konstantin & Chesnay,
> > > > > >
> > > > > > Thanks for driving this discussion, I am willing to volunteer as
> the
> > > > > > release manager for 1.16.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Godfrey
> > > > > >
> > > > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > With Flink 1.15 about to be released, the community has started
> > > > > planning
> > > > > > &
> > > > > > > developing features for the next release, Flink 1.16. As such,
> I
> > > would
> > > > > > like
> > > > > > > to start a discussion around managing this release.
> > > > > > >
> > > > > > > Specifically, Chesnay & myself would like to volunteer as
> release
> > > > > > managers.
> > > > > > > Our focus as release managers would be
> > > > > > > * to propose a release timeline
> > > > > > > * to provide an overview of all ongoing development threads and
> > > ideally
> > > > > > > their current status to the community
> > > > > > > * to keep an eye on build stability
> > > > > > > * facilitate release testing
> > > > > > > * to do the actual release incl. communication (blog post,
> etc.)
> > > > > > >
> > > > > > > Is anyone else interested in acting as a release manager for
> Flink
> > > > > 1.16?
> > > > > > If
> > > > > > > so, we are happy to make this a joint effort.
> > > > > > >
> > > > > > > Besides the question of who will act as a release manager, I
> > > think, we
> > > > > > can
> > > > > > > already use this thread to align on a timeline. 

Re: [DISCUSS] Keep the Community Member Information Up-to-Date

[Jingsong Li]

One value is that this page has avatars. :-)

Best,
Jingsong

On Thu, Apr 28, 2022 at 8:27 PM Chesnay Schepler  wrote:
>
> Personally I'm tempted to just link to
> https://projects.apache.org/committee.html?flink, if at all.
>
> I'm not sure overall whether this listing really provides value in the
> first place.
>
> On 28/04/2022 13:58, Xintong Song wrote:
> > Hi Flink Committers & PMC members,
> >
> > I just noticed that the list of community members on our website [1] is
> > quite out-of-date. According to the ASF roster [2], this project currently
> > has 87 committers, 39 of which are PMC members. However, there's only 62
> > people listed on our website, and many (e.g., me) have outdated roles.
> >
> > I believe the list on the website is supposed to be updated by each new
> > committer / PMC member. I remember reading somewhere that suggested new
> > committers to add themselves to this list as the first trying-out for
> > merging changes. Unfortunately I couldn't find it anymore.
> >
> > Do you think we should keep the page manually updated, or shall we
> > investigate some way to keep it automatically synchronized?
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1] https://flink.apache.org/community.html
> >
> > [2] https://whimsy.apache.org/roster/committee/flink
> >
>

Re: [DISCUSS] Planning Flink 1.16

[Jingsong Li]

Thanks for the check.

> Continue to strive for a 5 months release cycle and 1.5 months before to the 
> desired release date

Sounds good to me!

Best,
Jingsong

On Thu, Apr 28, 2022 at 7:06 PM Konstantin Knauf  wrote:
>
> Hi everyone,
>
> thank you for the feedback so far. I've checked the length of the previous
> last release cycles. Up until Flink 1.14, we've actually been incredibly
> good at maintaining a 5 months release cycle (see below). Interestingly,
> the community is officially targeting a 4 months release cycle [1].
>
> - 1.15.0 2022-05-01? (7 months, 2 days?)
> - 1.14.0: 2021-09-29 (4 months, 26 days)
> - 1.13.0: 2021-05-03 (4 months, 23 days)
> - 1.12.0: 2020-12-10 (5 months, 3 days)
> - 1.11.0: 2020-07-07 (4 months, 26 days)
> - 1.10.0: 2020-02-11
>
> The 1.15 release cycle has took significantly longer. In my opinion we
> should try to get back into the 5 months cadence with the next release.
> Since empirically we always often end up moving the the feature freeze by a
> week or two, and that we often need about a month for release testing &
> stabilization and releasing, I don't think, we should move the planned
> feature freeze to later than
> *mid August. *
> What do you think:
> 1. Should we continue to strive for a 5 months release cycle (and update
> [1] accordingly)?
> 2. Does it sound reasonable to target a feature freeze date, which is 1.5
> months before to the desired release date?
>
> Cheers,
>
> Konstantin
>
>  [1] https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
>
> Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
> jingsongl...@gmail.com>:
>
> > Thanks Konstantin and Chesnay for starting the discussion. And thanks
> > Konstantin, Chesnay, Godfrey, Martijn for volunteering.
> >
> > The 1.16 release at the end of July means that there are currently
> > only 3 months to fully commit to 1.16 development, and I'm a little
> > concerned that this is too short a time frame, which may result in
> > some features only reaching a halfway decent state.
> >
> > Best,
> > Jingsong
> >
> >
> > On Wed, Apr 27, 2022 at 7:10 PM David Morávek  wrote:
> > >
> > > Thanks Konstantin and Chesnay for starting the discussion and
> > volunteering.
> > > The timeline proposal sounds reasonable :+1:
> > >
> > > Best,
> > > D.
> > >
> > > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser  > >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks for starting this discussion. I would also volunteer to help
> > out as
> > > > a release manager for the 1.16 release.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn Visser
> > > > https://twitter.com/MartijnVisser82
> > > > https://github.com/MartijnVisser
> > > >
> > > >
> > > > On Tue, 26 Apr 2022 at 13:19, godfrey he  wrote:
> > > >
> > > > > Hi Konstantin & Chesnay,
> > > > >
> > > > > Thanks for driving this discussion, I am willing to volunteer as the
> > > > > release manager for 1.16.
> > > > >
> > > > >
> > > > > Best,
> > > > > Godfrey
> > > > >
> > > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > With Flink 1.15 about to be released, the community has started
> > > > planning
> > > > > &
> > > > > > developing features for the next release, Flink 1.16. As such, I
> > would
> > > > > like
> > > > > > to start a discussion around managing this release.
> > > > > >
> > > > > > Specifically, Chesnay & myself would like to volunteer as release
> > > > > managers.
> > > > > > Our focus as release managers would be
> > > > > > * to propose a release timeline
> > > > > > * to provide an overview of all ongoing development threads and
> > ideally
> > > > > > their current status to the community
> > > > > > * to keep an eye on build stability
> > > > > > * facilitate release testing
> > > > > > * to do the actual release incl. communication (blog post, etc.)
> > > > > >
> > > > > > Is anyone else interested in acting as a release manager for Flink
> > > > 1.16?
> > > > > If
> > > > > > so, we are happy to make this a joint effort.
> > > > > >
> > > > > > Besides the question of who will act as a release manager, I
> > think, we
> > > > > can
> > > > > > already use this thread to align on a timeline. For collecting
> > features
> > > > > and
> > > > > > everything else, we would start a dedicated threads shortly.
> > > > > >
> > > > > > Given Flink 1.15 will be released in the next days, and aiming for
> > a 4
> > > > > > months release cycle including stabilization, this would mean
> > *feature
> > > > > > freeze at the end of July*. The exact date could be determined
> > later.
> > > > Any
> > > > > > thoughts on the timeline.?
> > > > > >
> > > > > > Looking forward to your thoughts!
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Chesnay & Konstantin
> > > > >
> > > >
> >

Re: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

[Jing Zhang]

+1 (binding)

Best,
Jing Zhang

Martijn Visser  于2022年4月28日周四 20:11写道：

> +1 (Binding)
>
> On Thu, 28 Apr 2022 at 13:40, ron  wrote:
>
> > +1
> >
> > Best,
> > Ron
> >
> >
> > > -原始邮件-
> > > 发件人: "Jark Wu" 
> > > 发送时间: 2022-04-28 15:46:22 (星期四)
> > > 收件人: dev 
> > > 抄送:
> > > 主题: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for
> > migrating Hive dialect
> > >
> > > +1 (binding)
> > >
> > > Thank Yuxia, for driving this work.
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 28 Apr 2022 at 11:58, Jingsong Li 
> > wrote:
> > >
> > > > +1 (Binding)
> > > >
> > > > A very good step to move forward.
> > > >
> > > > Best.
> > > > Jingsong
> > > >
> > > > On Wed, Apr 27, 2022 at 9:33 PM yuxia 
> > wrote:
> > > > >
> > > > > Hi, everyone
> > > > >
> > > > > Thanks all for attention to FLIP-216: Introduce pluggable dialect
> and
> > > > plan for migrating Hive dialect [1] and participation in the
> > discussion in
> > > > the mail thread [2].
> > > > >
> > > > > I'd like to start a vote for it. The vote will be open for at least
> > 72
> > > > hours unless there is an objection or not enough votes.
> > > > >
> > > > > [1] [
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > > > |
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > > > ]
> > > > > [2] [
> > https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo
> > > > | https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo ]
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > >
> >
> >
> > --
> > Best,
> > Ron
> >
>

Re: [DISCUSS] Keep the Community Member Information Up-to-Date

[Chesnay Schepler]

Personally I'm tempted to just link to 
https://projects.apache.org/committee.html?flink, if at all.


I'm not sure overall whether this listing really provides value in the 
first place.


On 28/04/2022 13:58, Xintong Song wrote:

Hi Flink Committers & PMC members,

I just noticed that the list of community members on our website [1] is
quite out-of-date. According to the ASF roster [2], this project currently
has 87 committers, 39 of which are PMC members. However, there's only 62
people listed on our website, and many (e.g., me) have outdated roles.

I believe the list on the website is supposed to be updated by each new
committer / PMC member. I remember reading somewhere that suggested new
committers to add themselves to this list as the first trying-out for
merging changes. Unfortunately I couldn't find it anymore.

Do you think we should keep the page manually updated, or shall we
investigate some way to keep it automatically synchronized?

Thank you~

Xintong Song


[1] https://flink.apache.org/community.html

[2] https://whimsy.apache.org/roster/committee/flink

Re: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

[Martijn Visser]

+1 (Binding)

On Thu, 28 Apr 2022 at 13:40, ron  wrote:

> +1
>
> Best,
> Ron
>
>
> > -原始邮件-
> > 发件人: "Jark Wu" 
> > 发送时间: 2022-04-28 15:46:22 (星期四)
> > 收件人: dev 
> > 抄送:
> > 主题: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for
> migrating Hive dialect
> >
> > +1 (binding)
> >
> > Thank Yuxia, for driving this work.
> >
> > Best,
> > Jark
> >
> > On Thu, 28 Apr 2022 at 11:58, Jingsong Li 
> wrote:
> >
> > > +1 (Binding)
> > >
> > > A very good step to move forward.
> > >
> > > Best.
> > > Jingsong
> > >
> > > On Wed, Apr 27, 2022 at 9:33 PM yuxia 
> wrote:
> > > >
> > > > Hi, everyone
> > > >
> > > > Thanks all for attention to FLIP-216: Introduce pluggable dialect and
> > > plan for migrating Hive dialect [1] and participation in the
> discussion in
> > > the mail thread [2].
> > > >
> > > > I'd like to start a vote for it. The vote will be open for at least
> 72
> > > hours unless there is an objection or not enough votes.
> > > >
> > > > [1] [
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > > |
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > > ]
> > > > [2] [
> https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo
> > > | https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo ]
> > > >
> > > >
> > > > Best regards,
> > > > Yuxia
> > >
>
>
> --
> Best,
> Ron
>

[DISCUSS] Keep the Community Member Information Up-to-Date

[Xintong Song]

Hi Flink Committers & PMC members,

I just noticed that the list of community members on our website [1] is
quite out-of-date. According to the ASF roster [2], this project currently
has 87 committers, 39 of which are PMC members. However, there's only 62
people listed on our website, and many (e.g., me) have outdated roles.

I believe the list on the website is supposed to be updated by each new
committer / PMC member. I remember reading somewhere that suggested new
committers to add themselves to this list as the first trying-out for
merging changes. Unfortunately I couldn't find it anymore.

Do you think we should keep the page manually updated, or shall we
investigate some way to keep it automatically synchronized?

Thank you~

Xintong Song


[1] https://flink.apache.org/community.html

[2] https://whimsy.apache.org/roster/committee/flink

Re: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

[ron]

+1

Best,
Ron


> -原始邮件-
> 发件人: "Jark Wu" 
> 发送时间: 2022-04-28 15:46:22 (星期四)
> 收件人: dev 
> 抄送: 
> 主题: Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating 
> Hive dialect
> 
> +1 (binding)
> 
> Thank Yuxia, for driving this work.
> 
> Best,
> Jark
> 
> On Thu, 28 Apr 2022 at 11:58, Jingsong Li  wrote:
> 
> > +1 (Binding)
> >
> > A very good step to move forward.
> >
> > Best.
> > Jingsong
> >
> > On Wed, Apr 27, 2022 at 9:33 PM yuxia  wrote:
> > >
> > > Hi, everyone
> > >
> > > Thanks all for attention to FLIP-216: Introduce pluggable dialect and
> > plan for migrating Hive dialect [1] and participation in the discussion in
> > the mail thread [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > hours unless there is an objection or not enough votes.
> > >
> > > [1] [
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > |
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> > ]
> > > [2] [ https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo
> > | https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo ]
> > >
> > >
> > > Best regards,
> > > Yuxia
> >


--
Best,
Ron

Re: [DISCUSS] DockerHub repository maintainers

[Xintong Song]

I agree with you that doing QA for the image after the release has been
finalized doesn't feel right. IIUR, that is mostly because official image
PR needs 1) the binary release being deployed and propagated and 2) the
corresponding git commit being specified. I'm not completely sure about
this. Maybe we can improve the process by investigating more about the
feasibility of pre-verifying an official image PR before finalizing the
release. It's definitely a good thing to do if possible.

I also agree that QA from DockerHub folks is valuable to us.

I'm not against publishing official-images, and I'm not against working
closely with the DockerHub folks to improve the process of delivering the
official image. However, I don't think these should become reasons that we
don't release our own apache/flink images.

Taking the 1.12.0 as an example, admittedly it would be nice for us to
comply with the DockerHub folks' standards and not have a
just-for-kubernetes command in our entrypoint. However, this is IMO far
less important compared to delivering the image to our users timely. I
guess that's where the DockerHub folks and us have different
priorities, and that's why I think we should have a path that is fully
controlled by this community to deliver images. We could take their
valuable inputs and improve afterwards. Actually, that's what we did for
1.12.0 by starting to release to apache/flink.

Thank you~

Xintong Song



On Thu, Apr 28, 2022 at 6:30 PM Chesnay Schepler  wrote:

> I still think that's mostly a process issue.
> Of course we can be blind-sided if we do the QA for a release artifact
> after the release has been finalized.
> But that's a clearly broken process from the get-go.
>
> At the very least we should already open a PR when the RC is created to
> get earlier feedback.
>
> Moreover, nowadays the docker images are way slimmer and we are much
> more careful on what is actually added to the scripts.
>
> Finally, the problems they found did show that their QA is very valuable
> to us. And side-stepping that for such an essential piece of a release
> isn't a good idea imo.
>
> On 28/04/2022 11:31, Xintong Song wrote:
> > I'm overall against only releasing to official-images.
> >
> > We started releasing to apache/flink, in addition to the official-image,
> in
> > 1.12.0. That was because releasing the official-image needs approval from
> > the DockerHub folks, which is not under control of the Flink community.
> For
> > 1.12.0 there were unfortunately some divergences between us and the
> > DockerHub folks, and it ended-up taking us nearly 2 months to get that
> > official-image PR merged [1][2]. Many users, especially those who need
> > Flink's K8s & Native-K8s deployment modes, were asking for the image
> after
> > 1.12.0 was announced.
> >
> > One could argue that what happened for 1.12.0 is not a regular case.
> > However, I'd like to point out that the docker images are not something
> > nice-to-have, but a practically necessary piece of the release for the
> k8s
> > / native-k8s deployments to work. I'm strongly against a release process
> > where such an important piece depends on the approval of a 3rd party.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-20650
> >
> > [2] https://github.com/docker-library/official-images/pull/9249
> >
> >
> >
> > On Thu, Apr 28, 2022 at 2:43 PM Chesnay Schepler 
> wrote:
> >
> >> We could just stop releasing to apache/flink and only go for the
> >> official-images route.
> >>
> >> On 28/04/2022 07:43, Xintong Song wrote:
> >>> Forgot to mention that, we have also proposed to use one shared account
> >> and
> >>> limit its access to the PMC members, like what we do with the PyPI
> >> account.
> >>> Unfortunately, INFRA rejected this proposal [1].
> >>>
> >>>
> >>> Thank you~
> >>>
> >>> Xintong Song
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/INFRA-23208
> >>>
> >>> On Thu, Apr 28, 2022 at 1:39 PM Xintong Song 
> >> wrote:
>  Hi devs,
> 
>  I'd like to start a discussion about maintainers for DockerHub
>  repositories under the *apache* namespace [1].
> 
>  Currently, the Flink community maintains various repositories (flink,
>  flink-statefun, flink-statefun-playground, and
> >> flink-kubernetes-operator)
>  on DockerHub under the *apache* namespace. There's a limitation on how
> >> many
>  members the *apache* namespace can add, and recently INFRA is
> >> complaining
>  about Flink taking too many places [2][3]. They would like us to
> reduce
> >> our
>  maintainers from 20 now to 5.
> 
>  Jingsong and I would like to volunteer as two of the maintainers, and
> we
>  would like to learn who else wants to join us. While any committer may
>  serve as one of the maintainers, it's probably nice to also involve at
>  least one maintainer from statefun and one from kubernetes-operator.
> 
>  What do you think?
> 
>  

Re: [DISCUSS] Planning Flink 1.16

[Konstantin Knauf]

Hi everyone,

thank you for the feedback so far. I've checked the length of the previous
last release cycles. Up until Flink 1.14, we've actually been incredibly
good at maintaining a 5 months release cycle (see below). Interestingly,
the community is officially targeting a 4 months release cycle [1].

- 1.15.0 2022-05-01? (7 months, 2 days?)
- 1.14.0: 2021-09-29 (4 months, 26 days)
- 1.13.0: 2021-05-03 (4 months, 23 days)
- 1.12.0: 2020-12-10 (5 months, 3 days)
- 1.11.0: 2020-07-07 (4 months, 26 days)
- 1.10.0: 2020-02-11

The 1.15 release cycle has took significantly longer. In my opinion we
should try to get back into the 5 months cadence with the next release.
Since empirically we always often end up moving the the feature freeze by a
week or two, and that we often need about a month for release testing &
stabilization and releasing, I don't think, we should move the planned
feature freeze to later than
*mid August. *
What do you think:
1. Should we continue to strive for a 5 months release cycle (and update
[1] accordingly)?
2. Does it sound reasonable to target a feature freeze date, which is 1.5
months before to the desired release date?

Cheers,

Konstantin

 [1] https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases

Am Do., 28. Apr. 2022 um 05:20 Uhr schrieb Jingsong Li <
jingsongl...@gmail.com>:

> Thanks Konstantin and Chesnay for starting the discussion. And thanks
> Konstantin, Chesnay, Godfrey, Martijn for volunteering.
>
> The 1.16 release at the end of July means that there are currently
> only 3 months to fully commit to 1.16 development, and I'm a little
> concerned that this is too short a time frame, which may result in
> some features only reaching a halfway decent state.
>
> Best,
> Jingsong
>
>
> On Wed, Apr 27, 2022 at 7:10 PM David Morávek  wrote:
> >
> > Thanks Konstantin and Chesnay for starting the discussion and
> volunteering.
> > The timeline proposal sounds reasonable :+1:
> >
> > Best,
> > D.
> >
> > On Tue, Apr 26, 2022 at 1:37 PM Martijn Visser  >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for starting this discussion. I would also volunteer to help
> out as
> > > a release manager for the 1.16 release.
> > >
> > > Best regards,
> > >
> > > Martijn Visser
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > >
> > > On Tue, 26 Apr 2022 at 13:19, godfrey he  wrote:
> > >
> > > > Hi Konstantin & Chesnay,
> > > >
> > > > Thanks for driving this discussion, I am willing to volunteer as the
> > > > release manager for 1.16.
> > > >
> > > >
> > > > Best,
> > > > Godfrey
> > > >
> > > > Konstantin Knauf  于2022年4月26日周二 18:23写道：
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > With Flink 1.15 about to be released, the community has started
> > > planning
> > > > &
> > > > > developing features for the next release, Flink 1.16. As such, I
> would
> > > > like
> > > > > to start a discussion around managing this release.
> > > > >
> > > > > Specifically, Chesnay & myself would like to volunteer as release
> > > > managers.
> > > > > Our focus as release managers would be
> > > > > * to propose a release timeline
> > > > > * to provide an overview of all ongoing development threads and
> ideally
> > > > > their current status to the community
> > > > > * to keep an eye on build stability
> > > > > * facilitate release testing
> > > > > * to do the actual release incl. communication (blog post, etc.)
> > > > >
> > > > > Is anyone else interested in acting as a release manager for Flink
> > > 1.16?
> > > > If
> > > > > so, we are happy to make this a joint effort.
> > > > >
> > > > > Besides the question of who will act as a release manager, I
> think, we
> > > > can
> > > > > already use this thread to align on a timeline. For collecting
> features
> > > > and
> > > > > everything else, we would start a dedicated threads shortly.
> > > > >
> > > > > Given Flink 1.15 will be released in the next days, and aiming for
> a 4
> > > > > months release cycle including stabilization, this would mean
> *feature
> > > > > freeze at the end of July*. The exact date could be determined
> later.
> > > Any
> > > > > thoughts on the timeline.?
> > > > >
> > > > > Looking forward to your thoughts!
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chesnay & Konstantin
> > > >
> > >
>

[jira] [Created] (FLINK-27440) adds ignore-parse-errors configuration in avro format

[Jira]

陈磊 created FLINK-27440:
--

 Summary: adds ignore-parse-errors configuration in avro format
 Key: FLINK-27440
 URL: https://issues.apache.org/jira/browse/FLINK-27440
 Project: Flink
  Issue Type: New Feature
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.14.4
Reporter: 陈磊


In the current avro format, there is no configuration to ignore deserialization 
failures. Users expect to ignore the current deserialization failed data and 
continue to run the task. In fact, when deserialization fails, the task will 
directly restart or stop abnormally.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] DockerHub repository maintainers

[Chesnay Schepler]

I still think that's mostly a process issue.
Of course we can be blind-sided if we do the QA for a release artifact 
after the release has been finalized.

But that's a clearly broken process from the get-go.

At the very least we should already open a PR when the RC is created to 
get earlier feedback.


Moreover, nowadays the docker images are way slimmer and we are much 
more careful on what is actually added to the scripts.


Finally, the problems they found did show that their QA is very valuable 
to us. And side-stepping that for such an essential piece of a release 
isn't a good idea imo.


On 28/04/2022 11:31, Xintong Song wrote:

I'm overall against only releasing to official-images.

We started releasing to apache/flink, in addition to the official-image, in
1.12.0. That was because releasing the official-image needs approval from
the DockerHub folks, which is not under control of the Flink community. For
1.12.0 there were unfortunately some divergences between us and the
DockerHub folks, and it ended-up taking us nearly 2 months to get that
official-image PR merged [1][2]. Many users, especially those who need
Flink's K8s & Native-K8s deployment modes, were asking for the image after
1.12.0 was announced.

One could argue that what happened for 1.12.0 is not a regular case.
However, I'd like to point out that the docker images are not something
nice-to-have, but a practically necessary piece of the release for the k8s
/ native-k8s deployments to work. I'm strongly against a release process
where such an important piece depends on the approval of a 3rd party.

Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/FLINK-20650

[2] https://github.com/docker-library/official-images/pull/9249



On Thu, Apr 28, 2022 at 2:43 PM Chesnay Schepler  wrote:


We could just stop releasing to apache/flink and only go for the
official-images route.

On 28/04/2022 07:43, Xintong Song wrote:

Forgot to mention that, we have also proposed to use one shared account

and

limit its access to the PMC members, like what we do with the PyPI

account.

Unfortunately, INFRA rejected this proposal [1].


Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/INFRA-23208

On Thu, Apr 28, 2022 at 1:39 PM Xintong Song 

wrote:

Hi devs,

I'd like to start a discussion about maintainers for DockerHub
repositories under the *apache* namespace [1].

Currently, the Flink community maintains various repositories (flink,
flink-statefun, flink-statefun-playground, and

flink-kubernetes-operator)

on DockerHub under the *apache* namespace. There's a limitation on how

many

members the *apache* namespace can add, and recently INFRA is

complaining

about Flink taking too many places [2][3]. They would like us to reduce

our

maintainers from 20 now to 5.

Jingsong and I would like to volunteer as two of the maintainers, and we
would like to learn who else wants to join us. While any committer may
serve as one of the maintainers, it's probably nice to also involve at
least one maintainer from statefun and one from kubernetes-operator.

What do you think?

Thank you~

Xintong Song


[1] https://hub.docker.com/orgs/apache

[2] https://issues.apache.org/jira/browse/INFRA-23208

[3] https://issues.apache.org/jira/browse/INFRA-23213

[jira] [Created] (FLINK-27439) Fix ClassNotFoundException when using array type in table store

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-27439:
---

 Summary: Fix ClassNotFoundException when using array type in table 
store
 Key: FLINK-27439
 URL: https://issues.apache.org/jira/browse/FLINK-27439
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Affects Versions: table-store-0.1.0
 Environment: When using array data type with table store a 
{{ClassNotFound}} exception will be thrown. This is because FLINK-27408 
packaged codegen related classes into table store itself but missed out some 
classes.

To make sure that such bug will not happen again we also need an e2e test to 
check all supported data types.
Reporter: Caizhi Weng
 Fix For: table-store-0.1.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: Re: [DISCUSS] FLIP-168: Speculative execution for Batch Job

[Guowei Ma]

Hi, zhu

Many thanks to zhuzhu for initiating the FLIP discussion. Overall I think
it's ok, I just have 3 small questions

1. How to judge whether the Execution Vertex belongs to a slow task.
The current calculation method is: the current timestamp minus the
timestamp of the execution deployment. If the execution time of this
execution exceeds the baseline, then it is judged as a slow task. Normally
this is no problem. But if an execution fails, the time may not be
accurate. For example, the baseline is 59 minutes, and a task fails after
56 minutes of execution. In the worst case, it may take an additional 59
minutes to discover that the task is a slow task.

2. Speculative Scheduler's fault tolerance strategy.
The strategy in FLIP is: if the Execution Vertex can be executed, even if
the execution fails, the fault tolerance strategy will not be adopted.
Although currently `ExecutionTimeBasedSlowTaskDetector` can restart an
execution. But isn't this dependency a bit too strong? To some extent, the
fault tolerance strategy and the Slow task detection strategy are coupled
together.


3. The value of the default configuration
IMHO, prediction execution should only be required for relatively
large-scale, very time-consuming and long-term jobs.
If `slow-task-detector.execution-time.baseline-lower-bound` is too small,
is it possible for the system to always start some additional tasks that
have little effect? In the end, the user needs to reset this default
configuration. Is it possible to consider a larger configuration. Of
course, this part is best to listen to the suggestions of other community
users.

Best,
Guowei


On Thu, Apr 28, 2022 at 3:54 PM Jiangang Liu 
wrote:

> +1 for the feature.
>
> Mang Zhang  于2022年4月28日周四 11:36写道：
>
> > Hi zhu:
> >
> >
> > This sounds like a great job! Thanks for your great job.
> > In our company, there are already some jobs using Flink Batch,
> > but everyone knows that the offline cluster has a lot more load than
> > the online cluster, and the failure rate of the machine is also much
> higher.
> > If this work is done, we'd love to use it, it's simply awesome for
> our
> > flink users.
> > thanks again!
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > Best regards,
> > Mang Zhang
> >
> >
> >
> >
> >
> > At 2022-04-27 10:46:06, "Zhu Zhu"  wrote:
> > >Hi everyone,
> > >
> > >More and more users are running their batch jobs on Flink nowadays.
> > >One major problem they encounter is slow tasks running on hot/bad
> > >nodes, resulting in very long and uncontrollable execution time of
> > >batch jobs. This problem is a pain or even unacceptable in
> > >production. Many users have been asking for a solution for it.
> > >
> > >Therefore, I'd like to revive the discussion of speculative
> > >execution to solve this problem.
> > >
> > >Weijun Wang, Jing Zhang, Lijie Wang and I had some offline
> > >discussions to refine the design[1]. We also implemented a PoC[2]
> > >and verified it using TPC-DS benchmarks and production jobs.
> > >
> > >Looking forward to your feedback!
> > >
> > >Thanks,
> > >Zhu
> > >
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> > >[2]
> > https://github.com/zhuzhurk/flink/commits/1.14-speculative-execution-poc
> > >
> > >
> > >刘建刚  于2021年12月13日周一 11:38写道：
> > >
> > >> Any progress on the feature? We have the same requirement in our
> > company.
> > >> Since the soft and hard environment can be complex, it is normal to
> see
> > a
> > >> slow task which determines the execution time of the flink job.
> > >>
> > >>  于2021年6月20日周日 22:35写道：
> > >>
> > >> > Hi everyone,
> > >> >
> > >> > I would like to kick off a discussion on speculative execution for
> > batch
> > >> > job.
> > >> > I have created FLIP-168 [1] that clarifies our motivation to do this
> > and
> > >> > some improvement proposals for the new design.
> > >> > It would be great to resolve the problem of long tail task in batch
> > job.
> > >> > Please let me know your thoughts. Thanks.
> > >> >   Regards,
> > >> > wangwj
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> > >> >
> > >>
> >
>

[jira] [Created] (FLINK-27438) SQL validation failed when constructing a map array

[Wei Zhong (Jira)]

Wei Zhong created FLINK-27438:
-

 Summary: SQL validation failed when constructing a map array
 Key: FLINK-27438
 URL: https://issues.apache.org/jira/browse/FLINK-27438
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.15.0
Reporter: Wei Zhong


Exception: 
{code:java}
Exception in thread "main" org.apache.flink.table.api.ValidationException: SQL 
validation failed. Unsupported type when convertTypeToSpec: MAP
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:185)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110)
    at 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:237)
    at 
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105)
    at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:695)
    at 
org.apache.flink.table.planner.plan.stream.sql.LegacyTableFactoryTest.(LegacyTableFactoryTest.java:35)
    at 
org.apache.flink.table.planner.plan.stream.sql.LegacyTableFactoryTest.main(LegacyTableFactoryTest.java:49)
Caused by: java.lang.UnsupportedOperationException: Unsupported type when 
convertTypeToSpec: MAP
    at 
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1059)
    at 
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1081)
    at 
org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.castTo(SqlValidatorUtils.java:82)
    at 
org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForMultisetConstructor(SqlValidatorUtils.java:74)
    at 
org.apache.flink.table.planner.functions.utils.SqlValidatorUtils.adjustTypeForArrayConstructor(SqlValidatorUtils.java:39)
    at 
org.apache.flink.table.planner.functions.sql.SqlArrayConstructor.inferReturnType(SqlArrayConstructor.java:44)
    at org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:449)
    at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:531)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5716)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5703)
    at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1736)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1727)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:421)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4061)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3347)
    at 
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
    at 
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:997)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:975)
    at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:232)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:952)
    at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:180)
    ... 6 more {code}
How to reproduce:
{code:java}
tableEnv.executeSql("select array[map['A', 'AA'], map['B', 'BB'], map['C', 
CAST(NULL AS STRING)]] from (VALUES ('a'))").print(); {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] DockerHub repository maintainers

[Xintong Song]

I'm overall against only releasing to official-images.

We started releasing to apache/flink, in addition to the official-image, in
1.12.0. That was because releasing the official-image needs approval from
the DockerHub folks, which is not under control of the Flink community. For
1.12.0 there were unfortunately some divergences between us and the
DockerHub folks, and it ended-up taking us nearly 2 months to get that
official-image PR merged [1][2]. Many users, especially those who need
Flink's K8s & Native-K8s deployment modes, were asking for the image after
1.12.0 was announced.

One could argue that what happened for 1.12.0 is not a regular case.
However, I'd like to point out that the docker images are not something
nice-to-have, but a practically necessary piece of the release for the k8s
/ native-k8s deployments to work. I'm strongly against a release process
where such an important piece depends on the approval of a 3rd party.

Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/FLINK-20650

[2] https://github.com/docker-library/official-images/pull/9249



On Thu, Apr 28, 2022 at 2:43 PM Chesnay Schepler  wrote:

> We could just stop releasing to apache/flink and only go for the
> official-images route.
>
> On 28/04/2022 07:43, Xintong Song wrote:
> > Forgot to mention that, we have also proposed to use one shared account
> and
> > limit its access to the PMC members, like what we do with the PyPI
> account.
> > Unfortunately, INFRA rejected this proposal [1].
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-23208
> >
> > On Thu, Apr 28, 2022 at 1:39 PM Xintong Song 
> wrote:
> >
> >> Hi devs,
> >>
> >> I'd like to start a discussion about maintainers for DockerHub
> >> repositories under the *apache* namespace [1].
> >>
> >> Currently, the Flink community maintains various repositories (flink,
> >> flink-statefun, flink-statefun-playground, and
> flink-kubernetes-operator)
> >> on DockerHub under the *apache* namespace. There's a limitation on how
> many
> >> members the *apache* namespace can add, and recently INFRA is
> complaining
> >> about Flink taking too many places [2][3]. They would like us to reduce
> our
> >> maintainers from 20 now to 5.
> >>
> >> Jingsong and I would like to volunteer as two of the maintainers, and we
> >> would like to learn who else wants to join us. While any committer may
> >> serve as one of the maintainers, it's probably nice to also involve at
> >> least one maintainer from statefun and one from kubernetes-operator.
> >>
> >> What do you think?
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >> [1] https://hub.docker.com/orgs/apache
> >>
> >> [2] https://issues.apache.org/jira/browse/INFRA-23208
> >>
> >> [3] https://issues.apache.org/jira/browse/INFRA-23213
> >>
> >>
>
>

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

[Paul Lam]

Hi Martjin,

Thanks a lot for your reply! I agree that the scope may be a bit confusing,
please let me clarify.

The FLIP aims to add new SQL statements that are supported only in
sql-client, similar to
jar statements [1]. Jar statements can be parsed into jar operations, which
are used only in
CliClient in sql-client module and cannot be executed by TableEnvironment
(not available in
Table API program that contains SQL that you mentioned).

WRT the unchanged CLI client, I mean CliClient instead of the sql-client
module, which
currently contains the gateway codes (e.g. Executor). The FLIP mainly
extends
the gateway part, and barely touches CliClient and REST server (REST
endpoint in FLIP-91).

WRT the syntax, I don't have much experience with SQL standards, and I'd
like to hear
more opinions from the community. I prefer Hive-style syntax because I
think many users
are familiar with Hive, and there're on-going efforts to improve Flink-Hive
integration [2][3].
But my preference is not strong, I'm okay with other options too. Do you
think JOB/Task is
a good choice, or do you have other preferred keywords?

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/table/sql/jar/
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-152%3A+Hive+Query+Syntax+Compatibility
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-223%3A+Support+HiveServer2+Endpoint

Best,
Paul Lam

Martijn Visser  于2022年4月26日周二 20:14写道：

> Hi Paul,
>
> Thanks for creating the FLIP and opening the discussion. I did get a bit
> confused about the title, being "query lifecycle statements in SQL client".
> This sounds like you want to adopt the SQL client, but you want to expand
> the SQL syntax with lifecycle statements, which could be used from the SQL
> client, but of course also in a Table API program that contains SQL. GIven
> that you're highlighting the CLI client as unchanged, this adds to more
> confusion.
>
> I am interested if there's anything listed in the SQL 2016 standard on
> these types of lifecycle statements. I did a quick scan for "SHOW QUERIES"
> but couldn't find it. It would be great if we could stay as close as
> possible to such syntax. Overall I'm not in favour of using QUERIES as a
> keyword. I think Flink applications are not queries, but short- or long
> running applications. Why should we follow Hive's setup and indeed not
> others such as Snowflake, but also Postgres or MySQL?
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
>
> On Fri, 22 Apr 2022 at 12:06, Paul Lam  wrote:
>
> > Hi Shengkai,
> >
> > Thanks a lot for your opinions!
> >
> > > 1. I think the keyword QUERY may confuse users because the statement
> also
> > > works for the DML statement.
> >
> > I slightly lean to QUERY, because:
> >
> > Hive calls DMLs queries. We could be better aligned with Hive using
> QUERY,
> > especially given that we plan to introduce Hive endpoint.
> > QUERY is a more SQL-like concept and friendly to SQL users.
> >
> > In general, my preference: QUERY > JOB > TASK. I’m okay with JOB, but not
> > very good with TASK, as it conflicts with the task concept in Flink
> runtime.
> >
> > We could wait for more feedbacks from the community.
> >
> > > 2. STOP/CANCEL is not very straightforward for the SQL users to
> terminate
> > > their jobs.
> >
> > Agreed. I’m okay with DROP. And if we want to align with Hive, KILL might
> > an alternative.
> >
> > > 3. I think CREATE/DROP SAVEPOINTS statement is more SQL-like.
> >
> > Agreed. It’s more SQL-like and intuitive. I’m updating the syntax on the
> > FLIP.
> >
> > > 4. SHOW TASKS can just list the job id and use the DESCRIPE to get more
> > > detailed job infos.
> >
> > That is a more SQL-like approach I think. But considering the
> > ClusterClient APIs, we can fetch the names and the status along in one
> > request,
> > thus it may be more user friendly to return them all in the SHOW
> > statement?
> >
> > > It's better we can also get the infos about the cluster where the job
> is
> > > running on through the DESCRIBE statement.
> >
> > I think cluster info could be part of session properties instead. WDYT?
> >
> > Best,
> > Paul Lam
> >
> > > 2022年4月22日 11:14，Shengkai Fang  写道：
> > >
> > > Hi Paul
> > >
> > > Sorry for the late response. I propose my thoughts here.
> > >
> > > 1. I think the keyword QUERY may confuse users because the statement
> also
> > > works for the DML statement. I find the Snowflakes[1] supports
> > >
> > > - CREATE TASK
> > > - DROP TASK
> > > - ALTER TASK
> > > - SHOW TASKS
> > > - DESCRIPE TASK
> > >
> > > I think we can follow snowflake to use `TASK` as the keyword or use the
> > > keyword `JOB`?
> > >
> > > 2. STOP/CANCEL is not very straightforward for the SQL users to
> terminate
> > > their jobs.
> > >
> > > ```
> > > DROP TASK [IF EXISTS]  PURGE; -- Forcely stop the job with
> drain
> > >
> > > DROP TASK [IF EXISTS] ; -- Stop the task with savepoints
> > > 

[DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

[Mang Zhang]

Hi, everyone


I would like to open a discussion for support select clause in CREATE 
TABLE(CTAS),
With the development of business and the enhancement of flink sql capabilities, 
queries become more and more complex.
Now the user needs to use the Create Table statement to create the target table 
first, and then execute the insert statement.
However, the target table may have many columns, which will bring a lot of work 
outside the business logic to the user.
At the same time, ensure that the schema of the created target table is 
consistent with the schema of the query result.
Using a CTAS syntax like Hive/Spark can greatly facilitate the user.



You can find more details in FLIP-218[1]. Looking forward to your feedback.



[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-218%3A+Support+SELECT+clause+in+CREATE+TABLE(CTAS)




--

Best regards,
Mang Zhang

Re: Re: [DISCUSS] FLIP-168: Speculative execution for Batch Job

[Jiangang Liu]

+1 for the feature.

Mang Zhang  于2022年4月28日周四 11:36写道：

> Hi zhu:
>
>
> This sounds like a great job! Thanks for your great job.
> In our company, there are already some jobs using Flink Batch,
> but everyone knows that the offline cluster has a lot more load than
> the online cluster, and the failure rate of the machine is also much higher.
> If this work is done, we'd love to use it, it's simply awesome for our
> flink users.
> thanks again!
>
>
>
>
>
>
>
> --
>
> Best regards,
> Mang Zhang
>
>
>
>
>
> At 2022-04-27 10:46:06, "Zhu Zhu"  wrote:
> >Hi everyone,
> >
> >More and more users are running their batch jobs on Flink nowadays.
> >One major problem they encounter is slow tasks running on hot/bad
> >nodes, resulting in very long and uncontrollable execution time of
> >batch jobs. This problem is a pain or even unacceptable in
> >production. Many users have been asking for a solution for it.
> >
> >Therefore, I'd like to revive the discussion of speculative
> >execution to solve this problem.
> >
> >Weijun Wang, Jing Zhang, Lijie Wang and I had some offline
> >discussions to refine the design[1]. We also implemented a PoC[2]
> >and verified it using TPC-DS benchmarks and production jobs.
> >
> >Looking forward to your feedback!
> >
> >Thanks,
> >Zhu
> >
> >[1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> >[2]
> https://github.com/zhuzhurk/flink/commits/1.14-speculative-execution-poc
> >
> >
> >刘建刚  于2021年12月13日周一 11:38写道：
> >
> >> Any progress on the feature? We have the same requirement in our
> company.
> >> Since the soft and hard environment can be complex, it is normal to see
> a
> >> slow task which determines the execution time of the flink job.
> >>
> >>  于2021年6月20日周日 22:35写道：
> >>
> >> > Hi everyone,
> >> >
> >> > I would like to kick off a discussion on speculative execution for
> batch
> >> > job.
> >> > I have created FLIP-168 [1] that clarifies our motivation to do this
> and
> >> > some improvement proposals for the new design.
> >> > It would be great to resolve the problem of long tail task in batch
> job.
> >> > Please let me know your thoughts. Thanks.
> >> >   Regards,
> >> > wangwj
> >> > [1]
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+execution+for+Batch+Job
> >> >
> >>
>

Re: [VOTE] FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect

[Jark Wu]

+1 (binding)

Thank Yuxia, for driving this work.

Best,
Jark

On Thu, 28 Apr 2022 at 11:58, Jingsong Li  wrote:

> +1 (Binding)
>
> A very good step to move forward.
>
> Best.
> Jingsong
>
> On Wed, Apr 27, 2022 at 9:33 PM yuxia  wrote:
> >
> > Hi, everyone
> >
> > Thanks all for attention to FLIP-216: Introduce pluggable dialect and
> plan for migrating Hive dialect [1] and participation in the discussion in
> the mail thread [2].
> >
> > I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or not enough votes.
> >
> > [1] [
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> |
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect
> ]
> > [2] [ https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo
> | https://lists.apache.org/thread/66g79w5zlod2ylyv8k065j57pjjmv1jo ]
> >
> >
> > Best regards,
> > Yuxia
>

[jira] [Created] (FLINK-27437) Syntax error when running tpcds test

[Yao Zhang (Jira)]

Yao Zhang created FLINK-27437:
-

 Summary: Syntax error when running tpcds test
 Key: FLINK-27437
 URL: https://issues.apache.org/jira/browse/FLINK-27437
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.14.4, 1.15.0
 Environment: CentOS 7.4
Reporter: Yao Zhang


Run tpcds test with instruction from 
`flink-end-to-end-tests/flink-tpcds-test/README.md`

 
{code:java}
sh flink-end-to-end-tests/run-single-test.sh 
flink-end-to-end-tests/test-scripts/test_tpcds.sh {code}
Then a syntax error were thrown as follow:
{code:java}
Checking for availability of CI Maven mirror
Using Google mirror
Flink dist directory: 
/opt/zy/source_code/flink/flink-dist/target/flink-1.13.2-bin/flink-1.13.2
/path/to/flink/flink-end-to-end-tests/test-scripts/common.sh: line 257: syntax 
error near unexpected token `<'
/path/to/flink/flink-end-to-end-tests/test-scripts/common.sh: line 257: `    
done < <(grep "^server\." "${FLINK_DIR}/conf/zoo.cfg")' {code}
If bash is invoked with name sh, bash enters posix mode. Process substitution 
is not specified by POSIX, so not all POSIX shells support it, only some shells 
like bash.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27436) option `properties.group.id` is not effective in kafka connector for finksql

[Spongebob (Jira)]

Spongebob created FLINK-27436:
-

 Summary: option `properties.group.id` is not effective in kafka 
connector for finksql
 Key: FLINK-27436
 URL: https://issues.apache.org/jira/browse/FLINK-27436
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.14.3
Reporter: Spongebob


option `properties.group.id` is not effective in kafka connector for finksql.

when I run this sql, I can read message from specific topic normaly. But I 
could not

find the group named `test-group` in kafka server. 
{code:java}
// ddl like this
"CREATE TABLE ...
"WITH ('connector' = 'kafka',
 'properties.bootstrap.servers' = '...'," +
"'topic' = '...',"+
 "'scan.startup.mode'='latest-offset'," +
"'properties.group.id' = 'test-group'," +
 "'format' = 'debezium-json')"; {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Lijie Wang]

Hi Konstantin,

Thanks for your feedback. I will response your 4 remarks:


1) Thanks for reminding me of the controversy. I think “BlockList” is good
enough, and I will change it in FLIP.


2) Your suggestion for the REST API is a good idea. Based on the above, I
would change REST API as following:

POST/GET /blocklist/nodes

POST/GET /blocklist/taskmanagers

DELETE /blocklist/node/

DELETE /blocklist/taskmanager/


3) If a node is blocking/blocklisted, it means that all task managers on
this node are blocklisted. All slots on these TMs are not available. This
is actually a bit like TM losts, but these TMs are not really lost, they
are in an unavailable status, and they are still registered in this flink
cluster. They will be available again once the corresponding blocklist item
is removed. This behavior is the same in active/non-active clusters.
However in the active clusters, these TMs may be released due to idle
timeouts.


4) For the item timeout, I prefer to keep it. The reasons are as following:

a) The timeout will not affect users adding or removing items via REST API,
and users can disable it by configuring it to Long.MAX_VALUE .

b) Some node problems can recover after a period of time (such as machine
hotspots), in which case users may prefer that Flink can do this
automatically instead of requiring the user to do it manually.


Best,

Lijie

Konstantin Knauf  于2022年4月27日周三 19:23写道：

> Hi Lijie,
>
> I think, this makes sense and +1 to only support manually blocking
> taskmanagers and nodes. Maybe the different strategies can also be
> maintained outside of Apache Flink.
>
> A few remarks:
>
> 1) Can we use another term than "bla.cklist" due to the controversy around
> the term? [1] There was also a Jira Ticket about this topic a while back
> and there was generally a consensus to avoid the term blacklist & whitelist
> [2]? We could use "blocklist" "denylist" or "quarantined"
> 2) For the REST API, I'd prefer a slightly different design as verbs like
> add/remove often considered an anti-pattern for REST APIs. POST on a list
> item is generally the standard to add items. DELETE on the individual
> resource is standard to remove an item.
>
> POST /quarantine/items
> DELETE /quarantine/items/
>
> We could also consider to separate taskmanagers and nodes in the REST API
> (and internal data structures). Any opinion on this?
>
> POST/GET /quarantine/nodes
> POST/GET /quarantine/taskmanager
> DELETE /quarantine/nodes/
> DELETE /quarantine/taskmanager/
>
> 3) How would blocking nodes behave with non-active resource managers, i.e.
> standalone or reactive mode?
>
> 4) To keep the implementation even more minimal, do we need the timeout
> behavior? If items are added/removed manually we could delegate this to the
> user easily. In my opinion the timeout behavior would better fit into
> specific strategies at a later point.
>
> Looking forward to your thoughts.
>
> Cheers and thank you,
>
> Konstantin
>
> [1]
>
> https://en.wikipedia.org/wiki/Blacklist_(computing)#Controversy_over_use_of_the_term
> [2] https://issues.apache.org/jira/browse/FLINK-18209
>
> Am Mi., 27. Apr. 2022 um 04:04 Uhr schrieb Lijie Wang <
> wangdachui9...@gmail.com>:
>
> > Hi all,
> >
> > Flink job failures may happen due to cluster node issues (insufficient
> disk
> > space, bad hardware, network abnormalities). Flink will take care of the
> > failures and redeploy the tasks. However, due to data locality and
> limited
> > resources, the new tasks are very likely to be redeployed to the same
> > nodes, which will result in continuous task abnormalities and affect job
> > progress.
> >
> > Currently, Flink users need to manually identify the problematic node and
> > take it offline to solve this problem. But this approach has following
> > disadvantages:
> >
> > 1. Taking a node offline can be a heavy process. Users may need to
> contact
> > cluster administors to do this. The operation can even be dangerous and
> not
> > allowed during some important business events.
> >
> > 2. Identifying and solving this kind of problems manually would be slow
> and
> > a waste of human resources.
> >
> > To solve this problem, Zhu Zhu and I propose to introduce a blacklist
> > mechanism for Flink to filter out problematic resources.
> >
> >
> > You can find more details in FLIP-224[1]. Looking forward to your
> feedback.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-224%3A+Blacklist+Mechanism
> >
> >
> > Best,
> >
> > Lijie
> >
>

Re: [VOTE] Apache Flink Table Store 0.1.0, release candidate #1

[Jingsong Li]

Thanks Caizhi for the quick validation.

Let's cancel this RC and wait for your fix.

Best,
Jingsong

On Thu, Apr 28, 2022 at 2:55 PM Caizhi Weng  wrote:
>
> Hi all!
>
> -1 for this release. Currently we cannot write array and map types into
> table store due to this commit
> .
> Run the following SQL in SQL client and an exception will be thrown:
>
> create table if not exists tstore2 (
> a ARRAY,
> b MAP
> ) with (
> 'path' = 'hdfs:///tstore2',
> 'bucket' = '4'
> );
>
> insert into tstore2 values (array['hi','hello',cast(null as
> string),'test'], map[1,'A',2,'BB',100,'CCC']), (cast(null as
> array), cast(null as map));
>
> The exception stack is
>
> Exception in thread "main"
> org.apache.flink.table.client.SqlClientException: Unexpected exception.
> This is a bug. Please consider filing an issue.
>
> at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
>
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
>
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/table/planner/plan/utils/SortUtil$
>
> at
> org.apache.flink.table.planner.codegen.GenerateUtils$.generateCompare(GenerateUtils.scala:675)
>
> at
> org.apache.flink.table.planner.codegen.GenerateUtils$.$anonfun$generateRowCompare$1(GenerateUtils.scala:828)
>
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
>
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)
>
> at
> org.apache.flink.table.planner.codegen.GenerateUtils$.generateRowCompare(GenerateUtils.scala:802)
>
> at
> org.apache.flink.table.planner.codegen.sort.ComparatorCodeGenerator$.gen(ComparatorCodeGenerator.scala:53)
>
> at
> org.apache.flink.table.planner.codegen.sort.ComparatorCodeGenerator.gen(ComparatorCodeGenerator.scala)
>
> at
> org.apache.flink.table.store.codegen.CodeGeneratorImpl.generateRecordComparator(CodeGeneratorImpl.java:60)
>
> at
> org.apache.flink.table.store.codegen.CodeGenUtils.generateRecordComparator(CodeGenUtils.java:67)
>
> at
> org.apache.flink.table.store.file.FileStoreImpl.(FileStoreImpl.java:67)
>
> at
> org.apache.flink.table.store.connector.TableStore.buildFileStore(TableStore.java:220)
>
> at
> org.apache.flink.table.store.connector.TableStore.access$100(TableStore.java:81)
>
> at
> org.apache.flink.table.store.connector.TableStore$SinkBuilder.build(TableStore.java:415)
>
> at
> org.apache.flink.table.store.connector.sink.TableStoreSink.lambda$getSinkRuntimeProvider$0(TableStoreSink.java:143)
>
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.applySinkProvider(CommonExecSink.java:446)
>
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.createSinkTransformation(CommonExecSink.java:192)
>
> at
> org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.java:67)
>
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:148)
>
> at
> org.apache.flink.table.planner.delegation.BatchPlanner.$anonfun$translateToPlan$1(BatchPlanner.scala:82)
>
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
>
> at scala.collection.Iterator.foreach(Iterator.scala:937)
>
> at scala.collection.Iterator.foreach$(Iterator.scala:937)
>
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
>
> at scala.collection.IterableLike.foreach(IterableLike.scala:70)
>
> at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
>
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>
> at scala.collection.TraversableLike.map(TraversableLike.scala:233)
>
> at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
>
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>
> at
> org.apache.flink.table.planner.delegation.BatchPlanner.translateToPlan(BatchPlanner.scala:81)
>
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:181)
>
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1656)
>
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:782)
>
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeModifyOperations$4(LocalExecutor.java:222)
>
> at
> org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88)
>
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeModifyOperations(LocalExecutor.java:222)
>
> at
> org.apache.flink.table.client.cli.CliClient.callInserts(CliClient.java:600)
>
> at
> org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:589)
>
> at
> org.apache.flink.table.client.cli.CliClient.callOperation(CliClient.java:443)
>
> at
> 

Re: [VOTE] Apache Flink Table Store 0.1.0, release candidate #1

[Caizhi Weng]

Hi all!

-1 for this release. Currently we cannot write array and map types into
table store due to this commit
.
Run the following SQL in SQL client and an exception will be thrown:

create table if not exists tstore2 (
a ARRAY,
b MAP
) with (
'path' = 'hdfs:///tstore2',
'bucket' = '4'
);

insert into tstore2 values (array['hi','hello',cast(null as
string),'test'], map[1,'A',2,'BB',100,'CCC']), (cast(null as
array), cast(null as map));

The exception stack is

Exception in thread "main"
org.apache.flink.table.client.SqlClientException: Unexpected exception.
This is a bug. Please consider filing an issue.

at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)

at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)

Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/table/planner/plan/utils/SortUtil$

at
org.apache.flink.table.planner.codegen.GenerateUtils$.generateCompare(GenerateUtils.scala:675)

at
org.apache.flink.table.planner.codegen.GenerateUtils$.$anonfun$generateRowCompare$1(GenerateUtils.scala:828)

at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)

at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)

at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:194)

at
org.apache.flink.table.planner.codegen.GenerateUtils$.generateRowCompare(GenerateUtils.scala:802)

at
org.apache.flink.table.planner.codegen.sort.ComparatorCodeGenerator$.gen(ComparatorCodeGenerator.scala:53)

at
org.apache.flink.table.planner.codegen.sort.ComparatorCodeGenerator.gen(ComparatorCodeGenerator.scala)

at
org.apache.flink.table.store.codegen.CodeGeneratorImpl.generateRecordComparator(CodeGeneratorImpl.java:60)

at
org.apache.flink.table.store.codegen.CodeGenUtils.generateRecordComparator(CodeGenUtils.java:67)

at
org.apache.flink.table.store.file.FileStoreImpl.(FileStoreImpl.java:67)

at
org.apache.flink.table.store.connector.TableStore.buildFileStore(TableStore.java:220)

at
org.apache.flink.table.store.connector.TableStore.access$100(TableStore.java:81)

at
org.apache.flink.table.store.connector.TableStore$SinkBuilder.build(TableStore.java:415)

at
org.apache.flink.table.store.connector.sink.TableStoreSink.lambda$getSinkRuntimeProvider$0(TableStoreSink.java:143)

at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.applySinkProvider(CommonExecSink.java:446)

at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.createSinkTransformation(CommonExecSink.java:192)

at
org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.java:67)

at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:148)

at
org.apache.flink.table.planner.delegation.BatchPlanner.$anonfun$translateToPlan$1(BatchPlanner.scala:82)

at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)

at scala.collection.Iterator.foreach(Iterator.scala:937)

at scala.collection.Iterator.foreach$(Iterator.scala:937)

at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)

at scala.collection.IterableLike.foreach(IterableLike.scala:70)

at scala.collection.IterableLike.foreach$(IterableLike.scala:69)

at scala.collection.AbstractIterable.foreach(Iterable.scala:54)

at scala.collection.TraversableLike.map(TraversableLike.scala:233)

at scala.collection.TraversableLike.map$(TraversableLike.scala:226)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at
org.apache.flink.table.planner.delegation.BatchPlanner.translateToPlan(BatchPlanner.scala:81)

at
org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:181)

at
org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1656)

at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:782)

at
org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeModifyOperations$4(LocalExecutor.java:222)

at
org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88)

at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeModifyOperations(LocalExecutor.java:222)

at
org.apache.flink.table.client.cli.CliClient.callInserts(CliClient.java:600)

at
org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:589)

at
org.apache.flink.table.client.cli.CliClient.callOperation(CliClient.java:443)

at
org.apache.flink.table.client.cli.CliClient.executeOperation(CliClient.java:373)

at
org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:330)

at
org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:281)

at
org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:229)

at 

[VOTE] Apache Flink Table Store 0.1.0, release candidate #1

[Jingsong Li]

Hi everyone,

Please review and vote on the release candidate #1 for the version 0.1.0 of
Apache Flink Table Store, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

**Release Overview**

As an overview, the release consists of the following:
a) Table Store canonical source distribution, to be deployed to the
release repository at dist.apache.org
b) Maven artifacts to be deployed to the Maven Central Repository

**Staging Areas to Review**

The staging areas containing the above mentioned artifacts are as follows,
for your review:
* All artifacts for a) and b) can be found in the corresponding dev
repository at dist.apache.org [2]
* All artifacts for c) can be found at the Apache Nexus Repository [3]
* Pre Bundled Binaries Jar can work fine with quick start [4][5]

All artifacts are signed with the key
2C2B6A653B07086B65E4369F7C76245E0A318150 [6]

Other links for your review:
* JIRA release notes [7]
* source code tag "release-0.1.0-rc1" [8]
* PR to update the website Downloads page to include Table Store
links [9]

**Vote Duration**

The voting time will run for at least 72 hours.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Best,
Jingsong Lee

[1] 
https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release
[2] https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.1.0-rc1/
[3] https://repository.apache.org/content/repositories/orgapacheflink-1501/
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1501/org/apache/flink/flink-table-store-dist/0.1.0/flink-table-store-dist-0.1.0.jar
[5] 
https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/
[6] https://dist.apache.org/repos/dist/release/flink/KEYS
[7] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351234
[8] https://github.com/apache/flink-table-store/tree/release-0.1.0-rc1
[9] https://github.com/apache/flink-web/pull/531

Re: [DISCUSS] DockerHub repository maintainers

[Chesnay Schepler]

We could just stop releasing to apache/flink and only go for the 
official-images route.


On 28/04/2022 07:43, Xintong Song wrote:

Forgot to mention that, we have also proposed to use one shared account and
limit its access to the PMC members, like what we do with the PyPI account.
Unfortunately, INFRA rejected this proposal [1].


Thank you~

Xintong Song


[1] https://issues.apache.org/jira/browse/INFRA-23208

On Thu, Apr 28, 2022 at 1:39 PM Xintong Song  wrote:


Hi devs,

I'd like to start a discussion about maintainers for DockerHub
repositories under the *apache* namespace [1].

Currently, the Flink community maintains various repositories (flink,
flink-statefun, flink-statefun-playground, and flink-kubernetes-operator)
on DockerHub under the *apache* namespace. There's a limitation on how many
members the *apache* namespace can add, and recently INFRA is complaining
about Flink taking too many places [2][3]. They would like us to reduce our
maintainers from 20 now to 5.

Jingsong and I would like to volunteer as two of the maintainers, and we
would like to learn who else wants to join us. While any committer may
serve as one of the maintainers, it's probably nice to also involve at
least one maintainer from statefun and one from kubernetes-operator.

What do you think?

Thank you~

Xintong Song


[1] https://hub.docker.com/orgs/apache

[2] https://issues.apache.org/jira/browse/INFRA-23208

[3] https://issues.apache.org/jira/browse/INFRA-23213