date:20230112

Jane Chan created FLINK-30673:
-

 Summary: Add documentation for "EXPLAIN PLAN_ADVICE" statement
 Key: FLINK-30673
 URL: https://issues.apache.org/jira/browse/FLINK-30673
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.17.0
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30672) Support 'EXPLAIN PLAN_ADVICE' statement

Jane Chan created FLINK-30672:
-

 Summary: Support 'EXPLAIN PLAN_ADVICE' statement
 Key: FLINK-30672
 URL: https://issues.apache.org/jira/browse/FLINK-30672
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Client
Affects Versions: 1.17.0
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30671) Add AlgoOperator for ClusteringEvaluator

2023-01-12 Thread Zhipeng Zhang (Jira)

Zhipeng Zhang created FLINK-30671:
-

 Summary: Add AlgoOperator for ClusteringEvaluator
 Key: FLINK-30671
 URL: https://issues.apache.org/jira/browse/FLINK-30671
 Project: Flink
  Issue Type: Improvement
  Components: Library / Machine Learning
Affects Versions: ml-2.2.0
Reporter: Zhipeng Zhang






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30670) Ignore broadcast bytes when computing parallelism and input infos

2023-01-12 Thread Lijie Wang (Jira)

Lijie Wang created FLINK-30670:
--

 Summary: Ignore broadcast bytes when computing parallelism and 
input infos
 Key: FLINK-30670
 URL: https://issues.apache.org/jira/browse/FLINK-30670
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Lijie Wang
 Fix For: 1.17.0


Currently, we include the broadcast bytes in the 
"jobmanager.adaptive-batch-scheduler.avg-data-volume-per-task" when calculating 
the parallelism, and set a cap ratio(0.5) for the broadcast ratio. Considering 
that the broadcast bytes are generally relatively small, we can ignore the 
broadcast bytes to simplify the logic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Enabling dynamic partition discovery by default in Kafka source

2023-01-12 Thread 何军

>
> +1 for this idea, we have enabled kafka dynamic partition discovery in all
> jobs.
>
>

[RESULT][VOTE] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs

2023-01-12 Thread Junrui Lee

Hi all:
FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
[1] has been accepted. The FLIP was voted in this thread[2].

There are 3 bindings, and 2 non-bindings as follows:

Lijie Wang  (binding)
Zhu Zhu (binding)
yuxia (non-binding)
weijie guo (non-binding)
Xintong Song (binding)
There are no disapproving votes.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
[2] https://lists.apache.org/thread/gdymm7pr2slzy9gqkfo97vn73496w0cj

Best regards,
Junrui

[jira] [Created] (FLINK-30669) Update recent job status in FlinkDeployment resource object.

2023-01-12 Thread Mohemmad Zaid Khan (Jira)

Mohemmad Zaid Khan created FLINK-30669:
--

 Summary: Update recent job status in FlinkDeployment resource 
object.
 Key: FLINK-30669
 URL: https://issues.apache.org/jira/browse/FLINK-30669
 Project: Flink
  Issue Type: Bug
Reporter: Mohemmad Zaid Khan
 Attachments: image-2023-01-13-09-54-13-457.png, 
image-2023-01-13-09-54-54-280.png

User jar has code as  -
{code:java}
main() {
 init env
 pipelines.foreach{
  env.fromSource(pipeline.getSource())
 .map(pipeline.transform())
 .sinkTo(pipeline.getSink())
  env.execute(pipeline.getName())
 }
}{code}
and below configuration -
{code:java}
execution.runtime-mode: "BATCH"
execution.attached: "true"
$internal.pipeline.job-id: "" {code}

When this single jar executed in Application Mode by using 
flink-kubernetes-operator, multiple jobs are submitted sequentially and as per 
design only one of the JobStatus is always associated with FlinkDeployment k8s 
resource, this job status is periodically updated by operator. To update job 
status in k8s resource, it fetches all of the job status from job-manager rest 
endpoint and pick the first one and update that one. Problem is, job status 
list returned by job-manager rest api is not sorted on time.

!image-2023-01-13-09-53-18-494.png|width=587,height=326!
!image-2023-01-13-09-54-54-280.png|width=353,height=284!

As you can see in above example, job autoscaling-3 is first one in the rest 
response and same updated in FlinkDeployment resource, but FlinkDeployment 
should have status of job autoscaling-19 because that is the last job finished.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30668) Introduce ExplainFormat to Explainable and TableEnvironment

Jane Chan created FLINK-30668:
-

 Summary: Introduce ExplainFormat to Explainable and 
TableEnvironment
 Key: FLINK-30668
 URL: https://issues.apache.org/jira/browse/FLINK-30668
 Project: Flink
  Issue Type: Sub-task
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Enabling dynamic partition discovery by default in Kafka source

2023-01-12 Thread Qingsheng Ren

Hi devs,

I’d like to start a discussion about enabling the dynamic partition
discovery feature by default in Kafka source. Dynamic partition discovery
[1] is a useful feature in Kafka source especially under the scenario when
the consuming Kafka topic scales out, or the source subscribes to multiple
Kafka topics with a pattern. Users don’t have to restart the Flink job to
consume messages in the new partition with this feature enabled. Currently,
dynamic partition discovery is disabled by default and users have to
explicitly specify the interval of discovery in order to turn it on.

# Breaking changes

For Kafka table source:

- “scan.topic-partition-discovery.interval” will be set to 30 seconds by
default.
- As we need to provide a way for users to disable the feature,
“scan.topic-partition-discovery.interval” = “0” will be used to turn off
the discovery. Before this proposal, “0” means to enable partition
discovery with interval = 0, which is a bit senseless in practice.
Unfortunately we can't use negative values as the type of this option is
Duration.

For KafkaSource (DataStream API)

- Dynamic partition discovery in Kafka source will be enabled by default,
with discovery interval set to 30 seconds.
- To align with table source, only a positive value for option “
partition.discovery.interval.ms” could be used to specify the discovery
interval. Both negative and zero will be interpreted as disabling the
feature.

# Overhead of partition discovery

Partition discovery is made on KafkaSourceEnumerator, which asynchronously
fetches topic metadata from Kafka cluster and checks if there’s any new
topic and partition. This shouldn’t introduce performance issues on the
Flink side.

On the Kafka broker side, partition discovery makes MetadataRequest to
Kafka broker for fetching topic infos. Considering Kafka broker has its
metadata cache and the default request frequency is relatively low (per 30
seconds), this is not a heavy operation and the performance of the broker
won’t be affected a lot. It'll also be great to get some inputs from Kafka
experts.

Looking forward to your feedback!

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kafka/#dynamic-partition-discovery

Best regards,
Qingsheng

[jira] [Created] (FLINK-30667) remove the planner dependency in flink-connector-hive

Chen Qin created FLINK-30667:


 Summary:  remove the planner dependency in flink-connector-hive
 Key: FLINK-30667
 URL: https://issues.apache.org/jira/browse/FLINK-30667
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Affects Versions: 1.17.0
Reporter: Chen Qin
 Fix For: 1.17.0


There are some classes in flink-connector-hive reply on  planner, but 
fortunately, not too many.

It mainly rely on ParserImpl, PlannerContext, PlannerQueryOperation and so on.  
The dependency is mainly required to create RelNode.

To resolve this problem,  we need more abstraction for planner and provides 
public API for external dialects.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30666) Add document for Delete/Update API

luoyuxia created FLINK-30666:


 Summary: Add document for Delete/Update API
 Key: FLINK-30666
 URL: https://issues.apache.org/jira/browse/FLINK-30666
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30665) Add implementation for SupportsRowLevelUpdate

luoyuxia created FLINK-30665:


 Summary: Add implementation for SupportsRowLevelUpdate
 Key: FLINK-30665
 URL: https://issues.apache.org/jira/browse/FLINK-30665
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30664) [Connector/Hive] cleanup hive/haoop package ambiguous package dependencies

Chen Qin created FLINK-30664:


 Summary: [Connector/Hive] cleanup hive/haoop package ambiguous 
package dependencies
 Key: FLINK-30664
 URL: https://issues.apache.org/jira/browse/FLINK-30664
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Affects Versions: 1.17.0
Reporter: Chen Qin
 Fix For: 1.17.0


hive and hive-metastore combination introduced multiple versions of dependency 
packages, the goal is to ensure hive-connector has deterministic dependency 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30663) Add implementation for RowLevelDelete

luoyuxia created FLINK-30663:


 Summary: Add implementation for RowLevelDelete
 Key: FLINK-30663
 URL: https://issues.apache.org/jira/browse/FLINK-30663
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30662) Add implementation for SupportsDeletePushDown

luoyuxia created FLINK-30662:


 Summary: Add implementation for SupportsDeletePushDown
 Key: FLINK-30662
 URL: https://issues.apache.org/jira/browse/FLINK-30662
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30661) Introduce interfaces fro Delete/Update

luoyuxia created FLINK-30661:


 Summary: Introduce interfaces fro Delete/Update
 Key: FLINK-30661
 URL: https://issues.apache.org/jira/browse/FLINK-30661
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30660) move SQLClientHiveITCase and TestHiveCatalogFactory to flink-connector-hive e2e

Chen Qin created FLINK-30660:


 Summary: move SQLClientHiveITCase and TestHiveCatalogFactory to 
flink-connector-hive e2e
 Key: FLINK-30660
 URL: https://issues.apache.org/jira/browse/FLINK-30660
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive, Tests
Affects Versions: 1.17.0
Reporter: Chen Qin
 Fix For: 1.17.0


move SQLClientHiveITCase and TestHiveCatalogFactory to flink-connector-hive e2e

[https://github.com/apache/flink/pull/16532/files#]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30659) move Flink-sql-parser-hive to flink-connector-hive

Chen Qin created FLINK-30659:


 Summary: move Flink-sql-parser-hive to flink-connector-hive
 Key: FLINK-30659
 URL: https://issues.apache.org/jira/browse/FLINK-30659
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive, Table SQL / Planner
Affects Versions: 1.17.0
Reporter: Chen Qin
 Fix For: 1.17.0


Hive Parser should stay with hive connector and maintained together. During 
runtime, those package should load/unload together.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30658) remove Flink-sql-parser-hive dependency on table-planner

Chen Qin created FLINK-30658:


 Summary: remove Flink-sql-parser-hive dependency on table-planner
 Key: FLINK-30658
 URL: https://issues.apache.org/jira/browse/FLINK-30658
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 1.17.0
Reporter: Chen Qin
 Fix For: 1.17.0


In order to move Flink-sql-parser-hive out of Flink-table, we need to remove 
Flink-sql-parser-hive package dependency in Flink-table-planner.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30657) Disable Shared and Key_Shared related tests in Pulsar connector

2023-01-12 Thread Yufan Sheng (Jira)

Yufan Sheng created FLINK-30657:
---

 Summary: Disable Shared and Key_Shared related tests in Pulsar 
connector
 Key: FLINK-30657
 URL: https://issues.apache.org/jira/browse/FLINK-30657
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Pulsar
Affects Versions: 1.15.3, 1.16.0
Reporter: Yufan Sheng
 Fix For: 1.16.1, 1.15.4


As the [FLINK-30413|https://issues.apache.org/jira/browse/FLINK-30413] issue 
talked, we have dropped the Shared and Key_Shared supported in upcoming 
flink-connector-pulsar 4.0 release. The flaky tests of Shared and Key_Shared 
still matters the old Flink build.

Cause these tests are useless now, we can just disable them without any fix.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch Job

2023-01-12 Thread Jing Ge

+ 1(not binding)

Best Regards,
Jing

On Thu, Jan 12, 2023 at 10:01 AM Jing Zhang  wrote:

> +1 (binding)
>
> Best,
> Jing Zhang
>
> Lijie Wang  于2023年1月12日周四 16:39写道：
>
> > +1 (binding)
> >
> > Best,
> > Lijie
> >
> > Martijn Visser  于2023年1月12日周四 15:56写道：
> >
> > > +0 (binding)
> > >
> > > Op di 10 jan. 2023 om 13:11 schreef yuxia  >:
> > >
> > > > +1 (non-binding).
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > - 原始邮件 -
> > > > 发件人: "Zhu Zhu" 
> > > > 收件人: "dev" 
> > > > 发送时间: 星期二, 2023年 1 月 10日 下午 5:50:39
> > > > 主题: Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For
> Batch
> > > Job
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Biao Liu  于2023年1月5日周四 10:37写道：
> > > > >
> > > > > Hi Martijn,
> > > > >
> > > > > Sure, thanks for the reminder about the holiday period.
> > > > > Looking forward to your feedback!
> > > > >
> > > > > Thanks,
> > > > > Biao /'bɪ.aʊ/
> > > > >
> > > > >
> > > > >
> > > > > On Thu, 5 Jan 2023 at 03:07, Martijn Visser <
> > martijnvis...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Biao,
> > > > > >
> > > > > > To be honest, I haven't read the FLIP yet since this is still a
> > > holiday
> > > > > > period in Europe. I would like to read it in the next few days.
> Can
> > > you
> > > > > > keep the vote open a little longer?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Wed, Jan 4, 2023 at 1:31 PM Biao Liu 
> > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks for all the feedback!
> > > > > > >
> > > > > > > Based on the discussion[1], we seem to have a consensus. So I'd
> > > like
> > > > to
> > > > > > > start a vote on FLIP-281: Sink Supports Speculative Execution
> For
> > > > Batch
> > > > > > > Job[2]. The vote will last for 72 hours, unless there is an
> > > > objection or
> > > > > > > insufficient votes.
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-281+Sink+Supports+Speculative+Execution+For+Batch+Job
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Biao /'bɪ.aʊ/
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-12 Thread Jing Ge

Hi Biao,

Thanks for driving this. Like Martijn already pointed out. We will spend
effort to remove SinkFunction after we deprecate it. The more
functionality added into it, the bigger effort we will have to deprecate
and remove the SinkFunction. Commonly, It is not recommended to add new
features into an interface which we already decided to deprecate but do not
do yet. But, this FLIP is a special case and there are some reasons that
lead us to support this proposal.

First, the FLIP offered an equivalent solution for the new SinkV2, which
means the migration from SinkFunction to SinkV2 for this feature is
predictable and acceptable. The concern I raised above has been solved.

Second, since the SinkFunction is still marked as public now [1], it should
be fine to add new features into it (follow the rules), especially if the
requirement is urgent. Similar to [2] described for API graduation, it
should also take 8 months (two release cycles, ideal case is 8 months,
could be longer) to go from @Public to @Deprecated and to be removed.
Additionally, considering the SinkFunction is one core function whose
deletion will trigger a lot of further downstream deletions. The duration
will be increased to be 16 months (again, idea case) or even longer, e.g. 2
years.

Third, the SinkV2 is still marked as @PublicEvolving, which means a few
more months (8 months?) in addition before we can start the deprecation of
SinkFunction. It is not rational to say no features should be added into
SinkFunction during the upcoming 2 or 3 years.

After thinking about all these aspects, I would support this FLIP, so +1

This discussion leads us to another issue: we should graduate SinkV2
and deprecate and remove SinkFunction asap. The longer we keep
the SinkFunktion in the code base, the bigger effort we will have while
working on anything that might depend on sink or has impact on sink.

Best regards,
Jing

[1]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/SinkFunction.java
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process

On Thu, Jan 12, 2023 at 8:56 AM Martijn Visser 
wrote:

> Hi Biao,
>
> While I rather wouldn't add new features to (to-be) deprecated features, I
> would be +0 for this.
>
> Best regards,
>
> Martijn
>
> Op do 12 jan. 2023 om 08:42 schreef Biao Liu :
>
> > Hi Martijn,
> >
> > Thanks for your feedback!
> >
> > Yes, we propose to support speculative execution for SinkFunction.
> > 1. From the perspective of compatibility, SinkFunction is the most
> original
> > Sink implementation.There are lots of implementations based on
> > SinkFunction, not only in Flink official codebase but also in user's
> > private codebase. It's a more serious issue than Sink V1. Of course we
> hope
> > users could migrate the legacy implementation to the new interface.
> However
> > migration is always hard.
> > 2. From the perspective of cost, we don't need to do much extra work to
> > support speculative execution for SinkFunction. All we need to do is
> check
> > whether the SinkFunction implementation
> > inherits SupportsConcurrentExecutionAttempts or not. The other parts of
> > work are the same with Sink V2.
> >
> > To summarize, it's cheap to support speculative execution for
> SinkFunction.
> > And it may allow more existing scenarios to run with speculative
> execution.
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, 11 Jan 2023 at 21:22, Martijn Visser 
> > wrote:
> >
> > > Hi Biao,
> > >
> > > Apologies for the late jumping in. My only question is about
> > SinkFunction,
> > > does this imply that we want to add support for this to the
> SinkFunction?
> > > If so, I would not be in favour of that since we would like to
> deprecate
> > (I
> > > actually thought that was already the case) the SinkFunction in favour
> of
> > > SinkV2.
> > >
> > > Besides that, I have no other comments.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Jan 4, 2023 at 7:28 AM Jing Zhang 
> wrote:
> > >
> > > > Hi Biao,
> > > >
> > > > Thanks for explanation.
> > > >
> > > > +1 for the proposal.
> > > >
> > > > Best,
> > > > Jing Zhang
> > > >
> > > > Lijie Wang  于2023年1月4日周三 12:11写道：
> > > >
> > > > > Hi Biao,
> > > > >
> > > > > Thanks for the explanation of how SinkV2  knows the right subtask
> > > > > attempt. I have no more questions, +1 for the proposal.
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > Biao Liu  于2022年12月28日周三 17:22写道：
> > > > >
> > > > > > Thanks for all your feedback!
> > > > > >
> > > > > > To @Yuxia,
> > > > > >
> > > > > > > What the sink expect to do to isolate data produced by
> > speculative
> > > > > > > executions?  IIUC, if the taks failover, it also generate a new
> > > > > attempt.
> > > > > > > Does it make difference in isolating data produced?
> > > > > >
> > > > > >
> > > > > > Yes there is something different from the task failover scenario.
>

[jira] [Created] (FLINK-30656) Provide more logs for schema compatibility check

2023-01-12 Thread Hangxiang Yu (Jira)

Hangxiang Yu created FLINK-30656:


 Summary: Provide more logs for schema compatibility check
 Key: FLINK-30656
 URL: https://issues.apache.org/jira/browse/FLINK-30656
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Hangxiang Yu
Assignee: Hangxiang Yu


Currently, we have very few logs and exception info when checking schema 
compatibility.

It's difficult to see why the compatibility is not compatible, especially for 
some complicated nested serializers.

For example, for map serializer, when it's not compatible, we may only see 
below without other information:
{code:java}
Caused by: org.apache.flink.util.StateMigrationException: The new state 
serializer (org.apache.flink.api.common.typeutils.base.MapSerializer@e95e076a) 
must not be incompatible with the old state serializer 
(org.apache.flink.api.common.typeutils.base.MapSerializer@c33b100f). {code}
So I think we could add more infos when checking the compatibility.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30655) SqlClientITCase.testMatchRecognize failed with a 404

Matthias Pohl created FLINK-30655:
-

 Summary: SqlClientITCase.testMatchRecognize failed with a 404
 Key: FLINK-30655
 URL: https://issues.apache.org/jira/browse/FLINK-30655
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.17.0
Reporter: Matthias Pohl


We experience a test instability with SqlClientITCase.testMatchRecognize 
failing due to a 404 error:

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44755=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=15080]
{code:java}
Jan 12 14:51:42 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 208.308 s <<< FAILURE! - in SqlClientITCase
Jan 12 14:51:42 [ERROR] SqlClientITCase.testMatchRecognize  Time elapsed: 
49.435 s  <<< ERROR!
Jan 12 14:51:42 com.github.dockerjava.api.exception.NotFoundException: 
Jan 12 14:51:42 Status 404: {"message":"Could not find the file 
/flink/records-matchrecognize.out in container 
403f2ed027372780e0eb2b3ea33f300b93d33eeb1ecfb72ff7b19ce28b73a0ac"}
Jan 12 14:51:42 
Jan 12 14:51:42 at 
org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:241)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

2023-01-12 Thread Shammon FY

+1 (no-binding)


Best,
Shammon

On Thu, Jan 12, 2023 at 8:11 PM Shengkai Fang  wrote:

> +1(binding)
>
> Best,
> Shengkai
>
> Jark Wu  于2023年1月12日周四 19:22写道：
>
> > +1 (binding)
> > Thank you for driving this effort.
> >
> > Best,
> > Jark
> >
> > > 2023年1月9日 15:46，Jing Ge  写道：
> > >
> > > Hi,
> > >
> > > I'd like to start a vote on FLIP-279 Unified the max display column
> width
> > > for SqlClient and Table APi in both Streaming and Batch execMode. The
> > > discussion can be found at [1].
> > >
> > > The vote will last for at least 72 hours (Jan 12th at 9:00 GMT) unless
> > > there is an objection or insufficient votes.
> > >
> > > [1] https://lists.apache.org/thread/f9p622k8cgcjl0r0b44np5wm8krhtjjz
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-279+Unified+the+max+display+column+width+for+SqlClient+and+Table+APi+in+both+Streaming+and+Batch+execMode
> > >
> > > Best regards,
> > > Jing
> >
> >
>

[jira] [Created] (FLINK-30654) start cursor issue

2023-01-12 Thread likang (Jira)

likang created FLINK-30654:
--

 Summary: start cursor issue
 Key: FLINK-30654
 URL: https://issues.apache.org/jira/browse/FLINK-30654
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Pulsar
Affects Versions: pulsar-4.0.0
Reporter: likang


Pulsar does not take effect when the subscription is set to start position 
consumption. It is recommended to add an option to be determined by the user or 
to add a lastAck strategy to adapt to the scenario of whether to start 
consumption from the last consumption submission position

!image-2023-01-12-23-13-48-411.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30653) Triggere resource Events on autoscaler actions

2023-01-12 Thread Gyula Fora (Jira)

Gyula Fora created FLINK-30653:
--

 Summary: Triggere resource Events on autoscaler actions
 Key: FLINK-30653
 URL: https://issues.apache.org/jira/browse/FLINK-30653
 Project: Flink
  Issue Type: New Feature
  Components: Autoscaler, Kubernetes Operator
Reporter: Gyula Fora
Assignee: Matyas Orhidi
 Fix For: kubernetes-operator-1.4.0


The autoscaler currently doesn't trigger any events. We should use Kube events 
to notify users about scaling actions on a per-vertex level or in cases where 
the autoscaler blocked some scaling action due to past history.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

2023-01-12 Thread Matthias Pohl

Thanks Yang Wang for sharing your view on this. Please find my responses
below.

# HA data format in the HA backend(e.g. ZK, K8s ConfigMap)
> We have already changed the HA data format after introducing the multiple
> component leader election in FLINK-24038. For K8s HA,
> the num of ConfigMaps reduced from 4 to 2. Since we only have one leader
> elector, the K8s APIServer load should also be reduced.
> Why do we still need to change the format again? This just prevents the
> LAST_STATE upgrade mode in Flink-Kubernetes-Operator
> when the Flink version changed, even though it is a simple job and state is
> compatible.
>

The intention of this remark is that we could reduce the number of
redundant records (the ZooKeeperMultipleComponentLeaderElectionHaServices'
JavaDoc [1] visualizes the redundancy quite well since each of these
connection_info records would contain the very same information). We're
saving the same connection_info for each of the componentIds (e.g.
resource_manager, dispatcher, ...) right now. My rationale was that we only
need to save the connection info once per LeaderElectionDriver, i.e.
LeaderElectionService. It's an implementation detail of the
LeaderElectionService implementation to know what components it owns.
Therefore, I suggested that we would have a unique ID per
LeaderElectionService instance with a single connection_info that is used
by all the components that are registered to that service. If we decide to
have a separate LeaderElectionService for a specific component (e.g. the
resource manager) in the future, this would end up having a separate
ConfigMap in k8s or separate ZNode in ZooKeeper.

I added these details to the FLIP [2]. That part, indeed, was quite poorly
described there initally.

I don't understand how the leader election affects the LAST_STATE changes
in the Kubernetes Operator, though. We use a separate ConfigMap for the
checkpoint data [3]. Can you elaborate a little bit more on your concern?

[1]
https://github.com/apache/flink/blob/8ddfd590ebba7fc727e79db41b82d3d40a02b56a/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperMultipleComponentLeaderElectionHaServices.java#L47-L61
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-285%3A+Refactoring+LeaderElection+to+make+Flink+support+multi-component+leader+election+out-of-the-box#ha-backend-data-schema
[3]
https://github.com/apache/flink/blob/2770acee1bc4a82a2f4223d4a4cd6073181dc840/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesMultipleComponentLeaderElectionHaServices.java#L163


> # LeaderContender#initiateLeaderElection
> I do not get your point why we need the *initiateLeaderElection* in
> *LeaderContender*. AFAICS, the callback *onGrant/RevokeLeadership*
> could be executed as soon as the registration.
>

That's the change I'm not really happy about. I'm still not sure whether I
found the best solution here. The problem is the way the components are
initialized. The initial plan was to call the
LeaderElectionService.register(LeaderContender) from within the
LeaderContender constructor which would return the LeaderElection instance
that would be used as the adapter for the contender to confirm leadership.
Therefore, the LeaderContender has to own the LeaderElection instance to be
able to participate in the leader election handshake (i.e. grant leadership
-> confirm leadership). With that setup, we wouldn't be able to grant
leadership during the LeaderElectionService.register(LeaderContender) call
because that would require for the LeaderContender to confirm the
leadership. That is not possible because the LeaderElection wasn't created
and set within the LeaderContender, yet. Therefore, we have to have some
means to initialize the LeaderElection before triggering granting the
leadership from within the LeaderElectionService. ...and that's what this
method is for.

Best,
Matthias

[jira] [Created] (FLINK-30652) Use max busytime instead of average to compute true processing rate

2023-01-12 Thread Gyula Fora (Jira)

Gyula Fora created FLINK-30652:
--

 Summary: Use max busytime instead of average to compute true 
processing rate
 Key: FLINK-30652
 URL: https://issues.apache.org/jira/browse/FLINK-30652
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler, Kubernetes Operator
Affects Versions: kubernetes-operator-1.4.0
Reporter: Gyula Fora
Assignee: Gyula Fora
 Fix For: kubernetes-operator-1.4.0


Currently we use the some of busyTimes and processed records to estimate the 
true processing rate.

This computation however is not correct when any data skew is present as TPR is 
not fully additive. The first task to reach 100% utilization will set a limit 
to the pipeline processing rate through backpressure.

To avoid this we should use the max busyTime and compute the TPR from that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-280: Introduce EXPLAIN PLAN_ADVICE to provide SQL advice

2023-01-12 Thread Jane Chan

Hi all,

FLIP-280: Introduce EXPLAIN PLAN_ADVICE to provide SQL advice[1] has been
accepted. The FLIP was voted through this thread[2].

There are four binding votes as follows:

Godfrey  (binding)
Jark Wu  (binding)
Jingsong Li  (binding)
Lincoln Lee  (binding)


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-280%3A+Introduce+EXPLAIN+PLAN_ADVICE+to+provide+SQL+advice
[2] https://lists.apache.org/thread/bsgqvvs9wx1dkv7p3m9ctockh84rl11j

Best regards,
Jane Chan

[jira] [Created] (FLINK-30651) Move util methods to CatalogTest and remove CatalogTestUtils class

2023-01-12 Thread Samrat Deb (Jira)

Samrat Deb created FLINK-30651:
--

 Summary: Move util methods to CatalogTest and remove 
CatalogTestUtils class 
 Key: FLINK-30651
 URL: https://issues.apache.org/jira/browse/FLINK-30651
 Project: Flink
  Issue Type: Technical Debt
  Components: Table SQL / API, Tests
Reporter: Samrat Deb
 Fix For: 1.17.0


[CatalogTestUtils|https://github.com/apache/flink/blame/master/flink-table/flink-table-common/src/test/java/org/apache/flink/table/catalog/CatalogTestUtil.java#L43]
 class contains static utilities function. This functions/ methods can be moved 
to CatalogTest class and make code-flow easier to understand.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30650) FLIP-280: Introduce EXPLAIN PLAN_ADVICE to provide SQL advice

Jane Chan created FLINK-30650:
-

 Summary: FLIP-280: Introduce EXPLAIN PLAN_ADVICE to provide SQL 
advice
 Key: FLINK-30650
 URL: https://issues.apache.org/jira/browse/FLINK-30650
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Affects Versions: 1.17.0
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Apache Flink Table Store 0.3.0, release candidate #1

2023-01-12 Thread Jingsong Li

Thanks Yu, I have created table-store-0.3.1 and move these jiras to 0.3.1.

Best,
Jingsong

On Thu, Jan 12, 2023 at 7:41 PM Yu Li  wrote:
>
> Thanks for the quick action Jingsong! Here is my vote with the new staging
> directory:
>
> +1 (binding)
>
>
> - Checked release notes: *Action Required*
>
>   * The fix version of FLINK-30620 and FLINK-30628 are 0.3.0 but still
> open, please confirm whether this should be included or we should move it
> out of 0.3.0
>
> - Checked sums and signatures: *OK*
>
> - Checked the jars in the staging repo: *OK*
>
> - Checked source distribution doesn't include binaries: *OK*
>
> - Maven clean install from source: *OK*
>
> - Checked version consistency in pom files: *OK*
>
> - Went through the quick start: *OK*
>
>   * Verified with flink 1.14.6, 1.15.3 and 1.16.0
>
> - Checked the website updates: *OK*
>
> Best Regards,
> Yu
>
>
> On Thu, 12 Jan 2023 at 15:36, Jingsong Li  wrote:
>
> > Thanks Yu for your validation.
> >
> > I created a new staging directory [1]
> >
> > [1]
> > https://repository.apache.org/content/repositories/orgapacheflink-1577/
> >
> > Best,
> > Jingsong
> >
> > On Thu, Jan 12, 2023 at 3:07 PM Yu Li  wrote:
> > >
> > > Hi Jingsong,
> > >
> > > It seems the given staging directory [1] is not exposed, could you double
> > > check and republish if necessary? Thanks.
> > >
> > > Best Regards,
> > > Yu
> > >
> > > [1]
> > https://repository.apache.org/content/repositories/orgapacheflink-1576/
> > >
> > >
> > > On Tue, 10 Jan 2023 at 16:53, Jingsong Li 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > > > 0.3.0 of Apache Flink Table Store, as follows:
> > > >
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > **Release Overview**
> > > >
> > > > As an overview, the release consists of the following:
> > > > a) Table Store canonical source distribution to be deployed to the
> > > > release repository at dist.apache.org
> > > > b) Table Store binary convenience releases to be deployed to the
> > > > release repository at dist.apache.org
> > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > >
> > > > **Staging Areas to Review**
> > > >
> > > > The staging areas containing the above mentioned artifacts are as
> > follows,
> > > > for your review:
> > > > * All artifacts for a) and b) can be found in the corresponding dev
> > > > repository at dist.apache.org [2]
> > > > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> > > >
> > > > All artifacts are signed with the key
> > > > 2C2B6A653B07086B65E4369F7C76245E0A318150 [4]
> > > >
> > > > Other links for your review:
> > > > * JIRA release notes [5]
> > > > * source code tag "release-0.3.0-rc1" [6]
> > > > * PR to update the website Downloads page to include Table Store links
> > [7]
> > > >
> > > > **Vote Duration**
> > > >
> > > > The voting time will run for at least 72 hours.
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > votes.
> > > >
> > > > Best,
> > > > Jingsong Lee
> > > >
> > > > [1]
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release
> > > > [2]
> > > >
> > https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.3.0-rc1/
> > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1576/
> > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [5]
> > > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352111
> > > > [6] https://github.com/apache/flink-table-store/tree/release-0.3.0-rc1
> > > > [7] https://github.com/apache/flink-web/pull/601
> > > >
> >

[jira] [Created] (FLINK-30649) Run kubernetes session test (default input) failed with a timeout

Matthias Pohl created FLINK-30649:
-

 Summary: Run kubernetes session test (default input) failed with a 
timeout
 Key: FLINK-30649
 URL: https://issues.apache.org/jira/browse/FLINK-30649
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.17.0
Reporter: Matthias Pohl


{{Run kubernetes session test (default input)}} failed with a timeout.
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44748=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14=6317]

It appears that there was some issue with shutting down the pods of the 
MiniCluster:
{code:java}
2023-01-12T08:22:13.1388597Z timed out waiting for the condition on 
pods/flink-native-k8s-session-1-7dc9976688-gq788
2023-01-12T08:22:13.1390040Z timed out waiting for the condition on 
pods/flink-native-k8s-session-1-taskmanager-1-1 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

2023-01-12 Thread Shengkai Fang

+1(binding)

Best,
Shengkai

Jark Wu  于2023年1月12日周四 19:22写道：

> +1 (binding)
> Thank you for driving this effort.
>
> Best,
> Jark
>
> > 2023年1月9日 15:46，Jing Ge  写道：
> >
> > Hi,
> >
> > I'd like to start a vote on FLIP-279 Unified the max display column width
> > for SqlClient and Table APi in both Streaming and Batch execMode. The
> > discussion can be found at [1].
> >
> > The vote will last for at least 72 hours (Jan 12th at 9:00 GMT) unless
> > there is an objection or insufficient votes.
> >
> > [1] https://lists.apache.org/thread/f9p622k8cgcjl0r0b44np5wm8krhtjjz
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-279+Unified+the+max+display+column+width+for+SqlClient+and+Table+APi+in+both+Streaming+and+Batch+execMode
> >
> > Best regards,
> > Jing
>
>

[jira] [Created] (FLINK-30648) [Umbrella]FLIP-282 Introduce Delete/Update API

luoyuxia created FLINK-30648:


 Summary: [Umbrella]FLIP-282 Introduce Delete/Update API
 Key: FLINK-30648
 URL: https://issues.apache.org/jira/browse/FLINK-30648
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: luoyuxia






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-282: Introduce Delete & Update API

2023-01-12 Thread yuxia

Hi all; 
FLIP-282: Introduce Delete & Update API [1] has been accepted. The FLIP was 
voted in this thread[2]. 

There are 3 bindings, and 1 non-bindings as follows: 

Samrat Deb (no-binding) 
Jingsong Li (binding) 
Lincoln Lee (binding) 
Jark (binding) 
There are no disapproving votes. 

:) btw, sorry for the nosiy vote result email in vote thread[2] 

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061 
[2] https://lists.apache.org/thread/3m91zys8dgnzyypoq88gkb2t55lqvgh1 


Best regards, 
Yuxia

[jira] [Created] (FLINK-30647) Add human readable datetime of watermark column in the web for better debug experience

2023-01-12 Thread Yun Tang (Jira)

Yun Tang created FLINK-30647:


 Summary: Add human readable datetime of watermark column in the 
web for better debug experience
 Key: FLINK-30647
 URL: https://issues.apache.org/jira/browse/FLINK-30647
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Web Frontend
Reporter: Yun Tang
 Fix For: 1.17.0
 Attachments: image-2023-01-12-19-39-59-585.png

Currently, when we want to check the watermark of different tasks in the web 
ui, it reported as unix timestamp. However, this is not human readable and not 
easy to figure out the processing progress.

 !image-2023-01-12-19-39-59-585.png! 

We can introduce another column to parse the unix timestamp to offer better 
debug experience.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Apache Flink Table Store 0.3.0, release candidate #1

2023-01-12 Thread Yu Li

Thanks for the quick action Jingsong! Here is my vote with the new staging
directory:

+1 (binding)


- Checked release notes: *Action Required*

  * The fix version of FLINK-30620 and FLINK-30628 are 0.3.0 but still
open, please confirm whether this should be included or we should move it
out of 0.3.0

- Checked sums and signatures: *OK*

- Checked the jars in the staging repo: *OK*

- Checked source distribution doesn't include binaries: *OK*

- Maven clean install from source: *OK*

- Checked version consistency in pom files: *OK*

- Went through the quick start: *OK*

  * Verified with flink 1.14.6, 1.15.3 and 1.16.0

- Checked the website updates: *OK*

Best Regards,
Yu


On Thu, 12 Jan 2023 at 15:36, Jingsong Li  wrote:

> Thanks Yu for your validation.
>
> I created a new staging directory [1]
>
> [1]
> https://repository.apache.org/content/repositories/orgapacheflink-1577/
>
> Best,
> Jingsong
>
> On Thu, Jan 12, 2023 at 3:07 PM Yu Li  wrote:
> >
> > Hi Jingsong,
> >
> > It seems the given staging directory [1] is not exposed, could you double
> > check and republish if necessary? Thanks.
> >
> > Best Regards,
> > Yu
> >
> > [1]
> https://repository.apache.org/content/repositories/orgapacheflink-1576/
> >
> >
> > On Tue, 10 Jan 2023 at 16:53, Jingsong Li 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the version
> > > 0.3.0 of Apache Flink Table Store, as follows:
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > **Release Overview**
> > >
> > > As an overview, the release consists of the following:
> > > a) Table Store canonical source distribution to be deployed to the
> > > release repository at dist.apache.org
> > > b) Table Store binary convenience releases to be deployed to the
> > > release repository at dist.apache.org
> > > c) Maven artifacts to be deployed to the Maven Central Repository
> > >
> > > **Staging Areas to Review**
> > >
> > > The staging areas containing the above mentioned artifacts are as
> follows,
> > > for your review:
> > > * All artifacts for a) and b) can be found in the corresponding dev
> > > repository at dist.apache.org [2]
> > > * All artifacts for c) can be found at the Apache Nexus Repository [3]
> > >
> > > All artifacts are signed with the key
> > > 2C2B6A653B07086B65E4369F7C76245E0A318150 [4]
> > >
> > > Other links for your review:
> > > * JIRA release notes [5]
> > > * source code tag "release-0.3.0-rc1" [6]
> > > * PR to update the website Downloads page to include Table Store links
> [7]
> > >
> > > **Vote Duration**
> > >
> > > The voting time will run for at least 72 hours.
> > > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release
> > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.3.0-rc1/
> > > [3]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1576/
> > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [5]
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352111
> > > [6] https://github.com/apache/flink-table-store/tree/release-0.3.0-rc1
> > > [7] https://github.com/apache/flink-web/pull/601
> > >
>

Re: [VOTE] FLIP-282: Introduce Delete & Update API

2023-01-12 Thread yuxia

Hi all;
FLIP-282: Introduce Delete & Update API [1] has been accepted. The FLIP was 
voted in this thread[2].

There are 3 bindings, and 1 non-bindings as follows:

Samrat Deb (no-binding)
Jingsong Li (binding)
Lincoln Lee (binding)
Jark (binding)
There are no disapproving votes.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
[2] https://lists.apache.org/thread/3m91zys8dgnzyypoq88gkb2t55lqvgh1

Best regards,
Yuxia

- 原始邮件 -
发件人: "Jark Wu" 
收件人: "dev" 
发送时间: 星期二, 2023年 1 月 10日 下午 5:42:10
主题: Re: [VOTE] FLIP-282: Introduce Delete & Update API

+1 (binding)

Best,
Jark

> 2023年1月10日 17:39，Lincoln Lee  写道：
> 
> +1 (binding)
> 
> Best,
> Lincoln Lee
> 
> 
> Jingsong Li  于2023年1月10日周二 09:56写道：
> 
>> +1 binding
>> 
>> On Mon, Jan 9, 2023 at 6:14 PM Samrat Deb  wrote:
>>> 
>>> +1 (non binding )
>>> 
>>> thank you for driving
>>> 
>>> 
>>> 
>>> On Mon, 9 Jan 2023 at 3:36 PM, yuxia 
>> wrote:
>>> 
 Hi, all.

 I'd like to start a vote on FLIP-282: Introduce Delete & Update API[1].
 You can find the discussion on it in here[2].

 The vote will last for at least 72 hours (Jan 12th at 10:00 AM GMT )
 unless there is objection or insufficient votes.

 [1] [

>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
 |

>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
 ]
 [2] [ https://lists.apache.org/thread/6h64v7v6gj916pkvmc3ql3vxxccr46r3
>> |
 https://lists.apache.org/thread/6h64v7v6gj916pkvmc3ql3vxxccr46r3 ]

 Best regards,
 Yuxia

>>

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

2023-01-12 Thread Jark Wu

+1 (binding)
Thank you for driving this effort. 

Best,
Jark

> 2023年1月9日 15:46，Jing Ge  写道：
> 
> Hi,
> 
> I'd like to start a vote on FLIP-279 Unified the max display column width
> for SqlClient and Table APi in both Streaming and Batch execMode. The
> discussion can be found at [1].
> 
> The vote will last for at least 72 hours (Jan 12th at 9:00 GMT) unless
> there is an objection or insufficient votes.
> 
> [1] https://lists.apache.org/thread/f9p622k8cgcjl0r0b44np5wm8krhtjjz
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-279+Unified+the+max+display+column+width+for+SqlClient+and+Table+APi+in+both+Streaming+and+Batch+execMode
> 
> Best regards,
> Jing

[jira] [Created] (FLINK-30646) Table Store Hive catalog throws ClassNotFoundException when custom hive-site.xml is presented

2023-01-12 Thread Caizhi Weng (Jira)

Caizhi Weng created FLINK-30646:
---

 Summary: Table Store Hive catalog throws ClassNotFoundException 
when custom hive-site.xml is presented
 Key: FLINK-30646
 URL: https://issues.apache.org/jira/browse/FLINK-30646
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Affects Versions: table-store-0.3.0, table-store-0.4.0
Reporter: Caizhi Weng
Assignee: Caizhi Weng
 Fix For: table-store-0.3.0, table-store-0.4.0


For Hive 2.3.9, if a custom {{hive-site.xml}} is presented in 
{{$HIVE_HOME/conf}}, when creating Table Store Hive catalog in Flink, the 
following exception will be thrown.

{code}
Caused by: java.lang.ClassNotFoundException: Class 
org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not found
at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2273) 
~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2367) 
~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2393) 
~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.loadFilterHooks(HiveMetaStoreClient.java:250)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:145)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) ~[?:1.8.0_352]
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 ~[?:1.8.0_352]
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 ~[?:1.8.0_352]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 
~[?:1.8.0_352]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.shaded.org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:97)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.hive.HiveCatalog.createClient(HiveCatalog.java:415)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.hive.HiveCatalog.(HiveCatalog.java:82) 
~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:51)
 ~[flink-table-store-hive-catalog-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.file.catalog.CatalogFactory.createCatalog(CatalogFactory.java:106)
 ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.connector.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:66)
 ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.connector.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:57)
 ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.store.connector.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:31)
 ~[flink-table-store-dist-0.4-SNAPSHOT.jar:0.4-SNAPSHOT]
at 
org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:435)
 ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.createCatalog(TableEnvironmentImpl.java:1426)
 ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
at 
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1172)
 ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
at 
org.apache.flink.table.client.gateway.local.LocalExecutor.executeOperation(LocalExecutor.java:206)
 ~[flink-sql-client-1.16.0.jar:1.16.0]
... 10 more
{code}



--
This

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

2023-01-12 Thread Yanfei Lei

Hi all,

Thanks for the reminder.

@Matthias

any updates on the performance tests? ...or more specifically, any updates
on the script for alerting on performance regressions?


I create a PR for FLINK-27571[1] but it's still under review, would you
like to help take a look?

FLINK-27571 is just for the new benchmarks, for the old existing
benchmarks, their information is stored

in codespeed's database which can't be updated by URL request, so I also
logged into the Jenkins master

and modified the codespeed's database, currently "less is better" can be
displayed normally on the timeline[2].


Does it make sense to formalize/document the process?

Certainly, I'm preparing a draft to share my experience of finding commits
that caused regressions.

Originally, I wanted to wait for FLINK-27571 to be merged before starting a
discussion, and I will put

a draft of the document later.


This slack channel can only provide notice of regression and some
experience on how to locate regression,

but we also need some people to take action after the regression happens.
It is mainly a few people who volunteer to do these things,

like FLINK-30015[3] and FLINK-30623[4], many thanks for Martijn's
contribution.

As for whether to add the responsibilities to the release manager, I think
it needs to see other people's opinions.

@Martijn

Thanks for creating these tickets. For FLINK-30623 and FLINK-30624[5],
@Hangxiang and I have located the corresponding commit

and pinged the corresponding submitter. Regression may not be avoided, I
totally do agree that this work needs to be formalized as soon as possible
to fix regressions.


[1] https://issues.apache.org/jira/browse/FLINK-27571

[2]
http://codespeed.dak8s.net:8000/timeline/#/?ben=createScheduler.BATCH=on=on=off=2=200=1,3,5,6,8,9

[3] https://issues.apache.org/jira/browse/FLINK-30015

[4] https://issues.apache.org/jira/browse/FLINK-30623

[5] https://issues.apache.org/jira/browse/FLINK-30624


Best regards,

Yanfei

Martijn Visser  于2023年1月11日周三 01:11写道：

> Hi all,
>
> Related to Matthias' email, I've checked the notifications in the Slack
> channel and noticed three major benchmark regressions. In the end, I've
> decided to create Jira tickets for it [1] [2] [3] but I do agree that this
> work needs to be formalized as soon as possible to avoid regressions. It
> would also be great to include a process on how these regressions will be
> fixed, because I have no idea who to ping/notify that these regressions
> have occurred.
>
> Best regards,
>
> Martijn
>
> [1] https://issues.apache.org/jira/browse/FLINK-30623
> [2] https://issues.apache.org/jira/browse/FLINK-30624
> [3] https://issues.apache.org/jira/browse/FLINK-30625
>
> On Tue, Jan 10, 2023 at 1:56 PM Matthias Pohl
>  wrote:
>
> > Hi Yanfei,
> > any updates on the performance tests? ...or more specifically, any
> updates
> > on the script for alerting on performance regressions?
> >
> > Does it make sense to formalize/document the process? Currently, the
> > release management doesn't do anything in terms of performance
> > test monitoring. Therefore, performance regressions are not necessarily
> > identified actively (in contrast to CI instabilities). Or is this covered
> > by the PMC? It would be interesting to know whether there's someone to
> > reach out to who's monitoring the regression tests regularly. Would it
> make
> > sense for this person to join the release calls?
> >
> > Or shall we work on formalizing/documenting the process and integrating
> > this responsibility into what the release manager(s) are in charge of? My
> > concern with that approach is that contributors might be less willing to
> > volunteer in the release management if we collect everything in one role.
> > Alternatively, we could split the release manager role up into sub-roles
> > that contributors can volunteer for in a release (e.g. CI monitoring,
> > performance test monitoring, Jira maintenance, ... just coming up with
> > random tasks here).
> >
> > Alternatively, we could leave everything as is and just respond if
> there's
> > some complaint. I'm curious about your (and other's) opinions.
> >
> > Matthias
> >
> > On Tue, Nov 29, 2022 at 2:13 PM Yanfei Lei  wrote:
> >
> > > Hi Martijn,
> > >
> > > Thanks for bringing this up.
> > >
> > > In the past two months, this channel has helped us find many benchmark
> > fail
> > > issues, like FLINK-29883
> > > [1],
> > > FLINK-29886 [2],
> > > FLINK-30015 [3] and
> > > FLINK-30181 [4]. I
> > also
> > > have tried investigating several of the frequently reported regressions
> > and
> > > replied under the notification in slack channel(copy them here):
> > >
> > >1. serializerHeavyString
> > ><
> > >
> >
>

[DISCUSS] FLIP-286: Fix the API stability/scope annotation inconsistency in AbstractStreamOperator

2023-01-12 Thread Becket Qin

Hi flink devs,

I'd like to start a discussion thread for FLIP-286[1].

As a recap, currently while AbstractStreamOperator is a class marked as
@PublicEvolving, some classes exposed via its methods / fields are
marked as @Internal. This FLIP wants to fix this inconsistency of
stability / scope annotation.

Comments are welcome!

Thanks,

Jiangjie (Becket) Qin

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240880841

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

2023-01-12 Thread Yang Wang

Thanks Matthias for preparing this thorough FLIP, which has taken us
reviewing the multiple component leader election.

I am totally in favor of doing the code clean-up. The current
implementation does not have very good readability due to legacy
compatibility.
And I just have a few comments.

# HA data format in the HA backend(e.g. ZK, K8s ConfigMap)
We have already changed the HA data format after introducing the multiple
component leader election in FLINK-24038. For K8s HA,
the num of ConfigMaps reduced from 4 to 2. Since we only have one leader
elector, the K8s APIServer load should also be reduced.
Why do we still need to change the format again? This just prevents the
LAST_STATE upgrade mode in Flink-Kubernetes-Operator
when the Flink version changed, even though it is a simple job and state is
compatible.

# LeaderContender#initiateLeaderElection
I do not get your point why we need the *initiateLeaderElection* in
*LeaderContender*. AFAICS, the callback *onGrant/RevokeLeadership*
could be executed as soon as the registration.

# When to establish the HA backend connection
+1 for establish the connection beforehand


Best,
Yang

Matthias Pohl  于2023年1月5日周四 21:51写道：

> Hi everyone,
> I brought up FLINK-26522 [1] in the mailing list discussion about
> consolidating the HighAvailabilityServices interfaces [2], previously.
> There, it was concluded that the community still wants the ability to have
> per-component leader election and, therefore, keep the
> HighAvailabilityServices interface as is. I went back to work on
> FLINK-26522 [1] to figure out how we can simplify the current codebase
> keeping the decision in mind.
>
> I wanted to handle FLINK-26522 [1] as a follow-up cleanup task of
> FLINK-24038 [3]. But while working on it, I realized that even FLINK-24038
> [3] shouldn't have been handled without a FLIP. The per-process leader
> election which was introduced in FLINK-24038 [3] changed the ownership of
> certain components. This is actually a change that should have been
> discussed in the mailing list and deserved a FLIP. To overcome this
> shortcoming of FLINK-24038 [3], I decided to prepare FLIP-285 [4] to
> provide proper documentation of what happened in FLINK-24038 and what will
> be manifested with resolving its follow-up FLINK-26522 [1].
>
> Conceptually, this FLIP proposes moving away from Flink's support for
> single-contender LeaderElectionServices and introducing multi-contender
> support by disconnecting the HA-backend leader election lifecycle from the
> LeaderContender's lifecycle. This allows us to provide LeaderElection per
> component (as it was requested in [2]) but also enables us to utilize a
> single leader election for multiple components/contenders as well without
> the complexity of the code that was introduced by FLINK-24038 [3].
>
> I'm looking forward to your comments.
>
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-26522
> [2] https://lists.apache.org/thread/9oy2ml9s3j1v6r77h31sndhc3gw57cfm
> [3] https://issues.apache.org/jira/browse/FLINK-24038
> [4]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-285%3A+Refactoring+LeaderElection+to+make+Flink+support+multi-component+leader+election+out-of-the-box
>

[jira] [Created] (FLINK-30645) [FLIP-286] The scope/stability annotation in AbstractStreamOperator are inconsistent.

2023-01-12 Thread Jiangjie Qin (Jira)

Jiangjie Qin created FLINK-30645:


 Summary: [FLIP-286] The scope/stability annotation in 
AbstractStreamOperator are inconsistent.
 Key: FLINK-30645
 URL: https://issues.apache.org/jira/browse/FLINK-30645
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.16.0
Reporter: Jiangjie Qin


It looks that currently the {{{_}AbstractStreamOperator}}{_} API has some 
scope/stability annotation inconsistency. More specifically,
 * The {{AbstractStreamOperator}} class is marked as {{_@PublicEvolving_}}
 * {{AbstractStreamOperator.getInternalTimerService()}} returns a type of 
{{InternalTimerService}} __ which is marked as {{@Internal}}
 * {{InternalOperatorMetricGroup}} and {{InternalIOperatorIOMetricGroup}} __ 
are also available to the subclasses of {{AbstractStreamOperator}} but marked 
as {{{}@Internal{}}}.

FLIP-286 proposes to fix the above annotation inconsistency by marking the 
following classes \{{@PublicEvolving}}.
 * org.apache.flink.streaming.api.operators.InternalTimerService
 * org.apache.flink.runtime.metrics.groups.InternalOperatorMetricGroup
 * org.apache.flink.runtime.metrics.groups.InternalOperatorIOMetricGroup

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch Job

2023-01-12 Thread Jing Zhang

+1 (binding)

Best,
Jing Zhang

Lijie Wang  于2023年1月12日周四 16:39写道：

> +1 (binding)
>
> Best,
> Lijie
>
> Martijn Visser  于2023年1月12日周四 15:56写道：
>
> > +0 (binding)
> >
> > Op di 10 jan. 2023 om 13:11 schreef yuxia :
> >
> > > +1 (non-binding).
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Zhu Zhu" 
> > > 收件人: "dev" 
> > > 发送时间: 星期二, 2023年 1 月 10日 下午 5:50:39
> > > 主题: Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch
> > Job
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Biao Liu  于2023年1月5日周四 10:37写道：
> > > >
> > > > Hi Martijn,
> > > >
> > > > Sure, thanks for the reminder about the holiday period.
> > > > Looking forward to your feedback!
> > > >
> > > > Thanks,
> > > > Biao /'bɪ.aʊ/
> > > >
> > > >
> > > >
> > > > On Thu, 5 Jan 2023 at 03:07, Martijn Visser <
> martijnvis...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Biao,
> > > > >
> > > > > To be honest, I haven't read the FLIP yet since this is still a
> > holiday
> > > > > period in Europe. I would like to read it in the next few days. Can
> > you
> > > > > keep the vote open a little longer?
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Wed, Jan 4, 2023 at 1:31 PM Biao Liu 
> wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Thanks for all the feedback!
> > > > > >
> > > > > > Based on the discussion[1], we seem to have a consensus. So I'd
> > like
> > > to
> > > > > > start a vote on FLIP-281: Sink Supports Speculative Execution For
> > > Batch
> > > > > > Job[2]. The vote will last for 72 hours, unless there is an
> > > objection or
> > > > > > insufficient votes.
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-281+Sink+Supports+Speculative+Execution+For+Batch+Job
> > > > > >
> > > > > > Thanks,
> > > > > > Biao /'bɪ.aʊ/
> > > > > >
> > > > >
> > >
> >
>

[jira] [Created] (FLINK-30644) ChangelogCompatibilityITCase.testRestore fails due to CheckpointCoordinator being shutdown

Matthias Pohl created FLINK-30644:
-

 Summary: ChangelogCompatibilityITCase.testRestore fails due to 
CheckpointCoordinator being shutdown
 Key: FLINK-30644
 URL: https://issues.apache.org/jira/browse/FLINK-30644
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Runtime / State Backends
Affects Versions: 1.17.0
Reporter: Matthias Pohl


We observe a build failure in {{ChangelogCompatibilityITCase.testRestore}} due 
to the {{CheckpointCoordinator}} being shut down:

{code:java}
[...]
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: 
CheckpointCoordinator shutdown.
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:544)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2140)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2127)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoints(CheckpointCoordinator.java:2004)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoints(CheckpointCoordinator.java:1987)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingAndQueuedCheckpoints(CheckpointCoordinator.java:2183)
Jan 12 02:37:37 at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.shutdown(CheckpointCoordinator.java:426)
Jan 12 02:37:37 at 
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.onTerminalState(DefaultExecutionGraph.java:1329)
[...]{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44731=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=b78d9d30-509a-5cea-1fef-db7abaa325ae=9255



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30643) docs_check fails