[jira] [Created] (FLINK-32050) Bump Jackson to 2.14.3

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32050:
---

 Summary: Bump Jackson to 2.14.3
 Key: FLINK-32050
 URL: https://issues.apache.org/jira/browse/FLINK-32050
 Project: Flink
  Issue Type: Technical Debt
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin


There is FLINK-32032 with upgrade of flink-shaded where flink-shaded's jackson 
is bumping to 2.14.x.
It would make sense also bump transitive dep jackson



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Release Flink 1.17.1

[Gyula Fóra]

+1 for the release

Gyula

On Thu, 11 May 2023 at 05:35, Yun Tang  wrote:

> +1 for release flink-1.17.1
>
> The blocker issue might cause silent incorrect data, it's better to have a
> fix release ASAP.
>
>
> Best
> Yun Tang
> 
> From: weijie guo 
> Sent: Thursday, May 11, 2023 11:08
> To: dev@flink.apache.org ; tonysong...@gmail.com <
> tonysong...@gmail.com>
> Subject: [DISCUSS] Release Flink 1.17.1
>
> Hi all,
>
>
> I would like to discuss creating a new 1.17 patch release (1.17.1). The
> last 1.17 release is nearly two months old, and since then, 66 tickets have
> been closed [1], of which 14 are blocker/critical [2].  Some of them are
> quite important, such as FLINK-31293 [3] and  FLINK-32027 [4].
>
>
> I am not aware of any unresolved blockers and there are no in-progress
> tickets [5].
> Please let me know if there are any issues you'd like to be included in
> this release but still not merged.
>
>
> If the community agrees to create this new patch release, I could
> volunteer as the release manager
>  and Xintong can help with actions that require a PMC role.
>
>
> Thanks,
>
> Weijie
>
>
> [1]
>
> https://issues.apache.org/jira/browse/FLINK-32027?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC
>
> [2]
>
> https://issues.apache.org/jira/browse/FLINK-31273?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
>
> [3] https://issues.apache.org/jira/browse/FLINK-31293
>
> [4] https://issues.apache.org/jira/browse/FLINK-32027
>
> [5] https://issues.apache.org/jira/projects/FLINK/versions/12352886
>

Re: [DISCUSS] Release Flink 1.16.2

[Gyula Fóra]

+1 for the release

Gyula

On Thu, 11 May 2023 at 05:08, weijie guo  wrote:

> [1]
>
> https://issues.apache.org/jira/browse/FLINK-31092?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC
>
> [2]
>
> https://issues.apache.org/jira/browse/FLINK-31092?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
>
> [3] https://issues.apache.org/jira/browse/FLINK-31293
>
> [4] https://issues.apache.org/jira/browse/FLINK-32027
>
> [5] https://issues.apache.org/jira/projects/FLINK/versions/12352765
>
>
>
>
> weijie guo  于2023年5月11日周四 11:06写道：
>
> > Hi all,
> >
> >
> > I would like to discuss creating a new 1.16 patch release (1.16.2). The
> > last 1.16 release is over three months old, and since then, 99 tickets
> have
> >  been closed [1], of which 30 are blocker/critical [2].  Some
> > of them are quite important, such as FLINK-31293 [3] and FLINK-32027 [4].
> >
> >
> >
> > I am not aware of any unresolved blockers and there are no in-progress
> tickets [5].
> > Please let me know if there are any issues you'd like to be included in
> > this release but still not merged.
> >
> >
> >
> > If the community agrees to create this new patch release, I could
> volunteer as the release manager
> >  and Xintong can help with actions that require a PMC role.
> >
> > Best regards,
> >
> > Weijie
> >
>

Re: Re: [DISCUSS] FLIP-278: Hybrid Source Connector

[Ran Tao]

Hi, Илья.
Thanks for your opinions!

Your are right, and in fact, in addition to the different fields numbers,
the names may also be different.
Currently, we can also support inconsistent schema, which was discussed in
the previous design,
for example, we can provide a `schema.fields.mappings` parameter.

If we have different schema like below:
true batch fields is: a, f1, c, f3
true streaming fields is: f0, b, f2 (lack of 1 field)


1.about inconsistent field names

If user ddl is: f0, f1, f2
`schema.fields.mappings`='[{"f0":"a", f2":"C"},{"f1":"b"}]'

then in hybrid table source, we generate child batch schema is: a, f1, c,
streaming schema is: f0, b, f2 and pass them to final child table source.
(note: we not use batch f3 field, just skip is ok)

2.about inconsistent field numbers

If user ddl is: f0, f1, f2, f3
`schema.fields.mappings`='[{"f0":"a", f2":"C"},{"f1":"b"}]'

then in hybrid table source, we generate child batch schema is: a, f1, c,
f3,
streaming schema has 2 options to set:

1. set f0, b, f2, f3 and pass them to final child table source. (if child
source format is k-v mode, f3 will be null)

2. add an option, e.g.`schema.virtual-fields`='[[],["f3"]]' means
streaming's field f3 is not existed.
then hybrid table source set null for streaming's field f3 actively and
just pass f0, b, f2 to child source to call real data.

In a word, we can use `schema.fields.mappings` to deal with inconsistent
filed name
and pass more fields to child source to get null to deal with inconsistent
field numbers(or add a `schema.virtual-fields` option).

But in order to maintain consistency with the current DataStream api,
we currently support the case where the batch and streaming schemas are
consistent.
I will update the POC pr then you can re-run your case. WDYT?


Best Regards,
Ran Tao



Илья Соин  于2023年5月11日周四 03:12写道：

> Hi devs,
>
> I think for this approach to work, the internal record schema generated by
> Flink must be exactly the same for batch and stream records, because at
> runtime Flink will use the same serializer to send them downstream.
> However, it’s not always the case, because in batch mode Flink’s optimizer
> may realize that some fields are never actually used, so the records will
> not contain those fields. Such optimizations may not be done in the
> streaming mode, so records coming from the realtime source will have more
> fields. In that case, after switching to the realtime source, the job will
> fail, because record serializer expects records with the batch schema, but
> instead receives records with more fields and doesn’t know how to serialize
> them.
>
> Consider the following DDL:
> CREATE TABLE hybrid_table
> (
> trade ROW(
> `openTime` BIGINT,
> `closeTime` BIGINT),
> server  STRING,
> tradeTime as to_timestamp(from_unixtime(trade.openTime)),
> WATERMARK FOR tradeTime AS tradeTime - INTERVAL '1' MINUTE
> )
> WITH (
> 'connector' = 'hybrid',
> 'source-identifiers' = 'historical,realtime',
> 'historical.connector' = 'filesystem',
> 'historical.path' = 's3://path.to.daa',
> 'historical.format' = 'json',
> 'realtime.connector' = 'kafka',
> 'realtime.topic' = 'trades',
> 'realtime.properties.bootstrap.servers' = '...',
> 'realtime.properties.group.id <
> http://realtime.properties.group.id/>' = 'flink.tv ',
> 'realtime.format' = 'json',
> 'realtime.scan.startup.mode' = 'earliest-offset'
> )
> This query will fail:
>
> select server from hybrid_table
>
> But this query will work:
>
> select * from hybrid_table
>
> In the first query internal records in the batch source will only have 2
> fields: server and trade. But in the streaming source they will have all
> the fields described in the schema. When switching to the realtime source,
> the job fails because record serializer expects records with the same
> schema as in the batch source. The IllegalArgumentException happens here <
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/CopyingChainingOutput.java#L74>,
> saying “Row arity: 4, but serializer arity: 2"
>
> The second example works because all fields are accessed in the SQL query,
> so Flink doesn’t remove any of them from the internal records in batch, and
> record schemas in batch and streaming match exactly.
>
> __
> Best regards,
> Ilya Soin
>
> On 2023/05/09 07:09:53 Ran Tao wrote:
> > Hi, devs.
> >
> > I don't know if you have any other considerations for this FLIP. All
> > discussions are welcome.
> > If there are no other opinions in the near days, I will try to initiate a
> > vote. thank you all.
> >
> >
> > Best Regards,
> > Ran Tao
> >
> >
> > Ran Tao  于2023年4月10日周一 15:33写道：
> >
> > > Hi, devs. I want to reopen this discussion because some questions have
> > > been solved or need more discussions.
> > >
> > > In the previous discussion, there were 

Re: [DISCUSS] Release Flink 1.17.1

[Yun Tang]

+1 for release flink-1.17.1

The blocker issue might cause silent incorrect data, it's better to have a fix 
release ASAP.


Best
Yun Tang

From: weijie guo 
Sent: Thursday, May 11, 2023 11:08
To: dev@flink.apache.org ; tonysong...@gmail.com 

Subject: [DISCUSS] Release Flink 1.17.1

Hi all,


I would like to discuss creating a new 1.17 patch release (1.17.1). The
last 1.17 release is nearly two months old, and since then, 66 tickets have
been closed [1], of which 14 are blocker/critical [2].  Some of them are
quite important, such as FLINK-31293 [3] and  FLINK-32027 [4].


I am not aware of any unresolved blockers and there are no in-progress
tickets [5].
Please let me know if there are any issues you'd like to be included in
this release but still not merged.


If the community agrees to create this new patch release, I could
volunteer as the release manager
 and Xintong can help with actions that require a PMC role.


Thanks,

Weijie


[1]
https://issues.apache.org/jira/browse/FLINK-32027?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC

[2]
https://issues.apache.org/jira/browse/FLINK-31273?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC

[3] https://issues.apache.org/jira/browse/FLINK-31293

[4] https://issues.apache.org/jira/browse/FLINK-32027

[5] https://issues.apache.org/jira/projects/FLINK/versions/12352886

[DISCUSS] Release Flink 1.17.1

[weijie guo]

Hi all,


I would like to discuss creating a new 1.17 patch release (1.17.1). The
last 1.17 release is nearly two months old, and since then, 66 tickets have
been closed [1], of which 14 are blocker/critical [2].  Some of them are
quite important, such as FLINK-31293 [3] and  FLINK-32027 [4].


I am not aware of any unresolved blockers and there are no in-progress
tickets [5].
Please let me know if there are any issues you'd like to be included in
this release but still not merged.


If the community agrees to create this new patch release, I could
volunteer as the release manager
 and Xintong can help with actions that require a PMC role.


Thanks,

Weijie


[1]
https://issues.apache.org/jira/browse/FLINK-32027?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC

[2]
https://issues.apache.org/jira/browse/FLINK-31273?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC

[3] https://issues.apache.org/jira/browse/FLINK-31293

[4] https://issues.apache.org/jira/browse/FLINK-32027

[5] https://issues.apache.org/jira/projects/FLINK/versions/12352886

Re: [DISCUSS] Release Flink 1.16.2

[weijie guo]

[1]
https://issues.apache.org/jira/browse/FLINK-31092?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC

[2]
https://issues.apache.org/jira/browse/FLINK-31092?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC

[3] https://issues.apache.org/jira/browse/FLINK-31293

[4] https://issues.apache.org/jira/browse/FLINK-32027

[5] https://issues.apache.org/jira/projects/FLINK/versions/12352765




weijie guo  于2023年5月11日周四 11:06写道：

> Hi all,
>
>
> I would like to discuss creating a new 1.16 patch release (1.16.2). The
> last 1.16 release is over three months old, and since then, 99 tickets have
>  been closed [1], of which 30 are blocker/critical [2].  Some
> of them are quite important, such as FLINK-31293 [3] and FLINK-32027 [4].
>
>
>
> I am not aware of any unresolved blockers and there are no in-progress 
> tickets [5].
> Please let me know if there are any issues you'd like to be included in
> this release but still not merged.
>
>
>
> If the community agrees to create this new patch release, I could volunteer 
> as the release manager
>  and Xintong can help with actions that require a PMC role.
>
> Best regards,
>
> Weijie
>

[DISCUSS] Release Flink 1.16.2

[weijie guo]

Hi all,


I would like to discuss creating a new 1.16 patch release (1.16.2). The
last 1.16 release is over three months old, and since then, 99 tickets have
been closed [1], of which 30 are blocker/critical [2].  Some
of them are quite important, such as FLINK-31293 [3] and FLINK-32027 [4].


I am not aware of any unresolved blockers and there are no in-progress
tickets [5].
Please let me know if there are any issues you'd like to be included in
this release but still not merged.


If the community agrees to create this new patch release, I could
volunteer as the release manager
 and Xintong can help with actions that require a PMC role.

Best regards,

Weijie

RE: Re: [DISCUSS] FLIP-278: Hybrid Source Connector

[Илья Соин]

Hi devs,

I think for this approach to work, the internal record schema generated by 
Flink must be exactly the same for batch and stream records, because at runtime 
Flink will use the same serializer to send them downstream. However, it’s not 
always the case, because in batch mode Flink’s optimizer may realize that some 
fields are never actually used, so the records will not contain those fields. 
Such optimizations may not be done in the streaming mode, so records coming 
from the realtime source will have more fields. In that case, after switching 
to the realtime source, the job will fail, because record serializer expects 
records with the batch schema, but instead receives records with more fields 
and doesn’t know how to serialize them. 

Consider the following DDL:
CREATE TABLE hybrid_table
(
trade ROW(
`openTime` BIGINT,
`closeTime` BIGINT),
server  STRING,
tradeTime as to_timestamp(from_unixtime(trade.openTime)),
WATERMARK FOR tradeTime AS tradeTime - INTERVAL '1' MINUTE
)
WITH (
'connector' = 'hybrid',
'source-identifiers' = 'historical,realtime',
'historical.connector' = 'filesystem',
'historical.path' = 's3://path.to.daa',
'historical.format' = 'json',
'realtime.connector' = 'kafka',
'realtime.topic' = 'trades',
'realtime.properties.bootstrap.servers' = '...',
'realtime.properties.group.id ' = 
'flink.tv ',
'realtime.format' = 'json',
'realtime.scan.startup.mode' = 'earliest-offset'
)
This query will fail:

select server from hybrid_table

But this query will work:

select * from hybrid_table

In the first query internal records in the batch source will only have 2 
fields: server and trade. But in the streaming source they will have all the 
fields described in the schema. When switching to the realtime source, the job 
fails because record serializer expects records with the same schema as in the 
batch source. The IllegalArgumentException happens here 
,
 saying “Row arity: 4, but serializer arity: 2"

The second example works because all fields are accessed in the SQL query, so 
Flink doesn’t remove any of them from the internal records in batch, and record 
schemas in batch and streaming match exactly.

__
Best regards,
Ilya Soin

On 2023/05/09 07:09:53 Ran Tao wrote:
> Hi, devs.
> 
> I don't know if you have any other considerations for this FLIP. All
> discussions are welcome.
> If there are no other opinions in the near days, I will try to initiate a
> vote. thank you all.
> 
> 
> Best Regards,
> Ran Tao
> 
> 
> Ran Tao  于2023年4月10日周一 15:33写道：
> 
> > Hi, devs. I want to reopen this discussion because some questions have
> > been solved or need more discussions.
> >
> > In the previous discussion, there were some questions and problems.
> >
> > @Timo
> > 1.about option prefix, we decide to use identifiers. e.g.
> >
> > ```
> > create table hybrid_source(
> >  f0 varchar,
> >  f1 varchar,
> >  f2 bigint
> > ) with(
> >  'connector'='hybrid',
> >  'source-identifiers'='historical,realtime',
> >  'historical.connector'='filesystem'
> >  'historical.path' = '/tmp/a.csv',
> >  'historical.format' = 'csv',
> >  'realtime.connector'='kafka',
> >  'realtime.topic' = 'test',
> >  'realtime.properties.bootstrap.servers' = 'localhost:9092',
> >  'realtime.properties.group.id' = 'test',
> >  'realtime.scan.startup.mode' = 'earliest-offset',
> >  'realtime.format' = 'csv'
> > );
> > ```
> >
> > @Martijn Visser 
> > 1.table api usage
> >
> > I updated the FLIP about table api usage.
> >
> > 2.how dynamic switched start timestamp works?
> >
> > In this FLIP, we introduce 2 interfaces to support it.
> > If we open switched-start-position-enabled try to use dynamic switched
> > start timestamp, then first source split numerator needs to
> > implement SupportsGetEndTimestamp, next source needs to
> > implement SupportsSwitchedStartTimestamp.
> > We use SupportsGetEndTimestamp and SupportsSwitchedStartTimestamp to get
> > the previous bounded source end timestamp and apply it to the next
> > streaming source.
> >
> > @John Roesler
> > 1.source handoff
> >
> > We both support Fixed-Start-Position And Switched-start-Position. The
> > default is Fixed-Start-Position. Use option switched-start-position-enabled
> > to control it.
> > In Fixed-Start-Position, the next streaming source uses its own startup
> > strategy, e.g. in kafka, we use predefined kafka scan.startup.mode in user
> > sql.
> > In Switched-start-Position, this is the same question as `how dynamic
> > switched start timestamp works` from @Martijn above. We offer
> > SupportsGetEndTimestamp interface to extract first source split enumerator
> > endTimestamp
> > and pass it to the next source. and Next 

[jira] [Created] (FLINK-32049) CoordinatedSourceRescaleITCase.testDownscaling fails on AZP

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32049:
---

 Summary: CoordinatedSourceRescaleITCase.testDownscaling fails on 
AZP
 Key: FLINK-32049
 URL: https://issues.apache.org/jira/browse/FLINK-32049
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.17.1
Reporter: Sergey Nuyanzin


CoordinatedSourceRescaleITCase.testDownscaling fails with
{noformat}
May 08 03:19:14 [ERROR] Failures: 
May 08 03:19:14 [ERROR]   
CoordinatedSourceRescaleITCase.testDownscaling:75->resumeCheckpoint:107 
May 08 03:19:14 Multiple Failures (1 failure)
May 08 03:19:14 -- failure 1 --
May 08 03:19:14 [Any cause contains message 'successfully restored checkpoint'] 
May 08 03:19:14 Expecting any element of:
May 08 03:19:14   [org.apache.flink.runtime.client.JobExecutionException: Job 
execution failed.
May 08 03:19:14 at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
May 08 03:19:14 at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
May 08 03:19:14 at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
May 08 03:19:14 ...(45 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed),
May 08 03:19:14 org.apache.flink.runtime.JobException: Recovery is 
suppressed by NoRestartBackoffTimeStrategy
May 08 03:19:14 at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:139)
May 08 03:19:14 at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:83)
May 08 03:19:14 at 
org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:258)
May 08 03:19:14 ...(35 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed),
May 08 03:19:14 java.lang.IllegalStateException: This executor has been 
registered.
May 08 03:19:14 at 
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
May 08 03:19:14 at 
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl.registerSubtask(ChannelStateWriteRequestExecutorImpl.java:341)
May 08 03:19:14 at 
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorFactory.getOrCreateExecutor(ChannelStateWriteRequestExecutorFactory.java:63)
May 08 03:19:14 ...(17 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed)]
May 08 03:19:14 to satisfy the given assertions requirements but none did:
May 08 03:19:14 
May 08 03:19:14 org.apache.flink.runtime.client.JobExecutionException: Job 
execution failed.
May 08 03:19:14 at 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
May 08 03:19:14 at 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
May 08 03:19:14 at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
May 08 03:19:14 ...(45 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed)
May 08 03:19:14 error: 
May 08 03:19:14 Expecting throwable message:
May 08 03:19:14   "Job execution failed."
May 08 03:19:14 to contain:
May 08 03:19:14   "successfully restored checkpoint"
May 08 03:19:14 but did not.
May 08 03:19:14 

{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48772=logs=fc7981dc-d266-55b0-5fff-f0d0a2294e36=1a9b228a-3e0e-598f-fc81-c321539dfdbf=7191



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-cassandra 3.1.0, release candidate #2

[Ryan Skraba]

+1 (non-binding)

I've validated this RC: flink-connector-cassandra-3.1.0-rc2 at r61661
- The SHA512 hash is OK.
- The source file is signed correctly.
- The signature 0F79F2AFB2351BC29678544591F9C1EC125FD8DB is found in the
KEYS file.
- The source file is consistent with the Github tag v3.0.1-rc2, which
corresponds to commit
https://github.com/apache/flink-connector-cassandra/tree/83945fe41cb6e7c188dfbf656b04955142600bb2
  - The files explicitly excluded by create_pristine_sources (such as
.gitignore and the submodule tools/releasing/shared) are not present.
- Has a LICENSE file and NOTICE files.
- Does not contain any compiled binaries.
- Has 2 artifacts staged at
https://repository.apache.org/content/repositories/orgapacheflink-1631/

The bug that blocked RC1 (FLINK-31927) is no longer present.

Thanks!  Ryan

On Wed, May 10, 2023 at 11:51 AM Martijn Visser 
wrote:

> +1 (binding)
>
> - Validated hashes
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven
> - Verified licenses
> - Verified web PRs
>
> On Tue, May 9, 2023 at 11:43 PM Khanh Vu  wrote:
>
> > +1 (non-binding)
> >
> > - Verified sha512 checksum matches file archive.
> > - Verified file archive is signed and signature is authenticated.
> > - Verified no binaries exist in the source archive.
> > - Verified source archive is consistent with Github source code with tag
> > v3.1.0-rc2, at commit 83945fe41cb6e7c188dfbf656b04955142600bb2.
> > - Source built successfully with maven and integration tests passed.
> >
> > Best regards,
> > Khanh Vu
> >
> > On Tue, May 9, 2023 at 9:50 AM Etienne Chauchot 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > +1 (non-binding)
> > >
> > > I checked:
> > >
> > > - release notes
> > >
> > > - tag
> > >
> > > - tested the prod artifact with
> > https://github.com/echauchot/flink-samples
> > >
> > > Best
> > >
> > > Etienne
> > >
> > > Le 05/05/2023 à 11:39, Danny Cranmer a écrit :
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #2 for the version
> > 3.1.0,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > [2],
> > > > which are signed with the key with fingerprint
> > > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag v3.1.0-rc2 [5],
> > > > * website pull request listing the new release [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Danny
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353030
> > > > [2]
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.1.0-rc2
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > https://repository.apache.org/content/repositories/orgapacheflink-1631
> > > > [5]
> > https://github.com/apache/flink-connector-cassandra/tree/v3.1.0-rc2
> > > > [6] https://github.com/apache/flink-web/pull/642
> > > >
> > >
> >
>

[jira] [Created] (FLINK-32048) DecimalITCase.testAggMinGroupBy fails with "Insufficient number of network buffers"

[Lijie Wang (Jira)]

Lijie Wang created FLINK-32048:
--

 Summary: DecimalITCase.testAggMinGroupBy fails with "Insufficient 
number of network buffers"
 Key: FLINK-32048
 URL: https://issues.apache.org/jira/browse/FLINK-32048
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner, Tests
Affects Versions: 1.18.0
Reporter: Lijie Wang


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48855=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4]
{code:java}
May 10 09:37:41 Caused by: java.io.IOException: Insufficient number of network 
buffers: required 1, but only 0 available. The total number of network buffers 
is currently set to 2048 of 32768 bytes each. You can increase this number by 
setting the configuration keys 'taskmanager.memory.network.fraction', 
'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
May 10 09:37:41 at 
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.internalCreateBufferPool(NetworkBufferPool.java:495)
May 10 09:37:41 at 
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:456)
May 10 09:37:41 at 
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateFactory.lambda$createBufferPoolFactory$3(SingleInputGateFactory.java:330)
May 10 09:37:41 at 
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.setup(SingleInputGate.java:274)
May 10 09:37:41 at 
org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setup(InputGateWithMetrics.java:105)
May 10 09:37:41 at 
org.apache.flink.runtime.taskmanager.Task.setupPartitionsAndGates(Task.java:969)
May 10 09:37:41 at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:654)
May 10 09:37:41 at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
May 10 09:37:41 at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re:Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

[Mang Zhang]

Hi Ron,


First of all, thank you for your reply!
After our offline communication, what you said is mainly in the compilePlan 
scenario, but currently compilePlanSql does not support non INSERT statements, 
otherwise it will throw an exception.
>Unsupported SQL query! compilePlanSql() only accepts a single SQL statement of 
>type INSERT
But it's a good point that I will seriously consider. 
Non-atomic CTAS can be supported relatively easily;
But atomic CTAS needs more adaptation work, so I'm going to leave it as is and 
follow up with a separate issue to implement CTAS support for compilePlanSql.






--

Best regards,
Mang Zhang





At 2023-04-23 17:52:07, "liu ron"  wrote:
>Hi, Mang
>
>I have a question about the implementation details. For the atomicity case,
>since the target table is not created before the JobGraph is generated, but
>then the target table is required to exist when optimizing plan to generate
>the JobGraph. So how do you solve this problem?
>
>Best,
>Ron
>
>yuxia  于2023年4月20日周四 09:35写道：
>
>> Share some insights about the new TwoPhaseCatalogTable proposed after
>> offline discussion with Mang.
>> The main or important reason is that the TwoPhaseCatalogTable enables
>> external connectors to implement theirs own logic for commit / abort.
>> In FLIP-218, for atomic CTAS, the Catalog will then just drop the table
>> when the job fail. It's not ideal for it's too generic to work well.
>> For example, some connectors will need to clean some temporary files in
>> abort method. And the actual connector can know the specific logic for
>> aborting.
>>
>> Best regards,
>> Yuxia
>>
>>
>> 发件人: "zhangmang1" 
>> 收件人: "dev" , "Jing Ge" 
>> 抄送: "ron9 liu" , "lincoln 86xy" <
>> lincoln.8...@gmail.com>, luoyu...@alumni.sjtu.edu.cn
>> 发送时间: 星期三, 2023年 4 月 19日 下午 3:13:36
>> 主题: Re:Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS
>> SELECT(CTAS) statement
>>
>> hi, Jing
>> Thank you for your reply.
>> >1. It looks like you found another way to design the atomic CTAS with new
>> >serializable TwoPhaseCatalogTable instead of making Catalog serializable
>> as
>> >described in FLIP-218. Did I understand correctly?
>> Yes, when I was implementing the FLIP-218 solution, I encountered problems
>> with Catalog/CatalogTable serialization deserialization, for example, after
>> deserialization CatalogTable could not be converted to Hive Table. Also,
>> Catalog serialization is still a heavy operation, but it may not actually
>> be necessary, we just need Create Table.
>> Therefore, the TwoPhaseCatalogTable program is proposed, which also
>> facilitates the implementation of the subsequent data lake, ReplaceTable
>> and other functions.
>>
>> >2. I am a little bit confused about the isStreamingMode parameter of
>> >Catalog#twoPhaseCreateTable(...), since it is the selector argument(code
>> >smell) we should commonly avoid in the public interface. According to the
>> >FLIP,  isStreamingMode will be used by the Catalog to determine whether to
>> >support atomic or not. With this selector argument, there will be two
>> >different logics built within one method and it is hard to follow without
>> >reading the code or the doc carefully(another concern is to keep the doc
>> >and code alway be consistent) i.e. sometimes there will be no difference
>> by
>> >using true/false isStreamingMode, sometimes they are quite different -
>> >atomic vs. non-atomic. Another question is, before we call
>> >Catalog#twoPhaseCreateTable(...), we have to know the value of
>> >isStreamingMode. In case only non-atomic is supported for streaming mode,
>> >we could just follow FLIP-218 instead of (twistedly) calling
>> >Catalog#twoPhaseCreateTable(...) with a false isStreamingMode. Did I miss
>> >anything here?
>> Here's what I think about this issue, atomic CTAS wants to be the default
>> behavior and only fall back to non-atomic CTAS if it's completely
>> unattainable. Atomic CTAS will bring a better experience to users.
>> Flink is already a stream batch unified engine, In our company kwai, many
>> users are also using flink to do batch data processing, but still running
>> in Stream mode.
>> The boundary between stream and batch is gradually blurred, stream mode
>> jobs may also FINISH, so I added the isStreamingMode parameter, this
>> provides different atomicity implementations in Batch and Stream modes.
>> Not only to determine if atomicity is supported, but also to help select
>> different TwoPhaseCatalogTable implementations to provide different levels
>> of atomicity!
>>
>> Looking forward to more feedback.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Best regards,
>> Mang Zhang
>>
>>
>>
>>
>> At 2023-04-15 04:20:40, "Jing Ge"  wrote:
>> >Hi Mang,
>> >
>> >This is the FLIP I was looking forward to after FLIP-218. Thanks for
>> >driving it. I have two questions and would like to know your thoughts,
>> >thanks:
>> >
>> >1. It looks like you found another way to design the atomic CTAS with new
>> >serializable TwoPhaseCatalogTable 

[jira] [Created] (FLINK-32047) Fix args in JobSpec not being passed through to Flink in Standalone mode - 1.4.0

[Gil Shmaya (Jira)]

Gil Shmaya created FLINK-32047:
--

 Summary: Fix args in JobSpec not being passed through to Flink in 
Standalone mode - 1.4.0
 Key: FLINK-32047
 URL: https://issues.apache.org/jira/browse/FLINK-32047
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Gil Shmaya


This issue is related to a previously fixed bug in version 1.2.0 -  
[FLINK-29388|https://issues.apache.org/jira/browse/FLINK-29388]

I have noticed that while the args are successfully being passed when using 
version 1.2.0, this is not the case with version 1.4.0.

{+}Scenario{+}:

I added a log that prints the argument array length at the beginning of the 
main  function of the flink job:
!image-2023-04-30-18-54-22-291.png|width=659,height=102!

The result when running with 1.2.0:
!image-2023-04-30-19-56-30-150.png!

The result when running with 1.4.0:
!image-2023-04-30-19-56-57-680.png!
h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache flink-connector-gcp-pubsub v3.0.1 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-gcp-pubsub v3.0.1. This release is compatible with Flink
1.16.x and Flink 1.17.x

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352770

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[ANNOUNCE] Apache flink-connector-elasticsearch v3.0.1 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-elasticsearch v1.0.1. This release is compatible with Flink
1.16.x and Flink 1.17.x

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352521

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[ANNOUNCE] Apache flink-connector-opensearch v1.0.1 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-opensearch v1.0.1. This release is compatible with Flink
1.16.x and Flink 1.17.x

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352686

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[ANNOUNCE] Apache flink-connector-pulsar v4.0.0 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-pulsar v4.0.0. This release is compatible with Flink 1.17.x

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352653

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[ANNOUNCE] Apache flink-shaded v17.0 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-shaded v17.0.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352445

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[ANNOUNCE] Apache flink-connector-rabbitmq v3.0.1 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of Apache
flink-connector-rabbitmq v3.0.1. This release is compatible with Flink
1.16.x and Flink 1.17.x

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352699

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Release Manager

[RESULT][VOTE] Release flink-connector-gcp-pubsub v3.0.1, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:
* Ryan (non-binding)
* Martijn (binding)
* Leonard (binding)
* Khanh (non-binding)
* Danny (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

[RESULT][VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 3 of which are binding:
* Ryan (non-binding)
* Martijn (binding)
* Leonard (binding)
* Khanh (non-binding)
* Jing (non-binding)
* Danny (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

[RESULT][VOTE] Release flink-shaded v17.0, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 4 approving votes, 3 of which are binding:
* Matthias (non-binding)
* Danny (binding)
* Leonard (binding)
* Martijn (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

[RESULT][VOTE] Release flink-connector-pulsar v4.0.0, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 4 approving votes, 3 of which are binding:
* Yufan (non-binding)
* Martijn (binding)
* Danny (binding)
* Leonard (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

[RESULT][VOTE] Release flink-connector-elasticsearch v3.0.1, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:
* Andriy (non-binding)
* Martijn (binding)
* Danny (binding)
* Ahmed (non-binding)
* Leonard (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

[RESULT][VOTE] Release flink-connector-opensearch v1.0.1, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 3 of which are binding:
* Andrey (non-binding)
* Ryan (non-binding)
* Danny (binding)
* Martijn (binding)
* Khanh (non-binding)
* Leonard (binding)

There are no disapproving votes.

Thanks all!

Best regards,

Martijn

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.1, release candidate #1

[Martijn Visser]

Thanks all. This vote is now closed, I'll announce the results in a
separate thread.


On Tue, May 9, 2023 at 11:15 AM Danny Cranmer 
wrote:

> +1 (binding)
>
> - Release notes look good
> - Verified signature and hashes of source/maven binary
> - Verified there are no binaries in the source archive
> - Tag is present in Github
> - Reviewed web PR
> - Built from source, tests pass
>
> - Year in NOTICE file needs updating, not a blocker
>
> Thanks,
> Danny
>
> On Fri, May 5, 2023 at 8:15 AM Khanh Vu  wrote:
>
> > +1 (non-binding)
> >
> > - Verified sha512 checksum matches file archive.
> > - Verified file archive is signed and signature is authorized.
> > - Verified no binaries exist in the source archive.
> > - Verified source archive is consistent with Github source code with
> > tag v3.0.1-rc1, at commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
> > - Source built successfully with maven.
> > - Executed end to end tests successfully for flink versions: 1.15.4,
> > 1.16.1, 1.17.0
> >
> > Best regards,
> > Khanh Vu
> >
> >
> > On Thu, May 4, 2023 at 5:01 AM Leonard Xu  wrote:
> >
> > >  +1 (binding)
> > >
> > > - built from source code succeeded
> > > - verified signatures
> > > - verified hashsums
> > > - checked Github release tag
> > > - checked release notes
> > > - reviewed the web PR
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2023年4月20日 上午3:17，Martijn Visser  写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > - Validated hashes
> > > > - Verified signature
> > > > - Verified that no binaries exist in the source archive
> > > > - Build the source with Maven
> > > > - Verified licenses
> > > > - Verified web PRs
> > > >
> > > > On Fri, Apr 14, 2023 at 6:15 PM Ryan Skraba
> >  > > >
> > > > wrote:
> > > >
> > > >> Hello!  +1 (non-binding)
> > > >>
> > > >> I've validated the source for the RC1:
> > > >> flink-connector-gcp-pubsub-3.0.1-src.tgz
> > > >> * The sha512 checksum is OK.
> > > >> * The source file is signed correctly.
> > > >> * The signature A5F3BCE4CBE993573EC5966A65321B8382B219AF is found in
> > the
> > > >> KEYS file, and on https://keys.openpgp.org
> > > >> * The source file is consistent with the Github tag v3.0.1-rc1,
> which
> > > >> corresponds to commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
> > > >>   - The files explicitly excluded by create_pristine_sources (such
> as
> > > >> .gitignore and the submodule tools/releasing/shared) are not
> present.
> > > >> * Has a LICENSE file and a NOTICE filel
> > > >> * Does not contain any compiled binaries.
> > > >>
> > > >> * The sources can be compiled and unit tests pass with flink.version
> > > 1.17.0
> > > >> and flink.version 1.16.1
> > > >>
> > > >> * Nexus has three staged artifact ids for 3.0.1-1.16 and 3.0.1-1.17
> > > >> - flink-connector-gcp-pubsub (.jar, -javadoc.jar, -sources.jar and
> > .pom)
> > > >> - flink-connector-gcp-pubsub-e2e-tests (.jar, -sources.jar and .pom)
> > > >> - flink-connector-gcp-pubsub-parent (only .pom)
> > > >> * All 16 files have been signed with the same key as above, and have
> > > >> correct sha1 and md5 checksums.
> > > >>
> > > >> Simple smoke testing on an emulated Pub/Sub service works for both
> > flink
> > > >> versions.
> > > >>
> > > >> One really minor point: it looks like the
> > > >> org.apache.flink:flink-connector-gcp-pubsub-parent:3.0.1-1.17:pom
> has
> > > the
> > > >> flink-version set to 1.16.0.  This is a bit confusing, but all the
> > flink
> > > >> transitive dependencies are in the provided scope, so there's no
> > > >> consequence.  I guess we could argue that it is the "source"
> > > compatibility
> > > >> level for both versions!
> > > >>
> > > >> All my best and thanks for the release.
> > > >>
> > > >> Ryan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Apr 13, 2023 at 4:07 PM Martijn Visser <
> > > martijnvis...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>> Please review and vote on the release candidate #1 for the version
> > > 3.0.1,
> > > >>> as follows:
> > > >>> [ ] +1, Approve the release
> > > >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >>>
> > > >>> This version is compatible with Flink 1.16.x and Flink 1.17.x.
> > > >>>
> > > >>> The complete staging area is available for your review, which
> > includes:
> > > >>> * JIRA release notes [1],
> > > >>> * the official Apache source release to be deployed to
> > dist.apache.org
> > > >>> [2],
> > > >>> which are signed with the key with fingerprint
> > > >>> A5F3BCE4CBE993573EC5966A65321B8382B219AF [3],
> > > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > > >>> * source code tag v3.0.1-rc1 [5],
> > > >>> * website pull request listing the new release [6].
> > > >>>
> > > >>> The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >>> approval, with at least 3 PMC affirmative votes.
> > > >>>
> > > >>> Thanks,
> > > >>> Release Manager
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> 

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

[Martijn Visser]

Thank you Danny! :)

On Tue, May 9, 2023 at 11:22 AM Danny Cranmer 
wrote:

> Hey Martijn, I think you were missing 1 binding vote, so I have reviewed it
> in hindsight.
>
> +1 (binding)
>
> - Release notes look good
> - Verified signature and hashes of released source [1]
> - Verified there are no binaries in the source archive
> - Tag is present in Github
> - Built from source, tests pass
>
> [1]
>
> https://dist.apache.org/repos/dist/release/flink/flink-connector-rabbitmq-3.0.1/
>
> Thanks,
> Danny
>
> On Mon, May 8, 2023 at 1:05 PM Martijn Visser 
> wrote:
>
> > Thanks all. This vote is now closed, I'll announce the results in a
> > separate thread.
> >
> > On Sat, May 6, 2023 at 4:45 PM Jing Ge 
> wrote:
> >
> > > +1
> > >
> > > - built the source
> > > - verified signature
> > > - verified hash
> > > - contains no compiled binaries
> > > - checked tag
> > > - web PR looks good
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Fri, May 5, 2023 at 9:14 AM Khanh Vu  wrote:
> > >
> > > > Sorry, the above report is supposed for
> > flink-connector-gpc-pubsub-3.0.1
> > > >
> > > > -
> > > > Here is the check for this one:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > - Verified sha512 checksum matches file archive.
> > > > - Verified file archive is signed and signature is authenticated.
> > > > - Verified no binaries exist in the source archive.
> > > > - Verified source archive is consistent with Github source code with
> > tag
> > > > v3.0.1-rc1, at commit 9827e71662c8f155cda5efe5ebbac804fd0fd8e2
> > > > - Source built successfully with maven.
> > > > - (No end to end tests run for this connector)
> > > >
> > > > Best regards,
> > > > Khanh Vu
> > > >
> > > >
> > > > On Fri, May 5, 2023 at 7:55 AM Khanh Vu  wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - Verified sha512 checksum matches file archive.
> > > > > - Verified file archive is signed and signature is authorized.
> > > > > - Verified no binaries exist in the source archive.
> > > > > - Verified source archive is consistent with Github source code
> with
> > > > > tag v3.0.1-rc1, at commit 73e56edb2aa4513f6a73dc071545fb2508fd2d44
> > > > > - Source built successfully with maven.
> > > > > - Executed end to end tests successfully for flink versions:
> 1.15.4,
> > > > > 1.16.1, 1.17.0
> > > > >
> > > > > Best regards,
> > > > > Khanh Vu
> > > > >
> > > > >
> > > > > On Thu, May 4, 2023 at 3:47 AM Leonard Xu 
> wrote:
> > > > >
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> - built from source code succeeded
> > > > >> - verified signatures
> > > > >> - verified hashsums
> > > > >> - checked Github release tag
> > > > >> - reviewed the web PR
> > > > >>
> > > > >> Best,
> > > > >> Leonard
> > > > >>
> > > > >>
> > > > >> 2023年4月20日 上午3:29，Martijn Visser  写道：
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> - Validated hashes
> > > > >> - Verified signature
> > > > >> - Verified that no binaries exist in the source archive
> > > > >> - Build the source with Maven
> > > > >> - Verified licenses
> > > > >> - Verified web PRs
> > > > >>
> > > > >> On Mon, Apr 17, 2023 at 7:00 PM Ryan Skraba
> > > >  > > > >> >
> > > > >> wrote:
> > > > >>
> > > > >> Hello!  +1 (non-binding)
> > > > >>
> > > > >> I've validated the source for the RC1:
> > > > >> flink-connector-rabbitmq-3.0.1-src.tgz
> > > > >> * The sha512 checksum is OK.
> > > > >> * The source file is signed correctly.
> > > > >> * The signature A5F3BCE4CBE993573EC5966A65321B8382B219AF is found
> in
> > > the
> > > > >> KEYS file, and on https://keys.openpgp.org
> > > > >> * The source file is consistent with the Github tag v3.0.1-rc1,
> > which
> > > > >> corresponds to commit 9827e71662c8f155cda5efe5ebbac804fd0fd8e2
> > > > >>   - The files explicitly excluded by create_pristine_sources (such
> > as
> > > > >> .gitignore and the submodule tools/releasing/shared) are not
> > present.
> > > > >> * Has a LICENSE file and a NOTICE file.  The sql-connector has a
> > > > >> NOTICE file for bundled artifacts.
> > > > >> * Does not contain any compiled binaries.
> > > > >>
> > > > >> * The sources can be compiled and tests pass with flink.version
> > 1.17.0
> > > > and
> > > > >> flink.version 1.16.1
> > > > >>
> > > > >> * Nexus has three staged artifact ids for 3.0.1-1.16 and
> 3.0.1-1.17
> > > > >> - flink-connector-rabbitmq-parent (only .pom)
> > > > >> - flink-connector-rabbitmq (.jar, -sources.jar, -javadoc.jar and
> > .pom)
> > > > >> - flink-sql-connector-rabbitmq (.jar, -sources.jar and .pom)
> > > > >> * All 16 files have been signed with the same key as above, and
> have
> > > > >> correct sha1 and md5 checksums.
> > > > >>
> > > > >> I didn't run any additional smoke tests other than the integration
> > > test
> > > > >> cases.
> > > > >>
> > > > >> A couple minor points, but nothing that would block this release.
> > > > >>
> > > > >> - like flink-connector-gcp-pubsub-parent, the
> > > > >> flink-connector-rabbitmq-parent:3.0.1-1.17 

Re:Re: Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

[Mang Zhang]

Hi Jing,


Currently, we cannot determine in the planner whether the source is bounded or 
unbounded.
So when we design the API, we use the execution model to help determine if 
atomicity can be supported.
Thank you very much for your reply!




--

Best regards,
Mang Zhang





At 2023-04-28 21:51:02, "Jing Ge"  wrote:
>Hi Mang,
>
>Boundedness and execution modes are two orthogonal concepts. Since atomic
>CTAS will be only supported for bounded data, which means it does not
>depend on the execution modes. I was wondering if it is possible to only
>provide (or call) twoPhaseCreateTable for bounded data (in both streaming
>and batch mode) and let unbounded data use the non-atomic CTAS? In this
>way, we could avoid the selector argument code smell.
>
>Best regards,
>Jing
>
>On Tue, Apr 25, 2023 at 10:04 AM Mang Zhang  wrote:
>
>> Hi Jing,
>> Yes, the atomic CTAS will be only supported for bounded data, but the
>> execution modes can be stream or batch.
>> I introduced the isStreamingMode parameter in the twoPhaseCreateTable API
>> to make it easier for users to provide different levels of atomicity
>> implementation depending on the capabilities of the backend service.
>> For example, in the case of data synchronization, it is common to run the
>> job using Stream mode, but also expect the data to be visible to the user
>> only after the synchronization is complete.
>> flink cdc's synchronized data scenario, where the user must first write to
>> a temporary table and then manually rename it to the final table;
>> unfriendly to user experience.
>> Developers providing twoPhaseCreateTable capability in Catalog can decide
>> whether to support atomicity based on the execution mode, or they can
>> choose to provide lightweight atomicity support in Stream mode, such as
>> automatically renaming the table name for the user.
>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Mang Zhang
>>
>>
>>
>> At 2023-04-24 15:41:31, "Jing Ge"  wrote:
>> >Hi Mang,
>> >
>> >
>> >
>> >Thanks for clarifying it. I am trying to understand your thoughts. Do you
>> >actually mean the boundedness[1] instead of the execution modes[2]? I.e.
>> >the atomic CTAS will be only supported for bounded data.
>> >
>> >
>> >
>> >Best regards,
>> >
>> >Jing
>> >
>> >
>> >
>> >[1] https://flink.apache.org/what-is-flink/flink-architecture/
>> >
>> >[2]
>> >https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/#execution-mode-batchstreaming
>> >
>> >On Wed, Apr 19, 2023 at 9:14 AM Mang Zhang  wrote:
>> >
>> >> hi, Jing
>> >>
>> >> Thank you for your reply.
>> >>
>> >> >1. It looks like you found another way to design the atomic CTAS with new
>> >> >serializable TwoPhaseCatalogTable instead of making Catalog serializable 
>> >> >as
>> >> >described in FLIP-218. Did I understand correctly?
>> >> Yes, when I was implementing the FLIP-218 solution, I encountered 
>> >> problems with Catalog/CatalogTable serialization deserialization, for 
>> >> example, after deserialization CatalogTable could not be converted to 
>> >> Hive Table. Also, Catalog serialization is still a heavy operation, but 
>> >> it may not actually be necessary, we just need Create Table.
>> >> Therefore, the TwoPhaseCatalogTable program is proposed, which also 
>> >> facilitates the implementation of the subsequent data lake, ReplaceTable 
>> >> and other functions.
>> >>
>> >> >2. I am a little bit confused about the isStreamingMode parameter of
>> >> >Catalog#twoPhaseCreateTable(...), since it is the selector argument(code
>> >> >smell) we should commonly avoid in the public interface. According to the
>> >> >FLIP,  isStreamingMode will be used by the Catalog to determine whether 
>> >> >to
>> >> >support atomic or not. With this selector argument, there will be two
>> >> >different logics built within one method and it is hard to follow without
>> >> >reading the code or the doc carefully(another concern is to keep the doc
>> >> >and code alway be consistent) i.e. sometimes there will be no difference 
>> >> >by
>> >> >using true/false isStreamingMode, sometimes they are quite different -
>> >> >atomic vs. non-atomic. Another question is, before we call
>> >> >Catalog#twoPhaseCreateTable(...), we have to know the value of
>> >> >isStreamingMode. In case only non-atomic is supported for streaming mode,
>> >> >we could just follow FLIP-218 instead of (twistedly) calling
>> >> >Catalog#twoPhaseCreateTable(...) with a false isStreamingMode. Did I miss
>> >> >anything here?
>> >>
>> >> Here's what I think about this issue, atomic CTAS wants to be the default
>> >> behavior and only fall back to non-atomic CTAS if it's completely
>> >> unattainable. Atomic CTAS will bring a better experience to users.
>> >> Flink is already a stream batch unified engine, In our company kwai, many
>> >> users are also using flink to do batch data processing, but still running
>> >> in Stream mode.
>> >> The boundary between stream and batch is gradually blurred, stream mode
>> >> 

[jira] [Created] (FLINK-32046) OOM caused by SplitAssignmentTracker.uncheckpointedAssignments

[Peter Vary (Jira)]

Peter Vary created FLINK-32046:
--

 Summary: OOM caused by 
SplitAssignmentTracker.uncheckpointedAssignments
 Key: FLINK-32046
 URL: https://issues.apache.org/jira/browse/FLINK-32046
 Project: Flink
  Issue Type: Bug
  Components: API / Core
Reporter: Peter Vary


If the checkpointing is turned off then the 
{{SplitAssignmentTracker.uncheckpointedAssignments}} is never cleared and grows 
indefinitely. Eventually leading to OOM.

The only other place which would remove elements from this map is 
{{{}getAndRemoveUncheckpointedAssignment{}}}, but it is only for failure 
scenarios.

By my understanding this problem exists since the introduction of the new 
source code.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32045) optimize task deployment performance for large-scale jobs

[Weihua Hu (Jira)]

Weihua Hu created FLINK-32045:
-

 Summary: optimize task deployment performance for large-scale jobs
 Key: FLINK-32045
 URL: https://issues.apache.org/jira/browse/FLINK-32045
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Weihua Hu


h1. Background

In FLINK-21110, we cache shuffle descriptors on the job manager side and 
support using blob servers to offload these descriptors in order to reduce the 
cost of tasks deployment.

I think there is also some improvement we could do for large-scale jobs.
 # The default min size to enable distribution via blob server is 1MB. But for 
a large wordcount job with 2 parallelism, the size of serialized shuffle 
descriptors is only 300KB. It means users need to lower the 
"blob.offload.minsize", but the value is hard for users to decide.
 # The task executor side still needs to load blob files and deserialize 
shuffle descriptors for each task. Since these operations are running in the 
main thread, it may be pending other RPCs from the job manager.

h1. Propose
 # Enable distribute shuffle descriptors via blob server automatically. This 
could be decided by the edge number of the current shuffle descriptor. The blob 
offload will be enabled when the edge number exceeds an internal threshold.
 # Introduce cache of deserialized shuffle descriptors on the task executor 
side. This could reduce the cost of reading from local blob files and 
deserialization. Of course, the cache should have TTL to avoid occupying too 
much memory. And the cache should have the same switch mechanism as the blob 
server offload.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32044) Impove catalog name check to keep consistent about human-readable exception log in FunctionCatalog

[Ran Tao (Jira)]

Ran Tao created FLINK-32044:
---

 Summary: Impove catalog name check to keep consistent about 
human-readable exception log in FunctionCatalog 
 Key: FLINK-32044
 URL: https://issues.apache.org/jira/browse/FLINK-32044
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.17.0
Reporter: Ran Tao


{code:java}
Catalog catalog = catalogManager.getCatalog(catalogName).get(); {code}
 
-->
 
{code:java}
Catalog catalog =
catalogManager
.getCatalog(catalogName)
.orElseThrow(
() ->
new ValidationException(
String.format(
"Catalog %s not exists.", 
catalogName)));{code}
A trivial improvement to check optional#get and throw more friendly log to 
users. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-cassandra 3.1.0, release candidate #2

[Martijn Visser]

+1 (binding)

- Validated hashes
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven
- Verified licenses
- Verified web PRs

On Tue, May 9, 2023 at 11:43 PM Khanh Vu  wrote:

> +1 (non-binding)
>
> - Verified sha512 checksum matches file archive.
> - Verified file archive is signed and signature is authenticated.
> - Verified no binaries exist in the source archive.
> - Verified source archive is consistent with Github source code with tag
> v3.1.0-rc2, at commit 83945fe41cb6e7c188dfbf656b04955142600bb2.
> - Source built successfully with maven and integration tests passed.
>
> Best regards,
> Khanh Vu
>
> On Tue, May 9, 2023 at 9:50 AM Etienne Chauchot 
> wrote:
>
> > Hi everyone,
> >
> > +1 (non-binding)
> >
> > I checked:
> >
> > - release notes
> >
> > - tag
> >
> > - tested the prod artifact with
> https://github.com/echauchot/flink-samples
> >
> > Best
> >
> > Etienne
> >
> > Le 05/05/2023 à 11:39, Danny Cranmer a écrit :
> > > Hi everyone,
> > > Please review and vote on the release candidate #2 for the version
> 3.1.0,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > > which are signed with the key with fingerprint
> > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag v3.1.0-rc2 [5],
> > > * website pull request listing the new release [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Danny
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353030
> > > [2]
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.1.0-rc2
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1631
> > > [5]
> https://github.com/apache/flink-connector-cassandra/tree/v3.1.0-rc2
> > > [6] https://github.com/apache/flink-web/pull/642
> > >
> >
>

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Piotr Nowojski]

+1 (binding)

Thanks!

śr., 10 maj 2023 o 10:23 Yun Tang  napisał(a):

> +1 (binding)
>
> Thanks Zakelly for driving this topic.
>
> Best
> Yun Tang
> 
> From: Yu Li 
> Sent: Wednesday, May 10, 2023 15:44
> To: dev@flink.apache.org 
> Subject: Re: [VOTE] FLIP-306: Unified File Merging Mechanism for
> Checkpoints
>
> +1 (binding)
>
> Thanks Zakelly for driving this, and thanks everyone for the thorough
> discussion.
>
> On Wed, May 10, 2023 at 11:15 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks for driving this proposal, Zakelly.
> >
> > +1(binding)
> >
> > Best,
> > Rui Fan
> >
> > On Wed, May 10, 2023 at 11:04 AM Hangxiang Yu 
> wrote:
> >
> > > Hi Zakelly.
> > > Thanks for driving this.
> > > +1 (no-binding)
> > >
> > > On Wed, May 10, 2023 at 10:52 AM Yuan Mei 
> > wrote:
> > >
> > > > Thanks for driving this, Zakelly.
> > > >
> > > > As discussed in the thread,
> > > >
> > > > +1 for the proposal (binding)
> > > >
> > > > Best,
> > > >
> > > > Yuan
> > > >
> > > >
> > > >
> > > > On Wed, May 10, 2023 at 10:39 AM Zakelly Lan 
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Sorry for the 4 duplicate emails. There was a problem with the dev
> > > > > mailing list blocking the mails from Gmail. I thought it was a
> > network
> > > > > problem so I tried several times. The issue is addressed by
> > > > > INFRA-24572[1] and the piled up mails are delivered at one time.
> > > > >
> > > > > Based on the sending time, the vote will be open until May 12th at
> > > > > 11:00PM GMT. Please discuss and vote in the last thread (this one).
> > > > > Thanks!
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Zakelly
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/INFRA-24572
> > > > >
> > > > > On Wed, May 10, 2023 at 10:30 AM Yanfei Lei 
> > > wrote:
> > > > > >
> > > > > > +1 (no-binding)
> > > > > >
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > >
> > > > > > Jing Ge  于2023年5月10日周三 07:03写道：
> > > > > >
> > > > > > >
> > > > > > > Hi Zakelly,
> > > > > > >
> > > > > > > I saw you sent at least 4 same emails for voting FLIP-306. I
> > guess
> > > > > this one
> > > > > > > should be the last one and the right one for us to vote right?
> > BTW,
> > > > > based
> > > > > > > on the sending time, 72 hours means to open the discussion
> until
> > > May
> > > > > 12th.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan <
> > zakelly@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Thanks for all the feedback for FLIP-306: Unified File
> Merging
> > > > > > > > Mechanism for Checkpoints[1] on the discussion thread[2].
> > > > > > > >
> > > > > > > > I'd like to start a vote for it. The vote will be open for at
> > > least
> > > > > 72
> > > > > > > > hours (until May 11th, 12:00AM GMT) unless there is an
> > objection
> > > or
> > > > > an
> > > > > > > > insufficient number of votes.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zakelly
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> > > > > > > > [2]
> > > > https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
> > > > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> > >
> >
> --
> Best Regards,
> Yu
>

Re: Whether to handle removed partitions in KafkaSourceEnumerator ?

[Hongshun Wang]

Hi Ran Tao,

> task-side just continues to consume the removed partitions to the end.

I am curious about how task-side consumes removed partitions. Are you
saying that the task-side can still run without exceptions even if the
partition is removed from metadata?

If so, the "removed partitions" only affects the unassigned partitions. If
the partition has already been successfully unassigned, even if it no
longer exists, it will not affect the task-side.Therefore, I suggest
removing partitions from the pendingPartitionSplitAssignment so that
unassigned but removed partitions will not be assigned to the task side.
This assignment is redundant.

By the way, I wonder under what situations Kafka removes a partition?

Best

Hongshun

[jira] [Created] (FLINK-32043) SqlClient session unrecoverable once one wrong setting occurred

[lincoln lee (Jira)]

lincoln lee created FLINK-32043:
---

 Summary: SqlClient session unrecoverable once one wrong setting 
occurred
 Key: FLINK-32043
 URL: https://issues.apache.org/jira/browse/FLINK-32043
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.17.0
Reporter: lincoln lee


In sql client, it can not work normally once one wrong setting occurred

{code}

// wrong setting here

Flink SQL> SET table.sql-dialect = flink;
[INFO] Execute statement succeed.

Flink SQL> select '' AS f1, a from t1;
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalArgumentException: No enum constant 
org.apache.flink.table.api.SqlDialect.FLINK

Flink SQL> SET table.sql-dialect = default;
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalArgumentException: No enum constant 
org.apache.flink.table.api.SqlDialect.FLINK

{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yun Tang]

+1 (binding)

Thanks Zakelly for driving this topic.

Best
Yun Tang

From: Yu Li 
Sent: Wednesday, May 10, 2023 15:44
To: dev@flink.apache.org 
Subject: Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

+1 (binding)

Thanks Zakelly for driving this, and thanks everyone for the thorough
discussion.

On Wed, May 10, 2023 at 11:15 AM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks for driving this proposal, Zakelly.
>
> +1(binding)
>
> Best,
> Rui Fan
>
> On Wed, May 10, 2023 at 11:04 AM Hangxiang Yu  wrote:
>
> > Hi Zakelly.
> > Thanks for driving this.
> > +1 (no-binding)
> >
> > On Wed, May 10, 2023 at 10:52 AM Yuan Mei 
> wrote:
> >
> > > Thanks for driving this, Zakelly.
> > >
> > > As discussed in the thread,
> > >
> > > +1 for the proposal (binding)
> > >
> > > Best,
> > >
> > > Yuan
> > >
> > >
> > >
> > > On Wed, May 10, 2023 at 10:39 AM Zakelly Lan 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Sorry for the 4 duplicate emails. There was a problem with the dev
> > > > mailing list blocking the mails from Gmail. I thought it was a
> network
> > > > problem so I tried several times. The issue is addressed by
> > > > INFRA-24572[1] and the piled up mails are delivered at one time.
> > > >
> > > > Based on the sending time, the vote will be open until May 12th at
> > > > 11:00PM GMT. Please discuss and vote in the last thread (this one).
> > > > Thanks!
> > > >
> > > >
> > > > Best regards,
> > > > Zakelly
> > > >
> > > > [1] https://issues.apache.org/jira/browse/INFRA-24572
> > > >
> > > > On Wed, May 10, 2023 at 10:30 AM Yanfei Lei 
> > wrote:
> > > > >
> > > > > +1 (no-binding)
> > > > >
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > >
> > > > > Jing Ge  于2023年5月10日周三 07:03写道：
> > > > >
> > > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > I saw you sent at least 4 same emails for voting FLIP-306. I
> guess
> > > > this one
> > > > > > should be the last one and the right one for us to vote right?
> BTW,
> > > > based
> > > > > > on the sending time, 72 hours means to open the discussion until
> > May
> > > > 12th.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan <
> zakelly@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks for all the feedback for FLIP-306: Unified File Merging
> > > > > > > Mechanism for Checkpoints[1] on the discussion thread[2].
> > > > > > >
> > > > > > > I'd like to start a vote for it. The vote will be open for at
> > least
> > > > 72
> > > > > > > hours (until May 11th, 12:00AM GMT) unless there is an
> objection
> > or
> > > > an
> > > > > > > insufficient number of votes.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> > > > > > > [2]
> > > https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
> > > > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
> >
>
--
Best Regards,
Yu

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

[Yu Li]

+1 (binding)

Thanks Zakelly for driving this, and thanks everyone for the thorough
discussion.

On Wed, May 10, 2023 at 11:15 AM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks for driving this proposal, Zakelly.
>
> +1(binding)
>
> Best,
> Rui Fan
>
> On Wed, May 10, 2023 at 11:04 AM Hangxiang Yu  wrote:
>
> > Hi Zakelly.
> > Thanks for driving this.
> > +1 (no-binding)
> >
> > On Wed, May 10, 2023 at 10:52 AM Yuan Mei 
> wrote:
> >
> > > Thanks for driving this, Zakelly.
> > >
> > > As discussed in the thread,
> > >
> > > +1 for the proposal (binding)
> > >
> > > Best,
> > >
> > > Yuan
> > >
> > >
> > >
> > > On Wed, May 10, 2023 at 10:39 AM Zakelly Lan 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Sorry for the 4 duplicate emails. There was a problem with the dev
> > > > mailing list blocking the mails from Gmail. I thought it was a
> network
> > > > problem so I tried several times. The issue is addressed by
> > > > INFRA-24572[1] and the piled up mails are delivered at one time.
> > > >
> > > > Based on the sending time, the vote will be open until May 12th at
> > > > 11:00PM GMT. Please discuss and vote in the last thread (this one).
> > > > Thanks!
> > > >
> > > >
> > > > Best regards,
> > > > Zakelly
> > > >
> > > > [1] https://issues.apache.org/jira/browse/INFRA-24572
> > > >
> > > > On Wed, May 10, 2023 at 10:30 AM Yanfei Lei 
> > wrote:
> > > > >
> > > > > +1 (no-binding)
> > > > >
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > >
> > > > > Jing Ge  于2023年5月10日周三 07:03写道：
> > > > >
> > > > > >
> > > > > > Hi Zakelly,
> > > > > >
> > > > > > I saw you sent at least 4 same emails for voting FLIP-306. I
> guess
> > > > this one
> > > > > > should be the last one and the right one for us to vote right?
> BTW,
> > > > based
> > > > > > on the sending time, 72 hours means to open the discussion until
> > May
> > > > 12th.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > > On Tue, May 9, 2023 at 8:24 PM Zakelly Lan <
> zakelly@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks for all the feedback for FLIP-306: Unified File Merging
> > > > > > > Mechanism for Checkpoints[1] on the discussion thread[2].
> > > > > > >
> > > > > > > I'd like to start a vote for it. The vote will be open for at
> > least
> > > > 72
> > > > > > > hours (until May 11th, 12:00AM GMT) unless there is an
> objection
> > or
> > > > an
> > > > > > > insufficient number of votes.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Zakelly
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> > > > > > > [2]
> > > https://lists.apache.org/thread/56px3kvn3tlwpc7sl12kx6notfmk9g7q
> > > > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
> >
>
-- 
Best Regards,
Yu

Re: Whether to handle removed partitions in KafkaSourceEnumerator ?

[Ran Tao]

The removed partitions will not notice the task-side. As I say, the
task-side just continues to consume the removed partitions to the end.
It will not cause an exception because the task-side has the
remove-partitions's state and offset (even though these remove-partitions
are indicated removed in SourceEnumerator).

Best Regards,
Ran Tao


Hongshun Wang  于2023年5月10日周三 11:10写道：

> Hi Ran Tao,
>
> Thank you for your response. It's very helpful.
>
> I still have some questions:
>
> Currently, if a partition is removed, the reader will not be notified about
> it. Will this cause an exception with the split reader?If this happens, the
> system will never be able to restart successfully, repeatedly.
>
> Perhaps the reason why this issue has not been considered is because Kafka
> does not allow for scaling partitions?
>
>
> Yours
>
> Hongshun
>