[jira] [Created] (FLINK-25339) Moving to hadoop-free flink runtime.

2021-12-15 Thread Jira 
David Morávek created FLINK-25339:
-

 Summary: Moving to hadoop-free flink runtime.
 Key: FLINK-25339
 URL: https://issues.apache.org/jira/browse/FLINK-25339
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Coordination
Reporter: David Morávek


Only remaining reason for having hadoop dependencies (even though these are 
_provided_) in `flink-runtime` is the Security / Kerberos setup, which is 
already hidden behind a service loader. This should be fairly straightforward 
to move into the separate module.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25338) Improvement of connection from TM to JM in session cluster

2021-12-15 Thread Shammon (Jira) 
Shammon created FLINK-25338:
---

 Summary: Improvement of connection from TM to JM in session cluster
 Key: FLINK-25338
 URL: https://issues.apache.org/jira/browse/FLINK-25338
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.14.2, 1.13.5, 1.12.7
Reporter: Shammon


When taskmanager receives slot request from resourcemanager for the specify 
job, it will connect to the jobmaster with given job address. Taskmanager 
register itself, monitor the heartbeat of job and update task's state by this 
connection. There's no need to create connections in one taskmanager for each 
job, and when the taskmanager is busy, it will increase the latency of job. 

One idea is that taskmanager manages the connection to `Dispatcher`, sends 
events such as heartbeat, state update to `Dispatcher`,  and `Dispatcher` tell 
the local `JobMaster`. The main problem is that `Dispatcher` is an actor and 
can only be executed in one thread, it may be the performance bottleneck for 
deserialize event.

The other idea it to create a netty service in `SessionClusterEntrypoint`, it 
can receive and deserialize events from taskmanagers in a threadpool, and send 
the event to the `Dispatcher` or `JobMaster`. Taskmanagers manager the 
connection to the netty service when it start. Thus a service can also receive 
the result of a job from taskmanager later.

[~xtsong] What do you think? THX



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-12-15 Thread Yun Gao 
Hi Fabian,

Very thanks for the update! I think the latest version in general looks good 
from my side
and I think using separate feature interface would be much more easy to 
understand 
and extend in the future. I have some pending issues on the details though:

1. The first one is if we could support end-to-end exactly-once with 
post-committing
topology in the batch mode ? Since for the batch mode, currently we could only 
commit
 all the transactions after the whole job is finished, otherwise if there are 
JM failover or the 
writer / committer get restarted due to indeterminstic (A -> [B1, B2], A, B1 
have finished and
 B2 failed, if -> is rebalance / random / rescale,  all of A, B1, B2 would 
restarted), there might 
be repeat records. Previously one possible thought is to move committer and 
global committer
 to the operator coordinator, but if it is a topology, we might need some other 
kind of solutions?

2. I also want to have a dobule confirmation with the compatibility: since the 
old sink is also named 
with Sink, do we want to put the Sink v2 in a new package ? Besides, since we 
might want to keep 
only have one `sinkTo(Sink sink)` , perhaps we also need to make the Sink v1 
to be a subclass of 
Sink v2 and extends the stateful and two-phase-commit sinks, right? 

3. I'd like also have a confirmation on ours thoughts with the `DataStreamSink` 
returned by the sinkTo method: 
The main issue is how do we implement the method like `setParallelism` or 
`setMaxParallelism` since now the sink 
would be translated to multiple transformations? perhaps we could make it the 
default values for all the transformations 
for the sink? A related issue would be for iceberg sink, I think it would need 
to have only one committer to avoid the 
competition of the optimistic locks (which would cause performance 
degradation), then it might need to have N writers 
with 1 committers, to build such topology, perhaps we might need to add new 
methods to specify the parallelism of 
the writers and committers separately? 

Best,
Yun



 --Original Mail --
Sender:Fabian Paul 
Send Date:Mon Dec 13 23:59:43 2021
Recipients:dev 
Subject:Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small 
file compaction
Hi all,



After a lot of discussions with different, we received very fruitful

feedback and reworked the ideas behind this FLIP. Initially, we had

the impression that the compaction problem is solvable by a single

topology that we can reuse across different sinks. We now have a

better understanding that different external systems require different

compaction mechanism i.e. Hive requires compaction before finally

registering the file in the metastore or Iceberg compacts the files

after they have been registered and just lazily compacts them.



Considering all these different views we came up with a design that

builds upon what @guowei@gmail.com and @yungao...@aliyun.com have

proposed at the beginning. We allow inserting custom topologies before

and after the SinkWriters and Committers. Furthermore, we do not see

it as a downside. The Sink interfaces that will expose the DataStream

to the user reside in flink-streaming-java in contrast to the basic

Sink interfaces that reside fin flink-core deem it to be only used by

expert users.



Moreover, we also wanted to remove the global committer from the

unified Sink interfaces and replace it with a custom post-commit

topology. Unfortunately, we cannot do it without breaking the Sink

interface since the GlobalCommittables are part of the parameterized

Sink interface. Thus, we propose building a new Sink V2 interface

consisting of composable interfaces that do not offer the

GlobalCommitter anymore. We will implement a utility to extend a Sink

with post topology that mimics the behavior of the GlobalCommitter.

The new Sink V2 provides the same sort of methods as the Sink V1

interface, so a migration of sinks that do not use the GlobalCommitter

should be very easy.

We plan to keep the existing Sink V1 interfaces to not break

externally built sinks. As part of this FLIP, we migrate all the

connectors inside of the main repository to the new Sink V2 API.



The FLIP document is also updated and includes the proposed changes.



Looking forward to your feedback,

Fabian



https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction





On Thu, Dec 2, 2021 at 10:15 AM Roman Khachatryan  wrote:

>

> Thanks for clarifying (I was initially confused by merging state files

> rather than output files).

>

> > At some point, Flink will definitely have some WAL adapter that can turn 
> > any sink into an exactly-once sink (with some caveats). For now, we keep 
> > that as an orthogonal solution as it has a rather high price (bursty 
> > workload with high latency). Ideally, we can keep the compaction 
> > asynchronously...

>

> Yes, that

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Israel Ekpo 
Thanks for running and release Chesnay

I hope so too.

On Wed, Dec 15, 2021 at 9:56 PM Chesnay Schepler  wrote:

> The vote duration has passed and we have approved the releases.
>
> Binding votes:
> * Till
> * Yu
> * Stephan
>
> I will now finalize the release. Let's hope this time no new CVE pops up...
>
> On 15/12/2021 03:55, Chesnay Schepler wrote:
> > Hi everyone,
> >
> > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > It covers all 4 releases as they contain the same changes (upgrading
> > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > (Hence, if something is broken, it likely applies to all releases)
> >
> > Note: 1.11/1.12 are still missing the Python Mac releases.
> >
> >
> > Please review and vote on the release candidate #1 for the versions
> > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > [ ] +1, Approve the releases
> > [ ] -1, Do not approve the releases (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source releases and binary convenience releases
> > to be deployed to dist.apache.org [2], which are signed with the key
> > with fingerprint C2EED7B111D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tags [5],
> > * website pull request listing the new releases and adding
> > announcement blog post [6].
> >
> > The vote will be open for at least 24 hours. The minimum vote time has
> > been shortened as the changes are minimal and the matter is urgent.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> > votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> > 1.11:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> > 1.12:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> > 1.13:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> > 1.14:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> > [2]
> > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > 1.11:
> > https://repository.apache.org/content/repositories/orgapacheflink-1460
> > 1.12:
> > https://repository.apache.org/content/repositories/orgapacheflink-1462
> > 1.13:
> > https://repository.apache.org/content/repositories/orgapacheflink-1459
> > 1.14:
> > https://repository.apache.org/content/repositories/orgapacheflink-1461
> > [5]
> > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
> > [6] https://github.com/apache/flink-web/pull/489
> >
> >
>
> --
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://izzyacademy.com/

[jira] [Created] (FLINK-25337) Check whether the target table is valid when SqlToOperationConverter.convertSqlInsert

2021-12-15 Thread vim-wang (Jira) 
vim-wang created FLINK-25337:


 Summary: Check whether the target table is valid when 
SqlToOperationConverter.convertSqlInsert
 Key: FLINK-25337
 URL: https://issues.apache.org/jira/browse/FLINK-25337
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.14.0
Reporter: vim-wang


when I execute insert sql like "insert into t1 select ...", 

If the t1 is not defined，sql will not throw an exception after 
SqlToOperationConverter.convertSqlInsert(), I think this is unreasonable, why 
not use catalogManager to check whether the target table is valid?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25336) Kafka connector compatible problem in Flink sql

2021-12-15 Thread Yuan Zhu (Jira) 
Yuan Zhu created FLINK-25336:


 Summary: Kafka connector compatible problem in Flink sql
 Key: FLINK-25336
 URL: https://issues.apache.org/jira/browse/FLINK-25336
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.0
 Environment: Flink 1.14.0

Kafka 0.10.2.1
Reporter: Yuan Zhu
 Attachments: log.jpg

When I use sql to query kafka table, like
{code:java}
create table `kfk`
(
user_id VARCHAR
) with (
'connector' = 'kafka',
'topic' = 'test',
'properties.bootstrap.servers' = 'localhost:9092',
'format' = 'json', 
'scan.startup.mode' = 'timestamp',
'scan.startup.timestamp-millis' = '163941120',
'properties.group.id' = 'test'
);

CREATE TABLE print_table (user_id varchar) WITH ('connector' = 'print');

insert into print_table select user_id from kfk;{code}
It will encounter an exception:

org.apache.kafka.common.errors.UnsupportedVersionException: MetadataRequest 
versions older than 4 don't support the allowAutoTopicCreation field !log.jpg!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25335) Improvoment of task deployment by enable source split async enumerate

2021-12-15 Thread KevinyhZou (Jira) 
KevinyhZou created FLINK-25335:
--

 Summary: Improvoment of task deployment by enable source split 
async enumerate
 Key: FLINK-25335
 URL: https://issues.apache.org/jira/browse/FLINK-25335
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.12.1
Reporter: KevinyhZou


When submit olap query by flink client to Flink Session Cluster, the JobMaster 
will start scheduling and  enumerate the hive source split by 
`HiveSourceFileEnumerator`, and then deploy the query task and execute it. if 
the source
table has a lot of partition and the partition file is big, the source split 
enumerate will cost a lot of time, which would block the task deployment & 
execution for a long time, and the dashboard can not appear

 

JobMaster should async enumerate the hive split, and meanwhile deploy the query 
task and execute it. when the deployment is finished, source operator fetch 
split and read data, and the split enumeration is also going on.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Chesnay Schepler 

The vote duration has passed and we have approved the releases.

Binding votes:
* Till
* Yu
* Stephan

I will now finalize the release. Let's hope this time no new CVE pops up...

On 15/12/2021 03:55, Chesnay Schepler wrote:

Hi everyone,

This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 
1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading 
Log4j to 2.16.0) and were prepared simultaneously by the same person.

(Hence, if something is broken, it likely applies to all releases)

Note: 1.11/1.12 are still missing the Python Mac releases.


Please review and vote on the release candidate #1 for the versions 
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:

[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source releases and binary convenience releases 
to be deployed to dist.apache.org [2], which are signed with the key 
with fingerprint C2EED7B111D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding 
announcement blog post [6].


The vote will be open for at least 24 hours. The minimum vote time has 
been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative 
votes.


Thanks,
Chesnay

[1]
1.11: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
1.12: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
1.13: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
1.14: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059

[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11: 
https://repository.apache.org/content/repositories/orgapacheflink-1460
1.12: 
https://repository.apache.org/content/repositories/orgapacheflink-1462
1.13: 
https://repository.apache.org/content/repositories/orgapacheflink-1459
1.14: 
https://repository.apache.org/content/repositories/orgapacheflink-1461

[5]
1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
[6] https://github.com/apache/flink-web/pull/489

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Israel Ekpo 
The pull request for Flink-web looks good to me

+1 (non binding) from me



On Wed, Dec 15, 2021 at 8:19 PM Yun Gao 
wrote:

> > Ah actually we may have to merge another commit into the release
> > branches for that to be the case for 1.11/1.12.
>
> > Anyway, I still wouldn't block the release on that.
>
> Yes, it should only be cases for 1.11 / 1.12~ With a double check the
> other two versions contains the doc modification. Very thanks for the
> explanation~
>
> Best,
> Yun
>
>
>
> --
> From:Yun Gao 
> Send Time:2021 Dec. 16 (Thu.) 09:14
> To:dev ; Yun Gao ;
> Stephan Ewen 
> Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate
> #1
>
> Hi Chesnay,
>
> Very thanks for the explanation!
>
> Then +1 (non-binding)
>
> * Verifies the binary distributions could run successfully with the
> example jobs.
> * Verifies the new tags only contains the log4j version bump PRs.
> * The latest blog post and release notes looks good to me.
>
> Very thanks for driving the emergency releases!
>
> Best,
> Yun
>
>
>
> --
> From:Chesnay Schepler 
> Send Time:2021 Dec. 16 (Thu.) 09:03
> To:dev ; Yun Gao ;
> Stephan Ewen 
> Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate
> #1
>
> That's not a problem as far as I'm concerned, because the documentation
> of these releases is never actually published anywhere.
> The documentation at nightlies.apache.org will reference 2.16.0.
>
> On 16/12/2021 02:01, Yun Gao wrote:
> > Hi,
> >
> > May I have a double confirmation that it seems we still have log4j
> version written as
> > 2.15.0 in dev/project-configuration.md in the commit corresponding to
> release tags,
> > which seems to be not consistent with the PR in the github, is it
> expected and would it
> > have influence?
> >
> > Best,
> > Yun
> >
> >
> >
> >   --Original Mail --
> > Sender:Stephan Ewen 
> > Send Date:Thu Dec 16 08:34:10 2021
> > Recipients:dev 
> > Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release
> candidate #1
> > +1 (binding)
> >
> >
> >
> >   - Verified commit history, looks good
> >
> >   => stumbled over the changes in the "create_release_branch.sh ",
> >
> > which are present in each release commit. [1]
> >
> >   => agree that these are not an issue, because this is an out-of-band
> >
> > release
> >
> >   - Release notes for 1.14.2 are off, contain incorrect entry
> "FLINK-25222:
> >
> > Remove NetworkFailureProxy used for Kafka connector tests"
> >
> >   - Checked that released binaries and jars reference correct Scala
> versions
> >
> >   - Ran streaming examples against binary releases for 1.12.7, 1.13.5,
> >
> > 1.14.2. Execution logs look correct.
> >
> >   - Other checks (licenses, no binaries) carry over from previous
> releases
> >
> >
> >
> > [1]
> >
> >
> https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074
> >
> >
> >
> >
> >
> > On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:
> >
> >
> >
> >> +1 (non-binding)
> >> - Checked diff of all versions and verified dep upgrade
> >> - Verified checksum and signatures
> >> - Built 1.14 from source
> >> - checked blog post
> >> Seth
> >> On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:
> >>> +1
> >>> * Verified checksums and signatures
> >>> * Reviewed website PR
> >>> - Minor: left a comment to mention CVE-2021-45046
> >>> * Checked and confirmed new tags only contain log4j version bump
> >>> * Checked release notes and found no issues
> >>> - I've moved FLINK-25317 to 1.14.3
> >>> Thanks for driving these releases Chesnay!
> >>> Best Regards,
> >>> Yu
> >>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler
> >> wrote:
>  FYI; the publication of the python release for 1.11/1.12 will be
> >> delayed
>  because we hit the project size limit on pypi again, and increasing
> >> that
>  limit may take a while.
>  On the positive side, this gives us more time to fix the mac builds.
>  On 15/12/2021 03:55, Chesnay Schepler wrote:
> > Hi everyone,
> > This vote is for the emergency patch releases for 1.11, 1.12, 1.13
> >> and
> > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > It covers all 4 releases as they contain the same changes (upgrading
> > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > (Hence, if something is broken, it likely applies to all releases)
> > Note: 1.11/1.12 are still missing the Python Mac releases.
> > Please review and vote on the release candidate #1 for the versions
> > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > [ ] +1, Approve the releases
> > [ ] -1, Do not approve the releases (please provide specific
> >> comments)
> > The complete staging area is available for your review, which
> >> includes:
> > * JIRA release

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

2021-12-15 Thread Yang Wang 
I am afraid creating a dedicated StatefulSet for each TaskManager is too
expensive and using a shared StatefulSet for all
the TaskManagers is not flexible enough. Maybe setting a proper restart
policy for TaskManager pods could benefit from
this FLIP. But we might need to tackle some other issues, e.g. duplicated
registration, etc.

All in all, this is out of the scope of this FLIP. I agree we could leave
it in the future FLIPs.

I have no more concerns. +1


Best,
Yang

Till Rohrmann  于2021年12月15日周三 19:06写道：

> This is true. But this is not a new problem and I think that Flink should
> be susceptible to this problem already. One solution for this concrete case
> could be that the BlobServer stores some checksums and validates the file
> before serving it to the TM.
>
> Cheers,
> Till
>
> On Wed, Dec 15, 2021 at 11:59 AM Chesnay Schepler 
> wrote:
>
> > The issue with corrupted files is that some of them aren't read by the
> > component that stores them.
> > For example, a file can be corrupted in the blob server of the JM, but
> > that it is corrupted will only be noticed by the TaskExecutor.
> >
> > On 15/12/2021 11:36, Till Rohrmann wrote:
> > > Thanks everyone for your feedback. Let me try to address it by grouping
> > > some of the individual comments:
> > >
> > > ### Will this feature work for native Yarn and K8s deployments?
> > >
> > > The working directory is an optional feature that can be used to
> recover
> > > additional information. You can think of it like a cache. If the
> working
> > > directory is there, then Flink can do certain things a bit faster but
> in
> > > the worst case it will have to retrieve the required information from
> the
> > > JobManager or persistent storage.
> > >
> > > In order to make it work with native Yarn and K8s, we would have to
> > change
> > > these modes slightly. First of all, we would have to be able to map
> > working
> > > directories to processes and then set a deterministic resource ids for
> > the
> > > processes. For K8s this could be easily achievable by using a
> StatefulSet
> > > as the deployment mechanism for TaskExecutors. For Yarn, we probably
> > would
> > > have to remember the prior locations of a process. Both things are
> > > potential follow ups that I don't want to tackle in this FLIP.
> > >
> > > If one of the modes configures the working directory to be on a full or
> > > broken disk, then the process will fail. I think this is not all that
> > > different from the current state where some things in Flink will fail
> if
> > > they picked the wrong/full temporary directory (e.g. blob storage
> > > directory).
> > >
> > > ### Cleanup
> > >
> > > The working directory will be cleaned up if the Flink process is
> > gracefully
> > > shut down. This means that the JobManager process will clean it up if
> it
> > > runs in application mode and the job is terminated. SIGTERM and SIGKILL
> > > signals will be treated as an ungraceful shutdown and therefore they
> > won't
> > > clean up the working directory. This means that we probably also need a
> > > graceful way for shutting TaskManager processes down in the future
> > because
> > > right now they are in most cases killed in order to shut them down. If
> > the
> > > user uses the tmp directory, then any left-over working directories
> will
> > be
> > > cleaned up with the next system restart. This is somewhat similar to
> how
> > > RocksDB's working directory is currently cleaned up as well.
> > >
> > > ### Corrupted files
> > >
> > > The working directory itself won't give you any guarantees. It will be
> > the
> > > responsibility of the component that uses the working directory to make
> > > sure that it can deal with corrupted files. E.g. if the component
> cannot
> > > read the file, then it should delete it and fall back to the remote
> > > storage/ground truth to retrieve the required information.
> > >
> > > I hope this could answer your questions. Let me know if you have more
> > > feedback.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Dec 13, 2021 at 5:05 AM 刘建刚  wrote:
> > >
> > >> I like the idea. It can reuse the disk to do many things. Isn't it
> only
> > >> for inner failover? If not, the cleaning may be a problem. Also, many
> > >> resource components have their own disk schedule strategy.
> > >>
> > >> Chesnay Schepler  于2021年12月12日周日 19:59写道：
> > >>
> > >>> How do you intend to handle corrupted files, in particular due to
> > >>> process crashes during a write?
> > >>> Will all writes to a cached directory append some suffix (e.g.,
> > >>> ".pending") and do a rename?
> > >>>
> > >>> On 10/12/2021 17:54, Till Rohrmann wrote:
> >  Hi everyone,
> > 
> >  I would like to start a discussion about introducing an explicit
> > working
> >  directory for Flink processes that can be used to store information
> > [1].
> >  Per default this working directory will reside in the temporary
> > >>> directory
> >  of the node Flink runs on. However, if

[jira] [Created] (FLINK-25334) Manage and share gateways of taskmanagers between jobs in session cluster

2021-12-15 Thread Shammon (Jira) 
Shammon created FLINK-25334:
---

 Summary: Manage and share gateways of taskmanagers between jobs in 
session cluster
 Key: FLINK-25334
 URL: https://issues.apache.org/jira/browse/FLINK-25334
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.13.3, 1.12.4, 1.14.0
Reporter: Shammon


Flink session cluster launches a certain number of taskmanagers and they won't 
exit during the cluster running. When a job is submitted to the cluster, 
taskmanagers will offer slots to `JobMaster` with location info. Then 
`JobMaster` will get `InetAddress` from the location and connect the 
taskmanager. This cause some performance loss and it isn't necessary in session 
cluster. 

It's better to manager the gateways of taskmanagers in JM and share them 
between jobs. When a taskmanager's gateway is exist and valid, the `JobMaster` 
can use it directly instead of creating a new one.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Yun Gao 
> Ah actually we may have to merge another commit into the release
> branches for that to be the case for 1.11/1.12.

> Anyway, I still wouldn't block the release on that.

Yes, it should only be cases for 1.11 / 1.12~ With a double check the other two 
versions contains the doc modification. Very thanks for the explanation~

Best,
Yun



--
From:Yun Gao 
Send Time:2021 Dec. 16 (Thu.) 09:14
To:dev ; Yun Gao ; Stephan 
Ewen 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

Hi Chesnay,

Very thanks for the explanation!

Then +1 (non-binding)

* Verifies the binary distributions could run successfully with the example 
jobs.
* Verifies the new tags only contains the log4j version bump PRs.
* The latest blog post and release notes looks good to me.

Very thanks for driving the emergency releases!

Best,
Yun



--
From:Chesnay Schepler 
Send Time:2021 Dec. 16 (Thu.) 09:03
To:dev ; Yun Gao ; Stephan 
Ewen 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

That's not a problem as far as I'm concerned, because the documentation 
of these releases is never actually published anywhere.
The documentation at nightlies.apache.org will reference 2.16.0.

On 16/12/2021 02:01, Yun Gao wrote:
> Hi,
>
> May I have a double confirmation that it seems we still have log4j version 
> written as
> 2.15.0 in dev/project-configuration.md in the commit corresponding to release 
> tags,
> which seems to be not consistent with the PR in the github, is it expected 
> and would it
> have influence?
>
> Best,
> Yun
>
>
>
>   --Original Mail --
> Sender:Stephan Ewen 
> Send Date:Thu Dec 16 08:34:10 2021
> Recipients:dev 
> Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
> +1 (binding)
>
>
>
>   - Verified commit history, looks good
>
>   => stumbled over the changes in the "create_release_branch.sh ",
>
> which are present in each release commit. [1]
>
>   => agree that these are not an issue, because this is an out-of-band
>
> release
>
>   - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222:
>
> Remove NetworkFailureProxy used for Kafka connector tests"
>
>   - Checked that released binaries and jars reference correct Scala versions
>
>   - Ran streaming examples against binary releases for 1.12.7, 1.13.5,
>
> 1.14.2. Execution logs look correct.
>
>   - Other checks (licenses, no binaries) carry over from previous releases
>
>
>
> [1]
>
> https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074
>
>
>
>
>
> On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:
>
>
>
>> +1 (non-binding)
>> - Checked diff of all versions and verified dep upgrade
>> - Verified checksum and signatures
>> - Built 1.14 from source
>> - checked blog post
>> Seth
>> On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:
>>> +1
>>> * Verified checksums and signatures
>>> * Reviewed website PR
>>> - Minor: left a comment to mention CVE-2021-45046
>>> * Checked and confirmed new tags only contain log4j version bump
>>> * Checked release notes and found no issues
>>> - I've moved FLINK-25317 to 1.14.3
>>> Thanks for driving these releases Chesnay!
>>> Best Regards,
>>> Yu
>>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler
>> wrote:
 FYI; the publication of the python release for 1.11/1.12 will be
>> delayed
 because we hit the project size limit on pypi again, and increasing
>> that
 limit may take a while.
 On the positive side, this gives us more time to fix the mac builds.
 On 15/12/2021 03:55, Chesnay Schepler wrote:
> Hi everyone,
> This vote is for the emergency patch releases for 1.11, 1.12, 1.13
>> and
> 1.14 to address CVE-2021-44228/CVE-2021-45046.
> It covers all 4 releases as they contain the same changes (upgrading
> Log4j to 2.16.0) and were prepared simultaneously by the same person.
> (Hence, if something is broken, it likely applies to all releases)
> Note: 1.11/1.12 are still missing the Python Mac releases.
> Please review and vote on the release candidate #1 for the versions
> 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> [ ] +1, Approve the releases
> [ ] -1, Do not approve the releases (please provide specific
>> comments)
> The complete staging area is available for your review, which
>> includes:
> * JIRA release notes [1],
> * the official Apache source releases and binary convenience releases
> to be deployed to dist.apache.org [2], which are signed with the key
> with fingerprint C2EED7B111D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tags [5],
> * website pull request listing the new releases and adding
> announcement blog post [6].
>

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Yun Gao 
Hi Chesnay,

Very thanks for the explanation!

Then +1 (non-binding)

* Verifies the binary distributions could run successfully with the example 
jobs.
* Verifies the new tags only contains the log4j version bump PRs.
* The latest blog post and release notes looks good to me.

Very thanks for driving the emergency releases!

Best,
Yun



--
From:Chesnay Schepler 
Send Time:2021 Dec. 16 (Thu.) 09:03
To:dev ; Yun Gao ; Stephan 
Ewen 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

That's not a problem as far as I'm concerned, because the documentation 
of these releases is never actually published anywhere.
The documentation at nightlies.apache.org will reference 2.16.0.

On 16/12/2021 02:01, Yun Gao wrote:
> Hi,
>
> May I have a double confirmation that it seems we still have log4j version 
> written as
> 2.15.0 in dev/project-configuration.md in the commit corresponding to release 
> tags,
> which seems to be not consistent with the PR in the github, is it expected 
> and would it
> have influence?
>
> Best,
> Yun
>
>
>
>   --Original Mail --
> Sender:Stephan Ewen 
> Send Date:Thu Dec 16 08:34:10 2021
> Recipients:dev 
> Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
> +1 (binding)
>
>
>
>   - Verified commit history, looks good
>
>   => stumbled over the changes in the "create_release_branch.sh ",
>
> which are present in each release commit. [1]
>
>   => agree that these are not an issue, because this is an out-of-band
>
> release
>
>   - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222:
>
> Remove NetworkFailureProxy used for Kafka connector tests"
>
>   - Checked that released binaries and jars reference correct Scala versions
>
>   - Ran streaming examples against binary releases for 1.12.7, 1.13.5,
>
> 1.14.2. Execution logs look correct.
>
>   - Other checks (licenses, no binaries) carry over from previous releases
>
>
>
> [1]
>
> https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074
>
>
>
>
>
> On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:
>
>
>
>> +1 (non-binding)
>> - Checked diff of all versions and verified dep upgrade
>> - Verified checksum and signatures
>> - Built 1.14 from source
>> - checked blog post
>> Seth
>> On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:
>>> +1
>>> * Verified checksums and signatures
>>> * Reviewed website PR
>>> - Minor: left a comment to mention CVE-2021-45046
>>> * Checked and confirmed new tags only contain log4j version bump
>>> * Checked release notes and found no issues
>>> - I've moved FLINK-25317 to 1.14.3
>>> Thanks for driving these releases Chesnay!
>>> Best Regards,
>>> Yu
>>> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler
>> wrote:
 FYI; the publication of the python release for 1.11/1.12 will be
>> delayed
 because we hit the project size limit on pypi again, and increasing
>> that
 limit may take a while.
 On the positive side, this gives us more time to fix the mac builds.
 On 15/12/2021 03:55, Chesnay Schepler wrote:
> Hi everyone,
> This vote is for the emergency patch releases for 1.11, 1.12, 1.13
>> and
> 1.14 to address CVE-2021-44228/CVE-2021-45046.
> It covers all 4 releases as they contain the same changes (upgrading
> Log4j to 2.16.0) and were prepared simultaneously by the same person.
> (Hence, if something is broken, it likely applies to all releases)
> Note: 1.11/1.12 are still missing the Python Mac releases.
> Please review and vote on the release candidate #1 for the versions
> 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> [ ] +1, Approve the releases
> [ ] -1, Do not approve the releases (please provide specific
>> comments)
> The complete staging area is available for your review, which
>> includes:
> * JIRA release notes [1],
> * the official Apache source releases and binary convenience releases
> to be deployed to dist.apache.org [2], which are signed with the key
> with fingerprint C2EED7B111D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tags [5],
> * website pull request listing the new releases and adding
> announcement blog post [6].
> The vote will be open for at least 24 hours. The minimum vote time
>> has
> been shortened as the changes are minimal and the matter is urgent.
> It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> Thanks,
> Chesnay
> [1]
> 1.11:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> 1.12:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> 1.13:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Chesnay Schepler 
Ah actually we may have to merge another commit into the release 
branches for that to be the case for 1.11/1.12.


Anyway, I still wouldn't block the release on that.

On 16/12/2021 02:03, Chesnay Schepler wrote:
That's not a problem as far as I'm concerned, because the 
documentation of these releases is never actually published anywhere.

The documentation at nightlies.apache.org will reference 2.16.0.

On 16/12/2021 02:01, Yun Gao wrote:

Hi,

May I have a double confirmation that it seems we still have log4j 
version written as
2.15.0 in dev/project-configuration.md in the commit corresponding to 
release tags,
which seems to be not consistent with the PR in the github, is it 
expected and would it

have influence?

Best,
Yun



  --Original Mail --
Sender:Stephan Ewen 
Send Date:Thu Dec 16 08:34:10 2021
Recipients:dev 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release 
candidate #1

+1 (binding)



  - Verified commit history, looks good

  => stumbled over the changes in the "create_release_branch.sh ",

which are present in each release commit. [1]

  => agree that these are not an issue, because this is an out-of-band

release

  - Release notes for 1.14.2 are off, contain incorrect entry 
"FLINK-25222:


Remove NetworkFailureProxy used for Kafka connector tests"

  - Checked that released binaries and jars reference correct Scala 
versions


  - Ran streaming examples against binary releases for 1.12.7, 1.13.5,

1.14.2. Execution logs look correct.

  - Other checks (licenses, no binaries) carry over from previous 
releases




[1]

https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074 







On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:




+1 (non-binding)
- Checked diff of all versions and verified dep upgrade
- Verified checksum and signatures
- Built 1.14 from source
- checked blog post
Seth
On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:

+1
* Verified checksums and signatures
* Reviewed website PR
- Minor: left a comment to mention CVE-2021-45046
* Checked and confirmed new tags only contain log4j version bump
* Checked release notes and found no issues
- I've moved FLINK-25317 to 1.14.3
Thanks for driving these releases Chesnay!
Best Regards,
Yu
On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler

wrote:

FYI; the publication of the python release for 1.11/1.12 will be

delayed

because we hit the project size limit on pypi again, and increasing

that

limit may take a while.
On the positive side, this gives us more time to fix the mac builds.
On 15/12/2021 03:55, Chesnay Schepler wrote:

Hi everyone,
This vote is for the emergency patch releases for 1.11, 1.12, 1.13

and

1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading
Log4j to 2.16.0) and were prepared simultaneously by the same 
person.

(Hence, if something is broken, it likely applies to all releases)
Note: 1.11/1.12 are still missing the Python Mac releases.
Please review and vote on the release candidate #1 for the versions
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific

comments)

The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source releases and binary convenience 
releases

to be deployed to dist.apache.org [2], which are signed with the key
with fingerprint C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding
announcement blog post [6].
The vote will be open for at least 24 hours. The minimum vote time

has

been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative
votes.
Thanks,
Chesnay
[1]
1.11:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056 


1.12:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057 


1.13:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058 


1.14:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059 


[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11:

https://repository.apache.org/content/repositories/orgapacheflink-1460

1.12:

https://repository.apache.org/content/repositories/orgapacheflink-1462

1.13:

https://repository.apache.org/content/repositories/orgapacheflink-1459

1.14:

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Chesnay Schepler 
That's not a problem as far as I'm concerned, because the documentation 
of these releases is never actually published anywhere.

The documentation at nightlies.apache.org will reference 2.16.0.

On 16/12/2021 02:01, Yun Gao wrote:

Hi,

May I have a double confirmation that it seems we still have log4j version 
written as
2.15.0 in dev/project-configuration.md in the commit corresponding to release 
tags,
which seems to be not consistent with the PR in the github, is it expected and 
would it
have influence?

Best,
Yun



  --Original Mail --
Sender:Stephan Ewen 
Send Date:Thu Dec 16 08:34:10 2021
Recipients:dev 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+1 (binding)



  - Verified commit history, looks good

  => stumbled over the changes in the "create_release_branch.sh ",

which are present in each release commit. [1]

  => agree that these are not an issue, because this is an out-of-band

release

  - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222:

Remove NetworkFailureProxy used for Kafka connector tests"

  - Checked that released binaries and jars reference correct Scala versions

  - Ran streaming examples against binary releases for 1.12.7, 1.13.5,

1.14.2. Execution logs look correct.

  - Other checks (licenses, no binaries) carry over from previous releases



[1]

https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074





On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:




+1 (non-binding)
- Checked diff of all versions and verified dep upgrade
- Verified checksum and signatures
- Built 1.14 from source
- checked blog post
Seth
On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:

+1
* Verified checksums and signatures
* Reviewed website PR
- Minor: left a comment to mention CVE-2021-45046
* Checked and confirmed new tags only contain log4j version bump
* Checked release notes and found no issues
- I've moved FLINK-25317 to 1.14.3
Thanks for driving these releases Chesnay!
Best Regards,
Yu
On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler

wrote:

FYI; the publication of the python release for 1.11/1.12 will be

delayed

because we hit the project size limit on pypi again, and increasing

that

limit may take a while.
On the positive side, this gives us more time to fix the mac builds.
On 15/12/2021 03:55, Chesnay Schepler wrote:

Hi everyone,
This vote is for the emergency patch releases for 1.11, 1.12, 1.13

and

1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading
Log4j to 2.16.0) and were prepared simultaneously by the same person.
(Hence, if something is broken, it likely applies to all releases)
Note: 1.11/1.12 are still missing the Python Mac releases.
Please review and vote on the release candidate #1 for the versions
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific

comments)

The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source releases and binary convenience releases
to be deployed to dist.apache.org [2], which are signed with the key
with fingerprint C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding
announcement blog post [6].
The vote will be open for at least 24 hours. The minimum vote time

has

been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative
votes.
Thanks,
Chesnay
[1]
1.11:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056

1.12:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057

1.13:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058

1.14:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059

[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11:

https://repository.apache.org/content/repositories/orgapacheflink-1460

1.12:

https://repository.apache.org/content/repositories/orgapacheflink-1462

1.13:

https://repository.apache.org/content/repositories/orgapacheflink-1459

1.14:

https://repository.apache.org/content/repositories/orgapacheflink-1461

[5]
1.11:

https://github.com/apache/flink/releases/tag/release-1.11.6-rc1

1.12:

https://github.com/apache/flink/releases/tag/release-1.12.7-rc1

1.13:

Re: Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Yun Gao 
Hi,

May I have a double confirmation that it seems we still have log4j version 
written as
2.15.0 in dev/project-configuration.md in the commit corresponding to release 
tags,
which seems to be not consistent with the PR in the github, is it expected and 
would it
have influence? 

Best,
Yun



 --Original Mail --
Sender:Stephan Ewen 
Send Date:Thu Dec 16 08:34:10 2021
Recipients:dev 
Subject:Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
+1 (binding)



 - Verified commit history, looks good

 => stumbled over the changes in the "create_release_branch.sh ",

which are present in each release commit. [1]

 => agree that these are not an issue, because this is an out-of-band

release

 - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222:

Remove NetworkFailureProxy used for Kafka connector tests"

 - Checked that released binaries and jars reference correct Scala versions

 - Ran streaming examples against binary releases for 1.12.7, 1.13.5,

1.14.2. Execution logs look correct.

 - Other checks (licenses, no binaries) carry over from previous releases



[1]

https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074





On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:



> +1 (non-binding)

>

> - Checked diff of all versions and verified dep upgrade

> - Verified checksum and signatures

> - Built 1.14 from source

> - checked blog post

>

> Seth

>

> On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:

>

> > +1

> >

> > * Verified checksums and signatures

> > * Reviewed website PR

> > - Minor: left a comment to mention CVE-2021-45046

> > * Checked and confirmed new tags only contain log4j version bump

> > * Checked release notes and found no issues

> > - I've moved FLINK-25317 to 1.14.3

> >

> > Thanks for driving these releases Chesnay!

> >

> > Best Regards,

> > Yu

> >

> >

> > On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler 

> wrote:

> >

> > > FYI; the publication of the python release for 1.11/1.12 will be

> delayed

> > > because we hit the project size limit on pypi again, and increasing

> that

> > > limit may take a while.

> > > On the positive side, this gives us more time to fix the mac builds.

> > >

> > > On 15/12/2021 03:55, Chesnay Schepler wrote:

> > > > Hi everyone,

> > > >

> > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13

> and

> > > > 1.14 to address CVE-2021-44228/CVE-2021-45046.

> > > > It covers all 4 releases as they contain the same changes (upgrading

> > > > Log4j to 2.16.0) and were prepared simultaneously by the same person.

> > > > (Hence, if something is broken, it likely applies to all releases)

> > > >

> > > > Note: 1.11/1.12 are still missing the Python Mac releases.

> > > >

> > > >

> > > > Please review and vote on the release candidate #1 for the versions

> > > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:

> > > > [ ] +1, Approve the releases

> > > > [ ] -1, Do not approve the releases (please provide specific

> comments)

> > > >

> > > > The complete staging area is available for your review, which

> includes:

> > > > * JIRA release notes [1],

> > > > * the official Apache source releases and binary convenience releases

> > > > to be deployed to dist.apache.org [2], which are signed with the key

> > > > with fingerprint C2EED7B111D464BA [3],

> > > > * all artifacts to be deployed to the Maven Central Repository [4],

> > > > * source code tags [5],

> > > > * website pull request listing the new releases and adding

> > > > announcement blog post [6].

> > > >

> > > > The vote will be open for at least 24 hours. The minimum vote time

> has

> > > > been shortened as the changes are minimal and the matter is urgent.

> > > > It is adopted by majority approval, with at least 3 PMC affirmative

> > > > votes.

> > > >

> > > > Thanks,

> > > > Chesnay

> > > >

> > > > [1]

> > > > 1.11:

> > > >

> > >

> >

> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056

> > > > 1.12:

> > > >

> > >

> >

> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057

> > > > 1.13:

> > > >

> > >

> >

> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058

> > > > 1.14:

> > > >

> > >

> >

> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059

> > > > [2]

> > > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/

> > > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/

> > > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/

> > > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/

> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS

> > > > [4]

> > > > 1.11:

> > > >

>

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Stephan Ewen 
+1 (binding)

  - Verified commit history, looks good
  => stumbled over the changes in the "create_release_branch.sh ",
which are present in each release commit. [1]
  => agree that these are not an issue, because this is an out-of-band
release
  - Release notes for 1.14.2 are off, contain incorrect entry "FLINK-25222:
Remove NetworkFailureProxy used for Kafka connector tests"
  - Checked that released binaries and jars reference correct Scala versions
  - Ran streaming examples against binary releases for 1.12.7, 1.13.5,
1.14.2. Execution logs look correct.
  - Other checks (licenses, no binaries) carry over from previous releases

[1]
https://github.com/apache/flink/commit/6fd4b1c0ef2ddd12751889218445ce0e60ff6c80#diff-94c70ce1a0abddcd83314c83b46080d8edbcd919b737f316cd6f72006d464074


On Wed, Dec 15, 2021 at 5:54 PM Seth Wiesman  wrote:

> +1 (non-binding)
>
> - Checked diff of all versions and verified dep upgrade
> - Verified checksum and signatures
> - Built 1.14 from source
> - checked blog post
>
> Seth
>
> On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:
>
> > +1
> >
> > * Verified checksums and signatures
> > * Reviewed website PR
> >- Minor: left a comment to mention CVE-2021-45046
> > * Checked and confirmed new tags only contain log4j version bump
> > * Checked release notes and found no issues
> >- I've moved FLINK-25317 to 1.14.3
> >
> > Thanks for driving these releases Chesnay!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler 
> wrote:
> >
> > > FYI; the publication of the python release for 1.11/1.12 will be
> delayed
> > > because we hit the project size limit on pypi again, and increasing
> that
> > > limit may take a while.
> > > On the positive side, this gives us more time to fix the mac builds.
> > >
> > > On 15/12/2021 03:55, Chesnay Schepler wrote:
> > > > Hi everyone,
> > > >
> > > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13
> and
> > > > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > > > It covers all 4 releases as they contain the same changes (upgrading
> > > > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > > > (Hence, if something is broken, it likely applies to all releases)
> > > >
> > > > Note: 1.11/1.12 are still missing the Python Mac releases.
> > > >
> > > >
> > > > Please review and vote on the release candidate #1 for the versions
> > > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > > > [ ] +1, Approve the releases
> > > > [ ] -1, Do not approve the releases (please provide specific
> comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source releases and binary convenience releases
> > > > to be deployed to dist.apache.org [2], which are signed with the key
> > > > with fingerprint C2EED7B111D464BA [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tags [5],
> > > > * website pull request listing the new releases and adding
> > > > announcement blog post [6].
> > > >
> > > > The vote will be open for at least 24 hours. The minimum vote time
> has
> > > > been shortened as the changes are minimal and the matter is urgent.
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > > votes.
> > > >
> > > > Thanks,
> > > > Chesnay
> > > >
> > > > [1]
> > > > 1.11:
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> > > > 1.12:
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> > > > 1.13:
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> > > > 1.14:
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> > > > [2]
> > > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> > > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> > > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> > > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > > 1.11:
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1460
> > > > 1.12:
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1462
> > > > 1.13:
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1459
> > > > 1.14:
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1461
> > > > [5]
> > > > 1.11:
> https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> > > > 1.12:
> https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> > > > 1.13:
> https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> > > > 1.14:
>

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Seth Wiesman 
+1 (non-binding)

- Checked diff of all versions and verified dep upgrade
- Verified checksum and signatures
- Built 1.14 from source
- checked blog post

Seth

On Wed, Dec 15, 2021 at 10:22 AM Yu Li  wrote:

> +1
>
> * Verified checksums and signatures
> * Reviewed website PR
>- Minor: left a comment to mention CVE-2021-45046
> * Checked and confirmed new tags only contain log4j version bump
> * Checked release notes and found no issues
>- I've moved FLINK-25317 to 1.14.3
>
> Thanks for driving these releases Chesnay!
>
> Best Regards,
> Yu
>
>
> On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler  wrote:
>
> > FYI; the publication of the python release for 1.11/1.12 will be delayed
> > because we hit the project size limit on pypi again, and increasing that
> > limit may take a while.
> > On the positive side, this gives us more time to fix the mac builds.
> >
> > On 15/12/2021 03:55, Chesnay Schepler wrote:
> > > Hi everyone,
> > >
> > > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> > > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > > It covers all 4 releases as they contain the same changes (upgrading
> > > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > > (Hence, if something is broken, it likely applies to all releases)
> > >
> > > Note: 1.11/1.12 are still missing the Python Mac releases.
> > >
> > >
> > > Please review and vote on the release candidate #1 for the versions
> > > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > > [ ] +1, Approve the releases
> > > [ ] -1, Do not approve the releases (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source releases and binary convenience releases
> > > to be deployed to dist.apache.org [2], which are signed with the key
> > > with fingerprint C2EED7B111D464BA [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tags [5],
> > > * website pull request listing the new releases and adding
> > > announcement blog post [6].
> > >
> > > The vote will be open for at least 24 hours. The minimum vote time has
> > > been shortened as the changes are minimal and the matter is urgent.
> > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > >
> > > Thanks,
> > > Chesnay
> > >
> > > [1]
> > > 1.11:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> > > 1.12:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> > > 1.13:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> > > 1.14:
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> > > [2]
> > > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> > > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> > > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> > > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > > 1.11:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1460
> > > 1.12:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1462
> > > 1.13:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1459
> > > 1.14:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1461
> > > [5]
> > > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> > > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> > > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> > > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
> > > [6] https://github.com/apache/flink-web/pull/489
> > >
> > >
> >
> >
>

[jira] [Created] (FLINK-25333) Harden UID management when constructing statefun universe

2021-12-15 Thread Seth Wiesman (Jira) 
Seth Wiesman created FLINK-25333:


 Summary: Harden UID management when constructing statefun universe
 Key: FLINK-25333
 URL: https://issues.apache.org/jira/browse/FLINK-25333
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Seth Wiesman
Assignee: Seth Wiesman
 Fix For: statefun-3.2.0


As described in this stack overflow, we do not set UIDs on routers. Because of 
how the stream graph is generated, the uids may be non-deterministic. We should 
manually set all UIDs and enforce this via configuration. 

 

https://stackoverflow.com/questions/70316498/flink-statefun-high-availability-exception-java-lang-illegalstateexception-th



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25332) When using Pyflink Table API, 'where' clause seems to work incorrectly

2021-12-15 Thread TongMeng (Jira) 
TongMeng created FLINK-25332:


 Summary: When using Pyflink Table API, 'where' clause seems to 
work incorrectly
 Key: FLINK-25332
 URL: https://issues.apache.org/jira/browse/FLINK-25332
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.13.0
 Environment: Python 3.6.9, Pyflink 1.13.0, kafka2.12-2.4.0
Reporter: TongMeng


The UDF I used just returns a float, the first four data it returns 1.0, 2.0, 
3.0 and 4.0, then it returns 0.0. I use 'where' in the sql to filter the 0.0 
result. So the expected result I want to see in the kafka should be 1.0, 2.0, 
3.0 and 4.0. However kafka consumer gives four 0.0.

The sql is as follow:

"insert into algorithmsink select dt.my_result from(select udf1(a) AS my_result 
from mysource) AS dt where dt.my_result > 0.0"

After I  removed the 'where dt.my_result > 0.0' part, it workd well. Kafka gave 
1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0……



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Yu Li 
+1

* Verified checksums and signatures
* Reviewed website PR
   - Minor: left a comment to mention CVE-2021-45046
* Checked and confirmed new tags only contain log4j version bump
* Checked release notes and found no issues
   - I've moved FLINK-25317 to 1.14.3

Thanks for driving these releases Chesnay!

Best Regards,
Yu


On Wed, 15 Dec 2021 at 21:29, Chesnay Schepler  wrote:

> FYI; the publication of the python release for 1.11/1.12 will be delayed
> because we hit the project size limit on pypi again, and increasing that
> limit may take a while.
> On the positive side, this gives us more time to fix the mac builds.
>
> On 15/12/2021 03:55, Chesnay Schepler wrote:
> > Hi everyone,
> >
> > This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> > 1.14 to address CVE-2021-44228/CVE-2021-45046.
> > It covers all 4 releases as they contain the same changes (upgrading
> > Log4j to 2.16.0) and were prepared simultaneously by the same person.
> > (Hence, if something is broken, it likely applies to all releases)
> >
> > Note: 1.11/1.12 are still missing the Python Mac releases.
> >
> >
> > Please review and vote on the release candidate #1 for the versions
> > 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> > [ ] +1, Approve the releases
> > [ ] -1, Do not approve the releases (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source releases and binary convenience releases
> > to be deployed to dist.apache.org [2], which are signed with the key
> > with fingerprint C2EED7B111D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tags [5],
> > * website pull request listing the new releases and adding
> > announcement blog post [6].
> >
> > The vote will be open for at least 24 hours. The minimum vote time has
> > been shortened as the changes are minimal and the matter is urgent.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> > votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> > 1.11:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> > 1.12:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> > 1.13:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> > 1.14:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> > [2]
> > 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> > 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> > 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> > 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > 1.11:
> > https://repository.apache.org/content/repositories/orgapacheflink-1460
> > 1.12:
> > https://repository.apache.org/content/repositories/orgapacheflink-1462
> > 1.13:
> > https://repository.apache.org/content/repositories/orgapacheflink-1459
> > 1.14:
> > https://repository.apache.org/content/repositories/orgapacheflink-1461
> > [5]
> > 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> > 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> > 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> > 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
> > [6] https://github.com/apache/flink-web/pull/489
> >
> >
>
>

[jira] [Created] (FLINK-25331) Allow testcontainer tests to run on Java 11

2021-12-15 Thread Chesnay Schepler (Jira) 
Chesnay Schepler created FLINK-25331:


 Summary: Allow testcontainer tests to run on Java 11
 Key: FLINK-25331
 URL: https://issues.apache.org/jira/browse/FLINK-25331
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: Chesnay Schepler






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Chesnay Schepler 
FYI; the publication of the python release for 1.11/1.12 will be delayed 
because we hit the project size limit on pypi again, and increasing that 
limit may take a while.

On the positive side, this gives us more time to fix the mac builds.

On 15/12/2021 03:55, Chesnay Schepler wrote:

Hi everyone,

This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and 
1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading 
Log4j to 2.16.0) and were prepared simultaneously by the same person.

(Hence, if something is broken, it likely applies to all releases)

Note: 1.11/1.12 are still missing the Python Mac releases.


Please review and vote on the release candidate #1 for the versions 
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:

[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source releases and binary convenience releases 
to be deployed to dist.apache.org [2], which are signed with the key 
with fingerprint C2EED7B111D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding 
announcement blog post [6].


The vote will be open for at least 24 hours. The minimum vote time has 
been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative 
votes.


Thanks,
Chesnay

[1]
1.11: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
1.12: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
1.13: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
1.14: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059

[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11: 
https://repository.apache.org/content/repositories/orgapacheflink-1460
1.12: 
https://repository.apache.org/content/repositories/orgapacheflink-1462
1.13: 
https://repository.apache.org/content/repositories/orgapacheflink-1459
1.14: 
https://repository.apache.org/content/repositories/orgapacheflink-1461

[5]
1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
[6] https://github.com/apache/flink-web/pull/489

[jira] [Created] (FLINK-25330) Flink SQL doesn't retract all versions of Hbase data

2021-12-15 Thread Bruce Wong (Jira) 
Bruce Wong created FLINK-25330:
--

 Summary: Flink SQL doesn't retract all versions of Hbase data
 Key: FLINK-25330
 URL: https://issues.apache.org/jira/browse/FLINK-25330
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.14.0
Reporter: Bruce Wong
 Attachments: image-2021-12-15-20-00-22-767.png

h2. Background

When we use CDC to synchronize mysql data to HBase, we find that HBase deletes 
only the last version of the specified rowkey when deleting mysql data. The 
data of the old version still exists. You end up using the wrong data. And I 
think its a bug of HBase connector.

The following figure shows Hbase data changes before and after mysql data is 
deleted.

!image-2021-12-15-19-59-31-837.png|width=925,height=680!

 
h2.  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1

2021-12-15 Thread Stephan Ewen 
That's right, they are referenced in POMs published with the jars, though.
But that's minor.



On Wed, Dec 15, 2021 at 12:28 PM Chesnay Schepler 
wrote:

> AFAIK none of the jars we publish actually contains log4j.
> It's only bundled by the distribution/python binaries/docker images.
>
> Hence I don't think the jars help in this case.
>
> On 15/12/2021 10:42, Stephan Ewen wrote:
> > Given that these artifacts are published already, users can use them if
> > they want to update now:
> >
> > For example:
> > https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar
> >
> > Just for the users that really want to update now (rather than rely on
> the
> > mitigation via config) and are not as much concerned about the remaining
> > weakness in log4j 2.15.0
> >
> > On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman 
> wrote:
> >
> >> Thank you for managing these updates Chesnay!
> >>
> >>
> >>
> >> On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler 
> >> wrote:
> >>
> >>> Since the maven artifacts have already been published we will use the
> >>> next patch version for each release, i.e.:
> >>> 1.11.6
> >>> 1.12.7
> >>> 1.13.5
> >>> 1.14.2
> >>>
> >>> (We could technically just update the source/binaries, but that seems
> >>> fishy).
> >>>
> >>> On 14/12/2021 22:38, Chesnay Schepler wrote:
>  I'm canceling the release because the issue was not fully fixed in
>  Log4j 2.15.0; see CVE-2021-45046.
> 
>  I will start preparing new release candidates that use Log4j 2.16.0 .
> 
>  On 14/12/2021 21:28, Chesnay Schepler wrote:
> > The vote duration has passed and we have approved the releases.
> >
> > Binding votes:
> > * Stephan
> > * Till
> > * Xintong
> > * Zhu
> > * Gordon
> >
> > I will not finalize the release.
> >
> > On 13/12/2021 20:28, Chesnay Schepler wrote:
> >> Hi everyone,
> >>
> >> This vote is for the emergency patch releases for 1.11, 1.12, 1.13
> >> and 1.14 to address CVE-2021-44228.
> >> It covers all 4 releases as they contain the same changes (upgrading
> >> Log4j to 2.15.0) and were prepared simultaneously by the same
> person.
> >> (Hence, if something is broken, it likely applies to all releases)
> >>
> >> Please review and vote on the release candidate #1 for the versions
> >> 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows:
> >> [ ] +1, Approve the releases
> >> [ ] -1, Do not approve the releases (please provide specific
> >> comments)
> >> The complete staging area is available for your review, which
> >> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source releases and binary convenience
> >> releases to be deployed to dist.apache.org [2], which are signed
> >> with the key with fingerprint C2EED7B111D464BA [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >>  * *the jars for 1.13/1.14 are still being built*
> >> * source code tags [5],
> >> * website pull request listing the new releases and adding
> >> announcement blog post [6].
> >>
> >> The vote will be open for at least 24 hours. The minimum vote time
> >> has been shortened as the changes are minimal and the matter is
> >> urgent.
> >> It is adopted by majority approval, with at least 3 PMC affirmative
> >> votes.
> >>
> >> Thanks,
> >> Chesnay
> >>
> >> [1]
> >> 1.11:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350327
> >> 1.12:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350328
> >> 1.13:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350686
> >> 1.14:
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350512
> >> [2]
> >> 1.11:
> https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/
> >> 1.12:
> https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/
> >> 1.13:
> https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/
> >> 1.14:
> https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [4]
> >> 1.11/1.12:
> >>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1455
> >> 1.13:
> >>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1457
> >> 1.14:
> >>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1456
> >> [5]
> >> 1.11:
> >> https://github.com/apache/flink/releases/tag/release-1.11.5-rc1
> >> 1.12:
> >> https://github.com/apache/flink/releases/tag/release-1.12.6-rc1
> >> 1.13:
> >> https://github.com/apache/flink/releases/tag/release-1.13.4-rc1
> >> 1.14:
> >> https://github.com/apache/flink/releases/tag/release-1.14.1-rc1
> >> [6]

[jira] [Created] (FLINK-25329) Improvement of disk operations in flink session cluster for jobs

2021-12-15 Thread Shammon (Jira) 
Shammon created FLINK-25329:
---

 Summary: Improvement of disk operations in flink session cluster 
for jobs
 Key: FLINK-25329
 URL: https://issues.apache.org/jira/browse/FLINK-25329
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.13.3, 1.12.5, 1.14.0
Reporter: Shammon


Flink session cluster uses files to store info of jobs after they reach 
termination with `FileExecutionGraphInfoStore`, each job will generate one 
file. When the cluster executes many small jobs concurrently, there will be 
many disk related operations, which will

1> Increase the CPU usage of `Dispatcher`
2> Decrease the performance of the jobs in the cluster.

We hope to improve the disk operations in `FileExecutionGraphInfoStore` to 
increase the performance of session cluster.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1

2021-12-15 Thread Chesnay Schepler 

AFAIK none of the jars we publish actually contains log4j.
It's only bundled by the distribution/python binaries/docker images.

Hence I don't think the jars help in this case.

On 15/12/2021 10:42, Stephan Ewen wrote:

Given that these artifacts are published already, users can use them if
they want to update now:

For example:
https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar

Just for the users that really want to update now (rather than rely on the
mitigation via config) and are not as much concerned about the remaining
weakness in log4j 2.15.0

On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman  wrote:


Thank you for managing these updates Chesnay!



On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler 
wrote:


Since the maven artifacts have already been published we will use the
next patch version for each release, i.e.:
1.11.6
1.12.7
1.13.5
1.14.2

(We could technically just update the source/binaries, but that seems
fishy).

On 14/12/2021 22:38, Chesnay Schepler wrote:

I'm canceling the release because the issue was not fully fixed in
Log4j 2.15.0; see CVE-2021-45046.

I will start preparing new release candidates that use Log4j 2.16.0 .

On 14/12/2021 21:28, Chesnay Schepler wrote:

The vote duration has passed and we have approved the releases.

Binding votes:
* Stephan
* Till
* Xintong
* Zhu
* Gordon

I will not finalize the release.

On 13/12/2021 20:28, Chesnay Schepler wrote:

Hi everyone,

This vote is for the emergency patch releases for 1.11, 1.12, 1.13
and 1.14 to address CVE-2021-44228.
It covers all 4 releases as they contain the same changes (upgrading
Log4j to 2.15.0) and were prepared simultaneously by the same person.
(Hence, if something is broken, it likely applies to all releases)

Please review and vote on the release candidate #1 for the versions
1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows:
[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific

comments)

The complete staging area is available for your review, which

includes:

* JIRA release notes [1],
* the official Apache source releases and binary convenience
releases to be deployed to dist.apache.org [2], which are signed
with the key with fingerprint C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
 * *the jars for 1.13/1.14 are still being built*
* source code tags [5],
* website pull request listing the new releases and adding
announcement blog post [6].

The vote will be open for at least 24 hours. The minimum vote time
has been shortened as the changes are minimal and the matter is

urgent.

It is adopted by majority approval, with at least 3 PMC affirmative
votes.

Thanks,
Chesnay

[1]
1.11:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350327

1.12:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350328

1.13:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350686

1.14:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350512

[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11/1.12:


https://repository.apache.org/content/repositories/orgapacheflink-1455

1.13:


https://repository.apache.org/content/repositories/orgapacheflink-1457

1.14:


https://repository.apache.org/content/repositories/orgapacheflink-1456

[5]
1.11:

https://github.com/apache/flink/releases/tag/release-1.11.5-rc1

1.12:

https://github.com/apache/flink/releases/tag/release-1.12.6-rc1

1.13:

https://github.com/apache/flink/releases/tag/release-1.13.4-rc1

1.14:

https://github.com/apache/flink/releases/tag/release-1.14.1-rc1

[6] https://github.com/apache/flink-web/pull/489

[jira] [Created] (FLINK-25328) Improvement of share memory manager between jobs if they use the same slot in TaskManager for flink olap queries

2021-12-15 Thread Shammon (Jira) 
Shammon created FLINK-25328:
---

 Summary: Improvement of share memory manager between jobs if they 
use the same slot in TaskManager for flink olap queries
 Key: FLINK-25328
 URL: https://issues.apache.org/jira/browse/FLINK-25328
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.13.3, 1.12.5, 1.14.0
Reporter: Shammon


We submit batch jobs to flink session cluster as olap queries, and these 
jobs' subtasks in TaskManager are frequently created and destroyed because they 
finish their work quickly. Each slot in taskmanager manages `MemoryManager` for 
multiple tasks in one job, and the `MemoryManager` is closed when all the 
subtasks are finished. Join/Aggregate/Sort and etc. operators in the subtasks 
allocate `MemorySegment` via `MemoryManager` and these `MemorySegment` will be 
free when they are finished. 

It causes too much memory allocation and free of `MemorySegment` in 
taskmanager. For example, a TaskManager contains 50 slots, one job has 3 
join/agg operatos run in the slot, each operator will allocate 2000 segments 
and initialize them. If the subtasks of a job take 100ms to execute, then the 
taskmanager will execute 10 jobs' subtasks one second and it will allocate and 
free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too many 
segments from memory will cause two issues:

1) Increases the CPU usage of taskmanager
2) Increase the cost of subtasks in taskmanager, which will increase the 
latency of job and decrease the qps.

To improve the usage of memory segment between jobs in the same slot, 
we propose not drop memory manager when all the subtasks in the slot are 
finished. The slot will hold the `MemoryManager` and not free the allocated 
`MemorySegment` in it immediately. When some subtasks of another job are 
assigned to the slot, they don't need to allocate segments from memory and can 
reuse the `MemoryManager` and `MemorySegment` in it.  WDYT?  [~xtsong] THX



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25327) ApplicationMode "DELETE /cluster" REST call leads to exit code 2, instead of 0

2021-12-15 Thread Robert Metzger (Jira) 
Robert Metzger created FLINK-25327:
--

 Summary: ApplicationMode "DELETE /cluster" REST call leads to exit 
code 2, instead of 0
 Key: FLINK-25327
 URL: https://issues.apache.org/jira/browse/FLINK-25327
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Robert Metzger


FLINK-24113 introduced a mode to keep the Application Mode JobManager running 
after the Job has been cancelled. Cluster shutdown needs to be initiated for 
example using the DELETE /cluster REST endpoint.

The problem is that there can be a fatal error during the shutdown, making the 
JobManager exit with return code != 0 (making resource managers believe there 
was an error with the Flink application)

Error 
{code}
2021-12-15 08:09:55,708 ERROR 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Fatal error 
occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: Application failed unexpectedly.
at 
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$finishBootstrapTasks$1(ApplicationDispatcherBootstrap.java:177)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) 
~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278) 
~[?:1.8.0_312]
at 
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:125)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:284)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.util.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:696)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
 ~[?:1.8.0_312]
at 
org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
 ~[?:1.8.0_312]
at 
java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
 ~[?:1.8.0_312]
at 
org.apache.flink.util.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:693)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.util.concurrent.FutureUtils.runAfterwards(FutureUtils.java:660)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:281) 
~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
 ~[flink-dist-1.15-master-robert.jar:1.15-master-robert]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.lambda$terminate$0(AkkaRpcActor.java:580)
 ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
 ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:579)
 ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:191)
 ~[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) 
[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) 
[flink-rpc-akka_44e0316d-9cf7-4fc8-9b48-4f6084b0cc47.jar:1.15-master-robert]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

2021-12-15 Thread Till Rohrmann 
This is true. But this is not a new problem and I think that Flink should
be susceptible to this problem already. One solution for this concrete case
could be that the BlobServer stores some checksums and validates the file
before serving it to the TM.

Cheers,
Till

On Wed, Dec 15, 2021 at 11:59 AM Chesnay Schepler 
wrote:

> The issue with corrupted files is that some of them aren't read by the
> component that stores them.
> For example, a file can be corrupted in the blob server of the JM, but
> that it is corrupted will only be noticed by the TaskExecutor.
>
> On 15/12/2021 11:36, Till Rohrmann wrote:
> > Thanks everyone for your feedback. Let me try to address it by grouping
> > some of the individual comments:
> >
> > ### Will this feature work for native Yarn and K8s deployments?
> >
> > The working directory is an optional feature that can be used to recover
> > additional information. You can think of it like a cache. If the working
> > directory is there, then Flink can do certain things a bit faster but in
> > the worst case it will have to retrieve the required information from the
> > JobManager or persistent storage.
> >
> > In order to make it work with native Yarn and K8s, we would have to
> change
> > these modes slightly. First of all, we would have to be able to map
> working
> > directories to processes and then set a deterministic resource ids for
> the
> > processes. For K8s this could be easily achievable by using a StatefulSet
> > as the deployment mechanism for TaskExecutors. For Yarn, we probably
> would
> > have to remember the prior locations of a process. Both things are
> > potential follow ups that I don't want to tackle in this FLIP.
> >
> > If one of the modes configures the working directory to be on a full or
> > broken disk, then the process will fail. I think this is not all that
> > different from the current state where some things in Flink will fail if
> > they picked the wrong/full temporary directory (e.g. blob storage
> > directory).
> >
> > ### Cleanup
> >
> > The working directory will be cleaned up if the Flink process is
> gracefully
> > shut down. This means that the JobManager process will clean it up if it
> > runs in application mode and the job is terminated. SIGTERM and SIGKILL
> > signals will be treated as an ungraceful shutdown and therefore they
> won't
> > clean up the working directory. This means that we probably also need a
> > graceful way for shutting TaskManager processes down in the future
> because
> > right now they are in most cases killed in order to shut them down. If
> the
> > user uses the tmp directory, then any left-over working directories will
> be
> > cleaned up with the next system restart. This is somewhat similar to how
> > RocksDB's working directory is currently cleaned up as well.
> >
> > ### Corrupted files
> >
> > The working directory itself won't give you any guarantees. It will be
> the
> > responsibility of the component that uses the working directory to make
> > sure that it can deal with corrupted files. E.g. if the component cannot
> > read the file, then it should delete it and fall back to the remote
> > storage/ground truth to retrieve the required information.
> >
> > I hope this could answer your questions. Let me know if you have more
> > feedback.
> >
> > Cheers,
> > Till
> >
> > On Mon, Dec 13, 2021 at 5:05 AM 刘建刚  wrote:
> >
> >> I like the idea. It can reuse the disk to do many things. Isn't it only
> >> for inner failover? If not, the cleaning may be a problem. Also, many
> >> resource components have their own disk schedule strategy.
> >>
> >> Chesnay Schepler  于2021年12月12日周日 19:59写道：
> >>
> >>> How do you intend to handle corrupted files, in particular due to
> >>> process crashes during a write?
> >>> Will all writes to a cached directory append some suffix (e.g.,
> >>> ".pending") and do a rename?
> >>>
> >>> On 10/12/2021 17:54, Till Rohrmann wrote:
>  Hi everyone,
> 
>  I would like to start a discussion about introducing an explicit
> working
>  directory for Flink processes that can be used to store information
> [1].
>  Per default this working directory will reside in the temporary
> >>> directory
>  of the node Flink runs on. However, if configured to reside on a
> >>> persistent
>  volume, then this information can be used to recover from process/node
>  failures. Moreover, such a working directory can be used to
> consolidate
>  some of our other directories Flink creates under /tmp (e.g.
> >>> blobStorage,
>  RocksDB working directory).
> 
>  Here is a draft PR that outlines the required changes [2].
> 
>  Looking forward to your feedback.
> 
>  [1] https://cwiki.apache.org/confluence/x/ZZiqCw
>  [2] https://github.com/apache/flink/pull/18083
> 
>  Cheers,
>  Till
> 
> >>>
>
>

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

2021-12-15 Thread Chesnay Schepler 
The issue with corrupted files is that some of them aren't read by the 
component that stores them.
For example, a file can be corrupted in the blob server of the JM, but 
that it is corrupted will only be noticed by the TaskExecutor.


On 15/12/2021 11:36, Till Rohrmann wrote:

Thanks everyone for your feedback. Let me try to address it by grouping
some of the individual comments:

### Will this feature work for native Yarn and K8s deployments?

The working directory is an optional feature that can be used to recover
additional information. You can think of it like a cache. If the working
directory is there, then Flink can do certain things a bit faster but in
the worst case it will have to retrieve the required information from the
JobManager or persistent storage.

In order to make it work with native Yarn and K8s, we would have to change
these modes slightly. First of all, we would have to be able to map working
directories to processes and then set a deterministic resource ids for the
processes. For K8s this could be easily achievable by using a StatefulSet
as the deployment mechanism for TaskExecutors. For Yarn, we probably would
have to remember the prior locations of a process. Both things are
potential follow ups that I don't want to tackle in this FLIP.

If one of the modes configures the working directory to be on a full or
broken disk, then the process will fail. I think this is not all that
different from the current state where some things in Flink will fail if
they picked the wrong/full temporary directory (e.g. blob storage
directory).

### Cleanup

The working directory will be cleaned up if the Flink process is gracefully
shut down. This means that the JobManager process will clean it up if it
runs in application mode and the job is terminated. SIGTERM and SIGKILL
signals will be treated as an ungraceful shutdown and therefore they won't
clean up the working directory. This means that we probably also need a
graceful way for shutting TaskManager processes down in the future because
right now they are in most cases killed in order to shut them down. If the
user uses the tmp directory, then any left-over working directories will be
cleaned up with the next system restart. This is somewhat similar to how
RocksDB's working directory is currently cleaned up as well.

### Corrupted files

The working directory itself won't give you any guarantees. It will be the
responsibility of the component that uses the working directory to make
sure that it can deal with corrupted files. E.g. if the component cannot
read the file, then it should delete it and fall back to the remote
storage/ground truth to retrieve the required information.

I hope this could answer your questions. Let me know if you have more
feedback.

Cheers,
Till

On Mon, Dec 13, 2021 at 5:05 AM 刘建刚  wrote:


I like the idea. It can reuse the disk to do many things. Isn't it only
for inner failover? If not, the cleaning may be a problem. Also, many
resource components have their own disk schedule strategy.

Chesnay Schepler  于2021年12月12日周日 19:59写道：


How do you intend to handle corrupted files, in particular due to
process crashes during a write?
Will all writes to a cached directory append some suffix (e.g.,
".pending") and do a rename?

On 10/12/2021 17:54, Till Rohrmann wrote:

Hi everyone,

I would like to start a discussion about introducing an explicit working
directory for Flink processes that can be used to store information [1].
Per default this working directory will reside in the temporary

directory

of the node Flink runs on. However, if configured to reside on a

persistent

volume, then this information can be used to recover from process/node
failures. Moreover, such a working directory can be used to consolidate
some of our other directories Flink creates under /tmp (e.g.

blobStorage,

RocksDB working directory).

Here is a draft PR that outlines the required changes [2].

Looking forward to your feedback.

[1] https://cwiki.apache.org/confluence/x/ZZiqCw
[2] https://github.com/apache/flink/pull/18083

Cheers,
Till

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

2021-12-15 Thread Till Rohrmann 
Thanks everyone for your feedback. Let me try to address it by grouping
some of the individual comments:

### Will this feature work for native Yarn and K8s deployments?

The working directory is an optional feature that can be used to recover
additional information. You can think of it like a cache. If the working
directory is there, then Flink can do certain things a bit faster but in
the worst case it will have to retrieve the required information from the
JobManager or persistent storage.

In order to make it work with native Yarn and K8s, we would have to change
these modes slightly. First of all, we would have to be able to map working
directories to processes and then set a deterministic resource ids for the
processes. For K8s this could be easily achievable by using a StatefulSet
as the deployment mechanism for TaskExecutors. For Yarn, we probably would
have to remember the prior locations of a process. Both things are
potential follow ups that I don't want to tackle in this FLIP.

If one of the modes configures the working directory to be on a full or
broken disk, then the process will fail. I think this is not all that
different from the current state where some things in Flink will fail if
they picked the wrong/full temporary directory (e.g. blob storage
directory).

### Cleanup

The working directory will be cleaned up if the Flink process is gracefully
shut down. This means that the JobManager process will clean it up if it
runs in application mode and the job is terminated. SIGTERM and SIGKILL
signals will be treated as an ungraceful shutdown and therefore they won't
clean up the working directory. This means that we probably also need a
graceful way for shutting TaskManager processes down in the future because
right now they are in most cases killed in order to shut them down. If the
user uses the tmp directory, then any left-over working directories will be
cleaned up with the next system restart. This is somewhat similar to how
RocksDB's working directory is currently cleaned up as well.

### Corrupted files

The working directory itself won't give you any guarantees. It will be the
responsibility of the component that uses the working directory to make
sure that it can deal with corrupted files. E.g. if the component cannot
read the file, then it should delete it and fall back to the remote
storage/ground truth to retrieve the required information.

I hope this could answer your questions. Let me know if you have more
feedback.

Cheers,
Till

On Mon, Dec 13, 2021 at 5:05 AM 刘建刚  wrote:

> I like the idea. It can reuse the disk to do many things. Isn't it only
> for inner failover? If not, the cleaning may be a problem. Also, many
> resource components have their own disk schedule strategy.
>
> Chesnay Schepler  于2021年12月12日周日 19:59写道：
>
>> How do you intend to handle corrupted files, in particular due to
>> process crashes during a write?
>> Will all writes to a cached directory append some suffix (e.g.,
>> ".pending") and do a rename?
>>
>> On 10/12/2021 17:54, Till Rohrmann wrote:
>> > Hi everyone,
>> >
>> > I would like to start a discussion about introducing an explicit working
>> > directory for Flink processes that can be used to store information [1].
>> > Per default this working directory will reside in the temporary
>> directory
>> > of the node Flink runs on. However, if configured to reside on a
>> persistent
>> > volume, then this information can be used to recover from process/node
>> > failures. Moreover, such a working directory can be used to consolidate
>> > some of our other directories Flink creates under /tmp (e.g.
>> blobStorage,
>> > RocksDB working directory).
>> >
>> > Here is a draft PR that outlines the required changes [2].
>> >
>> > Looking forward to your feedback.
>> >
>> > [1] https://cwiki.apache.org/confluence/x/ZZiqCw
>> > [2] https://github.com/apache/flink/pull/18083
>> >
>> > Cheers,
>> > Till
>> >
>>
>>

Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs

2021-12-15 Thread godfrey he 
+1 (binding)

Best,
Godfrey

Ingo Bürk  于2021年12月15日周三 16:19写道：
>
> +1 (binding)
>
> Thanks for driving this much needed feature!
>
> On 14.12.21 17:45, Timo Walther wrote:
> > Hi everyone,
> >
> > I'd like to start a vote on FLIP-190: Support Version Upgrades for Table
> > API & SQL Programs [1] which has been discussed in this thread [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> > or not enough votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/KZBnCw
> > [2] https://lists.apache.org/thread/n8v32j6o3d50mpblxydbz82q1q436ob4
> >
> > Cheers,
> > Timo

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Chesnay Schepler 
I had to monkey patch the release scripts a bit; it's unfortunate that 
the changes ended up in the releases, but I don't think it's a blocker.


On 15/12/2021 11:01, Till Rohrmann wrote:

Thanks for driving these releases Chesnay!

+1

* Verified checksums and signatures
* Reviewed website PR
* Checked that build tags only contain log4j version bump

In the release-1.14.2-rc1 there are some changes to the
create_release_branch.sh script that don't seem necessary. But I think this
does not matter since we won't build upon release-1.14.2-rc1 in the future.

Cheers,
Till

On Wed, Dec 15, 2021 at 10:40 AM Yun Tang  wrote:


+ 1 (non-binding)


   *   Select several files of each release to verify the signature and
sha512sum.
   *   Reviewed the flink-web PR
   *   checked that those 4 release-branchs only contains fix to bump
log4j2 version to 2.15.0 and then to 2.16.0
   *   Checked the JIRA release notes and found that FLINK-25317 [1] was
tagged as 1.14.2 version to resolve by mistake.

[1] https://issues.apache.org/jira/browse/FLINK-25317

Best
Yun Tang

From: Chesnay Schepler 
Sent: Wednesday, December 15, 2021 10:55
To: dev@flink.apache.org 
Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

Hi everyone,

This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading
Log4j to 2.16.0) and were prepared simultaneously by the same person.
(Hence, if something is broken, it likely applies to all releases)

Note: 1.11/1.12 are still missing the Python Mac releases.


Please review and vote on the release candidate #1 for the versions
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source releases and binary convenience releases to
be deployed to dist.apache.org [2], which are signed with the key with
fingerprint C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding announcement
blog post [6].

The vote will be open for at least 24 hours. The minimum vote time has
been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Chesnay

[1]
1.11:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
1.12:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
1.13:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
1.14:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11:
https://repository.apache.org/content/repositories/orgapacheflink-1460
1.12:
https://repository.apache.org/content/repositories/orgapacheflink-1462
1.13:
https://repository.apache.org/content/repositories/orgapacheflink-1459
1.14:
https://repository.apache.org/content/repositories/orgapacheflink-1461
[5]
1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
[6] https://github.com/apache/flink-web/pull/489

[jira] [Created] (FLINK-25326) KafkaUtil.createKafkaContainer log levels are not set correctly

2021-12-15 Thread Alexander Preuss (Jira) 
Alexander Preuss created FLINK-25326:


 Summary: KafkaUtil.createKafkaContainer log levels are not set 
correctly
 Key: FLINK-25326
 URL: https://issues.apache.org/jira/browse/FLINK-25326
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.15.0
Reporter: Alexander Preuss


The internal kafka log levels set in KafkaUtils.createKafkaContainer method are 
wrong due to the order of the log hierarchy. If the test logger is set to e.g. 
'DEBUG' it means that `logger.isErrorEnabled()` already evaluated to true and 
therefore the log level gets set to ERROR instead.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25325) Migration Flink from Junit4 to Junit5

2021-12-15 Thread Jing Ge (Jira) 
Jing Ge created FLINK-25325:
---

 Summary: Migration Flink from Junit4 to Junit5
 Key: FLINK-25325
 URL: https://issues.apache.org/jira/browse/FLINK-25325
 Project: Flink
  Issue Type: New Feature
  Components: Tests
Affects Versions: 1.14.0
Reporter: Jing Ge
 Fix For: 1.15.0


Based on the consensus from the mailing list discussion[1][2], we have been 
starting working on the JUnit4 to JUnit5 migration. 

This is the umbrella ticket which describes the big picture of the migration 
with following steps:
 * AssertJ integration and guideline
 * Test Framework upgrade from JUnit4 to JUnit5
 * JUnit5 migration guideline(document and reference migration)
 * Optimization for issues found while writing new test in JUint5
 * [Long-term]Module based graceful migration of old tests in JUnit4 to JUnit5

 

[1] 

[[DISCUSS]Moving to 
JUnit5|https://lists.apache.org/thread/jsjvc2cqb91pyh47d4p6olk3c1vxqm3w]

[2] [[DISCUSS] Conventions on assertions to use in 
tests|https://lists.apache.org/thread/33t7hz8w873p1bc5msppk65792z08rgt]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25324) Migration Flink from Junit4 to Junit5

2021-12-15 Thread Jing Ge (Jira) 
Jing Ge created FLINK-25324:
---

 Summary: Migration Flink from Junit4 to Junit5
 Key: FLINK-25324
 URL: https://issues.apache.org/jira/browse/FLINK-25324
 Project: Flink
  Issue Type: New Feature
  Components: Tests
Affects Versions: 1.14.0
Reporter: Jing Ge
 Fix For: 1.15.0


Based on the consensus from the mailing list discussion[1][2], we have been 
starting working on the JUnit4 to JUnit5 migration. 

This is the umbrella ticket which describes the big picture of the migration 
with following steps:
 * AssertJ integration and guideline
 * Test Framework upgrade from JUnit4 to JUnit5
 * JUnit5 migration guideline(document and reference migration)
 * Optimization for issues found while writing new test in JUint5
 * [Long-term]Module based graceful migration of old tests in JUnit4 to JUnit5

 

[1] 

[[DISCUSS]Moving to 
JUnit5|https://lists.apache.org/thread/jsjvc2cqb91pyh47d4p6olk3c1vxqm3w]

[2] [[DISCUSS] Conventions on assertions to use in 
tests|https://lists.apache.org/thread/33t7hz8w873p1bc5msppk65792z08rgt]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Till Rohrmann 
Thanks for driving these releases Chesnay!

+1

* Verified checksums and signatures
* Reviewed website PR
* Checked that build tags only contain log4j version bump

In the release-1.14.2-rc1 there are some changes to the
create_release_branch.sh script that don't seem necessary. But I think this
does not matter since we won't build upon release-1.14.2-rc1 in the future.

Cheers,
Till

On Wed, Dec 15, 2021 at 10:40 AM Yun Tang  wrote:

>
> + 1 (non-binding)
>
>
>   *   Select several files of each release to verify the signature and
> sha512sum.
>   *   Reviewed the flink-web PR
>   *   checked that those 4 release-branchs only contains fix to bump
> log4j2 version to 2.15.0 and then to 2.16.0
>   *   Checked the JIRA release notes and found that FLINK-25317 [1] was
> tagged as 1.14.2 version to resolve by mistake.
>
> [1] https://issues.apache.org/jira/browse/FLINK-25317
>
> Best
> Yun Tang
> 
> From: Chesnay Schepler 
> Sent: Wednesday, December 15, 2021 10:55
> To: dev@flink.apache.org 
> Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1
>
> Hi everyone,
>
> This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
> 1.14 to address CVE-2021-44228/CVE-2021-45046.
> It covers all 4 releases as they contain the same changes (upgrading
> Log4j to 2.16.0) and were prepared simultaneously by the same person.
> (Hence, if something is broken, it likely applies to all releases)
>
> Note: 1.11/1.12 are still missing the Python Mac releases.
>
>
> Please review and vote on the release candidate #1 for the versions
> 1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
> [ ] +1, Approve the releases
> [ ] -1, Do not approve the releases (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source releases and binary convenience releases to
> be deployed to dist.apache.org [2], which are signed with the key with
> fingerprint C2EED7B111D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tags [5],
> * website pull request listing the new releases and adding announcement
> blog post [6].
>
> The vote will be open for at least 24 hours. The minimum vote time has
> been shortened as the changes are minimal and the matter is urgent.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Chesnay
>
> [1]
> 1.11:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
> 1.12:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
> 1.13:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
> 1.14:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
> [2]
> 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
> 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
> 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
> 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> 1.11:
> https://repository.apache.org/content/repositories/orgapacheflink-1460
> 1.12:
> https://repository.apache.org/content/repositories/orgapacheflink-1462
> 1.13:
> https://repository.apache.org/content/repositories/orgapacheflink-1459
> 1.14:
> https://repository.apache.org/content/repositories/orgapacheflink-1461
> [5]
> 1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
> 1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
> 1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
> 1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
> [6] https://github.com/apache/flink-web/pull/489
>
>
>

[jira] [Created] (FLINK-25323) Support side output late data for interval join

2021-12-15 Thread chenyuzhi (Jira) 
chenyuzhi created FLINK-25323:
-

 Summary: Support side output late data for interval join
 Key: FLINK-25323
 URL: https://issues.apache.org/jira/browse/FLINK-25323
 Project: Flink
  Issue Type: Improvement
Reporter: chenyuzhi


Now, Flink just discard late data when using interval-join:

[https://github.com/apache/flink/blob/83a2541475228a4ff9e9a9def4049fb742353549/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L231]
 

 

Maybe we could use two outputTags to side output late data for left/right 
stream.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

2021-12-15 Thread Stephan Ewen 
Congratulations, Matthias, and welcome to the Flink committers!

On Mon, Dec 13, 2021 at 4:39 AM 刘建刚  wrote:

> Congratulations!
>
> Best
> Liu Jiangang
>
> Nicholas Jiang  于2021年12月13日周一 11:23写道：
>
> > Congratulations, Matthias!
> >
> > Best,
> > Nicholas Jiang
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

2021-12-15 Thread Stephan Ewen 
Congrats and welcome!

On Mon, Dec 13, 2021 at 4:39 AM 刘建刚  wrote:

> Congratulations!
>
> Best
> Liu Jiangang
>
> Nicholas Jiang  于2021年12月13日周一 11:28写道：
>
> > Congratulations, Ingo!
> >
> > Best,
> > Nicholas Jiang
> >
>

Re: [CANCELLED] Release 1.11.5/1.12.6/1.13.4/1.14.1, release candidate #1

2021-12-15 Thread Stephan Ewen 
Given that these artifacts are published already, users can use them if
they want to update now:

For example:
https://search.maven.org/artifact/org.apache.flink/flink-core/1.14.1/jar

Just for the users that really want to update now (rather than rely on the
mitigation via config) and are not as much concerned about the remaining
weakness in log4j 2.15.0

On Tue, Dec 14, 2021 at 11:18 PM Seth Wiesman  wrote:

> Thank you for managing these updates Chesnay!
>
>
>
> On Tue, Dec 14, 2021 at 3:51 PM Chesnay Schepler 
> wrote:
>
> > Since the maven artifacts have already been published we will use the
> > next patch version for each release, i.e.:
> > 1.11.6
> > 1.12.7
> > 1.13.5
> > 1.14.2
> >
> > (We could technically just update the source/binaries, but that seems
> > fishy).
> >
> > On 14/12/2021 22:38, Chesnay Schepler wrote:
> > > I'm canceling the release because the issue was not fully fixed in
> > > Log4j 2.15.0; see CVE-2021-45046.
> > >
> > > I will start preparing new release candidates that use Log4j 2.16.0 .
> > >
> > > On 14/12/2021 21:28, Chesnay Schepler wrote:
> > >> The vote duration has passed and we have approved the releases.
> > >>
> > >> Binding votes:
> > >> * Stephan
> > >> * Till
> > >> * Xintong
> > >> * Zhu
> > >> * Gordon
> > >>
> > >> I will not finalize the release.
> > >>
> > >> On 13/12/2021 20:28, Chesnay Schepler wrote:
> > >>> Hi everyone,
> > >>>
> > >>> This vote is for the emergency patch releases for 1.11, 1.12, 1.13
> > >>> and 1.14 to address CVE-2021-44228.
> > >>> It covers all 4 releases as they contain the same changes (upgrading
> > >>> Log4j to 2.15.0) and were prepared simultaneously by the same person.
> > >>> (Hence, if something is broken, it likely applies to all releases)
> > >>>
> > >>> Please review and vote on the release candidate #1 for the versions
> > >>> 1.11.5, 1.12.6, 1.13.4 and 1.14.1, as follows:
> > >>> [ ] +1, Approve the releases
> > >>> [ ] -1, Do not approve the releases (please provide specific
> comments)
> > >>>
> > >>> The complete staging area is available for your review, which
> includes:
> > >>> * JIRA release notes [1],
> > >>> * the official Apache source releases and binary convenience
> > >>> releases to be deployed to dist.apache.org [2], which are signed
> > >>> with the key with fingerprint C2EED7B111D464BA [3],
> > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > >>> * *the jars for 1.13/1.14 are still being built*
> > >>> * source code tags [5],
> > >>> * website pull request listing the new releases and adding
> > >>> announcement blog post [6].
> > >>>
> > >>> The vote will be open for at least 24 hours. The minimum vote time
> > >>> has been shortened as the changes are minimal and the matter is
> urgent.
> > >>> It is adopted by majority approval, with at least 3 PMC affirmative
> > >>> votes.
> > >>>
> > >>> Thanks,
> > >>> Chesnay
> > >>>
> > >>> [1]
> > >>> 1.11:
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350327
> > >>> 1.12:
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350328
> > >>> 1.13:
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350686
> > >>> 1.14:
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350512
> > >>> [2]
> > >>> 1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.5-rc1/
> > >>> 1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.6-rc1/
> > >>> 1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.4-rc1/
> > >>> 1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.1-rc1/
> > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >>> [4]
> > >>> 1.11/1.12:
> > >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1455
> > >>> 1.13:
> > >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1457
> > >>> 1.14:
> > >>>
> https://repository.apache.org/content/repositories/orgapacheflink-1456
> > >>> [5]
> > >>> 1.11:
> https://github.com/apache/flink/releases/tag/release-1.11.5-rc1
> > >>> 1.12:
> https://github.com/apache/flink/releases/tag/release-1.12.6-rc1
> > >>> 1.13:
> https://github.com/apache/flink/releases/tag/release-1.13.4-rc1
> > >>> 1.14:
> https://github.com/apache/flink/releases/tag/release-1.14.1-rc1
> > >>> [6] https://github.com/apache/flink-web/pull/489
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

2021-12-15 Thread Yun Tang 

+ 1 (non-binding)


  *   Select several files of each release to verify the signature and 
sha512sum.
  *   Reviewed the flink-web PR
  *   checked that those 4 release-branchs only contains fix to bump log4j2 
version to 2.15.0 and then to 2.16.0
  *   Checked the JIRA release notes and found that FLINK-25317 [1] was tagged 
as 1.14.2 version to resolve by mistake.

[1] https://issues.apache.org/jira/browse/FLINK-25317

Best
Yun Tang

From: Chesnay Schepler 
Sent: Wednesday, December 15, 2021 10:55
To: dev@flink.apache.org 
Subject: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

Hi everyone,

This vote is for the emergency patch releases for 1.11, 1.12, 1.13 and
1.14 to address CVE-2021-44228/CVE-2021-45046.
It covers all 4 releases as they contain the same changes (upgrading
Log4j to 2.16.0) and were prepared simultaneously by the same person.
(Hence, if something is broken, it likely applies to all releases)

Note: 1.11/1.12 are still missing the Python Mac releases.


Please review and vote on the release candidate #1 for the versions
1.11.6, 1.12.7, 1.13.5 and 1.14.2, as follows:
[ ] +1, Approve the releases
[ ] -1, Do not approve the releases (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source releases and binary convenience releases to
be deployed to dist.apache.org [2], which are signed with the key with
fingerprint C2EED7B111D464BA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tags [5],
* website pull request listing the new releases and adding announcement
blog post [6].

The vote will be open for at least 24 hours. The minimum vote time has
been shortened as the changes are minimal and the matter is urgent.
It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Chesnay

[1]
1.11:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351056
1.12:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351057
1.13:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351058
1.14:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351059
[2]
1.11: https://dist.apache.org/repos/dist/dev/flink/flink-1.11.6-rc1/
1.12: https://dist.apache.org/repos/dist/dev/flink/flink-1.12.7-rc1/
1.13: https://dist.apache.org/repos/dist/dev/flink/flink-1.13.5-rc1/
1.14: https://dist.apache.org/repos/dist/dev/flink/flink-1.14.2-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
1.11: https://repository.apache.org/content/repositories/orgapacheflink-1460
1.12: https://repository.apache.org/content/repositories/orgapacheflink-1462
1.13: https://repository.apache.org/content/repositories/orgapacheflink-1459
1.14: https://repository.apache.org/content/repositories/orgapacheflink-1461
[5]
1.11: https://github.com/apache/flink/releases/tag/release-1.11.6-rc1
1.12: https://github.com/apache/flink/releases/tag/release-1.12.7-rc1
1.13: https://github.com/apache/flink/releases/tag/release-1.13.5-rc1
1.14: https://github.com/apache/flink/releases/tag/release-1.14.2-rc1
[6] https://github.com/apache/flink-web/pull/489

[jira] [Created] (FLINK-25322) Support no-claim mode in changelog state backend

2021-12-15 Thread Dawid Wysakowicz (Jira) 
Dawid Wysakowicz created FLINK-25322:


 Summary: Support no-claim mode in changelog state backend
 Key: FLINK-25322
 URL: https://issues.apache.org/jira/browse/FLINK-25322
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Checkpointing, Runtime / State Backends
Reporter: Dawid Wysakowicz
 Fix For: 1.15.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25321) standalone deploy on k8s,pod always OOM killed,actual heap memory usage is normal, gc is normal

2021-12-15 Thread Gao Fei (Jira) 
Gao Fei created FLINK-25321:
---

 Summary: standalone deploy on k8s,pod always OOM killed,actual 
heap memory usage is normal, gc is normal
 Key: FLINK-25321
 URL: https://issues.apache.org/jira/browse/FLINK-25321
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.11.3
 Environment: Flink 1.11.3

k8s v1.21.0

standlone deployment
Reporter: Gao Fei


Start a cluster on k8s, deploy in standalone mode, a jobmanager pod (1G) and a 
taskmanager pod (3372MB limit), the total memory configuration of the Flink TM 
process is 3072MB, and the managed configuration is 0, both of which are on the 
heap memory. Now the pod It will always be OOM killed, and the total process 
memory will always exceed 3072MB. I saw that the system has adopted jemlloc. 
There is no 64M problem. The application itself has not applied for direct 
memory. It is strange why the process is always killed by OOM after a period of 
time.

 

INFO  [] - Final TaskExecutor Memory configuration:
INFO  [] -   Total Process Memory:          3.000gb (3221225472 bytes)
INFO  [] -     Total Flink Memory:          2.450gb (2630667464 bytes)
INFO  [] -       Total JVM Heap Memory:     2.080gb (2233382986 bytes)
INFO  [] -         Framework:               128.000mb (134217728 bytes)
INFO  [] -         Task:                    1.955gb (2099165258 bytes)
INFO  [] -       Total Off-heap Memory:     378.880mb (397284478 bytes)
INFO  [] -         Managed:                 0 bytes
INFO  [] -         Total JVM Direct Memory: 378.880mb (397284478 bytes)
INFO  [] -           Framework:             128.000mb (134217728 bytes)
INFO  [] -           Task:                  0 bytes
INFO  [] -           Network:               250.880mb (263066750 bytes)
INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
INFO  [] -     JVM Overhead:                307.200mb (322122552 bytes)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-25320) The flink history web page shows the number of running jobs instead of just showing the number of completed jobs

2021-12-15 Thread Zhuang Liu (Jira) 
Zhuang Liu  created FLINK-25320:
---

 Summary: The flink history web page shows the number of running 
jobs instead of just showing the number of completed jobs
 Key: FLINK-25320
 URL: https://issues.apache.org/jira/browse/FLINK-25320
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Web Frontend
Affects Versions: shaded-14.0
Reporter: Zhuang Liu 






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs

2021-12-15 Thread Ingo Bürk 

+1 (binding)

Thanks for driving this much needed feature!

On 14.12.21 17:45, Timo Walther wrote:

Hi everyone,

I'd like to start a vote on FLIP-190: Support Version Upgrades for Table 
API & SQL Programs [1] which has been discussed in this thread [2].


The vote will be open for at least 72 hours unless there is an objection 
or not enough votes.


[1] https://cwiki.apache.org/confluence/x/KZBnCw
[2] https://lists.apache.org/thread/n8v32j6o3d50mpblxydbz82q1q436ob4

Cheers,
Timo

[jira] [Created] (FLINK-25319) Quickstarts Scala nightly end-to-end test failed due to akka rpc server failed to start

2021-12-15 Thread Yun Gao (Jira) 
Yun Gao created FLINK-25319:
---

 Summary: Quickstarts Scala nightly end-to-end test failed due to 
akka rpc server failed to start
 Key: FLINK-25319
 URL: https://issues.apache.org/jira/browse/FLINK-25319
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Yun Gao



{code:java}

Dec 14 17:36:15 at 
akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) 
~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 
~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 
~[?:1.8.0_292]
Dec 14 17:36:15 Caused by: java.lang.NullPointerException
Dec 14 17:36:15 at 
org.apache.flink.runtime.rpc.RpcEndpoint.getAddress(RpcEndpoint.java:283) 
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:177)
 ~[?:?]
Dec 14 17:36:15 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?]
Dec 14 17:36:15 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?]
Dec 14 17:36:15 at 
scala.PartialFunction.applyOrElse(PartialFunction.scala:123) 
~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at 
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) 
~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at 
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?]
Dec 14 17:36:15 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) 
~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) 
~[flink-scala_2.12-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
Dec 14 17:36:15 at akka.actor.Actor.aroundReceive(Actor.scala:537) 
~[?:?]
Dec 14 17:36:15 at akka.actor.Actor.aroundReceive$(Actor.scala:535) 
~[?:?]
Dec 14 17:36:15 at 
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?]
Dec 14 17:36:15 at 
akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) ~[?:?]
Dec 14 17:36:15 at akka.actor.ActorCell.invoke(ActorCell.scala:548) 
~[?:?]
Dec 14 17:36:15 at 
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[?:?]
Dec 14 17:36:15 at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[?:?]
Dec 14 17:36:15 at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) 
~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) 
~[?:1.8.0_292]
Dec 14 17:36:15 at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 
~[?:1.8.0_292]
Dec 14 17:36:15 2021-12-14 17:35:43,572 INFO  
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting 
StandaloneSessionClusterEntrypoint down with application status UNKNOWN. 
Diagnostics Cluster entrypoint has been closed externally..
Dec 14 17:36:15 2021-12-14 17:35:43,582 INFO  
org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB 
server at 0.0.0.0:42449
Dec 14 17:36:15 2021-12-14 17:35:43,584 INFO  
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint   [] - Shutting down 
rest endpoint.
Dec 14 17:36:15 2021-12-14 17:35:40,722 INFO  
org.apache.flink.runtime.taskexecutor.TaskManagerRunner  [] - 

Dec 14 17:36:15 2021-12-14 17:35:40,724 INFO  
org.apache.flink.runtime.taskexecutor.TaskManagerRunner  [] -  
Preconfiguration: 
Dec 14 17:36:15 2021-12-14 17:35:40,724 INFO  
org.apache.flink.runtime.taskexecutor.TaskManagerRunner  [] - 

{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=28117=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=3955



--
This message was sent by Atlassian Jira
(v8.20.1#820001)