[jira] [Created] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
Yang Wang created FLINK-17935:
-
Summary: Logs could not show up when deploying Flink on Yarn via
"--executor"
Key: FLINK-17935
URL: https://issues.apache.org/jira/browse/FLINK-17935
Project: Flink
Issue Type: Bug
Affects Versions: 1.11.0, 1.12.0
Reporter: Yang Wang
Fix For: 1.11.0
{code:java}
./bin/flink run -d -p 5 -e yarn-per-job examples/streaming/WindowJoin.jar{code}
When we use the {{-e/--executor}} to specify the deploy target to Yarn per-job,
the logs could not show up. The root cause is we do not set the logging files
in {{ExecutorCLI}}. We only do it in the {{FlinkYarnSessionCli}}.
If we use {{-m yarn-cluster}}, everything works well.
Maybe we should move the {{setLogConfigFileInConfig}} to
{{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17934) StreamingFileWriter should set chainingStrategy
Jingsong Lee created FLINK-17934:
Summary: StreamingFileWriter should set chainingStrategy
Key: FLINK-17934
URL: https://issues.apache.org/jira/browse/FLINK-17934
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Reporter: Jingsong Lee
Fix For: 1.11.0
We should let {{StreamingFileWriter}} should be eagerly chained whenever
possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Releasing Stateful Functions 2.1.0 soon?
Nice work, thanks for pushing this, Gordon! +1 also from my side for a quick release. I think it already warrants a release to have the 1.10.1 upgrade and the fix to not fail on savepoints that are triggered concurrently to a checkpoint. Even nicer that there are two cool new features included. On Fri, May 22, 2020 at 7:13 AM Tzu-Li (Gordon) Tai wrote: > Thanks for the positive feedback so far. > > Lets then set the feature freeze date for Stateful Functions 2.1.0 to be > next Wednesday (May 27th). > > We've made good progress over the past days, all mentioned features merged > besides the following: > - https://issues.apache.org/jira/browse/FLINK-17875 State TTL support for > remote functions > > Will keep track of that and hopefully cut the feature branch as scheduled. > > Cheers, > Gordon > > On Thu, May 21, 2020 at 7:22 PM Yuan Mei wrote: > > > faster iteration definitely helps early-stage projects. > > > > +1 > > > > Best, > > Yuan > > > > > > On Thu, May 21, 2020 at 4:14 PM Congxian Qiu > > wrote: > > > > > +1 from my side to have smaller and more frequent feature releases for > > the > > > project in its early phases. > > > > > > Best, > > > Congxian > > > > > > > > > Marta Paes Moreira 于2020年5月21日周四 下午12:49写道： > > > > > > > +1 for more frequent releases with a shorter (but feedback-informed) > > > > feature set. > > > > > > > > Thanks, Gordon (and Igal)! > > > > > > > > Marta > > > > > > > > On Thu, 21 May 2020 at 03:44, Yu Li wrote: > > > > > > > > > +1, it makes a lot of sense for stateful functions to evolve > faster. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Wed, 20 May 2020 at 23:36, Zhijiang > > > > .invalid> > > > > > wrote: > > > > > > > > > > > I also like this idea, considering stateful functions flexible > > enough > > > > to > > > > > > have a faster release cycle. +1 from my side. > > > > > > > > > > > > Best, > > > > > > Zhijiang > > > > > > > > > > > > > > > > > > > -- > > > > > > From:Seth Wiesman > > > > > > Send Time:2020年5月20日(星期三) 21:45 > > > > > > To:dev > > > > > > Subject:Re: [DISCUSS] Releasing Stateful Functions 2.1.0 soon? > > > > > > > > > > > > +1 for a fast release cycle > > > > > > > > > > > > Seth > > > > > > > > > > > > On Wed, May 20, 2020 at 8:43 AM Robert Metzger < > > rmetz...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > I like the idea of releasing Statefun more frequently to have > > > faster > > > > > > > feedback cycles! > > > > > > > > > > > > > > No objections for releasing 2.1.0 from my side. > > > > > > > > > > > > > > On Wed, May 20, 2020 at 2:22 PM Tzu-Li (Gordon) Tai < > > > > > tzuli...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > Since Stateful Functions 2.0 was released early April, > > > > > > > > we've been getting some good feedback from various channels, > > > > > > > > including the Flink mailing lists, JIRA issues, as well as > > Stack > > > > > > Overflow > > > > > > > > questions. > > > > > > > > > > > > > > > > Some of the discussions have actually translated into new > > > features > > > > > > > > currently being implemented into the project, such as: > > > > > > > > > > > > > > > >- State TTL for the state primitives in Stateful Functions > > > (for > > > > > both > > > > > > > >embedded/remote functions) > > > > > > > >- Transport for remote functions via UNIX domain sockets, > > > which > > > > > > would > > > > > > > be > > > > > > > >useful when remote functions are co-located with Flink > > > StateFun > > > > > > > workers > > > > > > > >(i.e. the "sidecar" deployment mode) > > > > > > > > > > > > > > > > > > > > > > > > Besides that, some critical shortcomings have already been > > > > addressed > > > > > > > since > > > > > > > > the last release: > > > > > > > > > > > > > > > >- After upgrading to Flink 1.10.1, failure recovery in > > > Stateful > > > > > > > >Functions now works properly with the new scheduler. > > > > > > > >- Support for concurrent checkpoints > > > > > > > > > > > > > > > > > > > > > > > > With these ongoing threads, while it's only been just short > of > > 2 > > > > > months > > > > > > > > since the last release, > > > > > > > > we (Igal Shilman and I) have been thinking about aiming to > > > already > > > > > > start > > > > > > > > the next feature release (2.1.0) soon. > > > > > > > > This is relatively shorter than the release cycle of what the > > > > > community > > > > > > > is > > > > > > > > used to in Flink (usually 3 months at least), > > > > > > > > but we think with the StateFun project in its early phases, > > > having > > > > > > > smaller > > > > > > > > and more frequent feature releases could potentially help > drive > > > > user > > > > > > > > adoption. > > > > > > > > > > > > > > > > So, what do you think about setting feature freeze for > StateFun > > > >
[jira] [Created] (FLINK-17933) TaskManager was terminated on Yarn - investigate
Roman Khachatryan created FLINK-17933: - Summary: TaskManager was terminated on Yarn - investigate Key: FLINK-17933 URL: https://issues.apache.org/jira/browse/FLINK-17933 Project: Flink Issue Type: Task Components: Deployment / YARN, Runtime / Task Affects Versions: 1.11.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan -- This message was sent by Atlassian Jira (v8.3.4#803005)
[VOTE] Release flink-shaded 11.0, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the version 11.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 11D464BA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-11.0-rc1" [5], * website pull request listing the new release [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Chesnay [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347784 [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-11.0-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1372/ [5] https://github.com/apache/flink-shaded/tree/release-11.0-rc1 [6] https://github.com/apache/flink-web/pull/340
[jira] [Created] (FLINK-17932) All hbase config Options can not work in HBaseTableSource
Leonard Xu created FLINK-17932: -- Summary: All hbase config Options can not work in HBaseTableSource Key: FLINK-17932 URL: https://issues.apache.org/jira/browse/FLINK-17932 Project: Flink Issue Type: Bug Components: Connectors / HBase Affects Versions: 1.10.0, 1.11.0, 1.12.0 Reporter: Leonard Xu All hbase config Options can not work in HBaseTableSource, because field `conf` in HBaseRowInputFormat it not serializable, and HBaseTableSource will fallback to load hbase-site.xml from classpath. I.E, many table config Options that user defined in DDL is useless and HbaseTableSource only load config options from classpath. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17931) Document fromValues
Dawid Wysakowicz created FLINK-17931: Summary: Document fromValues Key: FLINK-17931 URL: https://issues.apache.org/jira/browse/FLINK-17931 Project: Flink Issue Type: Sub-task Components: Documentation, Table SQL / API Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Semantics of our JIRA fields
Hi all, thanks a lot for the feedback. The majority of responses are very positive to my proposal. I have put my proposal into our wiki: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=154995514 Regarding the comments so far: @Jark: I clarified this in the wiki. @Israel: I have not considered build changing all 3000 resolved tickets to closed yet, but after consideration I don't think it is necessary. If others in the community would like to change them, please speak up in this thread :) @Flavio: I agree that we can not ask new or infrequent users to fully adhere to these definitions. I added a note in the Wiki. Using the resolved state for indicating "PR available" is problematic because there are plenty of cases where PRs are stale (and this ticket would then appear as a "resolved"). The Apache tools are adding a link to the PR, and some contributors are setting the ticket to "In Progress". I don't see a problem that we need to solve here. @Yun: Thank you for your comment. I added an example clarifying how I would handle such a case. It is slightly different from your proposal: You suggested using x.y.0 versions, I used "the next supported, unreleased version", because that's how we've done it so far (and I don't want to change things, I just want to document how the majority of the core contributors are using JIRA). Here are all the changes (in green, blue are just formatting changes) I made compared to my initial proposal: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=154995514=4=1 On Mon, May 25, 2020 at 2:28 PM Congxian Qiu wrote: > @ches...@apache.org Thanks for the confirmation > > Best, > Congxian > > > Zhu Zhu 于2020年5月25日周一 下午4:13写道： > > > This is very helpful! > > +1 > > > > Thanks, > > Zhu Zhu > > > > Yang Wang 于2020年5月25日周一 下午4:04写道： > > > > > +1 from this useful proposal. > > > > > > This makes me clearer about "Resolve" and "Close" since I used to be > > > confused by this two button. > > > > > > Best, > > > Yang > > > > > > Jingsong Li 于2020年5月25日周一 下午3:10写道： > > > > > > > +1 for the proposal. > > > > It makes me clearer. > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Mon, May 25, 2020 at 2:51 PM Zhijiang > > > .invalid> > > > > wrote: > > > > > > > > > Thanks for launching this discussion and giving so detailed infos, > > > > > Robert! +1 on my side for the proposal. > > > > > > > > > > For "Affects Version", I previously thought it was only for the > > already > > > > > released versions, so it can give a reminder that the fix should > also > > > > pick > > > > > into the related released branches for future minor versions. > > > > > I saw that Jark had somehow similar concerns for this field in > below > > > > > replies. Either way makes sense for me as long as we give a > > determined > > > > > rule in Wiki. > > > > > > > > > > Re Flavio' s comments, I agree that the Jira reporter can leave > most > > of > > > > > the fields empty if not confirmed of them, then the respective > > > component > > > > > maintainer or committer can update them accordingly later. > > > > > But the state of Jira should not be marked as "resolved" when the > PR > > is > > > > > detected, that is not fitting into the resolved semantic I guess. > If > > > > > possible, the Jira can be updated as "in progress" automatically if > > > > > the respective PR is ready, then it will save some time to stat > > > progress > > > > > of related issues during release process. > > > > > > > > > > Best, > > > > > Zhijiang > > > > > -- > > > > > From:Congxian Qiu > > > > > Send Time:2020年5月25日(星期一) 13:57 > > > > > To:dev@flink.apache.org > > > > > Subject:Re: [DISCUSS] Semantics of our JIRA fields > > > > > > > > > > Hi > > > > > > > > > > Currently, when I'm going to create an issue for Project-website. > I'm > > > not > > > > > very sure what the "Affect Version/s" should be set. The problem is > > > that > > > > > the current Dockfile[1] in flink-web is broken because of the EOL > of > > > > Ubuntu > > > > > 18.10[2], the project-web does not affect any release of Flink, it > > does > > > > > affect the process to build website, so what's the version should I > > set > > > > to? > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > > > > > [2] https://wiki.ubuntu.com/Releases > > > > > > > > > > Best, > > > > > Congxian > > > > > > > > > > > > > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > > > > > > > > > > > In my experience it's quite complicated for a normal reporter to > be > > > > able > > > > > to > > > > > > fill all the fields correctly (especially for new users). > > > > > > Usually you just wanto to report a problem, remember to add a new > > > > feature > > > > > > or improve code/documentation but you can't really give a > priority, > > > > >
[jira] [Created] (FLINK-17928) Incorrect state size reported when using unaligned checkpoints
Piotr Nowojski created FLINK-17928:
--
Summary: Incorrect state size reported when using unaligned
checkpoints
Key: FLINK-17928
URL: https://issues.apache.org/jira/browse/FLINK-17928
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.11.0
Reporter: Piotr Nowojski
Fix For: 1.11.0
Even when checkpoints on HDFS are between 100-300MBs, the reported state size
is in orders of magnitude larger with values like:
{noformat}
1GiB 1.5TiB 2.0TiB 2.1TiB 2.1TiB
148GiB 148GiB 148GiB 148GiB 148GiB 148GiB
{noformat}
it's probably because we have multiple {{Collection}},
and each of the individual handle returns the same value from
{{AbstractChannelStateHandle#getStateSize}} - the full size of the spilled
data, ignoring that only small portion of those data belong to a single input
channel/result subpartition. In other words {{
org.apache.flink.runtime.state.AbstractChannelStateHandle#getStateSize}} should
be taking the offsets into account and return only the size of the data that
belong exclusively to this handle.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17929) Fix invalid liquid expressions
Dawid Wysakowicz created FLINK-17929:
Summary: Fix invalid liquid expressions
Key: FLINK-17929
URL: https://issues.apache.org/jira/browse/FLINK-17929
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 1.11.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
The .ID expression in ops/deployment/docker.md should be escaped,
otherwise it is not properly rendered.
{code}
Generating...
Liquid Warning: Liquid syntax error (line 331): [:dot, "."] is not a valid
expression in "{{.ID}}" in ops/deployment/docker.md
Liquid Warning: Liquid syntax error (line 357): [:dot, "."] is not a valid
expression in "{{.ID}}" in ops/deployment/docker.md
Liquid Warning: Liquid syntax error (line 331): [:dot, "."] is not a valid
expression in "{{.ID}}" in ops/deployment/docker.zh.md
Liquid Warning: Liquid syntax error (line 357): [:dot, "."] is not a valid
expression in "{{.ID}}" in ops/deployment/docker.zh.md
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17927) Default value of "sink.partition-commit.trigger" should be "process-time"
Jingsong Lee created FLINK-17927: Summary: Default value of "sink.partition-commit.trigger" should be "process-time" Key: FLINK-17927 URL: https://issues.apache.org/jira/browse/FLINK-17927 Project: Flink Issue Type: Bug Components: Connectors / FileSystem Reporter: Jingsong Lee Fix For: 1.11.0 Users are hard to figure out what is wrong when they don't have watermark. We can set "sink.partition-commit.trigger" to "process-time" to have better out-of-box experience. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17926) Can't build flink-web docker image because of EOL of Ubuntu:18.10
Congxian Qiu(klion26) created FLINK-17926:
-
Summary: Can't build flink-web docker image because of EOL of
Ubuntu:18.10
Key: FLINK-17926
URL: https://issues.apache.org/jira/browse/FLINK-17926
Project: Flink
Issue Type: Bug
Components: Project Website
Reporter: Congxian Qiu(klion26)
Currently, the Dockerfile[1] in flink-web project is broken because of the EOL
of Ubuntu 18.10[2], will encounter the error such as bellow when executing
{{./run.sh}}
{code:java}
Err:3 http://security.ubuntu.com/ubuntu cosmic-security Release
404 Not Found [IP: 91.189.88.152 80]
Ign:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Ign:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Err:6 http://archive.ubuntu.com/ubuntu cosmic Release
404 Not Found [IP: 91.189.88.142 80]
Err:7 http://archive.ubuntu.com/ubuntu cosmic-updates Release
404 Not Found [IP: 91.189.88.142 80]
Err:8 http://archive.ubuntu.com/ubuntu cosmic-backports Release
404 Not Found [IP: 91.189.88.142 80]
Reading package lists...
{code}
The current LTS versions can be found in release website[2].
Apache Flink docker image uses fedora:28[3], so it unaffected.
As fedora does not have LTS release[4], I proposal to use Ubuntu for website
here, and change the version from {{18.10}} to the closest LTS version {{18.04,
tried locally, it works successfully.}}
[1]
[https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17]
[2] [https://wiki.ubuntu.com/Releases]
[3]https://github.com/apache/flink/blob/e92b2bf19bdf03ad3bae906dc5fa3781aeddb3ee/docs/docker/Dockerfile#L17
[4]
https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Semantics of our JIRA fields
@ches...@apache.org Thanks for the confirmation Best, Congxian Zhu Zhu 于2020年5月25日周一 下午4:13写道： > This is very helpful! > +1 > > Thanks, > Zhu Zhu > > Yang Wang 于2020年5月25日周一 下午4:04写道： > > > +1 from this useful proposal. > > > > This makes me clearer about "Resolve" and "Close" since I used to be > > confused by this two button. > > > > Best, > > Yang > > > > Jingsong Li 于2020年5月25日周一 下午3:10写道： > > > > > +1 for the proposal. > > > It makes me clearer. > > > > > > Best, > > > Jingsong Lee > > > > > > On Mon, May 25, 2020 at 2:51 PM Zhijiang > > .invalid> > > > wrote: > > > > > > > Thanks for launching this discussion and giving so detailed infos, > > > > Robert! +1 on my side for the proposal. > > > > > > > > For "Affects Version", I previously thought it was only for the > already > > > > released versions, so it can give a reminder that the fix should also > > > pick > > > > into the related released branches for future minor versions. > > > > I saw that Jark had somehow similar concerns for this field in below > > > > replies. Either way makes sense for me as long as we give a > determined > > > > rule in Wiki. > > > > > > > > Re Flavio' s comments, I agree that the Jira reporter can leave most > of > > > > the fields empty if not confirmed of them, then the respective > > component > > > > maintainer or committer can update them accordingly later. > > > > But the state of Jira should not be marked as "resolved" when the PR > is > > > > detected, that is not fitting into the resolved semantic I guess. If > > > > possible, the Jira can be updated as "in progress" automatically if > > > > the respective PR is ready, then it will save some time to stat > > progress > > > > of related issues during release process. > > > > > > > > Best, > > > > Zhijiang > > > > -- > > > > From:Congxian Qiu > > > > Send Time:2020年5月25日(星期一) 13:57 > > > > To:dev@flink.apache.org > > > > Subject:Re: [DISCUSS] Semantics of our JIRA fields > > > > > > > > Hi > > > > > > > > Currently, when I'm going to create an issue for Project-website. I'm > > not > > > > very sure what the "Affect Version/s" should be set. The problem is > > that > > > > the current Dockfile[1] in flink-web is broken because of the EOL of > > > Ubuntu > > > > 18.10[2], the project-web does not affect any release of Flink, it > does > > > > affect the process to build website, so what's the version should I > set > > > to? > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > > > > [2] https://wiki.ubuntu.com/Releases > > > > > > > > Best, > > > > Congxian > > > > > > > > > > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > > > > > > > > > In my experience it's quite complicated for a normal reporter to be > > > able > > > > to > > > > > fill all the fields correctly (especially for new users). > > > > > Usually you just wanto to report a problem, remember to add a new > > > feature > > > > > or improve code/documentation but you can't really give a priority, > > > > assign > > > > > the correct label or decide which releases will benefit of the > > fix/new > > > > > feature. This is something that only core developers could decide > > > (IMHO). > > > > > > > > > > My experiece says that it's better if normal users could just open > > > > tickets > > > > > with some default (or mark ticket as new) and leave them in such a > > > state > > > > > until an experienced user, one that can assign tickets, have the > time > > > to > > > > > read it and immediately reject the ticket or accept it and properly > > > > assign > > > > > priorities, fix version, etc. > > > > > > > > > > With respect to resolve/close I think that a good practice could be > > to > > > > mark > > > > > automatically a ticket as resolved once that a PR is detected for > > that > > > > > ticket, while marking it as closed should be done by the commiter > who > > > > merge > > > > > the PR. > > > > > > > > > > Probably this process would slightly increase the work of a > committer > > > but > > > > > this would make things a lot cleaner and will bring the benefit of > > > > having a > > > > > reliable and semantically sound JIRA state. > > > > > > > > > > Cheers, > > > > > Flavio > > > > > > > > > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha > > > scritto: > > > > > > > > > > > +1 for the proposal > > > > > > > > > > > > This will bring some consistency to the process > > > > > > > > > > > > Regarding Closed vs Resolved, should we go back and switch all > the > > > > > Resolved > > > > > > issues to Closed so that is is not confusing to new people to the > > > > project > > > > > > in the future that may not have seen this discussion? > > > > > > > > > > > > I would recommend changing it to Closed just to be consistent > since > > > > that > > > > > is > > > > > > the majority and the new process will be using Closed
Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis
If we still need to accept PRs for Flink-1.9/1.10, that could explain why we still need that command hint. Chesnay, thanks for your explanation. From: Chesnay Schepler Sent: Monday, May 25, 2020 18:17 To: dev@flink.apache.org ; Yun Tang Subject: Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis The travis bot commands must be retained so long as we accept PRs for 1.9/1.10 . On 25/05/2020 10:50, Yun Tang wrote: > I noticed that there still existed travis related bot commands in the github > PR page, and I think we should remove the command hint now. > > From: Robert Metzger > Sent: Thursday, April 23, 2020 15:44 > To: dev > Subject: Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / > switch off Travis > > FYI: I have moved the Flink PR and master builds from my personal Azure > account to a PMC controlled account: > https://dev.azure.com/apache-flink/apache-flink/_build > > On Fri, Apr 17, 2020 at 8:28 PM Robert Metzger wrote: > >> Thanks a lot for bringing up this topic again. >> The reason why I was hesitant to decommission Travis was that we were >> still facing some issues with the Azure infrastructure that I want to >> resolve, so that we have a strong test coverage. >> >> In the last few weeks, we had the following issues: >> - unstable e2e tests (we are running the e2e tests much more frequently, >> thus we see more failures (and discover actual bugs!)) >> - network issues (mostly around downloading maven artifacts. This is >> solved at the cost of slower builds. I'm preparing a fix to have stable & >> fast maven downloads) >> - the private builds were never really stable (this is work in progress. >> the situation is definitely better than the private Travis builds) >> - I haven't followed the overall master stability closely before February, >> but I have the feeling that April so far was a pretty unstable month on >> master. Piotr is regularly reverting commits that somehow broke master. The >> problem with unstable master is that is causes a "CI fatigue", were people >> assume that failing builds are not worth investigating anymore, leading to >> more instability. This is not a problem of the CI infrastructure itself, >> but it makes me less confident switching systems :) >> >> >> Unless something unexpected happens, I'm proposing to disable pull request >> processing on Travis next week. >> >> >> >> On Fri, Apr 17, 2020 at 10:05 AM Gary Yao wrote: >> >>> I am in favour of decommissioning Travis. >>> >>> Moreover, I wanted to use this thread to raise another issue with Travis >>> that I >>> have discovered recently; many of the builds running in my private Travis >>> account are timing out in the compilation stage (i.e., compilation takes >>> more >>> than 50 minutes). This means that I am not able to reliably run a full >>> build on >>> a CI server without creating a pull request. If other developers also >>> experience >>> this issue, it would speak for putting more effort into making Azure >>> Pipelines >>> the project-wide default. >>> >>> Best, >>> Gary >>> >>> On Thu, Mar 26, 2020 at 12:26 PM Yu Li wrote: >>> Thanks for the clarification Robert. Since the first step plan is to replace the travis PR runs, I checked >>> all PR builds from 2020-01-01 (PR#10735-11526) [1], and below is the result: * Travis FAILURE: 298 * Travis SUCCESS: 649 (68.5%) * Azure FAILURE: 420 * Azure SUCCESS: 571 (57.6%) Since the patch for each run is equivalent for Travis and Azure, there seems to be slightly higher failure rate (~10%) when running in Azure. However, with the just-merged fix for uploading logs (FLINK-16480), I believe the success rate of Azure could compete with Travis now >>> (uploading files contribute to 20% of the failures according to the report [2]). So I'm +1 to disable travis runs according to the numbers. Best Regards, Yu [1] >>> https://github.com/apache/flink/pulls?q=is%3Apr+created%3A%3E%3D2020-01-01 [2] >>> https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=4 On Thu, 26 Mar 2020 at 03:28, Robert Metzger >>> wrote: > Thank you for your responses. > > @Yu Li: In the current master, the log upload always fails, if the e2e job > failed. I just merged a PR that fixes this issue [1]. The problem was >>> not > really the network stability, rather a problem with the interaction of the > jobs in the pipeline (the e2e job did not set the right variables for >>> the > log upload) > Secondly, you are looking at the report of the "flink-ci.flink" >>> pipeline, > where pull requests are build. Naturally, pull request builds fail all the > time, because the PRs are not yet perfect. > > "flink-ci.flink-master" is the right pipeline to look
[jira] [Created] (FLINK-17925) Throws unsupported exception when using metastore commit policy for filesystem table
Jingsong Lee created FLINK-17925:
Summary: Throws unsupported exception when using metastore commit
policy for filesystem table
Key: FLINK-17925
URL: https://issues.apache.org/jira/browse/FLINK-17925
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Reporter: Jingsong Lee
Fix For: 1.11.0
Filesystem connector has an empty implementation in
\{{TableMetaStoreFactory}}. We should avoid user configuring this policy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17924) HBase connector：we can write data to HBase table, but con`t read data from HBase
pengnengsong created FLINK-17924:
Summary: HBase connector：we can write data to HBase table, but
con`t read data from HBase
Key: FLINK-17924
URL: https://issues.apache.org/jira/browse/FLINK-17924
Project: Flink
Issue Type: Bug
Reporter: pengnengsong
CREATE TABLE source(
rowkey INT,
f1 ROW,
f2 ROW
)WITH(
'connector.type' = 'hbase',
'connector.version' = '1.4.3',
'connector.table-name' = 'test_source:flink',
'connector.zookeeper.quorum' = ':2181',
'connector.zookeeper.znode.parent' = '/test/hbase'
)
Look at the source code of the Hbase connector, execute the configure method of
HBaseRowInputformat. Java, this.conf is always null. The default hbase
configuration information is created, and the configuration parameters in with
are not in effect.
private void connectToTable(){
if(this.conf ==null){
this.conf = HBaseConfiguration.create();
}
...
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)
+1 It would be nice to have these fixes! Best, Hequn On Mon, May 25, 2020 at 5:00 PM Zhijiang wrote: > Thanks for driving this, Chesnay! > +1 on my side. > > Best, > Zhijiang > -- > From:Till Rohrmann > Send Time:2020年5月25日(星期一) 16:28 > To:dev > Subject:Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0) > > +1 for the new flink-shaded release. > > Cheers, > Till > > On Mon, May 25, 2020 at 9:06 AM Chesnay Schepler > wrote: > > > Hello, > > > > I would like to do another flink-shaded release for 1.11.0, to include a > > zookeeper 3.4 security fix and resolve a shading issue when working with > > Gradle. > > > > > > > >
Re: Shall we avoid binding "unsafe ports" during random port selection?
I agree with Till. I think this should be a concern of the user configuring the port range. – Ufuk On Mon, May 25, 2020 at 10:27 AM Till Rohrmann wrote: > Hi Weike, > > would it be good enough if the user did not include unsafe ranges when > specifying `rest.bind-port`? My concern with excluding unsafe ports is that > it adds some invisible magic which can be hard to understand for the user. > I think over the past couple of years it has proven that auto magic often > leads to hard to understand features. > > Cheers, > Till > > On Sat, May 23, 2020 at 7:46 AM DONG, Weike > wrote: > > > Hi dev, > > > > Recently we have found that when* `rest.bind-port`* parameter is > specified > > as a port range (i.e. "5000-8000"), Flink may bind to some port (like > 6000) > > that is not allowed by Chrome (showing a "ERR_UNSAFE_PORT" message and > > preventing users to continue accessing the website), similarly Firefox > > blocks these unsafe port as well [1]. > > > > When I dig further into this issue, I do believe that this restriction is > > reasonable [2] as Flink may accidentally bind to some port that is > > generally considered to be used by other services, posing security risks > > and causing potential confusions to the network administrator. > > > > Here I propose that when Flink tries to do port selection in ` > > *NetUtils.getPortRangeFromString*`, we return an iterator that explicitly > > skips those unsafe ports, so that those unsafe ports would not be used > > unless users explicitly specify one in *`rest.port`* parameter. > > > > I would like to solicit opinions from the community on this matter, > thanks > > : ) > > > > Sincerely, > > Weike > > > > [1] https://www-archive.mozilla.org/projects/netlib/portbanning#portlist > > [2] > > > > > https://superuser.com/questions/188058/which-ports-are-considered-unsafe-by-chrome > > >
Re: [DISCUSS] FLIP-126: Unify (and separate) Watermark Assigners
The specifics of naming diverged a bit from the FLIP during
implementation but that should be fine. What matters in the end is the
intention of the FLIP and that the code that is committed in the end is
good and consistent in itself.
Best,
Aljoscha
On 24.05.20 05:12, Guanghui Zhang wrote:
Hi, @Aljoscha,
the function param currentTimestamp comment does not match the
recordTimestamp "long extractTimestamp(T element, long recordTimestamp)"
on wiki.
Best,
Zhangguanghui
Dawid Wysakowicz 于2020年5月13日周三 上午12:28写道：
Thank you for the update and sorry again for chiming in so late...
Best,
Dawid
On 12/05/2020 18:21, Aljoscha Krettek wrote:
Yes, I am also ok with a SerializableTimestampAssigner. This only
looks a bit clumsy in the API but as a user (that uses lambdas) you
should not see this. I pushed changes for this to my branch:
https://github.com/aljoscha/flink/tree/flink-xxx-wm-generators-rebased
And yes, recordTimestamp sounds good for the TimestampAssigner. I
admit I didn't read this well enough and only saw nativeTimestamp.
Best,
Aljoscha
On 12.05.20 17:16, Dawid Wysakowicz wrote:
I have similar thoughts to @Stephan
Ad. 1 I tried something like this on your branch:
/**
* Adds the given {@link TimestampAssigner} to this {@link
WatermarkStrategies}. For top-level classes that implement both
Serializable and TimestampAssigner
*/
public & Serializable>
WatermarkStrategies withTimestampAssigner(TA timestampAssigner) {
checkNotNull(timestampAssigner, "timestampAssigner");
this.timestampAssigner = timestampAssigner;
return this;
}
@FunctionalInterface
public interface SerializableTimestampAssigner extends
TimestampAssigner, Serializable {
}
/**
* Adds the given {@link TimestampAssigner} to this {@link
WatermarkStrategies}.
* Helper method for serializable lambdas.
*/
public WatermarkStrategies
withTimestampAssigner(SerializableTimestampAssigner
timestampAssigner) {
checkNotNull(timestampAssigner, "timestampAssigner");
this.timestampAssigner = timestampAssigner;
return this;
}
But I understand if that's too hacky. It's just a pity that we must
enforce limitations on an interface that are not strictly necessary.
Ad 2/3
I am aware the watermark assigner/timestamp extractor can be applied
further down the graph. Originally I also wanted to suggest
sourceTimestamp and SourceTimestampAssigner, but then I realized it can
be used also after the sources as you correctly pointed out. Even if the
TimestampAssigner is used after the source there might be some
native/record timestamp in the StreamRecord, that could've been
extracted by previous assigner.
Best,
Dawid
On 12/05/2020 16:47, Stephan Ewen wrote:
@Aljoscha
About (1) could we have an interface SerializableTimestampAssigner that
simply mixes in the java.io.Serializable interface? Or will this be too
clumsy?
About (3) RecordTimeStamp seems to fit both cases (in-source-record
timestamp, in stream-record timestamp).
On Tue, May 12, 2020 at 4:12 PM Aljoscha Krettek
wrote:
Definitely +1 to point 2) raised by Dawid. I'm not sure on points
1) and
3).
1) I can see the benefit of that but in reality most timestamp
assigners
will probably need to be Serializable. If you look at my (updated) POC
branch [1] you can see how a TimestampAssigner would be specified
on the
WatermarkStrategies helper class: [2]. The signature of this would
have
to be changed to something like:
public & Serializable>
WatermarkStrategies withTimestampAssigner(TA timestampAssigner)
Then, however, it would not be possible for users to specify a
lambda or
anonymous inner function for the TimestampAssigner like this:
WatermarkStrategy testWmStrategy = WatermarkStrategies
.forGenerator(new PeriodicTestWatermarkGenerator())
.withTimestampAssigner((event, timestamp) -> event)
.build();
3) This makes sense if we only allow WatermarkStrategies on sources,
where the previous timestamp really is the "native" timestamp.
Currently, we also allow setting watermark strategies at arbitrary
points in the graph. I'm thinking we probably should only allow
that in
sources but it's not the reality currently. I'm not against
renaming it,
just voicing those thoughts.
Best,
Aljoscha
[1]
https://github.com/aljoscha/flink/tree/flink-xxx-wm-generators-rebased
[2]
https://github.com/aljoscha/flink/blob/flink-xxx-wm-generators-rebased/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategies.java#L81
On 12.05.20 15:48, Stephan Ewen wrote:
+1 to all of Dawid's suggestions, makes a lot of sense to me
On Tue, May 12, 2020 at 2:32 PM Dawid Wysakowicz
Hi Aljoscha,
Sorry for adding comments during the vote, but I have some really
minor
suggestions that should not influence the voting thread imo.
1) Does it make sense to have the TimestampAssigner extend
Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis
The travis bot commands must be retained so long as we accept PRs for 1.9/1.10 . On 25/05/2020 10:50, Yun Tang wrote: I noticed that there still existed travis related bot commands in the github PR page, and I think we should remove the command hint now. From: Robert Metzger Sent: Thursday, April 23, 2020 15:44 To: dev Subject: Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis FYI: I have moved the Flink PR and master builds from my personal Azure account to a PMC controlled account: https://dev.azure.com/apache-flink/apache-flink/_build On Fri, Apr 17, 2020 at 8:28 PM Robert Metzger wrote: Thanks a lot for bringing up this topic again. The reason why I was hesitant to decommission Travis was that we were still facing some issues with the Azure infrastructure that I want to resolve, so that we have a strong test coverage. In the last few weeks, we had the following issues: - unstable e2e tests (we are running the e2e tests much more frequently, thus we see more failures (and discover actual bugs!)) - network issues (mostly around downloading maven artifacts. This is solved at the cost of slower builds. I'm preparing a fix to have stable & fast maven downloads) - the private builds were never really stable (this is work in progress. the situation is definitely better than the private Travis builds) - I haven't followed the overall master stability closely before February, but I have the feeling that April so far was a pretty unstable month on master. Piotr is regularly reverting commits that somehow broke master. The problem with unstable master is that is causes a "CI fatigue", were people assume that failing builds are not worth investigating anymore, leading to more instability. This is not a problem of the CI infrastructure itself, but it makes me less confident switching systems :) Unless something unexpected happens, I'm proposing to disable pull request processing on Travis next week. On Fri, Apr 17, 2020 at 10:05 AM Gary Yao wrote: I am in favour of decommissioning Travis. Moreover, I wanted to use this thread to raise another issue with Travis that I have discovered recently; many of the builds running in my private Travis account are timing out in the compilation stage (i.e., compilation takes more than 50 minutes). This means that I am not able to reliably run a full build on a CI server without creating a pull request. If other developers also experience this issue, it would speak for putting more effort into making Azure Pipelines the project-wide default. Best, Gary On Thu, Mar 26, 2020 at 12:26 PM Yu Li wrote: Thanks for the clarification Robert. Since the first step plan is to replace the travis PR runs, I checked all PR builds from 2020-01-01 (PR#10735-11526) [1], and below is the result: * Travis FAILURE: 298 * Travis SUCCESS: 649 (68.5%) * Azure FAILURE: 420 * Azure SUCCESS: 571 (57.6%) Since the patch for each run is equivalent for Travis and Azure, there seems to be slightly higher failure rate (~10%) when running in Azure. However, with the just-merged fix for uploading logs (FLINK-16480), I believe the success rate of Azure could compete with Travis now (uploading files contribute to 20% of the failures according to the report [2]). So I'm +1 to disable travis runs according to the numbers. Best Regards, Yu [1] https://github.com/apache/flink/pulls?q=is%3Apr+created%3A%3E%3D2020-01-01 [2] https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=4 On Thu, 26 Mar 2020 at 03:28, Robert Metzger wrote: Thank you for your responses. @Yu Li: In the current master, the log upload always fails, if the e2e job failed. I just merged a PR that fixes this issue [1]. The problem was not really the network stability, rather a problem with the interaction of the jobs in the pipeline (the e2e job did not set the right variables for the log upload) Secondly, you are looking at the report of the "flink-ci.flink" pipeline, where pull requests are build. Naturally, pull request builds fail all the time, because the PRs are not yet perfect. "flink-ci.flink-master" is the right pipeline to look at: https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=8=build We have a fairly high number of failures there, because we currently have some issues downloading the maven artifacts [2]. I'm working already with Chesnay on merging a fix for that. [1] https://github.com/apache/flink/commit/1c86b8b9dd05615a3b2600984db738a9bf388259 [2]https://issues.apache.org/jira/browse/FLINK-16720 On Wed, Mar 25, 2020 at 4:48 PM Chesnay Schepler wrote: The easiest way to disable travis for pushes is to remove all builds from the .travis.yml with a push/pr condition. On 25/03/2020 15:03, Robert Metzger wrote: Thank you for the feedback so far. Responses to the items Chesnay raised: - by virtue of maintaining the past
[jira] [Created] (FLINK-17923) It will throw MemoryAllocationException if rocksdb statebackend and Python UDF are used in the same slot
Dian Fu created FLINK-17923:
---
Summary: It will throw MemoryAllocationException if rocksdb
statebackend and Python UDF are used in the same slot
Key: FLINK-17923
URL: https://issues.apache.org/jira/browse/FLINK-17923
Project: Flink
Issue Type: Bug
Components: API / Python, Runtime / State Backends
Affects Versions: 1.10.0, 1.11.0
Reporter: Dian Fu
Fix For: 1.11.0
For the following job:
{code}
import logging
import os
import shutil
import sys
import tempfile
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import TableConfig, StreamTableEnvironment, DataTypes
from pyflink.table.udf import udf
def word_count():
content = "line Licensed to the Apache Software Foundation ASF under one " \
"line or more contributor license agreements See the NOTICE file
" \
"line distributed with this work for additional information " \
"line regarding copyright ownership The ASF licenses this file " \
"to you under the Apache License Version the " \
"License you may not use this file except in compliance " \
"with the License"
t_config = TableConfig()
env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(env, t_config)
# register Results table in table environment
tmp_dir = tempfile.gettempdir()
result_path = tmp_dir + '/result'
if os.path.exists(result_path):
try:
if os.path.isfile(result_path):
os.remove(result_path)
else:
shutil.rmtree(result_path)
except OSError as e:
logging.error("Error removing directory: %s - %s.", e.filename,
e.strerror)
logging.info("Results directory: %s", result_path)
sink_ddl = """
create table Results(
word VARCHAR,
`count` BIGINT
) with (
'connector' = 'blackhole'
)
"""
t_env.sql_update(sink_ddl)
@udf(input_types=[DataTypes.BIGINT()], result_type=DataTypes.BIGINT())
def inc(count):
return count + 1
t_env.register_function("inc", inc)
elements = [(word, 1) for word in content.split(" ")]
t_env.from_elements(elements, ["word", "count"]) \
.group_by("word") \
.select("word, count(1) as count") \
.select("word, inc(count) as count") \
.insert_into("Results")
t_env.execute("word_count")
if __name__ == '__main__':
logging.basicConfig(stream=sys.stdout, level=logging.INFO,
format="%(message)s")
word_count()
{code}
It will throw the following exception:
{code}
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for KeyedProcessOperator_c27dcf7b54ef6bfd6cff02ca8870b681_(1/1) from
any of the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:317)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:144)
... 9 more
Caused by: java.io.IOException: Failed to acquire shared cache resource for
RocksDB
at
org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.allocateSharedCachesIfConfigured(RocksDBOperationUtils.java:212)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:516)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:301)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
... 11 more
Caused by: org.apache.flink.runtime.memory.MemoryAllocationException: Could not
created the shared memory resource of size 536870920. Not enough memory left to
reserve from the slot's managed memory.
at
org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$8(MemoryManager.java:603)
at
org.apache.flink.runtime.memory.SharedResources.createResource(SharedResources.java:130)
at
org.apache.flink.runtime.memory.SharedResources.getOrAllocateSharedResource(SharedResources.java:72)
at
org.apache.flink.runtime.memory.MemoryManager.getSharedMemoryResourceForManagedMemory(MemoryManager.java:617)
at
[jira] [Created] (FLINK-17922) Refactor generic and hive table rules for HiveCatalog
Jingsong Lee created FLINK-17922:
Summary: Refactor generic and hive table rules for HiveCatalog
Key: FLINK-17922
URL: https://issues.apache.org/jira/browse/FLINK-17922
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Reporter: Jingsong Lee
Fix For: 1.12.0
Now the underlying rules are:
* HiveCatalog.createTable , hive table has false of {{isGeneric}} flag,
generic table has no flag.
* HiveCatalog.getTable , generic table has true of {{isGeneric}} flag, hive
table has no flag.
It is very difficult to understand and easy to cause bugs.
Whether it's {{createTable}} or {{getTable}} of {{HiveCatalog}}, there should
be two types of objects that are consistent: {{HiveCatalogTable}} and Flink
{{CatalogTableImpl}}.
Timo: From a logical perspective, the {{HiveCatalog}} should deal with this
additional property while storing and retrieving a {{CatalogTable}}. The
{{isGeneric}} should not travel through the stack of regular Flink connector
options. It is an internal property from the {{HiveCatalog}} through a custom
{{HiveCatalogTable}} to a custom {{HiveFactory}}. It should be stored as a
member variable in a {{HiveCatalogTable}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Please update the Requirements of PyPI
Thanks Ray and Till, Could you provide more information about the errors you encountered? Is it the same error message as reported in https://issues.apache.org/jira/browse/BEAM-8368? Best, Jincheng Till Rohrmann 于2020年5月25日周一 下午4:28写道： > Thanks for the pointer Ray. I've pulled in Jincheng who works on Flink's > Python integration. > > Cheers, > Till > > On Mon, May 25, 2020 at 8:41 AM Ray Chen > wrote: > > > Hi PyFlink Developers, > > > > > > I got some errors from the pyarrow depended on apache-flink v1.10.1. > > > > It seems updating pyarrow version greater than 0.14 will solve the > > problem. > > > > The version of Flink on PyPI depends on < 0.14, so cause the error. > > > > > > Best, > > Ray Chen > > >
[jira] [Created] (FLINK-17921) RpcGlobalAggregateManager#updateGlobalAggregate would cause akka.timeout
zhangminglei created FLINK-17921:
Summary: RpcGlobalAggregateManager#updateGlobalAggregate would
cause akka.timeout
Key: FLINK-17921
URL: https://issues.apache.org/jira/browse/FLINK-17921
Project: Flink
Issue Type: Improvement
Reporter: zhangminglei
As described in summary, {{RpcGlobalAggregateManager#updateGlobalAggregate}}
would cause akka.timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #1
Since the official image for 1.11 has not been release, I just want to
post how to
build the image with provided Flink binary. I think many users are running
Flink
in container environment(e.g. docker, Kubernetes, etc.) and hope this could
help
them with the testing.
git clone 'https://github.com/apache/flink-docker.git'
cd flink-docker/ && git checkout dev-master
export image_name="flink:1.11.0-rc1-scala_2.11"
./add-custom.sh -u
https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc1/flink-1.11.0-bin-scala_2.11.tgz
-n ${image_name}
docker build --no-cache --network="host" -t ${image_name}
dev/${image_name}-debian
docker push ${image_name}
Best,
Yang
Zhijiang 于2020年5月25日周一 下午12:09写道：
> Hi all,
>
> Apache Flink-1.11.0-RC1 has been created. It has all the artifacts that we
> would typically have for a release.
>
> This preview-only RC is created only to drive the current testing efforts,
> and no official vote will take place. It includes the following:
>
>* The preview source release and binary convenience releases [1], which
> are signed with the key with fingerprint
> 2DA85B93244FDFA19A6244500653C0A2CEA00D0E [2],
>* All artifacts that would normally be deployed to the Maven Central
> Repository [3]
>
> To test with these artifacts, you can create a settings.xml file with the
> content shown below [4]. This settings file can be referenced in your maven
> commands
> via --settings /path/to/settings.xml. This is useful for creating a
> quickstart project based on the staged release and also for building
> against the staged jars.
>
> Happy testing!
>
> Best,
> Zhijiang
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.0-rc1/
> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1370/
> [4]
>
>
> flink-1.11.0
>
>
>
> flink-1.11.0
>
>
> flink-1.11.0
>
> https://repository.apache.org/content/repositories/orgapacheflink-1370/
>
>
>
>archetype
>
> https://repository.apache.org/content/repositories/orgapacheflink-1370/
>
>
>
>
>
>
[jira] [Created] (FLINK-17920) Add the Python example of Time-windowed join in Table API doc
Huang Xingbo created FLINK-17920: Summary: Add the Python example of Time-windowed join in Table API doc Key: FLINK-17920 URL: https://issues.apache.org/jira/browse/FLINK-17920 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Reporter: Huang Xingbo Fix For: 1.11.0, 1.10.2 We have supported Time-windowed join in Python Table API since 1.10. So we should add the corresponding example in Table API doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)
Thanks for driving this, Chesnay! +1 on my side. Best, Zhijiang -- From:Till Rohrmann Send Time:2020年5月25日(星期一) 16:28 To:dev Subject:Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0) +1 for the new flink-shaded release. Cheers, Till On Mon, May 25, 2020 at 9:06 AM Chesnay Schepler wrote: > Hello, > > I would like to do another flink-shaded release for 1.11.0, to include a > zookeeper 3.4 security fix and resolve a shading issue when working with > Gradle. > > >
[jira] [Created] (FLINK-17919) KafkaConsumerThread should add ratelimiter functionality as well
zhangminglei created FLINK-17919:
Summary: KafkaConsumerThread should add ratelimiter functionality
as well
Key: FLINK-17919
URL: https://issues.apache.org/jira/browse/FLINK-17919
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Reporter: zhangminglei
Currently, {{KafkaConsumerThread}} within {{flink-connector-kafka-09}} has the
ability of rateLimiter. However, under {{flink-connector-kafka}} does not own
it. I would suggest we can add it as well if we could.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis
I noticed that there still existed travis related bot commands in the github PR page, and I think we should remove the command hint now. From: Robert Metzger Sent: Thursday, April 23, 2020 15:44 To: dev Subject: Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis FYI: I have moved the Flink PR and master builds from my personal Azure account to a PMC controlled account: https://dev.azure.com/apache-flink/apache-flink/_build On Fri, Apr 17, 2020 at 8:28 PM Robert Metzger wrote: > Thanks a lot for bringing up this topic again. > The reason why I was hesitant to decommission Travis was that we were > still facing some issues with the Azure infrastructure that I want to > resolve, so that we have a strong test coverage. > > In the last few weeks, we had the following issues: > - unstable e2e tests (we are running the e2e tests much more frequently, > thus we see more failures (and discover actual bugs!)) > - network issues (mostly around downloading maven artifacts. This is > solved at the cost of slower builds. I'm preparing a fix to have stable & > fast maven downloads) > - the private builds were never really stable (this is work in progress. > the situation is definitely better than the private Travis builds) > - I haven't followed the overall master stability closely before February, > but I have the feeling that April so far was a pretty unstable month on > master. Piotr is regularly reverting commits that somehow broke master. The > problem with unstable master is that is causes a "CI fatigue", were people > assume that failing builds are not worth investigating anymore, leading to > more instability. This is not a problem of the CI infrastructure itself, > but it makes me less confident switching systems :) > > > Unless something unexpected happens, I'm proposing to disable pull request > processing on Travis next week. > > > > On Fri, Apr 17, 2020 at 10:05 AM Gary Yao wrote: > >> I am in favour of decommissioning Travis. >> >> Moreover, I wanted to use this thread to raise another issue with Travis >> that I >> have discovered recently; many of the builds running in my private Travis >> account are timing out in the compilation stage (i.e., compilation takes >> more >> than 50 minutes). This means that I am not able to reliably run a full >> build on >> a CI server without creating a pull request. If other developers also >> experience >> this issue, it would speak for putting more effort into making Azure >> Pipelines >> the project-wide default. >> >> Best, >> Gary >> >> On Thu, Mar 26, 2020 at 12:26 PM Yu Li wrote: >> >> > Thanks for the clarification Robert. >> > >> > Since the first step plan is to replace the travis PR runs, I checked >> all >> > PR builds from 2020-01-01 (PR#10735-11526) [1], and below is the result: >> > >> > * Travis FAILURE: 298 >> > * Travis SUCCESS: 649 (68.5%) >> > * Azure FAILURE: 420 >> > * Azure SUCCESS: 571 (57.6%) >> > >> > Since the patch for each run is equivalent for Travis and Azure, there >> > seems to be slightly higher failure rate (~10%) when running in Azure. >> > >> > However, with the just-merged fix for uploading logs (FLINK-16480), I >> > believe the success rate of Azure could compete with Travis now >> (uploading >> > files contribute to 20% of the failures according to the report [2]). >> > >> > So I'm +1 to disable travis runs according to the numbers. >> > >> > Best Regards, >> > Yu >> > >> > [1] >> > >> https://github.com/apache/flink/pulls?q=is%3Apr+created%3A%3E%3D2020-01-01 >> > [2] >> > >> > >> https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=4 >> > >> > On Thu, 26 Mar 2020 at 03:28, Robert Metzger >> wrote: >> > >> > > Thank you for your responses. >> > > >> > > @Yu Li: In the current master, the log upload always fails, if the e2e >> > job >> > > failed. I just merged a PR that fixes this issue [1]. The problem was >> not >> > > really the network stability, rather a problem with the interaction of >> > the >> > > jobs in the pipeline (the e2e job did not set the right variables for >> the >> > > log upload) >> > > Secondly, you are looking at the report of the "flink-ci.flink" >> pipeline, >> > > where pull requests are build. Naturally, pull request builds fail all >> > the >> > > time, because the PRs are not yet perfect. >> > > >> > > "flink-ci.flink-master" is the right pipeline to look at: >> > > >> > > >> > >> https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=8=build >> > > We have a fairly high number of failures there, because we currently >> have >> > > some issues downloading the maven artifacts [2]. I'm working already >> with >> > > Chesnay on merging a fix for that. >> > > >> > > >> > > [1] >> > > >> > > >> > >> https://github.com/apache/flink/commit/1c86b8b9dd05615a3b2600984db738a9bf388259 >> > > [2]https://issues.apache.org/jira/browse/FLINK-16720 >> > > >> > > >> > > >> > >
Re: [DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)
+1 for the new flink-shaded release. Cheers, Till On Mon, May 25, 2020 at 9:06 AM Chesnay Schepler wrote: > Hello, > > I would like to do another flink-shaded release for 1.11.0, to include a > zookeeper 3.4 security fix and resolve a shading issue when working with > Gradle. > > >
Re: Please update the Requirements of PyPI
Thanks for the pointer Ray. I've pulled in Jincheng who works on Flink's Python integration. Cheers, Till On Mon, May 25, 2020 at 8:41 AM Ray Chen wrote: > Hi PyFlink Developers, > > > I got some errors from the pyarrow depended on apache-flink v1.10.1. > > It seems updating pyarrow version greater than 0.14 will solve the > problem. > > The version of Flink on PyPI depends on < 0.14, so cause the error. > > > Best, > Ray Chen >
Re: Shall we avoid binding "unsafe ports" during random port selection?
Hi Weike, would it be good enough if the user did not include unsafe ranges when specifying `rest.bind-port`? My concern with excluding unsafe ports is that it adds some invisible magic which can be hard to understand for the user. I think over the past couple of years it has proven that auto magic often leads to hard to understand features. Cheers, Till On Sat, May 23, 2020 at 7:46 AM DONG, Weike wrote: > Hi dev, > > Recently we have found that when* `rest.bind-port`* parameter is specified > as a port range (i.e. "5000-8000"), Flink may bind to some port (like 6000) > that is not allowed by Chrome (showing a "ERR_UNSAFE_PORT" message and > preventing users to continue accessing the website), similarly Firefox > blocks these unsafe port as well [1]. > > When I dig further into this issue, I do believe that this restriction is > reasonable [2] as Flink may accidentally bind to some port that is > generally considered to be used by other services, posing security risks > and causing potential confusions to the network administrator. > > Here I propose that when Flink tries to do port selection in ` > *NetUtils.getPortRangeFromString*`, we return an iterator that explicitly > skips those unsafe ports, so that those unsafe ports would not be used > unless users explicitly specify one in *`rest.port`* parameter. > > I would like to solicit opinions from the community on this matter, thanks > : ) > > Sincerely, > Weike > > [1] https://www-archive.mozilla.org/projects/netlib/portbanning#portlist > [2] > > https://superuser.com/questions/188058/which-ports-are-considered-unsafe-by-chrome >
Re: java.lang.NoSuchMethodError while writing to Kafka from Flink
Please double-check that your distribution and application jar were built against the same Flink version. This looks related to a binary-compatibility issues reporter in FLINK-13586 .
Re: [DISCUSS] Semantics of our JIRA fields
This is very helpful! +1 Thanks, Zhu Zhu Yang Wang 于2020年5月25日周一 下午4:04写道： > +1 from this useful proposal. > > This makes me clearer about "Resolve" and "Close" since I used to be > confused by this two button. > > Best, > Yang > > Jingsong Li 于2020年5月25日周一 下午3:10写道： > > > +1 for the proposal. > > It makes me clearer. > > > > Best, > > Jingsong Lee > > > > On Mon, May 25, 2020 at 2:51 PM Zhijiang > .invalid> > > wrote: > > > > > Thanks for launching this discussion and giving so detailed infos, > > > Robert! +1 on my side for the proposal. > > > > > > For "Affects Version", I previously thought it was only for the already > > > released versions, so it can give a reminder that the fix should also > > pick > > > into the related released branches for future minor versions. > > > I saw that Jark had somehow similar concerns for this field in below > > > replies. Either way makes sense for me as long as we give a determined > > > rule in Wiki. > > > > > > Re Flavio' s comments, I agree that the Jira reporter can leave most of > > > the fields empty if not confirmed of them, then the respective > component > > > maintainer or committer can update them accordingly later. > > > But the state of Jira should not be marked as "resolved" when the PR is > > > detected, that is not fitting into the resolved semantic I guess. If > > > possible, the Jira can be updated as "in progress" automatically if > > > the respective PR is ready, then it will save some time to stat > progress > > > of related issues during release process. > > > > > > Best, > > > Zhijiang > > > -- > > > From:Congxian Qiu > > > Send Time:2020年5月25日(星期一) 13:57 > > > To:dev@flink.apache.org > > > Subject:Re: [DISCUSS] Semantics of our JIRA fields > > > > > > Hi > > > > > > Currently, when I'm going to create an issue for Project-website. I'm > not > > > very sure what the "Affect Version/s" should be set. The problem is > that > > > the current Dockfile[1] in flink-web is broken because of the EOL of > > Ubuntu > > > 18.10[2], the project-web does not affect any release of Flink, it does > > > affect the process to build website, so what's the version should I set > > to? > > > > > > [1] > > > > > > > > > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > > > [2] https://wiki.ubuntu.com/Releases > > > > > > Best, > > > Congxian > > > > > > > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > > > > > > > In my experience it's quite complicated for a normal reporter to be > > able > > > to > > > > fill all the fields correctly (especially for new users). > > > > Usually you just wanto to report a problem, remember to add a new > > feature > > > > or improve code/documentation but you can't really give a priority, > > > assign > > > > the correct label or decide which releases will benefit of the > fix/new > > > > feature. This is something that only core developers could decide > > (IMHO). > > > > > > > > My experiece says that it's better if normal users could just open > > > tickets > > > > with some default (or mark ticket as new) and leave them in such a > > state > > > > until an experienced user, one that can assign tickets, have the time > > to > > > > read it and immediately reject the ticket or accept it and properly > > > assign > > > > priorities, fix version, etc. > > > > > > > > With respect to resolve/close I think that a good practice could be > to > > > mark > > > > automatically a ticket as resolved once that a PR is detected for > that > > > > ticket, while marking it as closed should be done by the commiter who > > > merge > > > > the PR. > > > > > > > > Probably this process would slightly increase the work of a committer > > but > > > > this would make things a lot cleaner and will bring the benefit of > > > having a > > > > reliable and semantically sound JIRA state. > > > > > > > > Cheers, > > > > Flavio > > > > > > > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha > > scritto: > > > > > > > > > +1 for the proposal > > > > > > > > > > This will bring some consistency to the process > > > > > > > > > > Regarding Closed vs Resolved, should we go back and switch all the > > > > Resolved > > > > > issues to Closed so that is is not confusing to new people to the > > > project > > > > > in the future that may not have seen this discussion? > > > > > > > > > > I would recommend changing it to Closed just to be consistent since > > > that > > > > is > > > > > the majority and the new process will be using Closed going forward > > > > > > > > > > Those are my thoughts for now > > > > > > > > > > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu < > qcx978132...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > +1 for the proposal. It's good to have a unified description and > > > write > > > > it > > > > > > down in the wiki, so that every contributor has the same > > > understanding > > > > of > > > > > > all the
Re: [DISCUSS] Semantics of our JIRA fields
+1 from this useful proposal. This makes me clearer about "Resolve" and "Close" since I used to be confused by this two button. Best, Yang Jingsong Li 于2020年5月25日周一 下午3:10写道： > +1 for the proposal. > It makes me clearer. > > Best, > Jingsong Lee > > On Mon, May 25, 2020 at 2:51 PM Zhijiang .invalid> > wrote: > > > Thanks for launching this discussion and giving so detailed infos, > > Robert! +1 on my side for the proposal. > > > > For "Affects Version", I previously thought it was only for the already > > released versions, so it can give a reminder that the fix should also > pick > > into the related released branches for future minor versions. > > I saw that Jark had somehow similar concerns for this field in below > > replies. Either way makes sense for me as long as we give a determined > > rule in Wiki. > > > > Re Flavio' s comments, I agree that the Jira reporter can leave most of > > the fields empty if not confirmed of them, then the respective component > > maintainer or committer can update them accordingly later. > > But the state of Jira should not be marked as "resolved" when the PR is > > detected, that is not fitting into the resolved semantic I guess. If > > possible, the Jira can be updated as "in progress" automatically if > > the respective PR is ready, then it will save some time to stat progress > > of related issues during release process. > > > > Best, > > Zhijiang > > -- > > From:Congxian Qiu > > Send Time:2020年5月25日(星期一) 13:57 > > To:dev@flink.apache.org > > Subject:Re: [DISCUSS] Semantics of our JIRA fields > > > > Hi > > > > Currently, when I'm going to create an issue for Project-website. I'm not > > very sure what the "Affect Version/s" should be set. The problem is that > > the current Dockfile[1] in flink-web is broken because of the EOL of > Ubuntu > > 18.10[2], the project-web does not affect any release of Flink, it does > > affect the process to build website, so what's the version should I set > to? > > > > [1] > > > > > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > > [2] https://wiki.ubuntu.com/Releases > > > > Best, > > Congxian > > > > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > > > > > In my experience it's quite complicated for a normal reporter to be > able > > to > > > fill all the fields correctly (especially for new users). > > > Usually you just wanto to report a problem, remember to add a new > feature > > > or improve code/documentation but you can't really give a priority, > > assign > > > the correct label or decide which releases will benefit of the fix/new > > > feature. This is something that only core developers could decide > (IMHO). > > > > > > My experiece says that it's better if normal users could just open > > tickets > > > with some default (or mark ticket as new) and leave them in such a > state > > > until an experienced user, one that can assign tickets, have the time > to > > > read it and immediately reject the ticket or accept it and properly > > assign > > > priorities, fix version, etc. > > > > > > With respect to resolve/close I think that a good practice could be to > > mark > > > automatically a ticket as resolved once that a PR is detected for that > > > ticket, while marking it as closed should be done by the commiter who > > merge > > > the PR. > > > > > > Probably this process would slightly increase the work of a committer > but > > > this would make things a lot cleaner and will bring the benefit of > > having a > > > reliable and semantically sound JIRA state. > > > > > > Cheers, > > > Flavio > > > > > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha > scritto: > > > > > > > +1 for the proposal > > > > > > > > This will bring some consistency to the process > > > > > > > > Regarding Closed vs Resolved, should we go back and switch all the > > > Resolved > > > > issues to Closed so that is is not confusing to new people to the > > project > > > > in the future that may not have seen this discussion? > > > > > > > > I would recommend changing it to Closed just to be consistent since > > that > > > is > > > > the majority and the new process will be using Closed going forward > > > > > > > > Those are my thoughts for now > > > > > > > > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu > > > > > wrote: > > > > > > > > > +1 for the proposal. It's good to have a unified description and > > write > > > it > > > > > down in the wiki, so that every contributor has the same > > understanding > > > of > > > > > all the fields. > > > > > > > > > > Best, > > > > > Congxian > > > > > > > > > > > > > > > Till Rohrmann 于2020年5月23日周六 上午12:04写道： > > > > > > > > > > > Thanks for drafting this proposal Robert. +1 for the proposal. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu > > > wrote: > > > > > > > > > > > > > Thanks bringing up this topic
[jira] [Created] (FLINK-17918) Jobs with two input operators are loosing data in unaligned checkpoints
Piotr Nowojski created FLINK-17918:
--
Summary: Jobs with two input operators are loosing data in
unaligned checkpoints
Key: FLINK-17918
URL: https://issues.apache.org/jira/browse/FLINK-17918
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.11.0
Reporter: Piotr Nowojski
Fix For: 1.11.0
After trying to enable unaligned checkpoints by default, a lot of Blink
streaming SQL/Table API tests containing joins or set operations are throwing
errors that are indicating we are loosing some data (full records, without
deserialisation errors). Example errors:
{noformat}
[ERROR] Failures:
[ERROR] JoinITCase.testFullJoinWithEqualPk:775 expected: but was:
[ERROR] JoinITCase.testStreamJoinWithSameRecord:391 expected: but was:
[ERROR] SemiAntiJoinStreamITCase.testAntiJoin:352 expected:<0> but was:<1>
[ERROR] SetOperatorsITCase.testIntersect:55 expected: but was:
[ERROR] JoinITCase.testJoinPushThroughJoin:1272 expected: but was:
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[ANNOUNCE] Weekly Community Update 2020/21
Dear community, happy to share this week's community update! Flink Development == * [releases] Zhijiang has published a first release candidate for Apache Flink 1.11.0. This is a "preview-only" release candidate to facilitate current testing efforts. There will not be a vote on this RC. [1] * [releases] Gordon proposes to release Apache Flink Stateful Function 2.1.0 soon. This would mean a faster release cycle compared to the main project, which seems to make sense given the early state of the project. The proposal received a lot positive feedback and the feature freeze date was set for this Wednesday (27th of May). [2] * [deployment] Klou proposes to remove support for shipping nested JARs during job submission, a so far hidden feature of Apache Flink from the early days of the project. [3] * [security] Weike Dong proposes to exclude "unsafe ports", i.e. ports considered to be unsafe by major browsers, as candidates for the Flink REST API, if no port was specified explicitly. [4] * [development process] Robert has started a thread to clarify the semantics and usage of JIRA fields in the Apache Flink project. If you are unsure about some fields, check out this thread and join the conversation. [5] [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-1-11-0-release-candidate-1-tp41840.html [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Stateful-Functions-2-1-0-soon-tp41734.html [3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-dependency-shipping-through-nested-jars-during-job-submission-tp41711.html [4] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Shall-we-avoid-binding-unsafe-ports-during-random-port-selection-tp41821.html [5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Semantics-of-our-JIRA-fields-tp41786p41806.html Notable Bugs == * [FLINK-17189] [1.10.1] HiveCatalog does not support processing time columns. [6] * [FLINK-17800] [1.10.1] There might be a state loss issue when using the Rocksdb with optimizeForPointLookup. The community is still investigating the bug report. [7] [6] https://issues.apache.org/jira/browse/FLINK-17189 [7] https://issues.apache.org/jira/browse/FLINK-17800 Events, Blog Posts, Misc === * Ankit Jhalaria of GoDaddy has published a blog post about their Flink-based streaming data platform on the Ververica Blog [8]. * If you have not heard about Lyft's streaming platform before, here is another great talk by Sherin Thomas and blog on InfoQ [9]. [8] https://www.ververica.com/blog/how-godaddy-uses-flink-to-run-real-time-streaming-pipelines [9] https://www.infoq.com/presentations/ml-streaming-lyft/ Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Semantics of our JIRA fields
+1 for the proposal. It makes me clearer. Best, Jingsong Lee On Mon, May 25, 2020 at 2:51 PM Zhijiang wrote: > Thanks for launching this discussion and giving so detailed infos, > Robert! +1 on my side for the proposal. > > For "Affects Version", I previously thought it was only for the already > released versions, so it can give a reminder that the fix should also pick > into the related released branches for future minor versions. > I saw that Jark had somehow similar concerns for this field in below > replies. Either way makes sense for me as long as we give a determined > rule in Wiki. > > Re Flavio' s comments, I agree that the Jira reporter can leave most of > the fields empty if not confirmed of them, then the respective component > maintainer or committer can update them accordingly later. > But the state of Jira should not be marked as "resolved" when the PR is > detected, that is not fitting into the resolved semantic I guess. If > possible, the Jira can be updated as "in progress" automatically if > the respective PR is ready, then it will save some time to stat progress > of related issues during release process. > > Best, > Zhijiang > -- > From:Congxian Qiu > Send Time:2020年5月25日(星期一) 13:57 > To:dev@flink.apache.org > Subject:Re: [DISCUSS] Semantics of our JIRA fields > > Hi > > Currently, when I'm going to create an issue for Project-website. I'm not > very sure what the "Affect Version/s" should be set. The problem is that > the current Dockfile[1] in flink-web is broken because of the EOL of Ubuntu > 18.10[2], the project-web does not affect any release of Flink, it does > affect the process to build website, so what's the version should I set to? > > [1] > > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > [2] https://wiki.ubuntu.com/Releases > > Best, > Congxian > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > > > In my experience it's quite complicated for a normal reporter to be able > to > > fill all the fields correctly (especially for new users). > > Usually you just wanto to report a problem, remember to add a new feature > > or improve code/documentation but you can't really give a priority, > assign > > the correct label or decide which releases will benefit of the fix/new > > feature. This is something that only core developers could decide (IMHO). > > > > My experiece says that it's better if normal users could just open > tickets > > with some default (or mark ticket as new) and leave them in such a state > > until an experienced user, one that can assign tickets, have the time to > > read it and immediately reject the ticket or accept it and properly > assign > > priorities, fix version, etc. > > > > With respect to resolve/close I think that a good practice could be to > mark > > automatically a ticket as resolved once that a PR is detected for that > > ticket, while marking it as closed should be done by the commiter who > merge > > the PR. > > > > Probably this process would slightly increase the work of a committer but > > this would make things a lot cleaner and will bring the benefit of > having a > > reliable and semantically sound JIRA state. > > > > Cheers, > > Flavio > > > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha scritto: > > > > > +1 for the proposal > > > > > > This will bring some consistency to the process > > > > > > Regarding Closed vs Resolved, should we go back and switch all the > > Resolved > > > issues to Closed so that is is not confusing to new people to the > project > > > in the future that may not have seen this discussion? > > > > > > I would recommend changing it to Closed just to be consistent since > that > > is > > > the majority and the new process will be using Closed going forward > > > > > > Those are my thoughts for now > > > > > > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu > > > wrote: > > > > > > > +1 for the proposal. It's good to have a unified description and > write > > it > > > > down in the wiki, so that every contributor has the same > understanding > > of > > > > all the fields. > > > > > > > > Best, > > > > Congxian > > > > > > > > > > > > Till Rohrmann 于2020年5月23日周六 上午12:04写道： > > > > > > > > > Thanks for drafting this proposal Robert. +1 for the proposal. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu > > wrote: > > > > > > > > > > > Thanks bringing up this topic @Robert, +1 to the proposal. > > > > > > > > > > > > It clarifies the JIRA fields well and should be a rule to follow. > > > > > > > > > > > > Best, > > > > > > Leonard Xu > > > > > > > 在 2020年5月22日，20:24，Aljoscha Krettek 写道： > > > > > > > > > > > > > > +1 That's also how I think of the semantics of the fields. > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > On 22.05.20 08:07, Robert Metzger wrote: > > > > > > >> Hi all, > > > > > > >> I have the feeling
Re: [DISCUSS] FLIP-84 Feedback Summary
+1
Personally, I would declare it as `collect(): ClosableIterator` to
avoid an additional class in the API and reuse existing Flink utils.
Regards,
Timo
On 17.05.20 10:21, Jingsong Li wrote:
+1
Best,
Jingsong Lee
On Sun, May 17, 2020 at 3:42 PM Kurt Young wrote:
+1 from my side.
Best,
Kurt
On Sun, May 17, 2020 at 12:41 PM godfrey he wrote:
hi everyone,
I would like to bring up another topic about the return value of
TableResult#collect() method.
Currently, the return type is `Iterator`, we meet some problems when
implementing FLINK-14807[1].
In current design, the sink operator has a buffer pool which buffers the
data from upstream,
and waits the client to consume the data. The client will pull the data
when `Iterator#next()` method is called.
If the client submits a select job, consumes a part of data and exits.
The
job will not be finished.
This will cause resource leak. We can't require the client must consume
all
data. for unbounded stream job, it's also impossible.
Currently, users can also cancel the job via
`TableResult.getJobClient().get().cancel()` method.
But this approach is not intuitive and convenient.
So, I want to change the return type from `Iterator` to
`CloseableRowIterator`,
the new method likes like:
public interface TableResult {
CloseableRowIterator collect();
}
public interface CloseableRowIterator extends Iterator,
AutoCloseable
{
}
Prefixing the name with "Closeable" is intended to remind the users that
this iterator should be closed,
users can conveniently use try-with-resources statement to close the
resources.
The resource leak problem is still there if users do not close the
iterator
or cancel the job through job client,
we just provide an easier way for users to avoid this.
I also notice that there is a `CloseableIterator` interface in
`org.apache.flink.util` package.
But I still tend to introduce `CloseableRowIterator`. My point of view
is:
1) `CloseableIterator` is in a util package, not a public interface.
2) `CloseableRowIterator` is more convenient, users do not need to define
generic type ``.
What do you think?
Best,
Godfrey
[1] https://issues.apache.org/jira/browse/FLINK-14807
Fabian Hueske 于2020年5月7日周四 下午3:59写道：
Thanks for the update Godfrey!
+1 to this approach.
Since there can be only one primary key, I'd also be fine to just use
`PRI` even if it is composite, but `PRI(f0, f5)` might be more
convenient
for users.
Thanks, Fabian
Am Do., 7. Mai 2020 um 09:31 Uhr schrieb godfrey he <
godfre...@gmail.com
:
Hi fabian,
Thanks for you suggestions.
Agree with you that `UNQ(f2, f3)` is more clear.
A table can have only ONE primary key,
this primary key can consist of single or multiple columns. [1]
if primary key consists of single column,
we can simply use `PRI` (or `PRI(xx)`) to represent it.
if primary key have multiple columns,
we should use `PRI(xx, yy, ...)` to represent it.
A table may have multiple unique keys,
each unique key can consist of single or multiple columns. [2]
if there is only one unique key and this unique key has only single
column,
we can simply use `UNQ` (or `UNQ(xx)`) to represent it.
otherwise, we should use `UNQ(xx, yy, ...)` to represent it.
(a corner case: two unique keys with same columns, like `UNQ(f2, f3)`,
`UNQ(f2, f3)`,
we can forbid this case or add a unique name for each key in the
future)
primary key and unique key with multiple columns example:
create table MyTable (
f0 BIGINT NOT NULL,
f1 ROW,
f2 VARCHAR<256>,
f3 AS f0 + 1,
f4 TIMESTAMP(3) NOT NULL,
f5 BIGINT NOT NULL,
* PRIMARY KEY (f0, f5)*,
*UNIQUE (f3, f2)*,
WATERMARK f4 AS f4 - INTERVAL '3' SECOND
) with (...)
++--+---++---+--+
| name | type
|
null | key | compute column | watermark
|
++--+---++---+--+
| f0 | BIGINT
|
false | PRI(f0, f5) | (NULL) | (NULL)
|
++--+---++---+--+
| f1 | ROW | true | (NULL)
|
(NULL) | (NULL) |
++--+---++---+--+
| f2 | VARCHAR<256> | true
|
UNQ(f2, f3) | (NULL) | (NULL)
|
++--+---++---+--+
| f3 | BIGINT
|
false | UNQ(f2, f3) | f0 + 1 | (NULL)
[DISCUSS] Release flink-shaded 11.0 (and include it in 1.11.0)
Hello, I would like to do another flink-shaded release for 1.11.0, to include a zookeeper 3.4 security fix and resolve a shading issue when working with Gradle.
Re: [DISCUSS] Semantics of our JIRA fields
Thanks for launching this discussion and giving so detailed infos, Robert! +1 on my side for the proposal. For "Affects Version", I previously thought it was only for the already released versions, so it can give a reminder that the fix should also pick into the related released branches for future minor versions. I saw that Jark had somehow similar concerns for this field in below replies. Either way makes sense for me as long as we give a determined rule in Wiki. Re Flavio' s comments, I agree that the Jira reporter can leave most of the fields empty if not confirmed of them, then the respective component maintainer or committer can update them accordingly later. But the state of Jira should not be marked as "resolved" when the PR is detected, that is not fitting into the resolved semantic I guess. If possible, the Jira can be updated as "in progress" automatically if the respective PR is ready, then it will save some time to stat progress of related issues during release process. Best, Zhijiang -- From:Congxian Qiu Send Time:2020年5月25日(星期一) 13:57 To:dev@flink.apache.org Subject:Re: [DISCUSS] Semantics of our JIRA fields Hi Currently, when I'm going to create an issue for Project-website. I'm not very sure what the "Affect Version/s" should be set. The problem is that the current Dockfile[1] in flink-web is broken because of the EOL of Ubuntu 18.10[2], the project-web does not affect any release of Flink, it does affect the process to build website, so what's the version should I set to? [1] https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 [2] https://wiki.ubuntu.com/Releases Best, Congxian Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > In my experience it's quite complicated for a normal reporter to be able to > fill all the fields correctly (especially for new users). > Usually you just wanto to report a problem, remember to add a new feature > or improve code/documentation but you can't really give a priority, assign > the correct label or decide which releases will benefit of the fix/new > feature. This is something that only core developers could decide (IMHO). > > My experiece says that it's better if normal users could just open tickets > with some default (or mark ticket as new) and leave them in such a state > until an experienced user, one that can assign tickets, have the time to > read it and immediately reject the ticket or accept it and properly assign > priorities, fix version, etc. > > With respect to resolve/close I think that a good practice could be to mark > automatically a ticket as resolved once that a PR is detected for that > ticket, while marking it as closed should be done by the commiter who merge > the PR. > > Probably this process would slightly increase the work of a committer but > this would make things a lot cleaner and will bring the benefit of having a > reliable and semantically sound JIRA state. > > Cheers, > Flavio > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha scritto: > > > +1 for the proposal > > > > This will bring some consistency to the process > > > > Regarding Closed vs Resolved, should we go back and switch all the > Resolved > > issues to Closed so that is is not confusing to new people to the project > > in the future that may not have seen this discussion? > > > > I would recommend changing it to Closed just to be consistent since that > is > > the majority and the new process will be using Closed going forward > > > > Those are my thoughts for now > > > > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu > > wrote: > > > > > +1 for the proposal. It's good to have a unified description and write > it > > > down in the wiki, so that every contributor has the same understanding > of > > > all the fields. > > > > > > Best, > > > Congxian > > > > > > > > > Till Rohrmann 于2020年5月23日周六 上午12:04写道： > > > > > > > Thanks for drafting this proposal Robert. +1 for the proposal. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu > wrote: > > > > > > > > > Thanks bringing up this topic @Robert, +1 to the proposal. > > > > > > > > > > It clarifies the JIRA fields well and should be a rule to follow. > > > > > > > > > > Best, > > > > > Leonard Xu > > > > > > 在 2020年5月22日，20:24，Aljoscha Krettek 写道： > > > > > > > > > > > > +1 That's also how I think of the semantics of the fields. > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > On 22.05.20 08:07, Robert Metzger wrote: > > > > > >> Hi all, > > > > > >> I have the feeling that the semantics of some of our JIRA fields > > > > (mostly > > > > > >> "affects versions", "fix versions" and resolve / close) are not > > > > defined > > > > > in > > > > > >> the same way by all the core Flink contributors, which leads to > > > cases > > > > > where > > > > > >> I spend quite some time on filling the fields correctly (at > least > > > > what
Please update the Requirements of PyPI
Hi PyFlink Developers, I got some errors from the pyarrow depended on apache-flink v1.10.1. It seems updating pyarrow version greater than 0.14 will solve the problem. The version of Flink on PyPI depends on < 0.14, so cause the error. Best, Ray Chen
Re: [DISCUSS] Semantics of our JIRA fields
Hi I like the idea to give clear description of JIRA fields in Flink community when creating tickets. After reading from Robert's explanation, I found my previous understanding differs from the field at "Affect Version/s", and is close to general understanding as JIRA official guide said[1]: Affects version is the version in which a bug or problem was found. If the bug is introduced and found in Flink-1.8.1 and still existing in current release-1.11 branch. I would like to fill in the first version and next major versions, eg. Flink-1.8.1, Flink-1.9.0, Flink-1.10.0 and Flink-1.11.0. Integrated with Robert's explanation, I think a better choice might be filling in the version which brought in and all currently supported and unreleased Flink versions known to be affected by this. I think it's okay to give different explanations on the same field for different communities. If so, I think we should provide this notice in the official web-site, JIRA template or any other place instead of just dev mail list to reach developer/users. [1] https://www.atlassian.com/agile/tutorials/versions Best Yun Tang From: Chesnay Schepler Sent: Monday, May 25, 2020 14:36 To: dev@flink.apache.org ; Congxian Qiu Subject: Re: [DISCUSS] Semantics of our JIRA fields flink-web is independent from any release, so there should be no affects/fix version. On 25/05/2020 07:56, Congxian Qiu wrote: > Hi > > Currently, when I'm going to create an issue for Project-website. I'm not > very sure what the "Affect Version/s" should be set. The problem is that > the current Dockfile[1] in flink-web is broken because of the EOL of Ubuntu > 18.10[2], the project-web does not affect any release of Flink, it does > affect the process to build website, so what's the version should I set to? > > [1] > https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 > [2] https://wiki.ubuntu.com/Releases > > Best, > Congxian > > > Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： > >> In my experience it's quite complicated for a normal reporter to be able to >> fill all the fields correctly (especially for new users). >> Usually you just wanto to report a problem, remember to add a new feature >> or improve code/documentation but you can't really give a priority, assign >> the correct label or decide which releases will benefit of the fix/new >> feature. This is something that only core developers could decide (IMHO). >> >> My experiece says that it's better if normal users could just open tickets >> with some default (or mark ticket as new) and leave them in such a state >> until an experienced user, one that can assign tickets, have the time to >> read it and immediately reject the ticket or accept it and properly assign >> priorities, fix version, etc. >> >> With respect to resolve/close I think that a good practice could be to mark >> automatically a ticket as resolved once that a PR is detected for that >> ticket, while marking it as closed should be done by the commiter who merge >> the PR. >> >> Probably this process would slightly increase the work of a committer but >> this would make things a lot cleaner and will bring the benefit of having a >> reliable and semantically sound JIRA state. >> >> Cheers, >> Flavio >> >> Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha scritto: >> >>> +1 for the proposal >>> >>> This will bring some consistency to the process >>> >>> Regarding Closed vs Resolved, should we go back and switch all the >> Resolved >>> issues to Closed so that is is not confusing to new people to the project >>> in the future that may not have seen this discussion? >>> >>> I would recommend changing it to Closed just to be consistent since that >> is >>> the majority and the new process will be using Closed going forward >>> >>> Those are my thoughts for now >>> >>> On Sat, May 23, 2020 at 7:48 AM Congxian Qiu >>> wrote: >>> +1 for the proposal. It's good to have a unified description and write >> it down in the wiki, so that every contributor has the same understanding >> of all the fields. Best, Congxian Till Rohrmann 于2020年5月23日周六 上午12:04写道： > Thanks for drafting this proposal Robert. +1 for the proposal. > > Cheers, > Till > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu >> wrote: >> Thanks bringing up this topic @Robert, +1 to the proposal. >> >> It clarifies the JIRA fields well and should be a rule to follow. >> >> Best, >> Leonard Xu >>> 在 2020年5月22日，20:24，Aljoscha Krettek 写道： >>> >>> +1 That's also how I think of the semantics of the fields. >>> >>> Aljoscha >>> >>> On 22.05.20 08:07, Robert Metzger wrote: Hi all, I have the feeling that the semantics of some of our JIRA fields > (mostly "affects versions", "fix versions" and resolve / close) are not > defined >> in the
Re: [DISCUSS] Semantics of our JIRA fields
flink-web is independent from any release, so there should be no affects/fix version. On 25/05/2020 07:56, Congxian Qiu wrote: Hi Currently, when I'm going to create an issue for Project-website. I'm not very sure what the "Affect Version/s" should be set. The problem is that the current Dockfile[1] in flink-web is broken because of the EOL of Ubuntu 18.10[2], the project-web does not affect any release of Flink, it does affect the process to build website, so what's the version should I set to? [1] https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17 [2] https://wiki.ubuntu.com/Releases Best, Congxian Flavio Pompermaier 于2020年5月24日周日 下午1:23写道： In my experience it's quite complicated for a normal reporter to be able to fill all the fields correctly (especially for new users). Usually you just wanto to report a problem, remember to add a new feature or improve code/documentation but you can't really give a priority, assign the correct label or decide which releases will benefit of the fix/new feature. This is something that only core developers could decide (IMHO). My experiece says that it's better if normal users could just open tickets with some default (or mark ticket as new) and leave them in such a state until an experienced user, one that can assign tickets, have the time to read it and immediately reject the ticket or accept it and properly assign priorities, fix version, etc. With respect to resolve/close I think that a good practice could be to mark automatically a ticket as resolved once that a PR is detected for that ticket, while marking it as closed should be done by the commiter who merge the PR. Probably this process would slightly increase the work of a committer but this would make things a lot cleaner and will bring the benefit of having a reliable and semantically sound JIRA state. Cheers, Flavio Il Dom 24 Mag 2020, 05:05 Israel Ekpo ha scritto: +1 for the proposal This will bring some consistency to the process Regarding Closed vs Resolved, should we go back and switch all the Resolved issues to Closed so that is is not confusing to new people to the project in the future that may not have seen this discussion? I would recommend changing it to Closed just to be consistent since that is the majority and the new process will be using Closed going forward Those are my thoughts for now On Sat, May 23, 2020 at 7:48 AM Congxian Qiu wrote: +1 for the proposal. It's good to have a unified description and write it down in the wiki, so that every contributor has the same understanding of all the fields. Best, Congxian Till Rohrmann 于2020年5月23日周六 上午12:04写道： Thanks for drafting this proposal Robert. +1 for the proposal. Cheers, Till On Fri, May 22, 2020 at 5:39 PM Leonard Xu wrote: Thanks bringing up this topic @Robert, +1 to the proposal. It clarifies the JIRA fields well and should be a rule to follow. Best, Leonard Xu 在 2020年5月22日，20:24，Aljoscha Krettek 写道： +1 That's also how I think of the semantics of the fields. Aljoscha On 22.05.20 08:07, Robert Metzger wrote: Hi all, I have the feeling that the semantics of some of our JIRA fields (mostly "affects versions", "fix versions" and resolve / close) are not defined in the same way by all the core Flink contributors, which leads to cases where I spend quite some time on filling the fields correctly (at least what I consider correctly), and then others changing them again to match their semantics. In an effort to increase our efficiency, and since I'm creating a lot of (test instability-related) tickets these days, I would like to discuss the semantics, come to a conclusion and document this in our Wiki. *Proposal:* *Priority:* "Blocker": needs to be resolved before a release (matched based on fix versions) "Critical": strongly considered before a release other priorities have no practical meaning in Flink. *Component/s:* Primary component relevant for this feature / fix. For test-related issues, add the component the test belongs to (for example "Connectors / Kafka" for Kafka test failures) + "Test". The same applies for documentation tickets. For example, if there's something wrong with the DataStream API, add it to the "API / DataStream" and "Documentation" components. *Affects Version/s:*Only for Bug / Task-type tickets: We list all currently supported and unreleased Flink versions known to be affected by this. Example: If I see a test failure that happens on "master" and "release-1.11", I set "affects version" to "1.12.0" and "1.11.0". *Fix Version/s:* For closed/resolved tickets, this field lists the released Flink versions that contain a fix or feature for the first time. For open tickets, it indicates that a fix / feature should be contained in the listed versions. Only blocker issues can block a release, all other tickets which have "fix version/s" set at the time of a release and are
[jira] [Created] (FLINK-17917) ResourceInformationReflector#getExternalResources should ignore the external resource with a value of 0
Yangze Guo created FLINK-17917:
--
Summary: ResourceInformationReflector#getExternalResources should
ignore the external resource with a value of 0
Key: FLINK-17917
URL: https://issues.apache.org/jira/browse/FLINK-17917
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Yangze Guo
Fix For: 1.11.0
*Background*: In FLINK-17390, we leverage
{{WorkerSpecContainerResourceAdapter.InternalContainerResource}} to handle
container matching logic. In FLINK-17407, we introduce external resources in
{{WorkerSpecContainerResourceAdapter.InternalContainerResource}}.
On containers returned by Yarn, we try to get the corresponding worker specs by:
- Convert the container to {{InternalContainerResource}}
- Get the WorkerResourceSpec from {{containerResourceToWorkerSpecs}} map.
Container mismatch could happen in the below scenario:
- Flink does not allocate any external resources, the {{externalResources}} of
{{InternalContainerResource}} is an empty map.
- The returned container contains all the resources (with a value of 0) defined
in Yarn's {{resource-types.xml}}. The {{externalResources}} of
{{InternalContainerResource}} has one or more entries with a value of 0.
- These two {{InternalContainerResource}} do not match.
To solve this problem, we could ignore all the external resources with a value
of 0 in "ResourceInformationReflector#getExternalResources".
cc [~trohrmann] Could you assign this to me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17915) TransitiveClosureITCase>JavaProgramTestBase.testJobWithObjectReuse:113 Error while calling the test program: Could not retrieve JobResult
Robert Metzger created FLINK-17915:
--
Summary:
TransitiveClosureITCase>JavaProgramTestBase.testJobWithObjectReuse:113 Error
while calling the test program: Could not retrieve JobResult
Key: FLINK-17915
URL: https://issues.apache.org/jira/browse/FLINK-17915
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2096=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=45cc9205-bdb7-5b54-63cd-89fdc0983323
{code}
2020-05-25T03:26:08.8891832Z [INFO] Tests run: 3, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 4.287 s - in
org.apache.flink.test.example.java.WordCountSimplePOJOITCase
2020-05-25T03:26:09.6452511Z Could not retrieve JobResult.
2020-05-25T03:26:09.6454291Z
org.apache.flink.runtime.client.JobExecutionException: Could not retrieve
JobResult.
2020-05-25T03:26:09.6482505Zat
org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:673)
2020-05-25T03:26:09.6483671Zat
org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:115)
2020-05-25T03:26:09.6625490Zat
org.apache.flink.examples.java.graph.TransitiveClosureNaive.main(TransitiveClosureNaive.java:120)
2020-05-25T03:26:09.6752644Zat
org.apache.flink.test.example.java.TransitiveClosureITCase.testProgram(TransitiveClosureITCase.java:51)
2020-05-25T03:26:09.6754368Zat
org.apache.flink.test.util.JavaProgramTestBase.testJobWithObjectReuse(JavaProgramTestBase.java:107)
2020-05-25T03:26:09.6756679Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-05-25T03:26:09.6757511Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-05-25T03:26:09.6759607Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-05-25T03:26:09.6760692Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-05-25T03:26:09.6761519Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-05-25T03:26:09.6762382Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-05-25T03:26:09.6763246Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-05-25T03:26:09.6778288Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-05-25T03:26:09.6779479Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-05-25T03:26:09.6780187Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-05-25T03:26:09.6780851Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-05-25T03:26:09.6781843Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-05-25T03:26:09.6782583Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-05-25T03:26:09.6783485Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-25T03:26:09.6784670Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-25T03:26:09.6785320Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-25T03:26:09.6786034Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-25T03:26:09.6786670Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-25T03:26:09.6787550Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-05-25T03:26:09.6788233Zat
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-05-25T03:26:09.6789050Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-05-25T03:26:09.6789698Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-25T03:26:09.6790701Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2020-05-25T03:26:09.6791797Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2020-05-25T03:26:09.6792592Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2020-05-25T03:26:09.6793535Zat
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2020-05-25T03:26:09.6794429Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-05-25T03:26:09.6795279Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-05-25T03:26:09.6796109Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-05-25T03:26:09.6797133Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2020-05-25T03:26:09.6798317Z Caused by:
[jira] [Created] (FLINK-17916) Separate KafkaShuffle read/write to different environments
Yuan Mei created FLINK-17916: Summary: Separate KafkaShuffle read/write to different environments Key: FLINK-17916 URL: https://issues.apache.org/jira/browse/FLINK-17916 Project: Flink Issue Type: Improvement Components: API / DataStream, Connectors / Kafka Affects Versions: 1.11.0 Reporter: Yuan Mei Fix For: 1.12.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17914) HistoryServerTest.testCleanExpiredJob:158->runArchiveExpirationTest:214 expected:<2> but was:<0>
Robert Metzger created FLINK-17914:
--
Summary:
HistoryServerTest.testCleanExpiredJob:158->runArchiveExpirationTest:214
expected:<2> but was:<0>
Key: FLINK-17914
URL: https://issues.apache.org/jira/browse/FLINK-17914
Project: Flink
Issue Type: Bug
Components: Runtime / Web Frontend
Affects Versions: 1.12.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2047=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
{code}
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.697 s
<<< FAILURE! - in org.apache.flink.runtime.webmonitor.history.HistoryServerTest
[ERROR] testCleanExpiredJob[Flink version less than 1.4:
false](org.apache.flink.runtime.webmonitor.history.HistoryServerTest) Time
elapsed: 0.483 s <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.flink.runtime.webmonitor.history.HistoryServerTest.runArchiveExpirationTest(HistoryServerTest.java:214)
at
org.apache.flink.runtime.webmonitor.history.HistoryServerTest.testCleanExpiredJob(HistoryServerTest.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
