Hi all, Thank you all for fixing issues of 1.8.3 release! The VOTE mail thread of the first RC of 1.8.3 has already been brought up. I would appreciate it if you can help to check the release and VOTE for the RC1.
Thanks, Hequn On Wed, Nov 27, 2019 at 11:36 AM Hequn Cheng <chenghe...@gmail.com> wrote: > Hi Jincheng, > > Thanks a lot for your timely help. I'm on my way to the release. > > Best, Hequn > > On Wed, Nov 27, 2019 at 7:36 AM jincheng sun <sunjincheng...@gmail.com> > wrote: > >> Hi Hequn, >> >> Thank you for your great job! Looking forward the first RC of 1.8.3 ! >> BTW: The version of 1.8.4 already created here: >> https://issues.apache.org/jira/projects/FLINK/versions/12346552 >> >> Best, >> Jincheng >> >> Hequn Cheng <chenghe...@gmail.com> 于2019年11月26日周二 下午8:18写道： >> >>> Hi all, >>> >>> I would like to share with you that all blockers are resolved now. If >>> there are no more critical issues, I will create the first RC tomorrow and >>> vote on it directly. >>> Hope everything goes well! >>> >>> Thank you all for the help of fixing, reviewing, driving and discussions! >>> >>> Best, Hequn >>> >>> On Tue, Nov 26, 2019 at 9:27 AM Hequn Cheng <chenghe...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> @Ufuk Celebi <u...@apache.org> Hi, we are very close now. There is one >>>> issue(FLINK-13995 <https://issues.apache.org/jira/browse/FLINK-13995>) >>>> left that I want to double-check with you guys. Once this is done, we can >>>> create the first RC. I already have some minor comments in the PR >>>> <https://github.com/apache/flink/pull/10195>. >>>> >>>> @Zhu Zhu <reed...@gmail.com> Glad to hear that it is not a blocker. >>>> Thank you. >>>> >>>> Best, Hequn >>>> >>>> On Mon, Nov 25, 2019 at 5:43 PM Ufuk Celebi <u...@apache.org> wrote: >>>> >>>>> @Hequn: flink-shaded:9.0 is available in Maven central now. I think >>>>> you can go ahead and create the first RC. :-) >>>>> >>>>> On Mon, Nov 25, 2019 at 7:47 AM Zhu Zhu <reed...@gmail.com> wrote: >>>>> >>>>>> Hi Hequn, >>>>>> >>>>>> Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. >>>>>> Given that this fix is for batch job only and batch is not very good >>>>>> in >>>>>> 1.8, I think it is a not blocker of release 1.8.3. >>>>>> So just don't be blocked by it and feel free to cut the RC when other >>>>>> blocking issues are resolved. >>>>>> >>>>>> Thanks, >>>>>> Zhu Zhu >>>>>> >>>>>> Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： >>>>>> >>>>>> > Hi Zhu Zhu, >>>>>> > >>>>>> > Thanks a lot for letting us know! >>>>>> > We can't cut the first RC right now due to the wait of the >>>>>> flink-shade >>>>>> > release, so go ahead. >>>>>> > >>>>>> > Theoretically, we will cut the first RC of 1.8.3 and vote for it >>>>>> once the >>>>>> > release of flink-shade is done, >>>>>> > but I will try my best to have it in 1.8.3. Hope we can get it on >>>>>> board on >>>>>> > time. :) >>>>>> > >>>>>> > Best, Hequn >>>>>> > >>>>>> > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: >>>>>> > >>>>>> >> Hi Jincheng & Hequn >>>>>> >> >>>>>> >> Thanks for driving the releasing of 1.8.3. >>>>>> >> >>>>>> >> I am now working on FLINK-14735. The fix avoids duplicated input >>>>>> >> checking when scheduling ALL-to-ALL >>>>>> >> connected downstream consumers with ALL input constraints. The >>>>>> duplicated >>>>>> >> checking can cause severe >>>>>> >> performance issues for large scale jobs. So I hope the fix could be >>>>>> >> released with 1.8.3. >>>>>> >> >>>>>> >> The fix is already merged into master, and is now in the process of >>>>>> >> backporting to 1.8. >>>>>> >> >>>>>> >> Thanks, >>>>>> >> Zhu Zhu >>>>>> >> >>>>>> >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： >>>>>> >> >>>>>> >>> Thanks Chesnay. >>>>>> >>> >>>>>> >>> I'm also +1 to release 1.8.3 asap without the changes for the >>>>>> Jackson >>>>>> >>> version bump and leave those for a future release. Realistically, >>>>>> the >>>>>> >>> flink-shaded release will take until mid next week or end of next >>>>>> week. >>>>>> >>> But >>>>>> >>> please correct me if you think that it should not take that long >>>>>> or it's >>>>>> >>> OK >>>>>> >>> to block the 1.8.3 release on the flink-shaded release. >>>>>> >>> >>>>>> >>> – Ufuk >>>>>> >>> >>>>>> >>> >>>>>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler < >>>>>> ches...@apache.org> >>>>>> >>> wrote: >>>>>> >>> >>>>>> >>> > I've kicked off a discussion about the next flink-shaded >>>>>> release, and >>>>>> >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. >>>>>> >>> > >>>>>> >>> > On 15/11/2019 13:54, Hequn Cheng wrote: >>>>>> >>> > > That's great, thank you very much! Ideally, we can kick off >>>>>> the >>>>>> >>> release >>>>>> >>> > > vote for the first RC of 1.8.3 within next week. :) >>>>>> >>> > > >>>>>> >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler < >>>>>> ches...@apache.org >>>>>> >>> > >>>>>> >>> > wrote: >>>>>> >>> > > >>>>>> >>> > >> I'm not aware of any more planned changes to flink-shaded; >>>>>> so we >>>>>> >>> could >>>>>> >>> > >> start the release right away. >>>>>> >>> > >> >>>>>> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >>>>>> >>> > >>> Hi, >>>>>> >>> > >>> >>>>>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in >>>>>> >>> approach >>>>>> >>> > for >>>>>> >>> > >>> 1.8/1.9. >>>>>> >>> > >>> @Ufuk Thank you for the nice summary. >>>>>> >>> > >>> >>>>>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a >>>>>> bit to >>>>>> >>> first >>>>>> >>> > >> do a >>>>>> >>> > >>> flink-shaded release. >>>>>> >>> > >>> BTW, @chesnay when would we plan to release the >>>>>> flink-shaded with >>>>>> >>> > >> upgraded >>>>>> >>> > >>> Jackson? >>>>>> >>> > >>> >>>>>> >>> > >>> Best, Hequn >>>>>> >>> > >>> >>>>>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >>>>>> >>> ches...@apache.org> >>>>>> >>> > >> wrote: >>>>>> >>> > >>>> One small modification: the flink-shaded upgrade does not >>>>>> have to >>>>>> >>> be >>>>>> >>> > >>>> part of the profile; since it is only intended for >>>>>> internal use >>>>>> >>> anyway >>>>>> >>> > >>>> (and thus has limited exposure) we can be pretty sure this >>>>>> doesn't >>>>>> >>> > break >>>>>> >>> > >>>> anything. >>>>>> >>> > >>>> >>>>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >>>>>> >>> > >>>>> Ufuk's summary is correct. >>>>>> >>> > >>>>> >>>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the >>>>>> >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >>>>>> >>> > >>>>> but I have no concerns about this change. >>>>>> >>> > >>>>> >>>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >>>>>> >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to >>>>>> include the >>>>>> >>> > >>>>>> profiles in >>>>>> >>> > >>>>>> 1.8 and 1.9 without changing the default versions >>>>>> (including the >>>>>> >>> > >> default >>>>>> >>> > >>>>>> version of flink-shaded). >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> As far as I can tell, the next steps would be: >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson >>>>>> >>> > >>>>>> 2a) Bump the flink-shaded version by default in master >>>>>> >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in >>>>>> profiles >>>>>> >>> > >>>>>> should also >>>>>> >>> > >>>>>> cover the upgrade to the most recent flink-shaded >>>>>> version) >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> @Chesnay: is this a correct summary? >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an >>>>>> >>> upside, we >>>>>> >>> > >>>>>> might >>>>>> >>> > >>>>>> get some additional feedback until the 1.10 release with >>>>>> these >>>>>> >>> > >>>>>> profiles in >>>>>> >>> > >>>>>> case users make use of them with 1.8/1.9. >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> – Ufuk >>>>>> >>> > >>>>>> >>>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >>>>>> >>> > ches...@apache.org >>>>>> >>> > >>>>>> wrote: >>>>>> >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / >>>>>> 1.9.2; on >>>>>> >>> master >>>>>> >>> > >>>>>>> (and >>>>>> >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >>>>>> >>> > >>>>>>> >>>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we >>>>>> usually do >>>>>> >>> not >>>>>> >>> > >> bump >>>>>> >>> > >>>>>>> dependencies in bugfix releases and b) it's a >>>>>> short-term change >>>>>> >>> > that >>>>>> >>> > >> we >>>>>> >>> > >>>>>>> aren't allowing to mature properly. >>>>>> >>> > >>>>>>> In contrast, the 1.10 release is significantly further >>>>>> away, >>>>>> >>> hence >>>>>> >>> > no >>>>>> >>> > >>>>>>> opt-in. >>>>>> >>> > >>>>>>> >>>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades >>>>>> being more >>>>>> >>> > >> common >>>>>> >>> > >>>>>>> in the future. >>>>>> >>> > >>>>>>> >>>>>> >>> > >>>>>>> We can certainly support every jackson version that >>>>>> fixes these >>>>>> >>> > >>>>>>> vulnerabilities; individual modules can always use a >>>>>> different >>>>>> >>> > >> version >>>>>> >>> > >>>>>>> (that hopefully includes the fixes). >>>>>> >>> > >>>>>>> Ideally of course we'd only be using 1 version, but >>>>>> that may >>>>>> >>> or may >>>>>> >>> > >> not >>>>>> >>> > >>>>>>> be feasible. >>>>>> >>> > >>>>>>> >>>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >>>>>> >>> > >>>>>>>> Hi Chesnay, >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. >>>>>> >>> Really a >>>>>> >>> > >> good >>>>>> >>> > >>>>>> job! >>>>>> >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 >>>>>> >>> > >>>>>>>> I had taken a quick look at the security >>>>>> vulnerabilities, >>>>>> >>> some of >>>>>> >>> > >> them >>>>>> >>> > >>>>>>>> seem can lead to high-security problems, thus from my >>>>>> point of >>>>>> >>> > view, >>>>>> >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, >>>>>> I would >>>>>> >>> like >>>>>> >>> > >> to >>>>>> >>> > >>>>>>>> trust your judgment as you are more professional at >>>>>> this >>>>>> >>> problem. >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9 >>>>>> >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. >>>>>> Another >>>>>> >>> > question >>>>>> >>> > >>>>>>>> here is whether do we plan to support multi jackson >>>>>> versions >>>>>> >>> that >>>>>> >>> > >> have >>>>>> >>> > >>>>>>>> eliminated the security vulnerabilities. If we only >>>>>> plan to >>>>>> >>> > support >>>>>> >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. >>>>>> As an >>>>>> >>> > option, >>>>>> >>> > >>>>>>>> users can downgrade the flink version if meet problems >>>>>> using >>>>>> >>> the >>>>>> >>> > new >>>>>> >>> > >>>>>>>> version. Of course, we will try our best to make the >>>>>> new >>>>>> >>> release >>>>>> >>> > out >>>>>> >>> > >>>>>>>> of question. >>>>>> >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it >>>>>> will >>>>>> >>> make >>>>>> >>> > our >>>>>> >>> > >>>>>>>> build unlikely convergence as more and more build >>>>>> options >>>>>> >>> will be >>>>>> >>> > >>>>>>>> added when we upgrade a commonly used lib like this >>>>>> one. >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> What do you think? >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> Best, Hequn >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >>>>>> >>> > >> ches...@apache.org >>>>>> >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> So here's the state of things: >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> The master of flink-shaded now uses jackson >>>>>> 2.10.1, >>>>>> >>> which >>>>>> >>> > >>>>>>>> eliminates a whole category of security >>>>>> >>> vulnerabilities. >>>>>> >>> > >>>>>>>> The flink master works perfectly fine with that >>>>>> >>> version; >>>>>> >>> > 1.9 >>>>>> >>> > >> will >>>>>> >>> > >>>>>>>> likely do so too and 1.8 would require a minor >>>>>> >>> adjustment. >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> Hence, there may be value in first doing a >>>>>> flink-shaded >>>>>> >>> > >>>>>>>> release so >>>>>> >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 >>>>>> and >>>>>> >>> 1.9.2 . >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> As for other jackson dependencies (coming from >>>>>> calcite, >>>>>> >>> > kafka, >>>>>> >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests >>>>>> of master >>>>>> >>> > >> yesterday >>>>>> >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, >>>>>> and they >>>>>> >>> > >> passed. I >>>>>> >>> > >>>>>>>> will open a PR soon-ish for making this change >>>>>> on >>>>>> >>> master. >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> The question now is whether we want to backport >>>>>> this >>>>>> >>> > change to >>>>>> >>> > >>>>>>>> 1.8/1.9 . >>>>>> >>> > >>>>>>>> Some code paths /may /not be covered by our >>>>>> tests, and >>>>>> >>> > >> transitive >>>>>> >>> > >>>>>>>> jackson users /might /run into issues. >>>>>> >>> > >>>>>>>> Alternatively, we could set this up as an opt-in >>>>>> >>> upgrade, >>>>>> >>> > by >>>>>> >>> > >>>>>>>> adding a separate profile that bumps the >>>>>> versions. This >>>>>> >>> > would >>>>>> >>> > >>>>>>>> present users/providers who are concerned about >>>>>> the >>>>>> >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk >>>>>> of >>>>>> >>> /some >>>>>> >>> > >> /things >>>>>> >>> > >>>>>>>> /maybe /not working. >>>>>> >>> > >>>>>>>> >>>>>> >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >>>>>> >>> > >>>>>>>>> Hi Chesnay, Jincheng >>>>>> >>> > >>>>>>>>> >>>>>> >>> > >>>>>>>>> Sure, I think it's good to have these fixes. >>>>>> >>> > >>>>>>>>> Thanks a lot for providing the information >>>>>> about the >>>>>> >>> > security >>>>>> >>> > >>>>>>>>> vulnerabilities! @Chesnay >>>>>> >>> > >>>>>>>>> >>>>>> >>> > >>>>>>>>> Best, Hequn >>>>>> >>> > >>>>>>>>> >>>>>> >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >>>>>> >>> > >>>>>> sunjincheng...@gmail.com> <mailto: >>>>>> sunjincheng...@gmail.com> >>>>>> >>> > >>>>>>>>> wrote: >>>>>> >>> > >>>>>>>>> >>>>>> >>> > >>>>>>>>>> +1 for try to eliminate the security >>>>>> vulnerabilities. >>>>>> >>> > Great >>>>>> >>> > >>>>>> thanks for >>>>>> >>> > >>>>>>>>>> doing this important work, Chesnay! >>>>>> >>> > >>>>>>>>>> What do you think Hequn ? >>>>>> >>> > >>>>>>>>>> >>>>>> >>> > >>>>>>>>>> Best, >>>>>> >>> > >>>>>>>>>> Jincheng >>>>>> >>> > >>>>>>>>>> >>>>>> >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >>>>>> >>> > >>>>>>>>>> <mailto:ches...@apache.org> >>>>>> >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： >>>>>> >>> > >>>>>>>>>>> It would be great if you could give me a day >>>>>> or 2 to >>>>>> >>> > check >>>>>> >>> > >> how >>>>>> >>> > >>>>>> easy it >>>>>> >>> > >>>>>>>>>>> would be to bump the various jackson >>>>>> dependencies to >>>>>> >>> > >>>>>>>>>>> eliminate a >>>>>> >>> > >>>>>> few >>>>>> >>> > >>>>>>>>>>> security vulnerabilities. >>>>>> >>> > >>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >>>>>> >>> > >>>>>>>>>>>> Hi Flink devs, >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> It has been more than 2 months since the >>>>>> 1.8.2 >>>>>> >>> > released. >>>>>> >>> > >> So, >>>>>> >>> > >>>>>> What do >>>>>> >>> > >>>>>>>>>> you >>>>>> >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> We already have many important bug fixes in >>>>>> the >>>>>> >>> > >> release-1.8 >>>>>> >>> > >>>>>> branch (29 >>>>>> >>> > >>>>>>>>>>>> resolved issues). >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> Most notable fixes are: >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers >>>>>> don't give >>>>>> >>> up >>>>>> >>> > >>>>>>>>>>>> leadership >>>>>> >>> > >>>>>> when AM >>>>>> >>> > >>>>>>>>>>> is >>>>>> >>> > >>>>>>>>>>>> shut down >>>>>> >>> > >>>>>>>>>>>> - FLINK-14315 NPE with >>>>>> >>> JobMaster.disconnectTaskManager >>>>>> >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in >>>>>> RowTypeInfo should >>>>>> >>> > >> consider >>>>>> >>> > >>>>>>>>>> fieldsNames >>>>>> >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager >>>>>> Acquires Too >>>>>> >>> Many >>>>>> >>> > >>>>>>>>>>>> Containers >>>>>> >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with >>>>>> the same >>>>>> >>> > >>>>>> AllocationID leads >>>>>> >>> > >>>>>>>>>> to >>>>>> >>> > >>>>>>>>>>>> inconsistent slot table >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> Furthermore, the following critical issues >>>>>> is in >>>>>> >>> > progress, >>>>>> >>> > >>>>>> maybe we can >>>>>> >>> > >>>>>>>>>>>> wait for it if it is not too much effort. >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor >>>>>> blocks the >>>>>> >>> > >>>>>> YarnResourceManager's >>>>>> >>> > >>>>>>>>>>> main >>>>>> >>> > >>>>>>>>>>>> thread >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> Please let me know what you think? >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >>>>>>>>>>>> Best, >>>>>> >>> > >>>>>>>>>>>> Jincheng >>>>>> >>> > >>>>>>>>>>>> >>>>>> >>> > >> >>>>>> >>> > >>>>>> >>> > >>>>>> >>> >>>>>> >> >>>>>> >>>>>
