Hi Hequn, Looks we are not able to merge fix of FLINK-14735 to 1.8 very soon. Given that this fix is for batch job only and batch is not very good in 1.8, I think it is a not blocker of release 1.8.3. So just don't be blocked by it and feel free to cut the RC when other blocking issues are resolved.
Thanks, Zhu Zhu Hequn Cheng <chenghe...@gmail.com> 于2019年11月23日周六 下午9:08写道： > Hi Zhu Zhu, > > Thanks a lot for letting us know! > We can't cut the first RC right now due to the wait of the flink-shade > release, so go ahead. > > Theoretically, we will cut the first RC of 1.8.3 and vote for it once the > release of flink-shade is done, > but I will try my best to have it in 1.8.3. Hope we can get it on board on > time. :) > > Best, Hequn > > On Sat, Nov 23, 2019 at 10:40 AM Zhu Zhu <reed...@gmail.com> wrote: > >> Hi Jincheng & Hequn >> >> Thanks for driving the releasing of 1.8.3. >> >> I am now working on FLINK-14735. The fix avoids duplicated input >> checking when scheduling ALL-to-ALL >> connected downstream consumers with ALL input constraints. The duplicated >> checking can cause severe >> performance issues for large scale jobs. So I hope the fix could be >> released with 1.8.3. >> >> The fix is already merged into master, and is now in the process of >> backporting to 1.8. >> >> Thanks, >> Zhu Zhu >> >> Ufuk Celebi <u...@apache.org> 于2019年11月15日周五 下午11:54写道： >> >>> Thanks Chesnay. >>> >>> I'm also +1 to release 1.8.3 asap without the changes for the Jackson >>> version bump and leave those for a future release. Realistically, the >>> flink-shaded release will take until mid next week or end of next week. >>> But >>> please correct me if you think that it should not take that long or it's >>> OK >>> to block the 1.8.3 release on the flink-shaded release. >>> >>> – Ufuk >>> >>> >>> On Fri, Nov 15, 2019 at 2:27 PM Chesnay Schepler <ches...@apache.org> >>> wrote: >>> >>> > I've kicked off a discussion about the next flink-shaded release, and >>> > have opened PRs for adding the opt-in profile to 1.8/1.9. >>> > >>> > On 15/11/2019 13:54, Hequn Cheng wrote: >>> > > That's great, thank you very much! Ideally, we can kick off the >>> release >>> > > vote for the first RC of 1.8.3 within next week. :) >>> > > >>> > > On Fri, Nov 15, 2019 at 8:47 PM Chesnay Schepler <ches...@apache.org >>> > >>> > wrote: >>> > > >>> > >> I'm not aware of any more planned changes to flink-shaded; so we >>> could >>> > >> start the release right away. >>> > >> >>> > >> On 15/11/2019 13:44, Hequn Cheng wrote: >>> > >>> Hi, >>> > >>> >>> > >>> @Chesnay Thanks a lot for the explanation. +1 to the opt-in >>> approach >>> > for >>> > >>> 1.8/1.9. >>> > >>> @Ufuk Thank you for the nice summary. >>> > >>> >>> > >>> Looks good so far except that we need to postpone 1.8.3 a bit to >>> first >>> > >> do a >>> > >>> flink-shaded release. >>> > >>> BTW, @chesnay when would we plan to release the flink-shaded with >>> > >> upgraded >>> > >>> Jackson? >>> > >>> >>> > >>> Best, Hequn >>> > >>> >>> > >>> On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler < >>> ches...@apache.org> >>> > >> wrote: >>> > >>>> One small modification: the flink-shaded upgrade does not have to >>> be >>> > >>>> part of the profile; since it is only intended for internal use >>> anyway >>> > >>>> (and thus has limited exposure) we can be pretty sure this doesn't >>> > break >>> > >>>> anything. >>> > >>>> >>> > >>>> On 15/11/2019 12:23, Chesnay Schepler wrote: >>> > >>>>> Ufuk's summary is correct. >>> > >>>>> >>> > >>>>> There's a slight caveat in that we'd also have to bump the >>> > >>>>> shade-plugin to 3.1.1 since it otherwise fails on jackson, >>> > >>>>> but I have no concerns about this change. >>> > >>>>> >>> > >>>>> On 15/11/2019 12:19, Ufuk Celebi wrote: >>> > >>>>>> The opt-in approach seems reasonable to me. +1 to include the >>> > >>>>>> profiles in >>> > >>>>>> 1.8 and 1.9 without changing the default versions (including the >>> > >> default >>> > >>>>>> version of flink-shaded). >>> > >>>>>> >>> > >>>>>> As far as I can tell, the next steps would be: >>> > >>>>>> >>> > >>>>>> 1) Release flink-shaded with upgraded Jackson >>> > >>>>>> 2a) Bump the flink-shaded version by default in master >>> > >>>>>> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles >>> > >>>>>> should also >>> > >>>>>> cover the upgrade to the most recent flink-shaded version) >>> > >>>>>> >>> > >>>>>> @Chesnay: is this a correct summary? >>> > >>>>>> >>> > >>>>>> Note this would block the 1.8.3 release on step 1. As an >>> upside, we >>> > >>>>>> might >>> > >>>>>> get some additional feedback until the 1.10 release with these >>> > >>>>>> profiles in >>> > >>>>>> case users make use of them with 1.8/1.9. >>> > >>>>>> >>> > >>>>>> – Ufuk >>> > >>>>>> >>> > >>>>>> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler < >>> > ches...@apache.org >>> > >>>>>> wrote: >>> > >>>>>>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on >>> master >>> > >>>>>>> (and >>> > >>>>>>> thus starting from 1.10.0) it's not opt-in. >>> > >>>>>>> >>> > >>>>>>> I have only proposed it as an opt-in because a) we usually do >>> not >>> > >> bump >>> > >>>>>>> dependencies in bugfix releases and b) it's a short-term change >>> > that >>> > >> we >>> > >>>>>>> aren't allowing to mature properly. >>> > >>>>>>> In contrast, the 1.10 release is significantly further away, >>> hence >>> > no >>> > >>>>>>> opt-in. >>> > >>>>>>> >>> > >>>>>>> Hence, I'm not concerned about such kind of ugprades being more >>> > >> common >>> > >>>>>>> in the future. >>> > >>>>>>> >>> > >>>>>>> We can certainly support every jackson version that fixes these >>> > >>>>>>> vulnerabilities; individual modules can always use a different >>> > >> version >>> > >>>>>>> (that hopefully includes the fixes). >>> > >>>>>>> Ideally of course we'd only be using 1 version, but that may >>> or may >>> > >> not >>> > >>>>>>> be feasible. >>> > >>>>>>> >>> > >>>>>>> On 15/11/2019 04:07, Hequn Cheng wrote: >>> > >>>>>>>> Hi Chesnay, >>> > >>>>>>>> >>> > >>>>>>>> Great to hear that jackson-2.10.1 works well on master. >>> Really a >>> > >> good >>> > >>>>>> job! >>> > >>>>>>>> - Whether backport this change to 1.8/1.9 >>> > >>>>>>>> I had taken a quick look at the security vulnerabilities, >>> some of >>> > >> them >>> > >>>>>>>> seem can lead to high-security problems, thus from my point of >>> > view, >>> > >>>>>>>> I'm in favor of adding the fix into 1.9/1.8. However, I would >>> like >>> > >> to >>> > >>>>>>>> trust your judgment as you are more professional at this >>> problem. >>> > >>>>>>>> >>> > >>>>>>>> - How to port this change to 1.8/1.9 >>> > >>>>>>>> I think providing an opt-in upgrade is a good idea. Another >>> > question >>> > >>>>>>>> here is whether do we plan to support multi jackson versions >>> that >>> > >> have >>> > >>>>>>>> eliminated the security vulnerabilities. If we only plan to >>> > support >>> > >>>>>>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an >>> > option, >>> > >>>>>>>> users can downgrade the flink version if meet problems using >>> the >>> > new >>> > >>>>>>>> version. Of course, we will try our best to make the new >>> release >>> > out >>> > >>>>>>>> of question. >>> > >>>>>>>> Another concern of making it an opt-in upgrade is, it will >>> make >>> > our >>> > >>>>>>>> build unlikely convergence as more and more build options >>> will be >>> > >>>>>>>> added when we upgrade a commonly used lib like this one. >>> > >>>>>>>> >>> > >>>>>>>> What do you think? >>> > >>>>>>>> >>> > >>>>>>>> Best, Hequn >>> > >>>>>>>> >>> > >>>>>>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler < >>> > >> ches...@apache.org >>> > >>>>>>>> <mailto:ches...@apache.org>> wrote: >>> > >>>>>>>> >>> > >>>>>>>> So here's the state of things: >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> The master of flink-shaded now uses jackson 2.10.1, >>> which >>> > >>>>>>>> eliminates a whole category of security >>> vulnerabilities. >>> > >>>>>>>> The flink master works perfectly fine with that >>> version; >>> > 1.9 >>> > >> will >>> > >>>>>>>> likely do so too and 1.8 would require a minor >>> adjustment. >>> > >>>>>>>> >>> > >>>>>>>> Hence, there may be value in first doing a flink-shaded >>> > >>>>>>>> release so >>> > >>>>>>>> we can eliminate these vulnerabilities in 1.8.3 and >>> 1.9.2 . >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> As for other jackson dependencies (coming from calcite, >>> > kafka, >>> > >>>>>>>> kinesis), I ran the unit and end-to-end tests of master >>> > >> yesterday >>> > >>>>>>>> will /all /jackson dependencies set to 2.10.1, and they >>> > >> passed. I >>> > >>>>>>>> will open a PR soon-ish for making this change on >>> master. >>> > >>>>>>>> >>> > >>>>>>>> The question now is whether we want to backport this >>> > change to >>> > >>>>>>>> 1.8/1.9 . >>> > >>>>>>>> Some code paths /may /not be covered by our tests, and >>> > >> transitive >>> > >>>>>>>> jackson users /might /run into issues. >>> > >>>>>>>> Alternatively, we could set this up as an opt-in >>> upgrade, >>> > by >>> > >>>>>>>> adding a separate profile that bumps the versions. This >>> > would >>> > >>>>>>>> present users/providers who are concerned about the >>> > >>>>>>>> vulnerabilities an easy workaround, at the risk of >>> /some >>> > >> /things >>> > >>>>>>>> /maybe /not working. >>> > >>>>>>>> >>> > >>>>>>>> On 14/11/2019 03:16, Hequn Cheng wrote: >>> > >>>>>>>>> Hi Chesnay, Jincheng >>> > >>>>>>>>> >>> > >>>>>>>>> Sure, I think it's good to have these fixes. >>> > >>>>>>>>> Thanks a lot for providing the information about the >>> > security >>> > >>>>>>>>> vulnerabilities! @Chesnay >>> > >>>>>>>>> >>> > >>>>>>>>> Best, Hequn >>> > >>>>>>>>> >>> > >>>>>>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< >>> > >>>>>> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> >>> > >>>>>>>>> wrote: >>> > >>>>>>>>> >>> > >>>>>>>>>> +1 for try to eliminate the security vulnerabilities. >>> > Great >>> > >>>>>> thanks for >>> > >>>>>>>>>> doing this important work, Chesnay! >>> > >>>>>>>>>> What do you think Hequn ? >>> > >>>>>>>>>> >>> > >>>>>>>>>> Best, >>> > >>>>>>>>>> Jincheng >>> > >>>>>>>>>> >>> > >>>>>>>>>> Chesnay Schepler<ches...@apache.org> >>> > >>>>>>>>>> <mailto:ches...@apache.org> >>> > >>>>>> 于2019年11月13日周三 下午5:17写道： >>> > >>>>>>>>>>> It would be great if you could give me a day or 2 to >>> > check >>> > >> how >>> > >>>>>> easy it >>> > >>>>>>>>>>> would be to bump the various jackson dependencies to >>> > >>>>>>>>>>> eliminate a >>> > >>>>>> few >>> > >>>>>>>>>>> security vulnerabilities. >>> > >>>>>>>>>>> >>> > >>>>>>>>>>> On 09/11/2019 05:10, jincheng sun wrote: >>> > >>>>>>>>>>>> Hi Flink devs, >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> It has been more than 2 months since the 1.8.2 >>> > released. >>> > >> So, >>> > >>>>>> What do >>> > >>>>>>>>>> you >>> > >>>>>>>>>>>> think about releasing Flink 1.8.3 soon? >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> We already have many important bug fixes in the >>> > >> release-1.8 >>> > >>>>>> branch (29 >>> > >>>>>>>>>>>> resolved issues). >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> Most notable fixes are: >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give >>> up >>> > >>>>>>>>>>>> leadership >>> > >>>>>> when AM >>> > >>>>>>>>>>> is >>> > >>>>>>>>>>>> shut down >>> > >>>>>>>>>>>> - FLINK-14315 NPE with >>> JobMaster.disconnectTaskManager >>> > >>>>>>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should >>> > >> consider >>> > >>>>>>>>>> fieldsNames >>> > >>>>>>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too >>> Many >>> > >>>>>>>>>>>> Containers >>> > >>>>>>>>>>>> - FLINK-14589 Redundant slot requests with the same >>> > >>>>>> AllocationID leads >>> > >>>>>>>>>> to >>> > >>>>>>>>>>>> inconsistent slot table >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> Furthermore, the following critical issues is in >>> > progress, >>> > >>>>>> maybe we can >>> > >>>>>>>>>>>> wait for it if it is not too much effort. >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the >>> > >>>>>> YarnResourceManager's >>> > >>>>>>>>>>> main >>> > >>>>>>>>>>>> thread >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> Please let me know what you think? >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> Best, >>> > >>>>>>>>>>>> Jincheng >>> > >>>>>>>>>>>> >>> > >> >>> > >>> > >>> >>
