Hi, @Chesnay Thanks a lot for the explanation. +1 to the opt-in approach for 1.8/1.9. @Ufuk Thank you for the nice summary.
Looks good so far except that we need to postpone 1.8.3 a bit to first do a flink-shaded release. BTW, @chesnay when would we plan to release the flink-shaded with upgraded Jackson? Best, Hequn On Fri, Nov 15, 2019 at 7:43 PM Chesnay Schepler <ches...@apache.org> wrote: > One small modification: the flink-shaded upgrade does not have to be > part of the profile; since it is only intended for internal use anyway > (and thus has limited exposure) we can be pretty sure this doesn't break > anything. > > On 15/11/2019 12:23, Chesnay Schepler wrote: > > Ufuk's summary is correct. > > > > There's a slight caveat in that we'd also have to bump the > > shade-plugin to 3.1.1 since it otherwise fails on jackson, > > but I have no concerns about this change. > > > > On 15/11/2019 12:19, Ufuk Celebi wrote: > >> The opt-in approach seems reasonable to me. +1 to include the > >> profiles in > >> 1.8 and 1.9 without changing the default versions (including the default > >> version of flink-shaded). > >> > >> As far as I can tell, the next steps would be: > >> > >> 1) Release flink-shaded with upgraded Jackson > >> 2a) Bump the flink-shaded version by default in master > >> 2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles > >> should also > >> cover the upgrade to the most recent flink-shaded version) > >> > >> @Chesnay: is this a correct summary? > >> > >> Note this would block the 1.8.3 release on step 1. As an upside, we > >> might > >> get some additional feedback until the 1.10 release with these > >> profiles in > >> case users make use of them with 1.8/1.9. > >> > >> – Ufuk > >> > >> On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <ches...@apache.org> > >> wrote: > >>> The opt-in approach would only be used for 1.8.3 / 1.9.2; on master > >>> (and > >>> thus starting from 1.10.0) it's not opt-in. > >>> > >>> I have only proposed it as an opt-in because a) we usually do not bump > >>> dependencies in bugfix releases and b) it's a short-term change that we > >>> aren't allowing to mature properly. > >>> In contrast, the 1.10 release is significantly further away, hence no > >>> opt-in. > >>> > >>> Hence, I'm not concerned about such kind of ugprades being more common > >>> in the future. > >>> > >>> We can certainly support every jackson version that fixes these > >>> vulnerabilities; individual modules can always use a different version > >>> (that hopefully includes the fixes). > >>> Ideally of course we'd only be using 1 version, but that may or may not > >>> be feasible. > >>> > >>> On 15/11/2019 04:07, Hequn Cheng wrote: > >>>> Hi Chesnay, > >>>> > >>>> Great to hear that jackson-2.10.1 works well on master. Really a good > >> job! > >>>> - Whether backport this change to 1.8/1.9 > >>>> I had taken a quick look at the security vulnerabilities, some of them > >>>> seem can lead to high-security problems, thus from my point of view, > >>>> I'm in favor of adding the fix into 1.9/1.8. However, I would like to > >>>> trust your judgment as you are more professional at this problem. > >>>> > >>>> - How to port this change to 1.8/1.9 > >>>> I think providing an opt-in upgrade is a good idea. Another question > >>>> here is whether do we plan to support multi jackson versions that have > >>>> eliminated the security vulnerabilities. If we only plan to support > >>>> 2.10.1, I would like to make it a non-opt-in upgrade. As an option, > >>>> users can downgrade the flink version if meet problems using the new > >>>> version. Of course, we will try our best to make the new release out > >>>> of question. > >>>> Another concern of making it an opt-in upgrade is, it will make our > >>>> build unlikely convergence as more and more build options will be > >>>> added when we upgrade a commonly used lib like this one. > >>>> > >>>> What do you think? > >>>> > >>>> Best, Hequn > >>>> > >>>> On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <ches...@apache.org > >>>> <mailto:ches...@apache.org>> wrote: > >>>> > >>>> So here's the state of things: > >>>> > >>>> > >>>> The master of flink-shaded now uses jackson 2.10.1, which > >>>> eliminates a whole category of security vulnerabilities. > >>>> The flink master works perfectly fine with that version; 1.9 will > >>>> likely do so too and 1.8 would require a minor adjustment. > >>>> > >>>> Hence, there may be value in first doing a flink-shaded > >>>> release so > >>>> we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 . > >>>> > >>>> > >>>> As for other jackson dependencies (coming from calcite, kafka, > >>>> kinesis), I ran the unit and end-to-end tests of master yesterday > >>>> will /all /jackson dependencies set to 2.10.1, and they passed. I > >>>> will open a PR soon-ish for making this change on master. > >>>> > >>>> The question now is whether we want to backport this change to > >>>> 1.8/1.9 . > >>>> Some code paths /may /not be covered by our tests, and transitive > >>>> jackson users /might /run into issues. > >>>> Alternatively, we could set this up as an opt-in upgrade, by > >>>> adding a separate profile that bumps the versions. This would > >>>> present users/providers who are concerned about the > >>>> vulnerabilities an easy workaround, at the risk of /some /things > >>>> /maybe /not working. > >>>> > >>>> On 14/11/2019 03:16, Hequn Cheng wrote: > >>>>> Hi Chesnay, Jincheng > >>>>> > >>>>> Sure, I think it's good to have these fixes. > >>>>> Thanks a lot for providing the information about the security > >>>>> vulnerabilities! @Chesnay > >>>>> > >>>>> Best, Hequn > >>>>> > >>>>> On Thu, Nov 14, 2019 at 10:07 AM jincheng sun< > >> sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> +1 for try to eliminate the security vulnerabilities. Great > >> thanks for > >>>>>> doing this important work, Chesnay! > >>>>>> What do you think Hequn ? > >>>>>> > >>>>>> Best, > >>>>>> Jincheng > >>>>>> > >>>>>> Chesnay Schepler<ches...@apache.org> > >>>>>> <mailto:ches...@apache.org> > >> 于2019年11月13日周三 下午5:17写道： > >>>>>>> It would be great if you could give me a day or 2 to check how > >> easy it > >>>>>>> would be to bump the various jackson dependencies to > >>>>>>> eliminate a > >> few > >>>>>>> security vulnerabilities. > >>>>>>> > >>>>>>> On 09/11/2019 05:10, jincheng sun wrote: > >>>>>>>> Hi Flink devs, > >>>>>>>> > >>>>>>>> It has been more than 2 months since the 1.8.2 released. So, > >> What do > >>>>>> you > >>>>>>>> think about releasing Flink 1.8.3 soon? > >>>>>>>> > >>>>>>>> We already have many important bug fixes in the release-1.8 > >> branch (29 > >>>>>>>> resolved issues). > >>>>>>>> > >>>>>>>> Most notable fixes are: > >>>>>>>> > >>>>>>>> - FLINK-14010 Dispatcher & JobManagers don't give up > >>>>>>>> leadership > >> when AM > >>>>>>> is > >>>>>>>> shut down > >>>>>>>> - FLINK-14315 NPE with JobMaster.disconnectTaskManager > >>>>>>>> - FLINK-12848 Method equals() in RowTypeInfo should consider > >>>>>> fieldsNames > >>>>>>>> - FLINK-12342 Yarn Resource Manager Acquires Too Many > >>>>>>>> Containers > >>>>>>>> - FLINK-14589 Redundant slot requests with the same > >> AllocationID leads > >>>>>> to > >>>>>>>> inconsistent slot table > >>>>>>>> > >>>>>>>> Furthermore, the following critical issues is in progress, > >> maybe we can > >>>>>>>> wait for it if it is not too much effort. > >>>>>>>> > >>>>>>>> - FLINK-13184 Starting a TaskExecutor blocks the > >> YarnResourceManager's > >>>>>>> main > >>>>>>>> thread > >>>>>>>> > >>>>>>>> Please let me know what you think? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Jincheng > >>>>>>>> > > > > > >
