Ufuk's summary is correct.
There's a slight caveat in that we'd also have to bump the shade-plugin
to 3.1.1 since it otherwise fails on jackson,
but I have no concerns about this change.
On 15/11/2019 12:19, Ufuk Celebi wrote:
The opt-in approach seems reasonable to me. +1 to include the profiles in
1.8 and 1.9 without changing the default versions (including the default
version of flink-shaded).
As far as I can tell, the next steps would be:
1) Release flink-shaded with upgraded Jackson
2a) Bump the flink-shaded version by default in master
2b) Create opt-in profiles for 1.8 and 1.9 (the opt-in profiles should also
cover the upgrade to the most recent flink-shaded version)
@Chesnay: is this a correct summary?
Note this would block the 1.8.3 release on step 1. As an upside, we might
get some additional feedback until the 1.10 release with these profiles in
case users make use of them with 1.8/1.9.
– Ufuk
On Fri, Nov 15, 2019 at 12:08 PM Chesnay Schepler <ches...@apache.org>
wrote:
The opt-in approach would only be used for 1.8.3 / 1.9.2; on master (and
thus starting from 1.10.0) it's not opt-in.
I have only proposed it as an opt-in because a) we usually do not bump
dependencies in bugfix releases and b) it's a short-term change that we
aren't allowing to mature properly.
In contrast, the 1.10 release is significantly further away, hence no
opt-in.
Hence, I'm not concerned about such kind of ugprades being more common
in the future.
We can certainly support every jackson version that fixes these
vulnerabilities; individual modules can always use a different version
(that hopefully includes the fixes).
Ideally of course we'd only be using 1 version, but that may or may not
be feasible.
On 15/11/2019 04:07, Hequn Cheng wrote:
Hi Chesnay,
Great to hear that jackson-2.10.1 works well on master. Really a good
job!
- Whether backport this change to 1.8/1.9
I had taken a quick look at the security vulnerabilities, some of them
seem can lead to high-security problems, thus from my point of view,
I'm in favor of adding the fix into 1.9/1.8. However, I would like to
trust your judgment as you are more professional at this problem.
- How to port this change to 1.8/1.9
I think providing an opt-in upgrade is a good idea. Another question
here is whether do we plan to support multi jackson versions that have
eliminated the security vulnerabilities. If we only plan to support
2.10.1, I would like to make it a non-opt-in upgrade. As an option,
users can downgrade the flink version if meet problems using the new
version. Of course, we will try our best to make the new release out
of question.
Another concern of making it an opt-in upgrade is, it will make our
build unlikely convergence as more and more build options will be
added when we upgrade a commonly used lib like this one.
What do you think?
Best, Hequn
On Thu, Nov 14, 2019 at 6:00 PM Chesnay Schepler <ches...@apache.org
<mailto:ches...@apache.org>> wrote:
So here's the state of things:
The master of flink-shaded now uses jackson 2.10.1, which
eliminates a whole category of security vulnerabilities.
The flink master works perfectly fine with that version; 1.9 will
likely do so too and 1.8 would require a minor adjustment.
Hence, there may be value in first doing a flink-shaded release so
we can eliminate these vulnerabilities in 1.8.3 and 1.9.2 .
As for other jackson dependencies (coming from calcite, kafka,
kinesis), I ran the unit and end-to-end tests of master yesterday
will /all /jackson dependencies set to 2.10.1, and they passed. I
will open a PR soon-ish for making this change on master.
The question now is whether we want to backport this change to
1.8/1.9 .
Some code paths /may /not be covered by our tests, and transitive
jackson users /might /run into issues.
Alternatively, we could set this up as an opt-in upgrade, by
adding a separate profile that bumps the versions. This would
present users/providers who are concerned about the
vulnerabilities an easy workaround, at the risk of /some /things
/maybe /not working.
On 14/11/2019 03:16, Hequn Cheng wrote:
Hi Chesnay, Jincheng
Sure, I think it's good to have these fixes.
Thanks a lot for providing the information about the security
vulnerabilities! @Chesnay
Best, Hequn
On Thu, Nov 14, 2019 at 10:07 AM jincheng sun<
sunjincheng...@gmail.com> <mailto:sunjincheng...@gmail.com>
wrote:
+1 for try to eliminate the security vulnerabilities. Great
thanks for
doing this important work, Chesnay!
What do you think Hequn ?
Best,
Jincheng
Chesnay Schepler<ches...@apache.org> <mailto:ches...@apache.org>
于2019年11月13日周三 下午5:17写道：
It would be great if you could give me a day or 2 to check how
easy it
would be to bump the various jackson dependencies to eliminate a
few
security vulnerabilities.
On 09/11/2019 05:10, jincheng sun wrote:
Hi Flink devs,
It has been more than 2 months since the 1.8.2 released. So,
What do
you
think about releasing Flink 1.8.3 soon?
We already have many important bug fixes in the release-1.8
branch (29
resolved issues).
Most notable fixes are:
- FLINK-14010 Dispatcher & JobManagers don't give up leadership
when AM
is
shut down
- FLINK-14315 NPE with JobMaster.disconnectTaskManager
- FLINK-12848 Method equals() in RowTypeInfo should consider
fieldsNames
- FLINK-12342 Yarn Resource Manager Acquires Too Many Containers
- FLINK-14589 Redundant slot requests with the same
AllocationID leads
to
inconsistent slot table
Furthermore, the following critical issues is in progress,
maybe we can
wait for it if it is not too much effort.
- FLINK-13184 Starting a TaskExecutor blocks the
YarnResourceManager's
main
thread
Please let me know what you think?
Best,
Jincheng
