Today's update: several changes related to the CVEs have been merged, along with a bugfix for Parquet. Thanks to all of you who helped on those changes. I believe there's only one Parquet change left for DRILL-7934: <https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this correct?
Laurent On Thu, May 27, 2021 at 10:48 AM Laurent Goujon <[email protected]> wrote: > Some fixes/improvements were made to the codebase since the last release, > and sadly an official release is needed to pick up those changes. Ray asked > the community more than a month ago. More recently, other people have been > asking too on the user mailing list. > > Like I said, it might be okay to change the scope but what I'm asking is a > little help/transparency here because it looks like I'm chasing a moving > target. If we can clarify which new issues have to be part of the release > and why (depending on the severity), and how long we think it will take, > I'd hope we can have some constructive discussion. > > As for the dependencies change: > - I actually wrote a pull request to address CVEs in both Hadoop and Jetty > - The Guava change will not address the most recent CVE. To address the > CVE, code must be changed, and it doesn't require a Guava update. The > change made to the Guava library was to deprecate the unsecure method... So > imho updating dependencies to address CVE without looking at the CVE itself > does not make things safer. So to address specifically the CVE, I opened a > new ticket (DRILL-7936 <https://issues.apache.org/jira/browse/DRILL-7936>) > and a pull request (https://github.com/apache/drill/pull/2240) > > > On Thu, May 27, 2021 at 9:30 AM Charles Givre <[email protected]> wrote: > >> Hi Laurent, >> I’m not sure what the rush is to get a release out. I would much rather >> do a quality release than just get something out the door for the sake of >> getting something out the door. >> >> In reference to Drill-7934 (Parquet), DRILL-7919 I am personally not in >> favor of putting out a release with known bugs, especially when these bugs >> affect parts of Drill that are in active use, we don’t do releases that >> frequently, and there is a PR that is awaiting merge. >> >> I’m also not in favor of a release that has known issues with >> dependencies, especially again when there are pending PRs that address >> these CVEs. If we did more frequent releases (which we have discussed and >> hope to do going forward), then fine, but we’ve been averaging 2 a year and >> I’d hate for users to have to wait 6 months for these fixes. >> >> — C >> >> >> >> > On May 27, 2021, at 12:19 PM, Laurent Goujon <[email protected]> >> wrote: >> > >> > Since I'm also a reviewer and that I see that the past comments I've >> been >> > addressed, and since I do not see another committer opposing the patch, >> > wouldn't I be able to give my +1 and that would clear that bar? >> > >> > As for the parquet issues, when we started the release discussion a >> month >> > ago, we agreed on a scope, and the parquet issues were not part of it. I >> > understand that scope can change but can we discuss it in this thread >> about >> > why this release should include it vs wait on the next release? We need >> to >> > draw a line somewhere. >> > >> > Laurent >> > >> > On Thu, May 27, 2021 at 8:05 AM Charles Givre <[email protected]> wrote: >> > >> >> Laurent, >> >> Per Apache policy, you need a +1 from a reviewer to merge a PR. Unless >> >> there is one, please do not merge. I'll reach out to Vitalii to see >> what >> >> the current status is. Also there are a few bug fixes for the Parquet >> >> which Vova submitted which looks like we should include as well. >> >> Best, >> >> -- C >> >> >> >>> On May 27, 2021, at 11:01 AM, Laurent Goujon <[email protected]> >> wrote: >> >>> >> >>> Sadly, I haven't heard from people regarding the patches. At the same >> >> time, >> >>> I think we held the window open for merging the changes for a very >> long >> >>> time. Unless there's objection, I'm planning to merge the Guava and >> >>> Jetty/Hadoop pull requests later today, and doing the first RC for >> Drill >> >>> 1.19.0 >> >>> >> >>> Here are the pull request links: >> >>> * https://github.com/apache/drill/pull/2202 >> >>> * https://github.com/apache/drill/pull/2236 >> >>> >> >>> Laurent >> >>> >> >>> >> >>> On Wed, May 26, 2021 at 11:59 AM Laurent Goujon <[email protected]> >> >> wrote: >> >>> >> >>>> After several retries, the Guava checks successfully passed: >> >>>> https://github.com/apache/drill/pull/2202 >> >>>> >> >>>> Charles, can we proceed on merging your change? >> >>>> >> >>>> Laurent >> >>>> >> >>>> On Tue, May 25, 2021 at 10:24 PM Laurent Goujon <[email protected]> >> >>>> wrote: >> >>>> >> >>>>> Just an update. There's a patch for updating both Jetty and Hadoop >> (at >> >>>>> the same time) as those changes are co-dependent: >> >>>>> https://github.com/apache/drill/pull/2236 >> >>>>> >> >>>>> As for the Guava patch, I'd be happy to help, but I'm not sure >> what's >> >>>>> left. As far as I can tell the shaded version of Guava has been >> >> updated, >> >>>>> but the build is failing. The security vulnerabilities for Guava are >> >>>>> moderate (and actually it seems a fix for CVE-2020-8908 would >> require a >> >>>>> code change instead of a Guava update. >> >>>>> >> >>>>> Since this has been almost a month since we started this release >> >> process, >> >>>>> I wonder if we still want to wait on this patch, or if we should >> move >> >> it to >> >>>>> the next release. >> >>>>> >> >>>>> Let me know what people think, >> >>>>> >> >>>>> On Tue, May 25, 2021 at 8:24 AM Laurent Goujon <[email protected]> >> >>>>> wrote: >> >>>>> >> >>>>>> Anything I can help with? >> >>>>>> >> >>>>>> On Tue, May 25, 2021 at 7:02 AM Charles Givre <[email protected]> >> >> wrote: >> >>>>>> >> >>>>>>> HI Laurent, >> >>>>>>> My apologies. I said Junit, when I was meaning to say to the >> Guava >> >> PR ( >> >>>>>>> https://github.com/apache/drill/pull/2202 < >> >>>>>>> https://github.com/apache/drill/pull/2202>). I think this one is >> >>>>>>> almost done as well. >> >>>>>>> -- C >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>>> On May 24, 2021, at 5:29 PM, Laurent Goujon <[email protected]> >> >>>>>>> wrote: >> >>>>>>>> >> >>>>>>>> Ok, I was hoping that some of the PRs could be merged, but if we >> are >> >>>>>>> in >> >>>>>>>> agreement, let's start the work :) >> >>>>>>>> >> >>>>>>>> On Sun, May 23, 2021 at 6:52 PM luoc <[email protected]> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Charles, >> >>>>>>>>> All right, we'll be expecting the update. >> >>>>>>>>> >> >>>>>>>>>> 2021年5月24日 上午12:13，Charles Givre <[email protected]> 写道： >> >>>>>>>>>> >> >>>>>>>>>> Hi Luoc, >> >>>>>>>>>> We still have a few PRs pending that we really should get into >> >> Drill >> >>>>>>>>> 1.19. The main one is the junit upgrade. There are a few >> critical >> >>>>>>> CVEs >> >>>>>>>>> associated with that, so I do think it is important to get that >> one >> >>>>>>>>> merged. I think Vitalii will have that one done in short order. >> >>>>>>>>>> Best, >> >>>>>>>>>> -- C >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>> On May 22, 2021, at 5:16 AM, luoc <[email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Hi Laurent, >> >>>>>>>>>>> It’s time to do a release with 1.19.0. >> >>>>>>>>>>> >> >>>>>>>>>>>> 2021年5月19日 上午2:20，Vitalii Diravka <[email protected]> 写道： >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi Laurent, >> >>>>>>>>>>>> DRILL-7871 requires additional time to be introduced and it >> is >> >>>>>>> better >> >>>>>>>>> to >> >>>>>>>>>>>> include it for the next release. >> >>>>>>>>>>>> DRILL-7904 is updated, I think it will be merged in a few >> days. >> >>>>>>> But it >> >>>>>>>>>>>> doesn't matter whether it is included in this release or in >> the >> >>>>>>> next >> >>>>>>>>> one. >> >>>>>>>>>>>> >> >>>>>>>>>>>> So we can plan to start the release process >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> Kind regards >> >>>>>>>>>>>> Vitalii >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon < >> >>>>>>> [email protected]> >> >>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Thanks Vitalii >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka < >> >>>>>>> [email protected]> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi Luoc! >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> They are almost ready. I plan to update PR for them today. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Kind regards >> >>>>>>>>>>>>>> Vitalii >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Sat, May 8, 2021 at 5:26 PM luoc <[email protected]> >> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi Vitalii, >> >>>>>>>>>>>>>>> Would you mind sharing that... Is DRILL-7904 ready to >> review >> >>>>>>> again? >> >>>>>>>>>>>>>> And what’s >> >>>>>>>>>>>>>>> the status on the DRILL-7871? thanks >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> 2021年5月4日 下午1:10，Ted Dunning <[email protected]> 写道： >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Laurent, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I don't have a stake here, so can't really comment about >> >>>>>>> specifics, >> >>>>>>>>> but >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>> process is looking good. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Mon, May 3, 2021 at 9:23 PM Laurent Goujon < >> >>>>>>> [email protected]> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks for all the answers >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> So the issues I found based on the feedback are: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> - DRILL-7878: Fix LGTM Alerts >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7878> >> >>>>>>>>>>>>>>> - DRILL-7871: StoragePluginStore instances for different >> >> users >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7871> >> >>>>>>>>>>>>>>> - DRILL-7908: Fix GitHub Actions CI >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7908> >> >>>>>>>>>>>>>>> - DRILL-7904: Update to 30-jre Guava version >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7904> >> >>>>>>>>>>>>>>> - DRILL-7826: Merge Pcap and Pcapng format plugin based on >> >> EVF >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7826> >> >>>>>>>>>>>>>>> - DRILL-7828: Refactor Pcap and Pcapng format plugin >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7828> >> >>>>>>>>>>>>>>> - DRILL-7910: Bumps commons-io from 2.4 to 2.7 >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7910> >> >>>>>>>>>>>>>>> - DRILL-7901: Bump junit from 4.12 to 4.13.1 >> >>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7901> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I wanted to propose Monday May 10th to do the first >> release >> >>>>>>>>> candidate, >> >>>>>>>>>>>>>> but >> >>>>>>>>>>>>>>> I have some concerns about some of the changes which may >> not >> >> be >> >>>>>>>>> ready >> >>>>>>>>>>>>> by >> >>>>>>>>>>>>>>> then considering they seem to involve some level of effort >> >> and >> >>>>>>> are >> >>>>>>>>> in >> >>>>>>>>>>>>>> very >> >>>>>>>>>>>>>>> early stage: The LGTM alert changes and the >> >> StoragePluginStore >> >>>>>>> model >> >>>>>>>>>>>>>>> change. JUnit version update might also become quite a >> large >> >>>>>>> change >> >>>>>>>>> if >> >>>>>>>>>>>>>>> instead of moving to 4.13.1, Drill is switching to JUnit5. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> What do people think? >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Sat, Apr 24, 2021 at 1:00 PM Vitalii Diravka < >> >>>>>>> [email protected] >> >>>>>>>>>> >> >>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi Laurent, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I want to include: >> >>>>>>>>>>>>>>> DRILL-7871 < >> https://issues.apache.org/jira/browse/DRILL-7871 >> >>> >> >>>>>>>>>>>>> (preparing >> >>>>>>>>>>>>>>> PR) >> >>>>>>>>>>>>>>> DRILL-7908 < >> https://issues.apache.org/jira/browse/DRILL-7908 >> >>> >> >>>>>>>>>>>>> (preparing >> >>>>>>>>>>>>>>> PR) >> >>>>>>>>>>>>>>> DRILL-7904 < >> https://issues.apache.org/jira/browse/DRILL-7904 >> >>> >> >>>>>>> (PR >> >>>>>>>>> is >> >>>>>>>>>>>>>>> opened, in review) >> >>>>>>>>>>>>>>> DRILL-7828 < >> https://issues.apache.org/jira/browse/DRILL-7828 >> >>> >> >>>>>>> (PR >> >>>>>>>>> is >> >>>>>>>>>>>>>>> opened, review is almost completed) >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> All these tasks are expected to be completed in a week >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Kind regards >> >>>>>>>>>>>>>>> Vitalii >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Fri, Apr 23, 2021 at 9:25 PM Charles Givre < >> >>>>>>> [email protected]> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi Laurent, >> >>>>>>>>>>>>>>> We have a few PRs pending which I'd like to see in the >> next >> >>>>>>> version >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> are: >> >>>>>>>>>>>>>>> 1. The update(s) and bug fixes to the Mongo plugin. >> >>>>>>>>>>>>>>> 2. There is an extended PR for bug fixes which clean up a >> >> lot >> >>>>>>> of >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> alerts >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> generated by LGTM >> >>>>>>>>>>>>>>> 3. There are a few other library updates which are >> pending. >> >>>>>>>>>>>>>>> 4. We have some work which changes the access model >> around >> >>>>>>> storage >> >>>>>>>>>>>>>>> plugins which would be good for this release >> >>>>>>>>>>>>>>> 5. The PCAP/PCAP-NG consolidation is awaiting review. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I think that's it. >> >>>>>>>>>>>>>>> -- C >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Apr 22, 2021, at 12:33 PM, Laurent Goujon < >> >>>>>>> [email protected]> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hello everyone, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> It has been more than 6 months since the last release, >> and I >> >>>>>>> believe >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> would be a good time to discuss the next one. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> As mentioned in a previous email thread, I am >> volunteering to >> >>>>>>> be the >> >>>>>>>>>>>>>>> release manager, and I'm looking forward working with the >> >>>>>>> whole >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> community >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> to make another great release. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> We have around 80 changes in master since the last >> release, >> >> and >> >>>>>>>>> there >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> are >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> several changes open for review too. It would be nice if >> >> people >> >>>>>>>>> could >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> reply >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> to this email and share issues which should be part of >> that >> >>>>>>> release, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> so >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> can decide on an initial cut-off date. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks in advance, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Laurent >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> >>>>>>> >> >> >> >> >> >>
