Thanks to Jon, Otto, and Nick for looking over RC1. Tonight I’ll fix the RAT glitch, build RC2, and put it to formal vote. Regards, --Matt
On 12/12/17, 5:14 AM, "Nick Allen" <n...@nickallen.org> wrote: RC1 is looking good to me. I validated the MD5s, built Metron, built the Bro plugin and reviewed the other artifacts like release notes. Running the RAT check on a 'clean' Metron does not produce any errors for me. It is only after building Metron, which pulls in additional Node dependencies, does the RAT check fail. On Sun, Dec 10, 2017 at 4:41 PM Matt Foley <ma...@apache.org> wrote: > Yes, but let’s see if anyone else find other issues. > > > > From: Otto Fowler <ottobackwa...@gmail.com> > Date: Saturday, December 9, 2017 at 6:16 AM > To: Matt Foley <mfo...@hortonworks.com>, "dev@metron.apache.org" < > dev@metron.apache.org> > Subject: Re: [DISCUSS] Upcoming Release > > > > So RC2 then? > > > > On December 8, 2017 at 20:43:21, Matt Foley (mfo...@hortonworks.com) > wrote: > > Hah, here it is: https://github.com/apache/metron/pull/743 > “This problem seems to only reproduce when one unrolls a tarball rather > than cloning from github.” > > Heh, the exclusion at > https://github.com/apache/metron/blob/master/pom.xml#L351 is still there, > but the hashcode in the bundle.css file name has changed from > a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04. Sigh. Did the version of Font > Awesome fonts change? > > > On 12/8/17, 5:26 PM, "Matt Foley" <ma...@apache.org> wrote: > > I remember having trouble with this bundle.css file on the last release, > but I can’t remember what we did about it. Anybody? > > On 12/8/17, 1:41 PM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > Steps > > - Downloaded tar.gz’s, asc files and KEYS > - Verified signing of both tar.gz’s > - searched for rouge 0.4.1 entries > - verified the main pom.xml > - built : > > mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T > 2C surefire:test@unit-tests && time mvn -q > surefire:test@integration-tests && time mvn -q test --projects > metron-interface/metron-config && time build_utils/verify_licenses.sh > > Found rat error: > > > ***************************************************** > Summary > ------- > Generated at: 2017-12-08T16:33:27-05:00 > > Notes: 3 > Binaries: 193 > Archives: 0 > Standards: 75 > > Apache Licensed: 74 > Generated Documents: 0 > > JavaDocs are generated, thus a license header is optional. > Generated files do not require license headers. > > 1 Unknown Licenses > > ***************************************************** > > Files with unapproved licenses: > > > /Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css > > ***************************************************** > > > > > > ***************************************************** > Summary > ------- > Generated at: 2017-12-08T16:33:27-05:00 > > Notes: 3 > Binaries: 193 > Archives: 0 > Standards: 75 > > Apache Licensed: 74 > Generated Documents: 0 > > JavaDocs are generated, thus a license header is optional. > Generated files do not require license headers. > > 1 Unknown Licenses > > ***************************************************** > > Files with unapproved licenses: > > > > /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css > > ***************************************************** > > > > On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org) wrote: > > Colleagues, > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/ > > Given the complexity of this RC, I’d appreciate if a couple people would be > willing to kick the tires before we put it up for a vote. > > I will myself be going thru the Verify Build process this weekend, as I > won’t be able to do it Friday. > > Thanks, > --Matt > > > On 12/4/17, 2:05 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote: > > Can we resolve the conversation regarding the second repo? I was waiting > to get more input/preferences from people There's also a documentation > update that fixes a few broken Stellar docs that already has aa +1, I just > need to merge it. > > Jon > > On Mon, Dec 4, 2017, 17:01 Casey Stella <ceste...@gmail.com> wrote: > > > I would be in favor of a release at this point. > > > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote: > > > > > Hey all, > > > I see METRON-1252 was resolved over the weekend. Shall I go ahead and > > > start the process with 0.4.2 release? > > > Does anyone have any commits they feel strongly should go in before > 0.4.2 > > > is done, or are we ready to call it good? > > > > > > I believe there is consensus the 0.4.2 release should include a release > > of > > > the current state of the metron-bro-plugin-kafka. I will continue the > > > discussion in that thread as to the process for accomplishing that, but > > > plan on it happening. > > > > > > Regards, > > > --Matt > > > > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote: > > > > > > Hope everyone (at least in the U.S.) had a great Thanksgiving > > holiday. > > > Regarding status of the release effort, still pending METRON-1252, so > > > not making the release branch yet. > > > > > > Regards, > > > --Matt > > > > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote: > > > > > > (With release manager hat on) > > > > > > The community has proposed a release of Metron in the near > > future, > > > focusing on Meta-alerts running in Elasticsearch. > > > Congrats on getting so many of the below already done. At this > > > point, only METRON-1252, and the discussion of how to handle joint > > release > > > of the Metron bro plugin, remain as gating items for the release. I > > > project these will be resolved next week, so let’s propose the > following: > > > > > > Sometime next week, after the last bits are done, I’ll start the > > > release process and create the release branch. > > > > > > The proposed new version will be 0.4.2, unless there are backward > > > incompatible changes that support making it 0.5.0. > > > Currently there are NO included Jiras labeled > > > ‘backward-incompatible’, nor having Docs Text indicating so. > > > ***If anyone knows that some of the commits included since 0.4.1 > > > introduce backward incompatibility, please say so now on this thread, > and > > > mark the Jira as such.*** > > > > > > The 90 or so jiras/commits already in master branch since 0.4.1 > > > are listed below. > > > Thanks, > > > --Matt > > > > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly > > > Filters Some Records (nickwallen) closes apache/metron#832 > > > METRON-1294 IP addresses are not formatted correctly in facet > > > and group results (merrimanr) closes apache/metron#827 > > > METRON-1291 Kafka produce REST endpoint does not work in a > > > Kerberized cluster (merrimanr) closes apache/metron#826 > > > METRON-1290 Only first 10 alerts are update when a MetaAlert > > > status is changed to inactive (justinleet) closes apache/metron#842 > > > METRON-1311 Service Check Should Check Elasticsearch Index > > > Templates (nickwallen) closes apache/metron#839 > > > METRON-1289 Alert fields are lost when a MetaAlert is created > > > (merrimanr) closes apache/metron#824 > > > METRON-1309 Change metron-deployment to pull the plugin from > > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837 > > > METRON-1310 Template Delete Action Deletes Search Indices > > > (nickwallen) closes apache/metron#838 > > > METRON-1275: Fix Metron Documentation closes > > > apache/incubator-metron#833 > > > METRON-1295 Unable to Configure Logging for REST API > > > (nickwallen) closes apache/metron#828 > > > METRON-1307 Force install of java8 since java9 does not > > appear > > > to work with the scripts (brianhurley via ottobackwards) closes > > > apache/metron#835 > > > METRON-1296 Full Dev Fails to Deploy Index Templates > > > (nickwallen via cestella) closes apache/incubator-metron#829 > > > METRON-1281 Remove hard-coded indices from the Alerts UI > > > (merrimanr) closes apache/metron#821 > > > METRON-1287 Full Dev Fails When Installing EPEL Repository > > > (nickwallen) closes apache/metron#820 > > > METRON-1267 Alerts UI returns a 404 when refreshing the > > > alerts-list page (iraghumitra via merrimanr) closes apache/metron#819 > > > METRON-1283 Install Elasticsearch template as a part of the > > > mpack startup scripts (anandsubbu via nickwallen) closes > > apache/metron#817 > > > METRON-1254: Conditionals as map keys do not function in > > > Stellar closes apache/incubator-metron#801 > > > METRON-1261 Apply bro security patch (JonZeolla via > > > ottobackwards) closes apache/metron#805 > > > METRON-1284 Remove extraneous dead query in ElasticsearchDao > > > (justinleet) closes apache/metron#818 > > > METRON-1270: fix for warnings missing @return tag argument in > > > metron-analytics/metron-profiler-common and metron-profiler-client > closes > > > apache/incubator-metron#810 > > > METRON-1272 Hide child alerts from searches and grouping if > > > they belong to meta alerts (justinleet) closes apache/metron#811 > > > METRON-1224 Add time range selection to search control > > > (iraghumitra via james-sirota) closes apache/metron#796 > > > METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects > > > (cestella via justinleet) closes apache/metron#816 > > > METRON-1243: Add a REST endpoint which allows us to get a > > list > > > of all indice closes apache/incubator-metron#797 > > > METRON-1196 Increment master version number to 0.4.2 for > > > on-going development (mattf-horton) closes apache/metron#767 > > > METRON-1278 Strip "Build Status" widget from root > > > README.md in site-book build (mattf-horton) closes apache/metron#815 > > > METRON-1274 Master has failure in > > > StormControllerIntegrationTest (merrimanr) closes apache/metron#813 > > > METRON-1266 Profiler - SASL Authentication Failed > > (nickwallen) > > > closes apache/metron#809 > > > METRON-1260 Include Alerts UI in Ambari Service Check > > > (nickwallen) closes apache/metron#804 > > > METRON-1251: Typo and formatting fixes for metron-rest README > > > closes apache/incubator-metron#800 > > > METRON-1241: Enable the REST API to use a cache for the > > > zookeeper config similar to the Bolts closes > apache/incubator-metron#795 > > > METRON-1267 Alerts UI returns a 404 when refreshing the > > > alerts-list page (merrimanr) closes apache/metron#808 > > > METRON-1262 Unable to add comment for a alert in a meta-alert > > > (merrimanr) closes apache/metron#806 > > > METRON-1263 Start Alerts UI service after Metron REST > > > (anandsubbu via nickwallen) closes apache/metron#807 > > > METRON-1255 MetaAlert search is not filtering on status > > > (merrimanr) closes apache/metron#802 > > > METRON-1249 Improve Metron MPack Service Checks (nickwallen) > > > closes apache/metron#799 > > > METRON-1237 address javadoc warnings in metron-maas-common > > > (dbist via james-sirota) closes apache/metron#792 > > > METRON-1240 address javadoc warnings in metron-platform and > > > metron-analytics (dbist via james-sirota) closes apache/metron#794 > > > METRON-1226 Searching Can Errantly Query the Wrong Indices > > > (nickwallen) closes apache/metron#793 > > > METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) > > > closes apache/metron#682 > > > METRON-1123 Add group by option using faceted search > > > capabilities of metron-rest-api (iraghumitra via james-sirota) closes > > > apache/metron#768 > > > METRON-1223 Add support to add comments for alerts > > > (iraghumitra via james-sirota) closes apache/metron#788 > > > METRON-1083 Add filters using faceted search capabilities of > > > metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710 > > > METRON-1232 Alert status changes are not reflected in list > > > view (iraghumitra via merrimanr) closes apache/metron#787 > > > METRON-1247 REST search and findOne endpoints return > > > unexpected or incorrect results for guids (justinleet) closes > > > apache/metron#798 > > > METRON-1235: Document the properties pulled from the global > > > configuration closes apache/incubator-metron#791 > > > METRON-1234: fix for WARNING 'dependencies.dependency.( > > > groupId:artifactId:type:classifier)' must be unique: > > > org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes > > > apache/metron#790 > > > METRON-1222: fix warning for The expression ${parent.version} > > > is deprecated. Please use ${project.parent.version} instead. (dbist via > > > mmiklavc) closes apache/metron#782 > > > METRON-1220 Create documentation around alert nested field > > > (justinleet) closes apache/metron#780 > > > METRON-1229 Management UI type is part of the declarations of > > > 2 modules (merrimanr) closes apache/metron#784 > > > METRON-1228: Configuration Management PUSH immediately does > > > DUMP after (mmiklavc via mmiklavc) closes apache/metron#783 > > > METRON-1218 Metron REST should return better error messages > > > (merrimanr) closes apache/metron#779 > > > METRON-1161 Add ability to edit parser command line options > > in > > > the management UI (merrimanr) closes apache/metron#737 > > > METRON-1209: Make stellar repl take logging properties, like > > > other CLI apps in metron closes apache/incubator-metron#772 > > > METRON-1059 address checkstyle warning AvoidStarImport in > > > metron-stellar (dbist via ottobackwards) closes apache/metron#664 > > > METRON-1204 UI does not time out after being idle, but stops > > > functioning (merrimanr) closes apache/metron#771 > > > METRON-1052: Add forensic similarity hash functions to > > Stellar > > > closes apache/incubator-metron#781 > > > METRON-632: Added validation of "shew.enrichmentType" and > > > "shew.keyColumns" closes apache/incubator-metron#732 > > > METRON-1194 Add Profiler Debug Functions to Profiler README > > > (nickwallen via ottobackwards) closes apache/metron#765 > > > METRON-1055 Metron 0.4.0 manual installation guide for CentOS > > > 6 updates (lvets via ottobackwards) closes apache/metron#661 > > > METRON-1079 STELLAR NaN should be a keyword (ottobackwards) > > > closes apache/metron#681 > > > METRON-1085 Add REST endpoint to save a user profile for the > > > Alerts UI (merrimanr) closes apache/metron#694 > > > METRON-1208 MPack for Alerts UI (merrimanr) closes > > > apache/metron#778 > > > METRON-1207 Make RPMs for Alerts UI (merrimanr) closes > > > apache/metron#777 > > > METRON-1215 Fix link to RPMs chapter (DimDroll via > > justinleet) > > > closes apache/metron#776 > > > METRON-1206 Make alerts UI conform to ops UI for install > > > (merrimanr) closes apache/metron#773 > > > METRON-1195 Meta alerts improperly handle updates to > > non-alert > > > fields (justinleet) closes apache/metron#766 > > > METRON-1189 Add alert escalation to the Alerts UI (merrimanr) > > > closes apache/metron#762 > > > METRON-1156 Simulate Triage Rules in the Stellar REPL > > > (nickwallen) closes apache/metron#733 > > > METRON-1198 Pycapa - No such configuration property > > > 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769 > > > METRON-1202 ElasticsearchDao Has extraneous sleep call > > > (justinleet) closes apache/metron#770 > > > METRON-938 "service metron-rest start <password>" does not > > > work on CentOS 7. (justinleet) closes apache/metron#757 > > > METRON-1182 Refactor Code in alert list to accommodate new > > > view types (iraghumitra via merrimanr) closes apache/metron#756 > > > METRON-1188: Ambari global configuration management > > (mmiklavc) > > > closes apache/metron#760 > > > METRON-1191 update public web site to point at 0.4.1 new > > > release (mattf-horton) closes apache/metron#764 > > > METRON-1063 address javadoc warnings in metron-stellar (dbist > > > via ottobackwards) closes apache/metron#668 > > > METRON-1190 Fix Meta Alert Type handling in calculation of > > > scores (justinleet) closes apache/metron#763 > > > METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup > > > Correctly (nickwallen) closes apache/metron#759 > > > METRON-1185: Stellar REPL does not work on a kerberized > > > cluster when calling functions interacting with HBase closes > > > apache/incubator-metron#755 > > > METRON-1186: Profiler Functions use classutils from shaded > > > storm closes apache/incubator-metron#758 > > > METRON-1173: Fix pointers to old stellar docs closes > > > apache/incubator-metron#746 > > > METRON-1179: Make STATS_ADD to take a list closes > > > apache/incubator-metron#750 > > > METRON-1180: Make Stellar Shell accept zookeeper quorum as a > > > CSV list and not require a port closes apache/incubator-metron#751 > > > METRON-1183 Improve KDC Setup Instructions (nickwallen) > > closes > > > apache/metron#753 > > > METRON-1177 Stale running topologies seen post-kerberization > > > and cause exceptions (nickwallen) closes apache/metron#748 > > > METRON-1158 Build backend for grouping alerts into meta > > alerts > > > (justinleet) closes apache/metron#734 > > > METRON-1146: Add ability to parse JSON string into JSONObject > > > for stellar closes apache/incubator-metron#727 > > > METRON-1176 REST: HDFS Service should support setting > > > permissions on files when writing (ottobackwards) closes > > apache/metron#749 > > > METRON-1114 Add group by capabilities to search REST endpoint > > > (merrimanr) closes apache/metron#702 > > > METRON-1167 Define Session Specific Global Configuration > > > Values in the REPL (nickwallen) closes apache/metron#740 > > > METRON-1171: Better validation for the SUBSTRING stellar > > > function closes apache/incubator-metron#745 > > > > > > > > > > > > On 11/17/17, 11:59 AM, "Nick Allen" <n...@nickallen.org> wrote: > > > > > > I just wanted to send an update on where we are at. We've > > > gotten a lot > > > done here recently as you can see below. > > > > > > ✓ DONE (1) First, METRON-1289 needs to go in. This one was > > > a fairly big > > > effort and I am hearing that we are pretty close. > > > > > > ✓ DONE (2) METRON-1294 fixes an issue in how field types > > are > > > looked-up. > > > > > > ✓ DONE (3) METRON-1290 is next. While this may have been > > > fixed in > > > M-1289, there may be some test cases we want from this PR. > > > > > > ✓ DONE (4) METRON-1301 addresses a problem with the sorting > > > logic. > > > > > > ✓ DONE (5) METRON-1291 fixes an issue with escalation of > > > metaalerts. > > > > > > (6) That leads us to Raghu's UI work in METRON-1252. This > > > introduces the > > > UI bits that depend on all the previous backend work. > > > > > > (7) At this point, we should have our best effort at > > running > > > Metaalerts > > > on Elasticsearch 2.x. I propose that we cut a release here. > > > > > > (8) After we cut the release, we can introduce the work for > > > ES 5.x in > > > METRON-939. I know we will need lots of help testing and > > > reviewing this > > > one. > > > > > > > > > > > > We also have an outstanding question that needs resolved > > > BEFORE we > > > release. We need to come to a consensus on how to release > > > having moved our > > > Bro Plugin to a separate repo. I don't think we've heard > > from > > > everyone on > > > this. I'd urge everyone to chime in so we can choose a path > > > forward. > > > > > > If anyone is totally confused in regards to that discussion, > > I > > > can try and > > > send an options summary again as a separate discuss thread. > > > The original > > > chain was somewhere around here [1]. > > > > > > [1] > > > https://lists.apache.org/thread.html/ > > > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@% > > > 3Cdev.metron.apache.org%3E > > > > > > > > > > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen < > > > n...@nickallen.org> wrote: > > > > > > > Hi Guys - > > > > > > > > I want to follow-up on this discussion. It sounds like > > most > > > people are in > > > > agreement with the general approach. > > > > > > > > A lot of people have been working hard on Metaalerts and > > > Elasticsearch. I > > > > have checked-in with those doing the heavy lifting and have > > > compiled a more > > > > detailed plan based on where we are at now. To the best of > > > my knowledge > > > > here is the plan of attack for finishing out this effort. > > > > > > > > (1) First, METRON-1289 needs to go in. This one was a > > > fairly big effort > > > > and I am hearing that we are pretty close. > > > > > > > > (2) METRON-1294 fixes an issue in how field types are > > > looked-up. > > > > > > > > (3) METRON-1290 is next. While this may have been fixed > > > in M-1289, > > > > there may be some test cases we want from this PR. > > > > > > > > (4) METRON-1301 addresses a problem with the sorting > > logic. > > > > > > > > (5) METRON-1291 fixes an issue with escalation of > > > metaalerts. > > > > > > > > (6) That leads us to Raghu's UI work in METRON-1252. > > This > > > introduces > > > > the UI bits that depend on all the previous backend work. > > > > > > > > (7) At this point, we should have our best effort at > > > running Metaalerts > > > > on Elasticsearch 2.x. I propose that we cut a release here. > > > > > > > > (8) After we cut the release, we can introduce the work > > > for ES 5.x in > > > > METRON-939. I know we will need lots of help testing and > > > reviewing this > > > > one. > > > > > > > > Please correct me if I am wrong. I will try and send out > > > updates as we > > > > make progress. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com < > > > zeo...@gmail.com> wrote: > > > > > > > >> I agree, I think it's very reasonable to move in line with > > > Nick's > > > >> proposal. I would also suggest that we outline what the > > > target versions > > > >> would be to add in the METRON-777 components, since it has > > > been functional > > > >> for a very long time but not reviewed and has some really > > > rockstar > > > >> improvements. > > > >> > > > >> Jon > > > >> > > > >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler < > > > ottobackwa...@gmail.com> > > > >> wrote: > > > >> > > > >> > I think the ES cutover should be the start of the 0.5.x > > > series, and we > > > >> > continue on with 0.4.x for the > > > >> > metadata improvements etc. We could chose to focus > > > 0.5.x’s first > > > >> releases > > > >> > on not only ES but > > > >> > getting a handle on kibana and the mpack situation as > > > well. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > On November 6, 2017 at 12:48:45, Michael Miklavcic ( > > > >> > michael.miklav...@gmail.com) wrote: > > > >> > > > > >> > I agree with your proposal, Nick. I think having a > > > stabilizing release > > > >> > prior to upgrading ES/Kibana makes sense. > > > >> > > > > >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen < > > > n...@nickallen.org> wrote: > > > >> > > > > >> > > I would like to start a discussion around upcoming > > > releases. We have a > > > >> > > couple separate significant tracks of work that we > > need > > > to reconcile > > > >> in > > > >> > our > > > >> > > release schedule. > > > >> > > > > > >> > > (1) We have had (and have in review) a good number of > > > bug fixes > > > >> required > > > >> > to > > > >> > > support Metaalerts on the existing Elasticsearch 2.x > > > infrastructure. > > > >> > > > > > >> > > > > > >> > > (2) We also have ongoing work to upgrade our > > > infrastructure to > > > >> > > Elasticsearch 5.x, which will not be backwards > > > compatible. > > > >> > > > > > >> > > > > > >> > > I would like to see a release that has our best work > > on > > > ES 2.x before > > > >> we > > > >> > > migrate to 5.x. I would propose the following. > > > >> > > > > > >> > > Release N+1: Introduce Metaalerts running on ES 2.x > > > >> > > > > > >> > > Release N+2: Cut-over to ES 5.x > > > >> > > > > > >> > > > > > >> > > (Q) Is it worth cutting a separate release for ES 2.x? > > > Is there a > > > >> better > > > >> > > way to handle the cut-over to 5.x? > > > >> > > > > > >> > > > > >> -- > > > >> > > > >> Jon > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Jon > > > > > >