I agree that it's probably time (more likely, overdue) for a release. Based off of looking at all of those changes I would also suggest going to at least 0.5.x.
It probably makes sense to take a look at Upgrading.md <https://github.com/apache/metron/blob/master/Upgrading.md> (and related docs) to make sure it's accurate as well. Jon On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Good call - I thought that made our last release, but this would be the 2nd > follow-on from when Nick originally posed a breakdown > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc via > mmiklavc) closes apache/metron#840 > > > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > We should also mention the Upgrade of ElasticSearch and Kibana > > > > Jon > > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <n...@nickallen.org> wrote: > > > > > Oh, and also the Solr work that is currently in a feature branch. We > > would > > > have to get the work finished up and merged though. Sounds like we are > > > real close on that. > > > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen <n...@nickallen.org> > wrote: > > > > > > > The next part of the conversation would be, what should the version > > > number > > > > be? To help with that, I have tried to summarize the changes in the > > > > release. Of course, this is going to be heavily biased towards my > own > > > > interests, so please feel free to chime in if I have missed anything. > > > > > > > > > > > > - Significant performance improvements in Parsing, Enrichments, > and > > > > Indexing > > > > > > > > > > > > - Support for Ubuntu installations > > > > > > > > > > > > - Support for Storm 1.1 and HDP 2.6 > > > > > > > > > > > > - Event time processing in the Profiler > > > > > > > > > > > > - Complex document support in the JSONMapParser > > > > > > > > > > > > - Support for the Elasticsearch X-Pack > > > > > > > > > > > > - Run Stellar in a Zeppelin Notebook > > > > > > > > > > > > - Oodles of bug fixes and usability improvements > > > > > > > > > > > > > > > > > > > > On Wed, May 9, 2018 at 12:14 PM, Nick Allen <n...@nickallen.org> > > wrote: > > > > > > > >> Something like this might be more digestible for these purposes. > > > >> > > > >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD > > > >> > > > >> 88 minutes ago METRON-1530 Default proxy config settings in > > > >> metron-contrib need to be updated (sardell via merrimanr) closes > > > >> apache/metron#998 > > > >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr) > > closes > > > >> apache/metron#1008 > > > >> 7 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor > > > Config > > > >> (nickwallen) closes apache/metron#1007 > > > >> 2 weeks ago METRON-1539: Specialized RENAME field transformer closes > > > >> apache/incubator-metron#1002 > > > >> 2 weeks ago METRON-1520: Add caching for stellar field > transformations > > > >> closes apache/incubator-metron#990 > > > >> 2 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config > > When > > > >> Run in Zeppelin REPL (nickwallen) closes apache/metron#997 > > > >> 2 weeks ago METRON-1511 Unable to Serialize Profiler Configuration > > > >> (nickwallen) closes apache/metron#982 > > > >> 3 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc > via > > > >> mmiklavc) closes apache/metron#996 > > > >> 3 weeks ago METRON-1445: Update performance tuning guide with more > > > >> explicit parameter instructions (mmiklavc via mmiklavc) closes > > > >> apache/metron#988 > > > >> 3 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) > > closes > > > >> apache/metron#974 > > > >> 3 weeks ago METRON-1527: Remove dead test file sitting in source > > folder > > > >> (mmiklavc via mmiklavc) closes apache/metron#994 > > > >> 3 weeks ago METRON-1499 Enable Configuration of Unified Enrichment > > > >> Topology via Ambari (nickwallen) closes apache/metron#984 > > > >> 3 weeks ago METRON-1515: Errors loading stellar functions currently > > bomb > > > >> the entire topology, they should be recoverable closes > > > >> apache/incubator-metron#985 > > > >> 4 weeks ago METRON-1522 Fix the typo errors at profile debugger > readme > > > >> (MohanDV via nickwallen) closes apache/metron#992 > > > >> 4 weeks ago METRON-1519 Indexing Error Topic Property Not Displayed > in > > > >> MPack (nickwallen) closes apache/metron#987 > > > >> 4 weeks ago METRON-1347: Indexing Topology should fail tuples > without > > a > > > >> source.type (cstella via mmiklavc) closes apache/metron#863 > > > >> 4 weeks ago Add support for Vagrant Cachier plugin if present > > > >> (simonellistonball via mmiklavc) closes apache/metron#993 > > > >> 4 weeks ago METRON-1521: JSONMapParser is no longer serializable > > closes > > > >> apache/incubator-metron#991 > > > >> 4 weeks ago METRON-1516 Support for Ansible 2.5.0 (ottobackwards) > > closes > > > >> apache/metron#989 > > > >> 4 weeks ago METRON-1494 Profiler Emits Messages to Kafka When Not > > Needed > > > >> (nickwallen) closes apache/metron#967 > > > >> 4 weeks ago METRON-1510 Update Metron website to include info about > > > >> github update subscription (anandsubbu) closes apache/metron#981 > > > >> 4 weeks ago METRON-1518 Build Failure When Using Profile HDP-2.5.0.0 > > > >> (nickwallen) closes apache/metron#986 > > > >> 4 weeks ago METRON-1465:Support for Elasticsearch X-pack (wardbekker > > via > > > >> mmiklavc) closes apache/metron#946 > > > >> 4 weeks ago METRON-1504: Enriching missing values does not match the > > > >> semantics between the new enrichment topology and old closes > > > >> apache/incubator-metron#976 > > > >> 5 weeks ago METRON-1505 Intermittent Profiler Integration Test > Failure > > > >> (nickwallen) closes apache/metron#977 > > > >> 5 weeks ago METRON-1449 Set Zookeeper URL for Stellar Running in > > > Zeppelin > > > >> Notebook (nickwallen) closes apache/metron#931 > > > >> 5 weeks ago METRON-1462: Separate ES and Kibana from Metron Mpack > > > >> (mmiklavc via mmiklavc) closes apache/metron#943 > > > >> 5 weeks ago METRON-1501 Parser messages that fail to validate are > > > dropped > > > >> silently (cestella via justinleet) closes apache/metron#972 > > > >> 5 weeks ago METRON-1497 Rest endpoint '/api/v1/search/search' needs > to > > > >> handle null when elastic search response return null for > > getAggregations > > > >> (MohanDV via justinleet) closes apache/metron#968 > > > >> 7 weeks ago METRON-1500 Enhance 'prepare-commit' to Support Feature > > > >> Branches (nickwallen) closes apache/metron#971 > > > >> 7 weeks ago METRON-1491: The indexing topology restart logic is > wrong > > > >> (cstella via mmiklavc) closes apache/metron#964 > > > >> 7 weeks ago METRON-590 Enable Use of Event Time in Profiler > > (nickwallen) > > > >> closes apache/metron#965 > > > >> 7 weeks ago METRON-1483: Create a tool to monitor performance of the > > > >> topologies closes apache/incubator-metron#958 > > > >> 8 weeks ago METRON-1487 Define Performance Benchmarks for Enrichment > > > >> Topology (nickwallen) closes apache/metron#961 > > > >> 8 weeks ago METRON-1493 Unhelpful Error Message When Assignment > > > >> Expressions Fail (nickwallen) closes apache/metron#966 > > > >> 8 weeks ago METRON-1397 Support for JSON Path and complex documents > in > > > >> JSONMapParser closes apache/incubator-metron#914 > > > >> 8 weeks ago METRON-1299 In MetronError tests, don't test for > HostName > > if > > > >> getHostName wouldn't work closes apache/incubator-metron#924 > > > >> 8 weeks ago METRON-1485 Upgrade vagrant for dev environments closes > > > >> apache/incubator-metron#959 > > > >> 8 weeks ago METRON-1488: user_settings hbase table does not have > acls > > > set > > > >> up for kerberos closes apache/metron#962 > > > >> 8 weeks ago METRON-1490: Better error message when user specifies an > > > >> enrichment type that doesn't exist closes apache/metron#963 > > > >> 9 weeks ago METRON-1468 Add support for > apache/metron-bro-plugin-kafka > > > to > > > >> prepare-commit (JonZeolla) closes apache/metron#948 > > > >> 9 weeks ago METRON-1471: Migrate shuffle connections to local or > > shuffle > > > >> closes apache/metron#949 > > > >> 9 weeks ago METRON-1467: Replace guava caches in places where the > > > >> keyspace might be large closes apache/metron#947 > > > >> 9 weeks ago METRON-1463: Adjust the groupings and shuffles in > > enrichment > > > >> to be more efficient closes apache/metron#944 > > > >> 9 weeks ago METRON-1460: Create a complementary non-split-join > > > enrichment > > > >> topology closes apache/metron#940 > > > >> 9 weeks ago METRON-1470 Update jquery to version 3+ (xyztdanid via > > > >> merrimanr) closes apache/metron#950 > > > >> 9 weeks ago METRON-1450 Add REST endpoint docs for index topology > > split > > > >> (MohanDV via nickwallen) closes apache/metron#936 > > > >> 9 weeks ago METRON-1337 List of facets should not be hardcoded > > > >> (merrimanr) closes apache/metron#853 > > > >> 9 weeks ago METRON-1452 Rebase Dev Environment on Latest CentOS 6 > > > >> (nickwallen) closes apache/metron#933 > > > >> 2 months ago METRON-1457 Move ASF links to main page in the Metron > > > >> website (anandsubbu) closes apache/metron#938 > > > >> 3 months ago METRON-1394 Create Rest endpoint to add the ACL for > > current > > > >> user to kafka topics (MohanDV via merrimanr) closes > apache/metron#895 > > > >> 3 months ago METRON-941 native PaloAlto parser corrupts message when > > > >> having a comma in the payload (ctramnitz via justinleet) closes > > > >> apache/metron#579 > > > >> 3 months ago METRON-1386: Fix Metron Website Required Links closes > > > >> apache/incubator-metron#935 > > > >> 3 months ago METRON-1455: Patch and Replace methods in the REST > > > >> UpdateController return 400 this closes apache/incubator-metron#937 > > > >> 3 months ago METRON-1318 Update MacOS Instructions for AWS > (wardbekker > > > >> via nickwallen) closes apache/metron#930 > > > >> 3 months ago METRON-1451 On Centos full dev, Metron Indexing shows > up > > as > > > >> stopped (anandsubbu via merrimanr) closes apache/metron#932 > > > >> 3 months ago METRON-1444: Add Ubuntu Repositories for Elasticsearch > to > > > >> the Mpack (mmiklavc via mmiklavc) closes apache/metron#928 > > > >> 3 months ago METRON-1273: Website documentation link should point to > > the > > > >> current site-book (JonZeolla via mmiklavc) closes apache/metron#812 > > > >> 3 months ago METRON-1447 Heap Size Not Set Correctly by MPack for ES > > 5.x > > > >> (nickwallen) closes apache/metron#927 > > > >> 3 months ago METRON-1446: Fix openjdk issue with Ubuntu (mmiklavc > via > > > >> mmiklavc) closes apache/metron#926 > > > >> 3 months ago METRON-1442: Split rest end points for indexing > topology > > > >> into random access indexing and batch indexing this closes > > > >> apache/incubator-metron#923 > > > >> 3 months ago METRON-1443 Missing Critical MPack Install Instruction > > for > > > >> Ubuntu (nickwallen) closes apache/metron#925 > > > >> 3 months ago METRON-1438 STELLAR: Move shell functions to common > from > > > >> metron-management (ottobackwards) closes apache/metron#920 > > > >> 3 months ago METRON-1435 Management UI cannot save json objects in > > > >> advanced config (merrimanr) closes apache/metron#917 > > > >> 3 months ago METRON-1439 Turn off git pager in platform-info script > > > >> (justinleet) closes apache/metron#919 > > > >> 3 months ago METRON-1091 STELLAR Shell: Stand Alone installation > > > >> (ottobackwards) closes apache/metron#690 > > > >> 3 months ago METRON-1427: Add support for storm 1.1 and hdp 2.6 > > (cstella > > > >> via mmiklavc) closes apache/metron#907 > > > >> 3 months ago METRON-1391 Typos in Documentation/Examples within > > > >> metron-management/README.md (havran via ottobackwards) closes > > > >> apache/metron#890 > > > >> 3 months ago METRON-1389 Zeppelin notebook import does not work with > > > >> Ambari 2.6 (anandsubbu) closes apache/metron#888 > > > >> 3 months ago METRON-1432 JDK Install Fails on Ubuntu Development > > > >> Environment (nickwallen) closes apache/metron#913 > > > >> 3 months ago METRON-1431 Add REGEXP_REPLACE function to Stellar > > (havran > > > >> via ottobackwards) closes apache/metron#912 > > > >> 3 months ago METRON-1410 Some more upgrade fallout... Can't > > restart > > > >> Metron Indexing. (ottobackwards) closes apache/metron#901 > > > >> 3 months ago METRON-1370 Create Full Dev Equivalent for Ubuntu > > > >> (nickwallen via cestella) closes apache/incubator-metron#903 > > > >> 3 months ago METRON-1430: Isolate jackson from being used as > arguments > > > or > > > >> returns from JSONUtils closes apache/metron#910 > > > >> 3 months ago METRON-1398 Exclude the basic-error-controller from > being > > > >> added to the swagger description (MohanDV via merrimanr) closes > > > >> apache/metron#898 > > > >> 3 months ago METRON-1392 Fix a test case to expect an Exception when > > > >> replication factor more than number of brokers while creating topic > > > >> (MohanDV via merrimanr) closes apache/metron#892 > > > >> 3 months ago METRON-1413 Add Metron Commit Tool (nickwallen) closes > > > >> apache/metron#902 > > > >> 3 months ago METRON-1429 SearchIntegrationTest refactor (merrimanr) > > > >> closes apache/metron#909 > > > >> 3 months ago METRON-1426: SensorIndexingConfigController > > IntegrationTest > > > >> fails intermittently closes apache/metron#906 > > > >> 4 months ago METRON-1417: Disable pcap-service by default in Monit > > > >> (mmiklavc via mmiklavc) closes apache/metron#905 > > > >> 4 months ago METRON-1400: Elasticsearch service check fails in > Ambari > > > >> (mmiklavc via mmiklavc) closes apache/metron#904 > > > >> 4 months ago METRON-1428: Travis build failing from metron-config > > > >> (mmiklavc via mmiklavc) closes apache/metron#908 > > > >> 4 months ago METRON-1302: Split up Indexing Topology into batch and > > > >> random access sections closes apache/incubator-metron#831 > > > >> 4 months ago METRON-1395 Documentation missing for Produce a message > > to > > > a > > > >> Kafka topic Rest API endpoint (MohanDV via nickwallen) closes > > > >> apache/metron#897 > > > >> 4 months ago METRON-1411 Fix sed command in Upgrading.md > (justinleet) > > > >> closes apache/metron#900 > > > >> 4 months ago METRON-1326: Metron deploy with Kerberos fails on > Ambari > > > 2.5 > > > >> during ES service stop (mmiklavc via mmiklavc) closes > > apache/metron#894 > > > >> 4 months ago METRON-1380: Create a typosquatting use-case closes > > > >> apache/incubator-metron#882 > > > >> 4 months ago METRON-1230: As a stopgap prior to METRON-777, add more > > > >> simplistic sideloading of custom Parsers closes > > > apache/incubator-metron#785 > > > >> 4 months ago METRON-1378: Create a summarizer closes > > > >> apache/incubator-metron#879 > > > >> 4 months ago METRON-1231 Separate Sensor name and topic in the > > > Management > > > >> UI (merrimanr) closes apache/metron#786 > > > >> 4 months ago METRON-1382 Run Stellar in a Zeppelin Notebook > > (nickwallen) > > > >> closes apache/metron#884 > > > >> 4 months ago METRON-1396 Fix .gitignore files to not ignore > themselves > > > >> (justinleet) closes apache/metron#896 > > > >> 4 months ago METRON-1366: Add an entropy stellar function (cstella > via > > > >> mmiklavc) closes apache/metron#872 > > > >> 4 months ago METRON-1390: Swagger UI for "Web Security Config" > > > Controller > > > >> needs request method (MohanDV via mmiklavc) closes apache/metron#889 > > > >> 4 months ago METRON-1393: Fix bro Elasticsearch template (mmiklavc > via > > > >> mmiklavc) closes apache/metron#893 > > > >> 4 months ago METRON-1379: Add an OBJECT_GET stellar function closes > > > >> apache/incubator-metron#880 > > > >> 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc > > via > > > >> mmiklavc) closes apache/metron#840 > > > >> 4 months ago METRON-1377: Stellar function to generate typosquatted > > > >> domains (similar to dnstwist) closes apache/incubator-metron#878 > > > >> 4 months ago METRON-1385 Missing "properties" in index > > > template > > > >> causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail > > > >> (merrimanr) closes apache/metron#886 > > > >> 4 months ago METRON-1388 update public web site to point at 0.4.2 > new > > > >> release (mattf-horton) closes apache/metron#887 > > > >> 4 months ago METRON-1362 Improve Metron Deployment README > (nickwallen) > > > >> closes apache/metron#869 > > > >> 4 months ago METRON-1384 Increment master version number to 0.4.3 > for > > > >> on-going development (mattf-horton via nickwallen) closes > > > apache/metron#885 > > > >> 4 months ago METRON-1381 Add Apache license to MD files and remove > the > > > >> Rat exclusion (justinleet) closes apache/metron#883 > > > >> 4 months ago METRON-1071 Create CONTRIBUTING.md (justinleet) closes > > > >> apache/metron#881 > > > >> 4 months ago METRON-1373 RAT failure for > > metron-interface/metron-alerts > > > >> (mattf-horton) closes apache/metron#875 > > > >> 4 months ago METRON-1351 Create Installable Packages for Ubuntu > Trusty > > > >> (nickwallen) closes apache/metron#868 > > > >> 5 months ago METRON-1376 RC Check Script should have named > parameters > > > >> (ottobackwards via nickwallen) closes apache/metron#877 > > > >> 5 months ago METRON-1365: Allow PROFILE_GET to return a default > value > > > for > > > >> a profile and entity that does not have a value written. closes > > > >> apache/incubator-metron#871 > > > >> 5 months ago METRON-1348 Metron Service Checks Use Wrong Hostname > > > >> (nickwallen) closes apache/metron#864 > > > >> 5 months ago METRON-1350: Add reservoir sampling functions to > Stellar > > > >> closes apache/incubator-metron#867 > > > >> 5 months ago METRON-1374 Script the RC checking process > > (ottobackwards) > > > >> closes apache/metron#876 > > > >> 5 months ago METRON-1372 Validate JIRA for Releases (nickwallen) > > closes > > > >> apache/metron#874 > > > >> 5 months ago METRON-1345: Update EC2 README for custom Ansible > > (mmiklavc > > > >> via mmiklavc) closes apache/metron#859 > > > >> 5 months ago METRON-1349 Full Dev Builds Metron Twice (nickwallen) > > > closes > > > >> apache/metron#866 > > > >> 5 months ago METRON-1343 Swagger UI for User Controller needs > request > > > >> method (MohanDV via ottobackwards) closes apache/metron#862 > > > >> 5 months ago METRON-1306: When index template install fails, we > should > > > >> fail the install closes apache/incubator-metron#834 > > > >> 5 months ago METRON-1341 Projection FieldTransformation > > > >> (simonellistonball via ottobackwards) closes apache/metron#861 > > > >> > > > >> > > > >> On Wed, May 9, 2018 at 11:57 AM, Michael Miklavcic < > > > >> michael.miklav...@gmail.com> wrote: > > > >> > > > >>> Is this what you mean Otto? > > > >>> https://github.com/apache/metron/blob/24822dddc68c264f59723f > > > >>> > > > > 5e17d423cd497f6807/dev-utilities/release-utils/validate-jira-for-release > > > >>> > > > >>> On Wed, May 9, 2018 at 9:52 AM, Casey Stella <ceste...@gmail.com> > > > wrote: > > > >>> > > > >>> > I wasn't aware we had a script for that..is that in > > > >>> > dev-utilities/release-utils? > > > >>> > > > > >>> > On Wed, May 9, 2018 at 11:41 AM Otto Fowler < > > ottobackwa...@gmail.com > > > > > > > >>> > wrote: > > > >>> > > > > >>> > > Can you run the issues included script and post that for us to > > see? > > > >>> > > > > > >>> > > > > > >>> > > On May 9, 2018 at 11:14:11, Casey Stella (ceste...@gmail.com) > > > wrote: > > > >>> > > > > > >>> > > Is it about time for a release? I know we got some substantial > > > >>> > performance > > > >>> > > changes in since the last release. I think we might have a > > > >>> justification > > > >>> > > for a release. > > > >>> > > > > > >>> > > Casey > > > >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >> > > > >> > > > > > > > > > -- > > > > Jon > > > -- Jon
