GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/720
METRON-1136 Initial Merge of Metron Extensions and Parser Extensions This pull request will land the Metron Extensions and Parser Extensions functionality in the new feature branch. It represents the total combined work on this functionality: - [METRON-777](https://github.com/apache/metron/pull/530) - [METRON-942](https://github.com/apache/metron/pull/580) - [METRON-947](https://github.com/apache/metron/pull/626) This feature branch work is now being tracked is [CONFLUENCE](https://cwiki.apache.org/confluence/display/METRON/Metron+Extension+System+and+Parser+Extensions#) and [JIRA](https://issues.apache.org/jira/browse/METRON-1136) ## Status [x] Creation of Metron Bundles ( adaptation and refactoring of Nifi NAR ) [x] Adaption of nar maven plugin to bundles maven plugin [x] Archetype for creating parser extensions [x] metron-parsers reduced to library for interfaces, base types, and base parsers [x] Metron parser types moved to parser extensions [x] Metron parser extensions packaged by rpm, installed by ambari including hdfs deployment [x] Metron parser extensions loaded by bundle from parser bolt, from hdfs [x] Refactored grok building to support variable metron hdfs apps root, parsers with directories of groks [x] Rest api for parser extensions ( install, delete, list, find ) [x] Zookeeper registration for extensions and their metadata, including default configurations [x] Stellar management for parser extension configurations [x] Guide to adding parsers to the metron system > Currently there is no work in Metron Config UI for installing or managing parser extensions ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? > see contributing PR's - [x] Have you included steps or a guide to how the change may be verified and tested manually? > see contributing PR's - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` #### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron ext-feature Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/720.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #720 ---- commit d68019dd48b17f6961fb6be1a87afada9ece57ab Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-08-29T13:06:12Z Squashed commit of the following: commit 2b54f853fadd1ffb4b7c70cfb70d1c28d8220eaf Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 11:46:28 2017 -0400 conform to logging norms commit 2c1eab5e266b5cab94708a05ec292fb4d28cc9de Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 11:11:50 2017 -0400 more test tweaks commit 76affe12d52e7dea30a417dc7b8a1ac8292943ca Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 10:04:49 2017 -0400 do not need the resource copy commit 5c38f00ba1d7cbdf72908105d469fe6ad4797f0b Merge: dcf9cee4c c8e84fa3b Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 09:40:51 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit dcf9cee4c2ef9cc5945b92f085baa3465f029418 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 09:18:25 2017 -0400 copy resources before class commit e18fe49305dfe06c637b6d491fd098a43a7aeb0f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 08:44:12 2017 -0400 javadoc and some saftey options for ResourceCopier commit 799b0b73a57cc127ea9b25cb3f92b02f97d07f73 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 28 00:02:54 2017 -0400 integrate with 777 + 777-grok commit 49788639e91e1649eacd7b8c4f36475a44f0ceac Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 26 00:20:59 2017 -0400 fix up imports commit 239df5f5e8d21253edaf6127e839b9b293d2a19f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 9 10:26:28 2017 -0400 missed a fix, rename folder commit bacd11508eaac0635d62afe3825dc40eb35ebf93 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 9 10:00:16 2017 -0400 adding the metron management function for parser extension configs from METRON-947, no reason to maintain two prs commit 94d223ed4badbc9f109dfbe17027cefacc1ee1ae Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 7 16:43:26 2017 -0400 merge fixes commit 97edf2f869ff234613db8b74830b57e84c0231a9 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 30 16:33:24 2017 -0400 change documenation to reflect uninstall commit b0fc2e4cc8b0637f9b619ac5c48e4b716245d16e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 30 16:31:19 2017 -0400 After merge, fix tests. Problems re-activating kafka after it was taken out in master. Instead - do not delete kafka topic or storm topology, since this service doesn't create them. commit a02b6fc2b80fe0a69ddef61db9ac73b6aab3c1bf Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 6 10:24:17 2017 -0400 better readme for usage commit beee5d94e10f8a46fd3c0a632b2ae29c7e05d6e8 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 10 09:25:24 2017 -0400 Add endpoints to readme commit 2d0edf6d6ef557370e517ec09e18df966bc05cbd Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 9 09:50:23 2017 -0400 Use the ambari managed spring configuration to properly get the grok pattern dir for hdfs Be sure to rollback on install exception, and delete grok dir for installed parsers commit fda824e38d36b12cc394cbdad40c3323f37ad9aa Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri May 5 10:43:25 2017 -0400 Do not install an extension if it is already installed commit e536adf08d499e4c0408c97f0042fc82206f3907 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri May 5 10:17:09 2017 -0400 fix extensions alt lib path in config commit cf3ad6991a6eea3987e8eeae62890d28abeed882 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 4 14:58:29 2017 -0400 cleanup a couple of imports commit 40048dbe953fa11dac20415d7ee3d6f57dd3e005 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 4 14:53:27 2017 -0400 implementation of delete commit 1912795bfd5a6395d55cc7222b73bf37370aa432 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 3 07:59:49 2017 -0400 async delete call commit 2e2c7a2d462eb4aab0b4cc2581fd52ab47cb0b54 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 3 07:15:36 2017 -0400 fix to return th correct httpstatus of created for install, use async dispatch in mock tests to get right result commit a3a5d47032e02a1ac6a474bfeb7326311c21ce62 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 23:54:48 2017 -0400 added delete test, need to implement commit 69378114704060d20dd5312d7b78c1b071b89fad Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 23:44:10 2017 -0400 fix json path issue for test of getAll commit c2839078e730226f9fdb491b6181fff7729a63f0 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 18:29:55 2017 -0400 test for getall() the call works, but I cannot get the jsonpath verification right a single statement works in a verification tool, but not here commit f9429494417fad9181400c08d4804a5f8780ee83 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 13:20:13 2017 -0400 test for getAll change ConfigurationType to allow changing zookeeper root generate a new extension name, must not have 0.0.0 version at end... will be the archive.tar.gz file name, with . -> _ and dropping the archive.tar.gz this foo-0.4.0-archive.tar.gz -> foo-0_5_0 as the extension name commit ade08a9505ce899bef502c05848a5ce777de424c Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 10:37:32 2017 -0400 update goal comment commit b193fe8604b4a75a84b2d122d9e43feed4db204f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 2 10:11:33 2017 -0400 support for putting the default es templates in the extension config fixed bug where we were calling wrong function in ConfigurationUtils commit e62b0216fce099f8641074ef0f7f33a2fea93d3c Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon May 1 17:51:11 2017 -0400 grok fix, after some manual testing. need to provide an override for the patterns dir in hdfs while testing commit 98f0c40273b790514b5cdad88cb0f998a33a8dd1 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon May 1 16:39:11 2017 -0400 Write out files to grok. At this time, the location of the hdfs grok directory is hard coded to the default If it is changed in ambari, this will not work. commit 098a288fd7754c9164c97f7aa8ee97ed4e3110ec Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jul 4 00:46:15 2017 -0400 track extension installs, along with metadata and defaults Needs more tests, but this should allow implmementation of delete() commit 180215a7393ddf33d095257baf27e7284dbbc32d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Apr 30 11:22:11 2017 -0400 Create an actual context object and build it out with more information, including bundle information The goal is to have enough information to fill out a configuration object that is in zk, that will allow for some kind of removal/uninstall of a parser commit 7ca11b63bed268cf333db9407fc431eb88a441a2 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Apr 29 13:09:38 2017 -0400 do not assume empty content, unless you are going to clear out everything commit 6f5257311cec9c9169d99065b4f4d56ba4668ef5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Apr 29 00:11:26 2017 -0400 do not need to add this, with the right intellij project ref to metron-common this works fine commit 37def88edf8599b930c0f7f19855a61af00498bb Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Apr 28 23:41:27 2017 -0400 start of controller test commit cd2f589da0ab1f97978043c8ecbd1372a35197d3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Apr 28 10:50:15 2017 -0400 correct path to write bundle to commit 6f9d170e7ff46539ffbd237b3d1d48a4bd97ef6f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Apr 28 10:39:31 2017 -0400 first pass implementation, no tests commit c0c8312582d0423afb8cd85a1e326e0afce82386 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Apr 27 23:49:24 2017 -0400 use context to track paths that are created, delete tmp files on exit commit d4183732019084383dccb5249ec65db3a0dada18 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Apr 27 23:08:47 2017 -0400 wire up the services we will use commit 6e0ec09301e8c1c6748a764f5e2979758536634b Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Apr 27 23:00:38 2017 -0400 start of implementation of service, validation of extensions commit 5dc7e2fa044b7485600533f70f9f80f7aa710f72 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Apr 27 17:02:49 2017 -0400 create controller class shell commit 1f71847dfec1f261d03f75238c091b0af393cc75 Merge: ae3a07ad3 219012b49 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Aug 27 12:48:15 2017 -0400 Merge branch '777-grok' into METRON-777 commit ae3a07ad3d89d5fd0a54692fb722f3b8ac26f857 Merge: a53aadc20 a2bae0bce Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 26 00:12:07 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 219012b4952d5695c867df9eec701f7ab6df77f8 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 15:21:18 2017 -0400 remove debug and fixup docs and tests for ResourceLoader handle paths missing / better commit 97cf0d738e381a1587e9a5b0ff71fb27cb6ea772 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 12:08:04 2017 -0400 cleanup SesnsorParserConfigServiceImpl commit 4fa2423be52784591977ccaadc09b1196c0126c6 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 12:01:50 2017 -0400 cleanup BundleSystemConfig commit 152100ced5a011008a6396dafdf513f1c9321463 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 11:41:00 2017 -0400 correct the tests commit 5f7e52b7495cdc271c8d5905effa2d649196da23 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 09:30:18 2017 -0400 fix grok tests, add new test for save statement commit b2eec55443fd9eae693758abadf97d895ac861b2 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 25 07:43:51 2017 -0400 working integegration with parser extensions as bundles and rest/ui cleanup and refactor todo commit 9c7f98f3e05f81ca2b8239085918bc74ea9b909d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 22 16:20:05 2017 -0400 refactor hdfs fixups so they are not required commit 5c98a1573655826cc7740ff211143eeba39e1f31 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 22 02:36:33 2017 -0400 first pass integration of rest with bundles and extensions commit f0923a6ca4ca0cbfa5fe541311ebe20126a28bad Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 16 16:35:20 2017 -0400 refactor hdfs and pattern loading Create GrokBuilder and ResourceLoader for re-usable and consistant file loading and path fixups tests create the correct ./target environment for rules to be loaded from ParserBolt injects the global configuration into the parser configuration Global configuration has new key for metron hdfs root new key is setup by ambari Now, give a parser configuration with a simple grokPath "/patterns/yaf" - it will be fixed to work in hdfs or local for tests - it will support yaf as a directory not just a file commit a53aadc2005d60dbea459821050ea56f9cc3a10a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 23 02:13:43 2017 -0400 fix test, missing comma from merge commit 4dc1650021034882459b0fbe0d9549c9227f1169 Merge: b25d5f1f1 d64afbc3f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 22 16:25:47 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit b25d5f1f1c976cf5716dc8af4786a75e1cba4e7c Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Aug 17 00:13:38 2017 -0400 Changes for synchronization and other things per latest review commit c0fd02d0c8296f79e8a8a479d1a6099e31900583 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 16 16:40:37 2017 -0400 fix indentation error commit 8455bbed2f3c5fb0935e17b2c574c4fd4fa65bab Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 19:04:25 2017 -0400 update doc per review commit 13942173cf4cd174d790a3ca835e8756c1a686e5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 19:01:59 2017 -0400 sigh, fix tests commit c5f8c34e4de8e6d456b97edd6f8a0d33b4819d69 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 18:36:10 2017 -0400 address review comments around exceptions, singltons commit 9526fed864aff3209dce7b319eda59961681f03d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 15:09:50 2017 -0400 javadoc for extension mapping commit d652fa093a89916bd84347984e3ec0d6a00c42e1 Merge: 88c005fd8 adb378f35 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 14:54:00 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 88c005fd8a63b13cc28ec27c66b5925566618d7a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 14:50:00 2017 -0400 per review - add comments about extensiable extensions namespaces and difference with nar commit fd887c1bfb107fa4633a0f50f48083e45b36bd63 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 14 14:40:37 2017 -0400 add dummy implementation of FileObject for system bundle commit 72abee73f00085ddd2019dd84f689b81d81da4d3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Aug 13 13:11:26 2017 -0400 document usage of BundleSystem commit b672ef3d31c2a1255566e120d2654b615b24009f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Aug 13 12:50:10 2017 -0400 javadoc additions commit ce8d98dcb69fc7e249fbfce5edd4afadaf2a14af Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Aug 13 12:14:17 2017 -0400 simplify, the extensionmanager has the classes, no need to do the init dance with the BundleThreadContextClassLoader commit d1b9871dd6f85afcfa9536b3bfd0ae1f650336f4 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 22:47:23 2017 -0400 test fix: need to clear after class commit 5cb6bdf4cf69396664489f5d620e057c386c6a3b Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 22:05:28 2017 -0400 javadoc and formatting commit 5bcc8c6f066f5339578c329b24d92d746a803562 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 22:01:45 2017 -0400 unit test and some fixups commit 97295d72470dbf5fccc68cb0073f123fe3a16391 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 21:44:32 2017 -0400 returning a value would work better commit 70f2106f9b58d04bee3d7cee981535194d2285c5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 18:53:57 2017 -0400 simplified the call interface for creating classes commit 103f869f4901de0ae1c007702b526a213793b9c5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 17:38:33 2017 -0400 we do not need to do the mapping, we are not using them commit 2f93bfc3356230a3362989c1d4c0d825236e31a3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 17:11:16 2017 -0400 more javadoc detail commit 3edafdb5ff2b09b58bddd900ff1951347f0679ff Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 16:58:09 2017 -0400 enforce immutability in the BundleClassLoader as well commit 4cf9cb4c7aad3458ed40880dd5695b1d2acb0d44 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 12 13:15:28 2017 -0400 based on review, refactor the ExtensionManager for Immutability and Thread Safety commit 97ef4f0b509ad9b110165665382223373392e25a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 11 22:21:49 2017 -0400 per review, add clarification commit 22ba18b569b05bf040d1fca1f331602210e2e5ec Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 11 22:19:13 2017 -0400 per review fix spelling commit 571168d897cdd10bc616cef92b613763f76a1842 Merge: ea6240a0e 86b0f137e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 11 14:15:26 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit ea6240a0e17c5ac7261b6aa41ffc75f512573cca Merge: 8b5d66582 b3148a182 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Aug 11 08:51:50 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 8b5d66582ccfe391ff139ecd8f4bbc58d128542f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Aug 10 17:30:42 2017 -0400 remove reference to nifi commit 27c7de4e563820d2722ac7d7a7e79105882da400 Merge: 0b583e0ff 7d554444e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Aug 10 10:12:51 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 0b583e0ffb31c04b75d9b7375c4058999a3930ce Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 9 15:36:47 2017 -0400 per review, added some validation, also renamed variables and accessors commit b402517e858554761a27f7624c65a2538e5c0c89 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 8 16:57:49 2017 -0400 the headers, the headers commit 6690398f242ab9279786eaa83f5a1e67cb6e4103 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 8 16:52:02 2017 -0400 add header commit c6e9a6abc147a2c07a7e2a884a054c4f7fd29679 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 8 16:50:20 2017 -0400 remove unused import commit 4bad9483cc00b6632c13c482deeba7b2cc349bf4 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 8 16:48:16 2017 -0400 fix working dir reference commit f42eef2146b539e47914f625df4b3808bcc9491f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 8 16:43:06 2017 -0400 Changes per reviews commit 4f8409abc06191c3453e30756afa50941870a1d8 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 7 16:09:12 2017 -0400 fix post merge bits commit 54602cf3a706beb75bb5da17e71c08ce20b7057d Merge: 0ffac1616 813adf2da Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Aug 7 15:35:37 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 0ffac16167576bb99128892868f99f852ea0810e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 5 08:47:26 2017 -0400 remove notice about versions matching commit b1a96448c44e20768638fb925191eb66f2ca7d7f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 5 08:41:59 2017 -0400 change plugin inclusion in archetype pom, this resolves requirment for version to match metron version commit bf5ec2f0e4df5688c12813da17651e6d0d2c3080 Merge: 04727ca8c df25d81c4 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Aug 5 08:00:32 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 04727ca8c448e2845e48a8a95812e6bf27b2c13d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Aug 3 13:30:29 2017 -0400 update document to remove references to the working dir commit 3212c1bc9a0c31c607c7652a27519ddc97879999 Merge: c78592921 b2375a1f1 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Aug 3 08:11:57 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit c78592921d6be4c2f694b93245cea77f72e3581c Merge: 4b9a1245a 27ab28923 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Aug 2 17:24:52 2017 -0400 Merge branch 'master' of https://github.com/apache/metron into METRON-777 commit 4b9a1245a24f7d4b6dbac95fa40334c0fb0bea69 Merge: 7a97e41de ec959d20e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 1 12:00:14 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 7a97e41de770e4c95e3715feae080dc96f20a9c7 Merge: afe4b9e29 7b9f20c93 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Aug 1 09:24:49 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit afe4b9e29a1fb711cf170d8a844275ca22929965 Merge: 5bb6a33b3 c99df8ff1 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Jul 31 21:12:28 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 5bb6a33b36962a8cec8a7a3f3f1941f87dc068bc Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 23:15:16 2017 -0400 refactor/rename VFSUtil to FileSystemManagerFactory, which is what it is commit 2e48f087a9c320ab2094f606005a87030da1e153 Merge: f1cdcd793 ba46fa734 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 13:00:51 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit f1cdcd79374e603dd3a90e4e067c119d1bb8bf7a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 12:17:34 2017 -0400 for BundleThreadContextClassloaderTest, fix last path. in travis if the folder is empty it may not be there? commit 98545d1526e61564bbabf2268f4743f7a2b7c646 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 12:03:39 2017 -0400 more travis fix attempts commit 868cf6cb103775ef30cf45a0f4f042b50ebdd9f3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 11:45:01 2017 -0400 attempt to fix travis only failure commit 66a0c8205c778469d99dd1108f7735ca5668c3f6 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jul 29 10:46:37 2017 -0400 Based on feedback from reviews and discussion with mattf, this commit represent a major refactoring for the bundle functionality. While trying to resolve the ugliness bought about by working around VFS's read only support for HDFS, I have come up with a different approach. While this approach makes bundles-lib significantly different than nar-utils, it makes it much more simple, and may arguably be an approach worth concidering. The original bundles-lib maintained the nar system's method of unpacking the bundle into a 'working' directory, and then using having the classloader load from that directory structure. Trying to avoid too far a departure ( at the time ) from nifi, bundle-lib maintained the 'create folders and files as nec' scheme found in nifi. Since nifi is always working with local disk and File objects, this is ok. But with VFS and HDFS this did not work. The work around was to create a poop static 'class cluster' type step with multiple implemenations of FileUtils ( VFS and HDFS ). The new approach, which is a more willing departure from the nifi approach, is to avoid having to ever do writes ( file or folder creation ) from bundles-lib at all. This allows for a simplier, single implementation of the file utils and the removal of the anti-pattern. It also removes the need for a 'working' area. The new approach is to better utilize the capabilities of VFS such that we never have to unpack the bundle at all. VFS can load JAR files as Filesystems. I have created a new Classloader ( based on the VFS classloader implemenation from the VFS project ) that is bundle aware. It can load the bundle and it's dependencies as resources and make them available. This is what removes the requirement to unpack the bundles into a working directory. As such, the bundle-lib has been refactored to move from the 'working directory' to only needing to work with the bundle in the lib. I have also removed the doc directory support since we are not using it. We no longer need multiple file utils, the hdfs working directory, and other things. Thanks to mattf for the time and discussion. commit 9fffd6187b2a36448a3a4caf7472d1b5da10b426 Merge: bbc022857 9355a0465 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 23:32:23 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit bbc022857ffbc553cdb9519910b3a466765c0c62 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 19:00:19 2017 -0400 use MessageParser for consistantcy commit 05797505d5ef4f0e562a2e1621ff455b709bd515 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 18:40:57 2017 -0400 remove more nar-isms. base tests on archetype build parser projects instead of old nifi processors commit 7c84fc49f0010b517d72f87843efc7de6a93d527 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 14:16:46 2017 -0400 refactore type property to packageType per review commit 4650125713bb9bc0e4d3184b9434eb216b750524 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 08:25:40 2017 -0400 remove nar references per review commit f84efb72047e19d0c9bc8e66e96c62bba627d878 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 08:18:42 2017 -0400 remove nar references commit c4f346867c6aa0e46a6f6a596c81eb3457cf0804 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 08:12:07 2017 -0400 refactor package per review commit fc38b3df915b47495d9f7791736255aa0fbb6df6 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 08:06:52 2017 -0400 more incubator instances commit 132763c75435dfc804b435d16580bb1d8a48a273 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 27 08:04:10 2017 -0400 remove incubator references commit 8f274d835e0fb787bcb74a53c98a3ba53bc7e2ba Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 26 11:24:30 2017 -0400 reformat based on review commit a6dffb246599b79abbfc03c63755f01f221b7c54 Merge: 0d9a43a93 badc6cf97 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 26 11:23:37 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 0d9a43a9393a20309845c668a206005f45ea4617 Merge: fcb8ef72e 309d8097a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Jul 24 10:05:28 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit fcb8ef72e243c34e7e2f2191b968431f3b01fe0a Merge: 7ee99ef3a cf7043c59 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jul 20 10:44:54 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 7ee99ef3aa42b016f87fe4b6b69a507712aa0102 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jul 18 09:52:54 2017 -0400 reverting last change, it worked in other travis repo commit 1b095e5ffce8ca9add693e223470ce4f42660874 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jul 18 09:49:06 2017 -0400 try to resolve travis not building bundle commit 06e4d7626bb9b40f704816109735b88973f85fcf Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jul 14 08:58:32 2017 -0400 need to install bundle now that we moved the other install phase commit 3ef2059c72f00108e52a4f106cda3280e170b355 Merge: 6d4be0f4a 954b5ea13 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jul 14 08:54:09 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 6d4be0f4a25bdb7e87b89a2a3622e913425d97c3 Merge: b673fc7b9 dcf768297 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 12 10:43:40 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit b673fc7b933cdfefc64db81f7853275161d4bc7d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jul 7 15:59:16 2017 -0400 fix version in new poms commit 8b70e08071738eeb7406f8df39581753068daca9 Merge: d12d3fcb5 a8f35363f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jul 7 15:35:30 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit d12d3fcb5a72437f87e53f0d2a34e84d4ffba354 Merge: efecf3a9a f7a94f2e6 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jul 7 12:20:26 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit efecf3a9a3240c0404be547fc153bdf7825add23 Merge: 31d52d436 df270feea Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 5 12:56:01 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 31d52d43632faf2720bc9dfecd1f0d5cb230db71 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 5 11:02:58 2017 -0400 add documentation of multiple configurations of multiple parsers in one extension commit 3e9c3972c053e96cf0feea380028c69dae062870 Merge: 8a53de80a cc7bbc948 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jul 5 09:10:24 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 8a53de80a4bf146759eabff3e58aab6e9a28d684 Merge: e67a55edf a5b13777a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jul 4 00:43:25 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit e67a55edfd4c0734dcde3e65c9ec660ff2d38e59 Merge: 679d56fd7 df94ed405 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 30 12:36:20 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 Fix integration tests to work with extension paths commit 679d56fd7df8819f8ebf631bf045f34b23f27290 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 29 20:58:54 2017 -0400 per review, changed version in plugin to 2.4 also updated metron version in the archetype test files commit 2a7e8d3acba321174330b160fa98bef072632251 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 29 07:57:58 2017 -0400 fix extra project node per review add ASF headers to other poms commit 6d984823fec9696854b94d0c94c42aabb0c32739 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 29 00:08:59 2017 -0400 removed deployed pattern directory and common pattern files for parsers that do not use patterns commit 2db581469e40a5dbb000ba9c5dc430586f68ee5e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 28 23:23:27 2017 -0400 fix spacing, fix name commit b8e65fbe8cc7847731bcb22b3e80be68bc9dee1c Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 28 23:11:13 2017 -0400 deployment readme commit 86190f830e25846d1b4f89f7d2a10377b5d59e74 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 28 22:19:31 2017 -0400 readme changes per review commit a8f34eff42e575f110e4260e050ab95393bfbff0 Merge: b3daa9ed2 095be23dc Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 28 22:06:44 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit b3daa9ed211da74007ec22b63184002e60672b19 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 27 18:13:28 2017 -0400 start of deployment documenation commit 50290332ffed4027766578e7ec6195ffe4ee9a08 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 27 13:07:08 2017 -0400 remove reporting tag, we will inherit here - per review commit 16f79994cb7147e8a3e60d97def49199b99cf397 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 27 11:29:14 2017 -0400 more documentation work around packaging and terms commit d4f28ed8f5bc23b1cb3dac29834265831a9843ff Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 27 08:05:25 2017 -0400 fix readme links commit 72eed83899ebeebd2d63aaf9e324579a310f9865 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Jun 26 18:15:46 2017 -0400 start documentation of packaging and terms commit 33682eba5fa0e7883f57184c9c97752a05fd6661 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 22 09:40:26 2017 -0400 more fge requirements commit 269f739dcfb47038497b2249ef315037b6d0cc5f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 22 09:16:28 2017 -0400 put the fge json stuff back in the parsers that use them in test, as opposed to adding fge as a shipped dependency commit bfafd30d356766f9408b6e34ac81d7bf7053f81b Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 21 16:22:20 2017 -0400 create a testing pom project to hold testing only dependencies - per review remove some nar references commit ee6fbd4b8307f5ae7032db88dd7fd1935414b000 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 21 09:35:55 2017 -0400 Per review, remove .gitignore left over from nifi plugin as separate project commit ced63804f40438eb8e3bef44acddc1e2d57ca90d Merge: 96957271c b76bcd5f2 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 21 09:30:59 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 96957271c8ed849b786b18373054347bbcb09bd6 Merge: 22c0cba89 de2c871ab Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Jun 19 11:09:46 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 22c0cba895625a7020b49ddf6875b8b05c8ad700 Merge: 02ef46ec7 f523c1795 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 16 10:56:02 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 02ef46ec72161b952b56b72faa4d8067418486c3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue Jun 13 14:21:34 2017 -0400 per review, move metron-extensions under metron-platform associated fixes for path related things commit a6408061e2122a0c61ca6d82b2dca3a2de3745d3 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Jun 11 17:00:03 2017 -0400 add readme for bundles-lib this readme specifically references the relationship to apache nifi nars this readme is based on the nifi developer guide for nars commit df78af106ff74f97675950145736f5c73282b739 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Jun 11 10:52:41 2017 -0400 refactor based on feedback to use the metron integration test classes this should be OK because that jar does not pull in other metron jars and is test only commit 9e012a5e37b94737ff4faa5a26aa27b4a0c52e0f Merge: 9099f3682 30d0e2a6f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 9 16:07:08 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 9099f3682e1397f9901e568f1efab85d0bcb3356 Merge: e9b8b9bbf 7d6121bfd Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Jun 9 07:01:34 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit e9b8b9bbf3756920eaaddee6a47891993a4e6aa6 Merge: 350ac94e4 61cbab46e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 8 08:54:00 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 350ac94e43652d7b3916bd07b61dba88f12090af Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 7 23:36:39 2017 -0400 fix markdown so site-book does not crash commit 145465fce07a734637aafca0416227e94c1c841e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 7 23:13:50 2017 -0400 Added a guide for adding parsers to the system commit 572212ff4dc6c1c63c1b1fd37030b2fa6c4d392e Merge: 9bee17ce9 5b72da7be Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 7 12:45:44 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 9bee17ce99e89477a67eacf0d181d38301b3d884 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 7 09:12:55 2017 -0400 per review: use inherited reporting in pom commit d8b09c6c31a44e7342dd2b7b3ecf294a670d2c9c Merge: b25358f60 605a3cc6d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed Jun 7 09:07:42 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit b25358f6044d68b3a49a70aae1cd8abd63638f3b Merge: bf726813d d0e1ba508 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Jun 5 21:43:47 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit bf726813dbb6a0f899c77f103af967ce0e0a47b6 Merge: 48ce8e26b e31ed2f7d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Jun 3 08:58:45 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 48ce8e26bcecae166856834fc323a431727a63b2 Merge: 23e2a0bce 85872bd68 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Jun 1 11:34:03 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 23e2a0bce63ffcd0b20666c25cde5dc3b03e82b9 Merge: f12922d6d 61105c715 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 31 10:20:37 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit f12922d6de8485c21d28ca12b85f72ed37357feb Merge: 0dd8cf8c9 a2634f31d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 25 22:01:56 2017 -0400 Merge branch 'jz-metron-777' into METRON-777 commit a2634f31d2b16f9ece00dbae49a8b0265f091cfd Merge: 0dd8cf8c9 2e66ea397 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 25 22:00:07 2017 -0400 Merge branch 'METRON-777' of https://github.com/JonZeolla/metron into jz-metron-777 commit 0dd8cf8c9ab89b43d543de8b5237010a948540a9 Merge: 382a29e18 08f95dc8f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 25 21:58:05 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 2e66ea397ffc82caf6a76d47ba3651754361dd51 Author: Jon Zeolla <zeo...@gmail.com> Date: Thu May 25 18:07:23 2017 -0400 Trivial documentation changes commit 90a51deb14078b57966f3ad8e2f7e000cbb74ca3 Merge: 7ef929ae4 382a29e18 Author: Jon Zeolla <zeo...@gmail.com> Date: Thu May 25 17:52:55 2017 -0400 Merge branch 'METRON-777' of https://github.com/ottobackwards/metron into METRON-777 commit 382a29e18f9ab161f0a21d92c1ba3ce6f2601eeb Merge: eb2062be4 e2197316d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon May 22 08:01:38 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit eb2062be4c14d0549a6ce2504dbbe29ea5510141 Merge: b73f35fc2 356881ad2 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon May 22 07:58:50 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit b73f35fc25982556ad469de409e15c72b0305a8a Merge: 97c1be132 ce381f3e0 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri May 19 11:33:30 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 97c1be132c114a0f3f729e5f9887adb5aa49f582 Merge: 225fb4e38 be0307659 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 16 22:22:33 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 225fb4e38d6e5ec5eff877be2a84c8fd29af0c3e Merge: 5565ad661 345eba8d5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri May 12 14:12:03 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 5565ad661f4143234b67230f7cfd420ea1bce91a Merge: ab0aad849 e375936a1 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 10 15:47:58 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit ab0aad849319117dd3d964247861cf09bc3e7822 Merge: 6c0201b85 8532e2848 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 10 08:57:21 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 6c0201b853ffd079f02589011c57cb6a960475e8 Merge: e3b71aed7 e08b7d47f Author: Otto Fowler <ottobackwa...@gmail.com> Date: Tue May 9 09:55:31 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit e3b71aed74271cf255be5fad072a99506f553d2e Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat May 6 12:08:58 2017 -0400 use simple json and not the JSONUtil Instance, there are issues in storm loading commit 1e0e30562b4b526d9512bb1dc06077674d6b3277 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri May 5 14:18:32 2017 -0400 update archetype based on changes to loading and configuration commit a3e63555e4e76486be58ddd6f818006ff42cdd1f Merge: 5f74fc3a1 716bda326 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 4 23:42:54 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 5f74fc3a15fe5e56f9e3483fa4c25961ffb284dd Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 4 16:25:18 2017 -0400 be sure to clear out before and after, we cannot assume order of maven build commit 995a4d746fa21400a55e9571957878ef2d5b48d8 Merge: de4884563 38d26d432 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu May 4 09:46:58 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit de48845634ae1cecb258175326592a28f5b3f8fe Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 3 22:40:55 2017 -0400 second attempt to fix plugin errors with jacoco commit 9cafe970b40d551a846fc442ac121591ab1d0d6d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 3 22:18:29 2017 -0400 specify the plugin to fix dependency problem commit fd021e42a2cdb80676426b65ae227ac9f44fcd6a Merge: ac92d7b88 1d27a32d5 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Wed May 3 10:34:31 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit ac92d7b88404f8b8bc825c71a4aeee5aa015d757 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Apr 30 09:44:58 2017 -0400 instead of hard-coding in metron-parsers-extensions, overload in the archetype to use metronVersion commit e21efb89a37d034e36cb82dab6c0b862cad3777b Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Apr 30 09:16:41 2017 -0400 make dependency explicit, I think parent version is not what we think it is when running from archetype, causing the extension versio to be used for this dependency commit 158f463f121cbca2cf2c4d3c1cd9cf7b707e1d23 Merge: c25585d10 47e5aa70a Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Apr 29 08:58:41 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit c25585d10529ca2048d2ed0df4d8dbd904d3db17 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Apr 29 00:07:11 2017 -0400 use the configured extension list, do not hardcode commit bc7cc2d823d54be58fae13ef94f671457d4c10f0 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sat Apr 29 00:05:41 2017 -0400 use correct testing bundle commit 7760425b02142f01f37d58876aa09fb88b197c82 Merge: c18cbe3a8 5dd87886d Author: Otto Fowler <ottobackwa...@gmail.com> Date: Thu Apr 27 06:45:37 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit c18cbe3a8847fbed7b5f6ba9182b58f01d142c2c Merge: e8caaf51e 41b5b1050 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Sun Apr 23 12:14:15 2017 -0400 merge apache/master commit e8caaf51e3f3d640e9ec1c3c548bd722d595687d Merge: 850154cb0 e6992d568 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Mon Apr 17 08:19:02 2017 -0400 Merge remote-tracking branch 'apache/master' into METRON-777 commit 850154cb086bd9f825daa6797b575cde95392a87 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Apr 14 08:55:37 2017 -0400 fix package names commit 864d320d91c522dfc2eb63fc12341f316a3f8952 Author: Otto Fowler <ottobackwa...@gmail.com> Date: Fri Mar 17 00:56:49 2017 -0400 Metron Extension system Based on Apache Nifi Nars NAR changes * new lib , rebrand to bundles from NAR * port to VFS/FileObject from File based * ability to set property values * Rework FileUtils so that you can derive and override * added initializers to set 'classes' that we care about instead of hard coding them, still needs defaults * added components nec. for integration tests ( do not want dep. on metron-* ) * VFSClassloader for NarClassLoader * Hdfs based integration test version of unpacknars tests * HDFS ( filesystem ) based fileutilities to cover for writes to hdfs, since VFS is currently R/O HDFS * modified plugin to support configuration of outputs * use class index not service loader ( both subclass and annotated supported ) Archetype * Parser Extension archetyp * incudes all configuration * creates tar.gz with bundle and configuration * class index support ( automatic generation ) Extensions * new extensions modules * parser * archetype built module for each parser type * support for configuration only parsers with tests Parsers * moved all but json, csv, grok to extensions * Bolt now loads from bundle properties Deployment * rpms for parsers * create extension directories * ambari initializes zookeeper per parser * amabri creates hdfs directories * ISSUE: Writing to hdfs Rest-API * only test against parsers in metron-parsers * still needs integration ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---