When things change in hdfs, how do we know

At the moment, when a grok file or something changes in HDFS, how do we know? Do we have to restart the parser topology to pick it up? Just trying to clarify for myself. ottO

Re: [GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Maybe we need an adding support for a new platform doc On January 25, 2018 at 19:37:47, nickwallen (g...@git.apache.org) wrote: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 > @lvets: Just for my understanding, but why Ubuntu Trusty? In April that

Re: Metron User Community Meeting Call

Thanks! I'll be there. Excited to hear Ahmed's successes and challenges. -Kyle On Thu, Jan 25, 2018 at 7:44 PM zeo...@gmail.com wrote: > Thanks Otto, I'm in to attend at that time/place. > > Jon > > On Thu, Jan 25, 2018, 14:45 Otto Fowler wrote: > >>

Re: Metron User Community Meeting Call

Thanks Otto, I'm in to attend at that time/place. Jon On Thu, Jan 25, 2018, 14:45 Otto Fowler wrote: > I would like to propose a Metron user community meeting. I propose that we > set the meeting next week, and will throw out Wednesday, January 31st at > 09:30AM PST,

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 I am liking a combination of the suggestions from @lvets and @cestella. Something like this maybe? * `dev-on-centos6` * `dev-on-ubuntu14` I like the name because of points made

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 > @lvets: Just for my understanding, but why Ubuntu Trusty? In April that will be 2 full Ubuntu LTS versions behind the then current one... Because that's the requirement that I need to

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/903 @lvets trusty is 14.04. As far as I'm aware the only newer LTS is 16.04, with a new one expected in April. https://wiki.ubuntu.com/Releases ---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user lvets commented on the issue: https://github.com/apache/metron/pull/903 - Wouldn't `dev-on-centos` and `dev-on-ubuntu` or even `metron-dev-centos` and `metron-dev-ubuntu` make more sense? I don't think we want people to expect that this VM is sufficient to run in

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user cestella commented on the issue: https://github.com/apache/metron/pull/903 Honestly my only beef with this is the directory naming. I think I'd prefer `metron-deployment/vagrant/{centos_$VERSION,ubuntu_$VERSION}` because it's possible that we may want 2 separate versions

[GitHub] metron pull request #902: METRON-1413 Add Metron Commit Tool

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/902 ---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 Thanks @ottobackwards . I'll see if we can get any more reviewers before I merge this. ---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/903 +1 -> ran up both images, everything checked out Ship it ---

[GitHub] metron issue #901: METRON-1410 [MPACK] Check for existing HBASE tables befor...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/901 I'm going to wait for @lvets a chance to try his scenario ---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 > @ottobackwards : Do we expect there to be issues with 2.6? Is this PR and Casey's 2.6 pr going to conflict or have issues? Yes, we will need to retest one or the other. I am open to

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 I spun this up again on both Ubuntu and CentOS. Both worked successfully. I am happy with it now @ottobackwards . Give her another go when you can. Thanks. ---

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/910#discussion_r163967410 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java --- @@ -140,9 +147,9 @@ default Document

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/910#discussion_r163962696 --- Diff: metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java --- @@ -140,9 +147,9 @@ default Document

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/902 +1 ---

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/910 Yeah, this is the first step to moving it out, I'd say. This was useful independent of #907 because it made some things a lot cleaner (namely fewer `new TypeReference>() {}`

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/902 I merged with master, so I expect Travis to be happy now. Just need +1s and I'll get this in to allow for any follow-ons. ---

[GitHub] metron issue #901: METRON-1410 [MPACK] Check for existing HBASE tables befor...

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/901 +1 Looks good @ottobackwards. Thanks for fixing this! I have not tested this myself, but it looks solid. Let me know if you'd prefer me to spin this up to get a second test run in.

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/910 @ottobackwards I was envisioning a metron-deps project of some sort. Jackson and Guava are perennial offenders in the Hadoop stack, and a facade of some sort would be useful to insulate us from the

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

Sure it helps, but I am not sure I answered __your__ questions? As I mentioned, we already use Map rawMap = JSONUtils.INSTANCE.load(originalString, new TypeReference>() { }); So, using JSONPath which is using the same object mapper operation under the covers

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

Heh, as I said, I was looking in Python. For SAX-like JSON parsers I found numerous libraries, most built on top of an underlying Python library named ijson, which is itself based on a C library called yajl. The yajl page (http://lloyd.github.io/yajl/ ) lists a double handful of language

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/910 This looks awesome at first look. I'm a big fan of doing this regardless of the shading issue. The only thought that comes to mind is that there is a tipping point where a *Utils class isn't

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/903 @nickwallen let me know when you feel ok about it, I'll run it through again. ---

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/910 METRON-1430: Isolate jackson from being used as arguments or returns from JSONUtils ## Contributor Comments Currently jackson is used as part of our internal API to JSONUtils. The problem

Metron User Community Meeting Call

I would like to propose a Metron user community meeting. I propose that we set the meeting next week, and will throw out Wednesday, January 31st at 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting will be held over a web-ex, the details of which will be included in the

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/903 > @ottobackwards : Failure during vagrant up for metron-on-ubuntu Thanks, Otto. Yep, I messed that up. I pushed the fix, but I am going to run through full CentOS and Ubuntu deployments

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

In other words, I don’t believe the issue is parsing, but rather searching and extracting. I have used SAX with xml as well, can you point me to the json equivalent you found? On January 25, 2018 at 13:01:58, Otto Fowler (ottobackwa...@gmail.com) wrote: JSONPath is indeed what nifi uses. I

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

JSONPath is indeed what nifi uses. I used their implementation as a guide. I believe starting with a path would be a good minimum viable, a good start. We could support multiple paths of course. Beside the fact that I knew NiFi used this approach, I believe that JSONPath provides a flexible

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

Hi Otto, Oddly, I had reason a couple weeks ago to try to figure out a streaming parser for very large json objects -- altho it was in Python rather than Java. Search showed two basic approaches, both unsurprisingly modeled on xml processing: - SAX-like parsing - XPath-like parsing Both are

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/903 Failure during vagrant up for metron-on-ubuntu ``` 2018-01-25 10:41:49,302 p=37541 u=ottofowler | fatal: [node1]: FAILED! => {"changed": true, "cmd":

[DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

While it would be preferred if all data streamed into the parsers is already in ‘stream’ form, as opposed to ‘batched’ form, it may not always be possible, or possible at every step of system development. I was wondering if it would be worth adding optional support to the JSONMap Parser to

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/903 Trying this now. Only comment on the content here is there is a _lot_ going on in this pr. A lot of while I'm here I might as well work. It might have been better to have kept this

[GitHub] metron pull request #901: METRON-1410 Check for existing HBASE tables before...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/901#discussion_r163868868 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py ---

[GitHub] metron pull request #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/907 METRON-1427: Add support for storm 1.1 and hdp 2.6 ## Contributor Comments Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 because of some classpath issues and not

[GitHub] metron pull request #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user cestella closed the pull request at: https://github.com/apache/metron/pull/907 ---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/909 @cestella You're right about that - the general goodness. We (I) had made the mistake of doing some test refactoring in the ES upgrade branch that would have been better done in a separate PR,

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/909 +1 pending travis ---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user cestella commented on the issue: https://github.com/apache/metron/pull/909 Sounds good, +1 Good work here, that test was confusing. ---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/909 I think this is useful outside of any Solr work and I intended for it to go into master. ---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/909 @cestella This is going into master per the merge notes, right? ---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

Github user cestella commented on the issue: https://github.com/apache/metron/pull/909 Should this be against master or should this be committed against the Solr branch? It *seems* like this is general purpose goodness and maybe fits in master, but I wanted to double check. ---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/909#discussion_r163859481 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java --- @@ -724,6 +503,52 @@ public void

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/909#discussion_r163859462 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java --- @@ -724,6 +503,52 @@ public void

[GitHub] metron pull request #906: METRON-1426: SensorIndexingConfigControllerIntegra...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/906 ---

[GitHub] metron issue #906: METRON-1426: SensorIndexingConfigControllerIntegrationTes...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/906 +1 ---

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/902 I have another script I want to add for tracking master in feature branches after this as well ---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/907 Hi @cestella , I did a 12-node deploy on CentOS 7 with this patch. Post-kerberization, I noticed the following errors in Metron REST. Is this a related issue or a different one? ```