Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Michael Miklavcic
"METRON-685" - wow, that one was a long time coming. On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > +1 binding > > I validated the release tarball, ran the full test suite and validated the > CentOS 6 development environment. Everything looks solid. Let's ship it. > > On Wed, May 8, 2019

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Nick Allen
> I suppose we could always allow this to be overridden, also. I like an on/off switch for the "original string" functionality. If on, you get the original string in pristine condition. If off, no original string is appended for those who care more about storage space. I can't think of a

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Otto Fowler
What about parser chaining? Should the original string be from kafka, or the last parsed? On May 10, 2019 at 19:03:39, Simon Elliston Ball ( si...@simonellistonball.com) wrote: The only scenario I can think of where a parser might treat original string differently, or even need to know about

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Michael Miklavcic
I think that's an excellent idea. Can anyone think of a situation where we wouldn't want to add this the same way for all parsers? I suppose we could always allow this to be overridden, also. On Fri, May 10, 2019 at 3:43 PM Nick Allen wrote: > I think maintaining the integrity of the original

[DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Michael Miklavcic
When adding the capability for parsing messages in the JsonMapParser using JSON Path expressions the original behavior for managing original strings was changed.

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Nick Allen
I think maintaining the integrity of the original data makes a lot of sense for any parser. And ideally the original string should be what came out of Kafka with only the minimally necessary processing. With that in mind, we could solve this one level up. Instead of relying on each parser to do

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Otto Fowler
+1 On May 10, 2019 at 13:57:55, Michael Miklavcic (michael.miklav...@gmail.com) wrote: When adding the capability for parsing messages in the JsonMapParser using JSON Path expressions the original behavior for managing original strings was changed.

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Simon Elliston Ball
My understanding is that chaining preserves (correctly to my mind) the original original string. In other words: unless the message strategy is raw message, the original string is just passed through. Original string therefore comes from outside Metron, and is preserved throughout Metron

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Otto Fowler
The original string would be the string specified as the message body, thus each message in the chain produced would just be the bytes passed in, from a specific field in the incoming message. On May 10, 2019 at 19:55:28, Simon Elliston Ball ( si...@simonellistonball.com) wrote: My

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Nick Allen
I really enjoyed the retro, 3-digit vibe on that one. On Fri, May 10, 2019 at 4:38 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > "METRON-685" - wow, that one was a long time coming. > > On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > > > +1 binding > > > > I validated the

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Michael Miklavcic
+1 binding Validated same as Nick. Mike On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > +1 binding > > I validated the release tarball, ran the full test suite and validated the > CentOS 6 development environment. Everything looks solid. Let's ship it. > > On Wed, May 8, 2019 at 6:50 PM

Re: [DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

2019-05-10 Thread Ali Nazemian
Oops. Turns out I totally missed this email. Thanks, Mike for your reply. Spark native support of Kubernetes has been added very recently and it is not really at the stage that can provide all the aforementioned features. There is no doubt that Spark is a powerful tool and it is been widely used