Having 2 different patterns for configuring field name transformations on
read vs write is confusing to me. I agree with both of you that
normalizing on '.' and not having to do the translation at all would be
ideal. Like you both suggested, we would need some utility or script to
convert
+1 (non-binding)
* mvn clean package and build RPMs
* Brought up Metron on 12-node CentOS 7 cluster using generated RPMs and mpacks.
* Bro, YAF and snort - ingest and validated events in Alerts UI
* Add squid telemetry, ingest and validated events in Alerts UI
* Management UI, Alerts UI and
Well, on write it is a transformation, on read it's a translation. This is
to say that you're providing a mapping on read to translate field names
given the index you're using. The other approach that I was considering
last night is a field transformation REST call which translates field names
To be clear, I'm not even suggesting that we create any tooling here. I'd
say just a reference to the ES docs and a call-out in Upgrading.md would
suffice as long as we have some strong reason to believe it'll work. As
far as I'm concerned, the sooner we're out of the business of transforming
PutMetronEnrichementRecords* ;)
On June 5, 2018 at 10:32:43, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
Do we, the community, think it would be a good idea to create a
PutMetronEnrichment NiFi processor for this use case? It seems a number of
people want to use NiFi to manage
I just don't know if telling users to do a bulk upgrade of their indices is
sufficient enough of an upgrade path. I would expect some to have
downstream processes dependent on those field names, which would also need
to be updated.
Although, we could tell users to do any field name conversions
I agree completely. I will leave this thread open for a day or two to give
others a chance to weigh in. If no one opposes, I will creates Jiras for
removing field transformations and transforming existing data.
On Tue, Jun 5, 2018 at 8:21 AM, Casey Stella wrote:
> Well, on write it is a
The problem, as you correctly diagnosed, is the key in HBase. We construct
the key very specifically in Metron, so it's unlikely to work out of the
box with the NiFi processor unfortunately. The key that we use is formed
here in the codebase:
I'd be in strong support of that, Simon. I think we should have some other
NiFi components in Metron to enable users to interact with our
infrastructure from NiFi (e.g. being able to transform via stellar, etc).
On Tue, Jun 5, 2018 at 10:32 AM Simon Elliston Ball <
si...@simonellistonball.com>
I would definitely agree that the transformation should be removed. We have
now however added a complex generic solution in the backend, which is going
to be noop for most people. This was done I believe for the sake of
backward compatibility. I would argue however, that there is no need to
Agreed, we should definitely have a clear picture about how to do that,
maybe even a worked example in the use-cases that we can reference. I'm
just saying we don't need to migrate ES docs into Metron, but rather
reference them as much as we possibly can.
On Tue, Jun 5, 2018 at 11:38 AM Otto
I am in favor of removing the `FieldNameConverter` abstraction as an end
state. Although, I don't agree with Simon that we could have just done
that directly without providing a backwards compatible solution as was done
in #1022. There are too many touch points that rely on that conversion and
It is still our user list and dev list that will have the burden of talking
folks through that.
On June 5, 2018 at 09:58:32, Casey Stella (ceste...@gmail.com) wrote:
To be clear, I'm not even suggesting that we create any tooling here. I'd
say just a reference to the ES docs and a call-out in
Are there consequences with Kibana as well? queries, visualizations,
templates they may have?
On June 5, 2018 at 12:03:44, Nick Allen (n...@nickallen.org) wrote:
I just don't know if telling users to do a bulk upgrade of their indices is
sufficient enough of an upgrade path. I would expect
Yes, anything using elastic would need the field names changed. That said,
people who are on such an old version (eol) will need to not the bullet with ES
compatibility as some point.
Simon
> On 5 Jun 2018, at 17:17, Otto Fowler wrote:
>
> Are there consequences with Kibana as well?
Similar to Bro, we may need to release out of cycle.
On June 5, 2018 at 13:17:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
Do you mean in the sense of a separate module, or are you suggesting we go
as far as a sub-project?
On 5 June 2018 at 10:08, Otto Fowler wrote:
> If we
If we do that, we should have it as a separate component maybe.
On June 5, 2018 at 12:42:57, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
@otto, well, of course we would use the record api... it's great.
@casey, I have actually written a stellar processor, which applies stellar
to
Aren’t people who are on an old version of ES everyone pre 0.5.0? Like all
the metron users?
On June 5, 2018 at 12:31:30, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
Yes, anything using elastic would need the field names changed. That said,
people who are on such an old version
Do you mean in the sense of a separate module, or are you suggesting we go
as far as a sub-project?
On 5 June 2018 at 10:08, Otto Fowler wrote:
> If we do that, we should have it as a separate component maybe.
>
>
> On June 5, 2018 at 12:42:57, Simon Elliston Ball (
>
@otto, well, of course we would use the record api... it's great.
@casey, I have actually written a stellar processor, which applies stellar
to all FlowFile attributes outputting the resulting stellar variable space
to either attributes or as json in the content.
Is it worth us creating an
Nifi releases more often then Metron does, that might be an issue.
On June 5, 2018 at 14:07:22, Simon Elliston Ball (
si...@simonellistonball.com) wrote:
To be honest, I would expect this to be heavily linked to the Metron
releases, since it's going to use other metron classes and dependencies
I agree with Simon here, the benefit of providing NiFi tooling is to enable
NiFi to use our infrastructure (e.g. our parsers, MaaS, stellar
enrichments, etc). This would tie it to Metron pretty closely.
On Tue, Jun 5, 2018 at 3:12 PM Otto Fowler wrote:
> Nifi releases more often then Metron
Having it in it’s own repo doesn’t tie it to Metron any less functional
wise, but allows
for a new release with Nifi only changes to be produced, or multiple
streams of releases
across nifi versions ( 1.7.x, 1.8.x ) to be produced.
On June 5, 2018 at 15:14:38, Casey Stella (ceste...@gmail.com)
The vote has passed. Including my +1, the voting was:
3 binding +1’s
1 non-binding +1’s
no 0’s
no -1’s.
Release process is underway and will be announced to dev@ and user@ lists.
Justin
Also, the bundle would be part of the metron project I expect, so the NiFi
release shouldn’t matter much, now NiFi can version only processors
independently.
Simon
> On 5 Jun 2018, at 20:14, Casey Stella wrote:
>
> I agree with Simon here, the benefit of providing NiFi tooling is to enable
25 matches
Mail list logo
