After some discussion with David, I am going to rebase my work off of
PR-436 ( ansible deploys with mpack )
On February 27, 2017 at 10:31:26, Otto Fowler (ottobackwa...@gmail.com)
wrote:
Sure,
I am not doing the ansible work for any other reason than ansible is what
we have now, and what
Sure,
I am not doing the ansible work for any other reason than ansible is what
we have now, and what works.
I realized that deployment of the current parsers needs to work with ambari
etc, and it is next. I just did ansible first, because that is what works
for quick-dev etc.
As far as how to
Hi Otto,
It will have a pretty major effect. We agreed a bit back that we wanted, as
a community, to reduce reliance on Ansible, so I think an Ansible-based
parser loader would be sub-optimal. You may be working on this a bit in
early. There are some some major changes on the way that will could
Not sure how https://github.com/apache/incubator-metron/pull/436 is going
to effect this, but it will.
On February 26, 2017 at 23:04:32, Otto Fowler (ottobackwa...@gmail.com)
wrote:
OK, Current status
Complete:
* Parsers broken out into common, base ( raw types - csv, json, grok ) and
OK, Current status
Complete:
* Parsers broken out into common, base ( raw types - csv, json, grok ) and
module for each type
* Ansible modified to install new parsers, adding new parser should just be
adding a new name to the array
* Parsers install into /usr/metron/ver/telemetry/{NAME}/ with
More thoughts
(1) We should do a treatment for each area
(2) We can use the telemetry stuff as an incubator, itself to be replaced
with something better that is developed after
(3) That is a nice idea - ‘live packaging’ ( i’m getting the TM and a
website as we speak )
(4) sure, but we may
Update: https://github.com/ottobackwards/incubator-metron/tree/METRON-258
I have all the parsers broken out, including squid and yaf which are
configuration only.
The idea there being we could and should have tests and other collateral
even for configuration only parsers.
I have created:
Yes, this exactly. Similar to the NAR format for NiFi. At this early
point, we can start with the tar.gz’s that we produce from assembly.
Now, if someone reads this and thinks “you are just re-inventing FOO, and I
love FOO!” get, that is an implementation detail anyways.
The important thing to
Your mention of a "package mechanism" sparked some half-baked ideas on my
part.
Be forewarned these are probably tangents from your immediate goal (sorry),
but maybe these ideas might help shape how you want to take this forward.
(1)
We should consider that eventually each "function" of Metron
Awesome write up and ideas Otto, I also strongly support this idea. As
someone who has the development of a few parsers quickly approaching the
top of their to do list, I will happily beta test this for you when it's
far enough along for that. Until then I will attempt to take a look at
your
I like the idea of having each parser as its own maven module and having an
archetype for it. In my vision when you click to create a maven "metronParser"
archetype what you would get is a module consisting of a blank parser template
with a parse() method that a dev would have to fill in, the
Thanks for taking the time Matt,
It is likely that I am not seeing your point clearly, could you elaborate
how Spring or Guice would be applicable to this proposal if there is no
intent to change the parser’s composition or run-time functionality, but
rather it’s deployment and external
Outstanding write-up, Otto! As Casey said, don’t expect this to be a coherent
response, but some possibly useful thoughts:
1. It’s clear that because parsers, enrichers, and indexers are all specialized
per sensor, that “adding a new sensor” is necessarily a complex operation.
You’ve thrown
RE:
* One Module - yes, I think grouping for the base parsers is good, I just
don’t want them to stay in -common, it should ‘live’ in the metron lib. I
think a grouped set of the primitive parsers is correct, still it’s own.
* ES Templates - they don’t *have* to be there, but if they are they
Ok, This is a long one, so don't expect a coherent response just yet, but I
will give some initial impressions:
- I strongly agree with the premise of this idea. Making Metron
extensible is and should be among the top of our priorities and at the
moment, it's painful to develop a new
The ability for implementors and developers building on the project to
‘side load’, that is to build, maintain, and install, telemetry sources
into the system without having to actually develop within METRON itself is
very important.
If done properly it gives developers and easier and more
16 matches
Mail list logo
