On August 15, 2018 at 09:30:47, Justin Leet (justinjl...@gmail.com) wrote:
As an exercise, let me summarize the points of contention I've seen and lay
out the tradeoffs as I see them. That way we can prioritize what's
important to us in a NiFi implementation and better work towards a
favorable sol
As an exercise, let me summarize the points of contention I've seen and lay
out the tradeoffs as I see them. That way we can prioritize what's
important to us in a NiFi implementation and better work towards a
favorable solution (basically, I want to requirements we have for an MVP).
My opinions/co
Yep, I'm wondering whether our parser interface should have the ability to
create schema either like that, or well, that, which would be helpful
within Metron as well.
@Otto, the one thing missing from the record reader api, is that if you
don't emit any records at all for a flow file, it errors,
Also, If we are doing the record readers, we can have a reader for a
parser type and explicitly set the schema, as seen here :
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/ja
If we can do the record readers ourselves ( with the parsers inside them )
we can handle the returns.
I’ll be doing the net flow 5 readers once the net flow 5 processor PR ( not
mine ) is in.
I don’t think having a generic class loading parsers foo and having to
manage all that is preferable to ha
Maybe the edge use case will clarify the config issue a little. The reason
I would want to be able to push Metron parsers into NiFi would be so I can
pre-parse and filter on the edge to save bandwidth from remote locations. I
would expect to be able to parse at the edge and use NiFi to prioritise o
I would say that
- For each configuration parameter we want to pull in, it should be
explicitly configured through a property as well as through a controller
service that accesses the metron zk
- Transformations should not be conflated with parsing in those processors
or readers
There is no on th
That's definitely good info, thanks for reaching out to them about it.
In terms of exposing/sharing, I don't think we have to couple them tightly
(in fact, I think we should loosen the coupling as much as possible without
forcing reimplementation of things). I think there's definitely a way to do
I think the benefits are clear. What is unclear is if the goal is to
expose or share or re-use Metron capabilities ( stellar, parsing ) in nifi
in a way that is native to nifi ( configured and managed in nifi ), where
you may not even need metron ( say you just want to parse asa ) or if the
goal i
I reached out to the nifi list about the Record api ‘stability'
On August 9, 2018 at 09:54:22, Bryan Bende (bbe...@gmail.com) wrote:
I don't think there are any stability issues with the record API, it
is definitely recommended to use the record approach where it makes
sense.
That comment was pr
I'll add onto Mike's discussion with the original set of requirements I had
in mind (and apply feedback on these as necessary!). This is largely
overlap with what Mike said, but I want to make sure it's clear where my
proposal was coming from, so we can improve on it as needed. James and
Mike are
I think it also provides customers greater control over their architecture
by giving them the flexibility to choose where/how to host their parsers.
To Justin's point about the API, my biggest concern about the RecordReader
approach is that it is not stable. We already have a similar problem in
ha
Integration with NiFi would be useful for parsing low-volume telemetries at the
edge. This is a much more resource friendly way to do it than setting up
dedicated storm topologies. The integration would be that the NiFi processor
parses the data and pushes it straight into the enrichment topic
Why do we start over. We are going back and forth on implementation, and I
don’t think we have the same goals or concerns.
What would be the requirements or goals of metron integration with Nifi?
How many levels or options for integration do we have?
What are the approaches to choose from?
Who ar
So how does the MetronRecordReader roll into everything? It seems like it'd
be more useful on the reader per format approach, but otherwise it doesn't
really seem like we gain much, and it requires getting everything linked up
properly to be used. Assuming we looked at doing it that way, is the ide
A Metron Processor itself isn’t really necessary. A MetronRecordReader (
either the megalithic or a reader per format ) would be a good approach.
Then have StellarTransformRecord processor that can do Stellar on _any_
record, regardless of source.
On August 7, 2018 at 11:06:22, Justin Leet (ju
Thanks for the comments, Otto, this is definitely great feedback. I'd love
to respond inline, but the email's already starting to lose it's
formatting, so I'll go with the classic "wall of text". Let me know if I
didn't address everything.
Loading modules (or jars or whatever) outside of our Pro
I think this is a good idea. As I mentioned in the other thread I’ve been
doing a lot of work on Nifi recently.
I think the important thing is that what is done should be done the NiFi
way, not bolting the Metron composition
onto Nifi. Think of it like the Tao of Unix, the parsers and components
Hi all,
There's interest in being able to run Metron parsers in NiFi, rather than
inside Storm. I dug into this a bit, and have some thoughts on how we could
go about this. I'd love feedback on this, along with anything we'd
consider must haves as well as future enhancements.
1. Separate metr
19 matches
Mail list logo
