[
https://issues.apache.org/jira/browse/METRON-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878691#comment-15878691
]
Otto Fowler commented on METRON-735:
------------------------------------
I think having a configurable strategy of changing the data to suite the
indexer should be a base feature for all indexer's, and just turned on by
default ( or forced on ) for Solr if it will not work without it.
I think this ties into our conversations on the list around 'deriving' indexing
templates from parsers+enrichments automatically as well. In other words,
would that system exist, then it may have to tie together with this
> Support Complex Data Types in Telemetry Messages
> ------------------------------------------------
>
> Key: METRON-735
> URL: https://issues.apache.org/jira/browse/METRON-735
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
>
> Up until now, we have been working under the assumption that all telemetry
> messages in Metron must not contain complex data types like lists or maps.
> Complex data types were 'flattened' to remove any complex data types from the
> telemetry messages.
> Most parsers today produce flat telemetry messages. That is messages with no
> complex data types. The problem is that even if I use a parser that
> generates only flattened data, I could create an enrichment (like using
> GEO_GET) that appends non-flat data to a message. Thus we have non-flat data
> coursing through the veins of Metron. ;)
> I think the original idea of flattening data was because one of our Indexers
> could not handle non-flat, complex data types. At the time, we just decided,
> well don't create any non-flat data.
> But now, since we have a completely 'programmable' system, I don't think it
> is safe to assume that the data will always be non-flat. A user could create
> their own Stellar function to use during enrichment. Should we force on them
> the burden of flattening the data?
> It makes way more sense in my mind, to make the indexer transform the data
> however it needs to , to correctly index the data. If the current issue is
> with the Solr Indexer, then we should fix that to flatten any data that it
> needs to. There would be one touch point to address this issue rather than
> many.
> This is a blanket JIRA that might result in multiple sub-tasks of changes
> needed to allow telemetry messages to contain complex data types like lists
> and maps.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
