Nick Allen created METRON-735:
---------------------------------
Summary: Support Complex Data Types in Telemetry Messages
Key: METRON-735
URL: https://issues.apache.org/jira/browse/METRON-735
Project: Metron
Issue Type: Improvement
Reporter: Nick Allen
Up until now, we have been working under the assumption that all telemetry
messages in Metron must not contain complex data types like lists or maps.
Complex data types were 'flattened' to remove any complex data types from the
telemetry messages.
Most parsers today produce flat telemetry messages. That is messages with no
complex data types. The problem is that even if I use a parser that generates
only flattened data, I could create an enrichment (like using GEO_GET) that
appends non-flat data to a message. Thus we have non-flat data coursing through
the veins of Metron. ;)
I think the original idea of flattening data was because one of our Indexers
could not handle non-flat, complex data types. At the time, we just decided,
well don't create any non-flat data.
But now, since we have a completely 'programmable' system, I don't think it is
safe to assume that the data will always be non-flat. A user could create their
own Stellar function to use during enrichment. Should we force on them the
burden of flattening the data?
It makes way more sense in my mind, to make the indexer transform the data
however it needs to , to correctly index the data. If the current issue is with
the Solr Indexer, then we should fix that to flatten any data that it needs to.
There would be one touch point to address this issue rather than many.
This is a blanket JIRA that might result in multiple sub-tasks of changes
needed to allow telemetry messages to contain complex data types like lists and
maps.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
