Hi, Are you doing this cleansing all in the parser or are you using any Stellar to do it? Can you create a jira?
On April 26, 2017 at 08:59:16, Ali Nazemian ([email protected]) wrote: Hi all, We are facing certain use cases in Metron production that happen to be related to noisy stream. For example, a wrong timestamp, duplicate hostname/IP address, etc. To deal with the normalization we have added an additional step for the corresponding parsers to do the data cleaning. Clearly, parsing is a standard factor which is mostly related to the device that is generating the data and can be used for the same type of device everywhere, but normalization is very production dependent and there is no point of mixing normalization with parsing. It would be nice to have a sperate bolt in a parsing topologies to dedicate to production related cleaning process. In that case, eveybody can easily contribute to Metron community with additional parsers without being worried about mixing parsers and data cleaning process. Regards, Ali
