[DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2018-12-19 Thread Michael Miklavcic
When a message is filtered by the message filtering mechanism, we explicitly drop the message (and presumably ack it in Storm), as explained here - https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#filtered. When using the REGEX_SELECT field transformation (see here -

Re: [DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2018-12-19 Thread Casey Stella
We absolutely should be acking the dropped messages otherwise they'll be in a replay loop. Not acking is a flat-out bug IMO. On Wed, Dec 19, 2018 at 2:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > When a message is filtered by the message filtering mechanism, we > explicitly

Site-book broken in master

2018-12-19 Thread Michael Miklavcic
FYI, site-book is currently broken in master. I think I've found the source of the issue and will be issuing a fix shortly. Best, Mike

Re: [DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2018-12-19 Thread Michael Miklavcic
Completely agreed on the acking. The reason I posed the question to begin with was because, while I believe dropping+acking is the correct functionality, I could see a few alternative patterns for handling this: 1. Require filtering to be handled by the message filter infrastructure and

Re: Site-book broken in master

2018-12-19 Thread Michael Miklavcic
I'd also like to submit a call for suggestions for making this process simpler. I think we've made some excellent progress towards getting an html-generated version of our documentation, but it's also proving to be somewhat error-prone. Github and Doxia have different expectations as to

Re: Site-book broken in master

2018-12-19 Thread Michael Miklavcic
Fix is out in this PR: https://github.com/apache/metron/pull/1309 Jira: https://issues.apache.org/jira/browse/METRON-1950 It ended up being a bit bigger than I hoped because we had some remaining incompatibilities between our Github markdown and the Doxia-expected formatting. This caused a

Re: Site-book broken in master

2018-12-19 Thread Otto Fowler
Maybe we should consider ascii doc, it shows in github, and can generate multiple formats. On December 19, 2018 at 22:49:26, Otto Fowler (ottobackwa...@gmail.com) wrote: What would help is an automated test the tests all the links as well. On December 19, 2018 at 19:55:03, Michael Miklavcic

Re: Site-book broken in master

2018-12-19 Thread Otto Fowler
What would help is an automated test the tests all the links as well. On December 19, 2018 at 19:55:03, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: I'd also like to submit a call for suggestions for making this process simpler. I think we've made some excellent progress towards