Re: [DISCUSS] Writer class refactor

2019-01-22 Thread Ryan Merriman
Thanks Mike, very helpful to have all that context. I'm in agreement with everything you've said. Accepting duplicates may be a tradeoff we accept to keep performance high. Your comments are centered around Kafka but how does this apply to other writers? Since we're now handling multiple

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread zeo...@gmail.com
Totally on board with everybody's comments above this point. Jon On Fri, Jan 18, 2019, 6:07 PM Michael Miklavcic wrote: > Thanks for the write up, Ryan. I had to touch on some of this when > refactoring the kafka writer away from the async model so we could > guarantee delivery. We had

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread Michael Miklavcic
Thanks for the write up, Ryan. I had to touch on some of this when refactoring the kafka writer away from the async model so we could guarantee delivery. We had potential to drop messages before that change because of the async producer calls, which would ack the Storm tuple as soon as the writer

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread Otto Fowler
Agreed On January 18, 2019 at 14:52:32, Ryan Merriman (merrim...@gmail.com) wrote: I am on board with that. In that case, I think it's even more important that we get the Writer interfaces right. On Fri, Jan 18, 2019 at 1:34 PM Otto Fowler wrote: > I think that the writers should be loaded

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread Ryan Merriman
I am on board with that. In that case, I think it's even more important that we get the Writer interfaces right. On Fri, Jan 18, 2019 at 1:34 PM Otto Fowler wrote: > I think that the writers should be loaded as, and act as extension points, > such that it is possible to have 3rd party writers,

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread Otto Fowler
I think that the writers should be loaded as, and act as extension points, such that it is possible to have 3rd party writers, and would structure them as such. On January 18, 2019 at 13:55:00, Ryan Merriman (merrim...@gmail.com) wrote: Recently there was a bug reported by a user where a

[DISCUSS] Writer class refactor

2019-01-18 Thread Ryan Merriman
Recently there was a bug reported by a user where a parser that emits multiple messages from a single tuple doesn't work correctly: https://issues.apache.org/jira/browse/METRON-1968. This has exposed a problem with how the writer classes work. The fundamental issue is this: the writer classes