I believe that "better" is relative most of the time. Optimizing a failure case costs something, unless of course the code is doing something horrible and there truly is a better way to do it that uses less memory, cpu, etc. If so please put up a pull request and lets fix the thing. However most of the time getting better performance costs something. If we want to avoid going back to kafka for a failed tuple then we need to store it somewhere. The simplest way to do it would be to cache the tuple in memory, but that will cost as much memory as all of the outstanding tuples for the spout. It is not a clear win if failures are not common. How frequently should failures happen? I don't think we have anything like that documented, because it depends on a number of things, but most of the time it indicates that there is something that needs to be fixed about the topology. For me if I see any failures in a topology and we are not in the middle of a rolling upgrade I am concerned. A lot of teams ignore errors because they are running a lambda architecture and will have batch stomp on the top of it later, but I still get very nervous when I see it.
- Bobby On Monday, July 10, 2017, 2:33:52 PM CDT, chandan singh <cks07...@gmail.com> wrote: Sorry about being cryptic there. What I meant is that it will be much better if we don't make assumptions about frequency of failure rates in topologies. I know it is more of a commonsense but out of curiosity, can you point me to any Storm documentation which makes a comment on preferable failure rates. I was suggesting if we can offer the user an optimization through clean API, the user will be free to decide on the rationale of using it. On Tue, Jul 11, 2017 at 12:06 AM, Bobby Evans <ev...@yahoo-inc.com.invalid> wrote: > I'm not sure what assumptions you want to make that this is preventing, or > why they would be helpful. > > - Bobby > > > On Monday, July 10, 2017, 12:14:53 PM CDT, chandan singh < > cks07...@gmail.com> wrote: > > Hi Stig & Bobby > > Thanks for confirming my understanding. > > 1) Ensuring that calls to nexTuple(), ack() and fail() are non-blocking > has been a guideline on http://storm.apache.org/ > releases/1.1.0/Concepts.html > for long. Copying verbatim here : "The main method on spouts is nextTuple. > nextTuple either emits a new tuple into the topology or simply returns if > there are no new tuples to emit. It is imperative that nextTuple does not > block for any spout implementation, because Storm calls all the spout > methods on the same thread." I admit that there is some chance my > interpretation is partially incorrect but I have been following it in a > custom spout till now. Even though the objective is different, there is a > similar hint on Kafka official documentation. Please see under heading "2. > Decouple Consumption and Processing" on > https://kafka.apache.org/0110/javadoc/index.html?org/apache/ > kafka/clients/consumer/KafkaConsumer.html. > Essentially, a thread polls Kafka and spout thread gets the messages > through a shared queue. If pre-fetching is present in Kafka (I will read > about it further), I assume we do not have to fetch in another thread but I > am not sure how does the pre-fetching behave with re-seeking before every > poll. > > 2) @Bobby, you are correct in pointing what needs to be optimized but the > facts, sometimes, prevent us from making assumptions. We do optimize our > retry loop such that we don't poll the messages again. I especially see > problems when combined with exponential back off. I am not sure how > difficult or clean will it be to expose some sort of configuration to allow > such optimization? Do you think it will be worth trying out something? > > Thanks > Chandan >
