Hi there, here's my tuppence! 1. Something to look at first:
If you write two different values to the same column quickly in succession, if both writes go out with the same timestamp, then it is indeterminate which one wins i.e. write order doesn't necessarily matter. 2. Don't trust PayPal (anyone using PayPal should really read this) We are / were relying on IPNs to manage our website's recurring subscriptions list. We experienced this weird thing where the recurring_payment_profile_created IPN was missing, and got thought maybe Cassandra was losing it because PayPal is a financial system and it couldn't possibly fail to generate an IPN, right!!? Anyway, it turns out that after exhaustive discussions with PayPal engineers, and having proved this from the PayPal logs, that sometimes IPNs fail to get generated. Yup. Read that again!!!! Sometimes the fail to get generated and in fact this is happening to us quite regularly now. They justify this (while acknowledging this issue should be in their documentation) by saying that because HTTP delivery is unreliable (hmmm isn't this what the retry queue is for..) we shouldn't be relying entirely on IPNs and should regularly download the logs and run them through scripts to catch problems (this is idiotic, since the angry customer will get on our case immediately when they pay and membership doesn't start) Not sure whether PayPal or database failing is best option. Look forward to hearing resolution. Best, Dominic On 22 August 2011 17:49, Kyle Gibson <kyle.gib...@frozenonline.com> wrote: > I made some changes to my code base that uses cassandra. I went back > to using the "processed" column, but instead of using "0" or "1" I > decided to use "no" and "yes" > > You can view the code here: http://pastebin.com/gRBC16e7 > > As you can see from the code, I perform an insert, get, check the > result, if it didn't work, I try to insert again, and check the get. > Each time I do a print out to see what the result is. Each operation > is a CL.ALL. > > A few successful IPNs did come through before this one was generated: > > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce > OrderedDict([..., (u'processed', u'no'), ...]) > Failed to set processed to yes > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert 1314012603578714 > Failed to set processed to yes > IPN-5943a4adc8eab68cdbc9d9eff7fa7dc669fa0bce insert2 1314012603586201 > > As expected, this IPN was processed twice. > > On Sat, Aug 20, 2011 at 5:37 PM, Peter Schuller > <peter.schul...@infidyne.com> wrote: > >> Do you mean the cassandra log, or just logging in the script itself? > > > > The script itself. I.e, some "independent" verification that the line > > of code after the insert is in fact running, just in case there's some > > kind of silent failure. > > > > Sounds like you've tried to address it though with the E-Mail:s. > > > > I suppose it boils down to: Either there is something wrong in your > > environment/code, or Cassandra does have a bug. If the latter, it > > would probably be helpful if you could try to reproduce it in your > > environment in a way which can be shared - such as a script that does > > writes and reads back to confirm the write made it. Or maybe just > > adding more explicit logging to your script (even if it causes some > > log flooding) to "prove" that a write truly happened. > > > > -- > > / Peter Schuller (@scode on twitter) > > >
