Hi Waleed, This is likely related to this bug: http://code.google.com/p/googleappengine/issues/detail?id=6323
Though I'm a bit surprised that it is impacting you this much since you don't seem to be using much CPU. Cheers, Brian On Tue, Nov 22, 2011 at 8:29 AM, Waleed Abdulla <[email protected]> wrote: > Bump! > Nick, I sent the info you requested. > I'm still trying to find an explanation for why using multithreading > increased my response time by 10X. Any ideas? > Waleed > > > On Tue, Nov 15, 2011 at 7:00 PM, Waleed Abdulla <[email protected]> wrote: >> >> Hi Nick, >> Here are more details, and I'm attaching AppStats screen shots. >> - Along with the switch to Python 2.7, we also switched to NDB, and we >> removed our use of memcache in favor of the built-in caching in NDB. So when >> we noticed the delays, the first thing we suspected was that NDB caching >> wasn't working, so we re-implemented our use of memcache. That didn't help. >> - We checked if there are "pending_ms" values in the logs. There weren't >> any. >> - The instance and latency in "Application Settings" are all set to >> "automatic". >> - To keep things simple, we tested with one of the simple requests (it >> fetches a Blob from memcache and serves it as an image, then updates values >> in memcache). That request takes 1s to 3s when threadsafe is True, and >> 100-300ms when threadsafe is False. >> - I just enabled appstats after your email and collected data for both >> cases. Most of the delay seems to be happening in RPC calls. For example, >> memcache calls seem to take 200ms+. I'm attaching a couple of screen shots >> for a simple request when threadsafe is True and False. >> Hope this helps. >> Waleed >> >> >> >> On Tue, Nov 15, 2011 at 4:49 PM, Nick Johnson <[email protected]> >> wrote: >>> >>> Hi Waleed, >>> It's impossible to say what the cause of the latency might be without >>> more details. You say you spent a lot of time investigating it; can you show >>> us some of the data you gathered? Did you use appstats? >>> -Nick Johnson >>> >>> On Wed, Nov 16, 2011 at 11:37 AM, Waleed Abdulla <[email protected]> >>> wrote: >>>> >>>> I just switched one of my apps to Python 2.7 and the average latency >>>> went up from 125ms to about 1.3 seconds (about 10x). My app is 'not' >>>> CPU-bound. It has a mix of different use patterns (some requests use >>>> urlfetch, others load data from datastore or memcache). That variety should >>>> make it a good candidate to benefit from multi-threading, but that doesn't >>>> seem to be working as expected. While the number of instances went down >>>> from >>>> 9 instances to 6, the 10X increase in latency is not acceptable. >>>> I spent a lot of time investigating it, and finally changed threadsafe >>>> to false and that fixed the problem (and, of course, raised my instance >>>> count to it's original level). Like the original poster, I was hoping that >>>> multithreading will reduce my instance cost, but that plan failed. >>>> Am I doing something wrong? Or is multi-threading not ready for live >>>> traffic yet? >>>> Waleed >>>> app id: bitpixelshr >>>> >>>> >>>> >>>> >>>> On Tue, Nov 15, 2011 at 3:10 AM, Jeff Schnitzer <[email protected]> >>>> wrote: >>>>> >>>>> Dunno about the Python library, but the standard Java jBCrypt library >>>>> uses a random salt per-user by default. >>>>> I'm also very suspicious of this idea that the attacker doesn't have >>>>> the source code. Python is trivially easy to decompile. Also, where do >>>>> you >>>>> keep your source code? In Github? Opens up a whole new set of attack >>>>> vectors, including disgruntled Github employees. >>>>> >>>>> Jeff >>>>> On Mon, Nov 14, 2011 at 10:52 PM, Brandon Wirtz <[email protected]> >>>>> wrote: >>>>>> >>>>>> Nick, >>>>>> >>>>>> >>>>>> >>>>>> I agree, that my threat model assumes they didn’t get my source code. >>>>>> That “Somebody else’s problem” works under the assumption people are >>>>>> going >>>>>> to get my data, not my source code because I don’t ever write my own DB >>>>>> server code I am stuck using someone else’s which means the vulnerability >>>>>> that I am most likely to face is that somebody else’s screw up will be >>>>>> where >>>>>> my problem lies. >>>>>> >>>>>> >>>>>> >>>>>> Granted this is a better strategy if you are running compiled code, >>>>>> since my code lives on the Google Server I’m at the mercy of Google’s >>>>>> Security, where as if I were running compiled code it would be less >>>>>> likely >>>>>> someone would get the code. >>>>>> >>>>>> >>>>>> >>>>>> I would say that unique salt per user, is a good thing. The most >>>>>> common way to attack a large password database is to look at the most >>>>>> common >>>>>> entries and compare against the most common passwords from other sources. >>>>>> If you know the 15 most used passwords and the 15 most often occurring >>>>>> database results you are a long ways towards knowing what those 15 values >>>>>> are and calculating the salt. You aren’t crunching millions of >>>>>> combinations >>>>>> you are crunching 1000’s and once you have the salt, you take your >>>>>> already >>>>>> deciphered list of the most common passwords and you calculate the top 5k >>>>>> using bcrypt and you now have about 50% of the data in fewer than 10k >>>>>> operations. >>>>>> >>>>>> >>>>>> >>>>>> Compare that with my scenario. You have data. You don’t have the >>>>>> source code. The UserID or other “spoiler” is in every salt so the >>>>>> reoccurrence of a hash doesn’t correspond to a duplicate password, and >>>>>> now >>>>>> the computation is nearly impossible even if you have the source code, >>>>>> because you have to calculate every value for every user anyway. >>>>>> >>>>>> >>>>>> >>>>>> Would Brcypt(Pass+UserID+Salt) be the best? Yes. But >>>>>> MD5(Pass+UserID+Salt) is going to still going to be orders of magnitude >>>>>> more >>>>>> difficult than Bcrypt(Pass+salt), because I can’t use knowledge of >>>>>> frequency >>>>>> tables to predict likely outcomes or detect duplicate passwords. >>>>>> >>>>>> >>>>>> >>>>>> -Brandon >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> From: [email protected] >>>>>> [mailto:[email protected]] On Behalf Of Nick Johnson >>>>>> Sent: Monday, November 14, 2011 6:21 PM >>>>>> >>>>>> To: [email protected] >>>>>> Subject: Re: [google-appengine] Help resolve massive performance >>>>>> regression in 2.7 vs 2.5 runtime >>>>>> >>>>>> >>>>>> >>>>>> Hi Brandon, >>>>>> >>>>>> >>>>>> >>>>>> What you say is fine if your threat model only includes "script >>>>>> kiddies" who don't have your source code. If either of those is not true >>>>>> - >>>>>> you have an adversary with some level of independent skill, or your >>>>>> source >>>>>> code is compromised - any method that relies on obscurity for its >>>>>> security >>>>>> will fare very poorly. >>>>>> >>>>>> >>>>>> >>>>>> One thing to bear in mind is that if your app is ever compromised, >>>>>> your password database and/or source may be posted publicly; at that >>>>>> point, >>>>>> you no longer have to worry about just the initial attacker, but anyone >>>>>> with >>>>>> sufficient motivation. >>>>>> >>>>>> >>>>>> >>>>>> Of course, using federated login like OpenID or the Users API obviates >>>>>> the need to store passwords at all, making it Someone Else's Problem. :) >>>>>> >>>>>> >>>>>> >>>>>> -Nick >>>>>> >>>>>> On Tue, Nov 15, 2011 at 1:07 PM, Brandon Wirtz <[email protected]> >>>>>> wrote: >>>>>> >>>>>> If I know your salt, I can “De-Hash” bcrypts faster than I can any of >>>>>> the “weird” combinations. Because there are libraries for doing so on ATI >>>>>> cards. >>>>>> >>>>>> >>>>>> >>>>>> If you do something weird a script kiddie can’t just pull code off the >>>>>> web and attack it. >>>>>> >>>>>> >>>>>> >>>>>> You want to see who can offline crack a set of 1M users? Your bcrypt >>>>>> list vs my “Weird” You don’t even have to give me the salt I’ll have >>>>>> 10k >>>>>> of those cracked in the first 72 hours. 10 to 1 odds you won’t get >>>>>> through >>>>>> mine without my source code in my life time. >>>>>> >>>>>> >>>>>> >>>>>> -Brandon Wirtz >>>>>> >>>>>> >>>>>> >>>>>> PS >>>>>> I don’t usually do the “trust me I’m far more evil” but FBI, Homeland >>>>>> Security, and the CIA have been to my doorstep for things I have >>>>>> defeated, >>>>>> documented, or built to keep from being defeated. The first time I was >>>>>> in >>>>>> 3rd grade. >>>>>> >>>>>> >>>>>> >>>>>> From: [email protected] >>>>>> [mailto:[email protected]] On Behalf Of Nick Johnson >>>>>> Sent: Monday, November 14, 2011 3:56 PM >>>>>> >>>>>> To: [email protected] >>>>>> Subject: Re: [google-appengine] Help resolve massive performance >>>>>> regression in 2.7 vs 2.5 runtime >>>>>> >>>>>> >>>>>> >>>>>> No! Please, please don't do this. Obscurity is no substitute for >>>>>> security. >>>>>> >>>>>> >>>>>> >>>>>> 1) Bcrypt or similar is not 'overkill' no matter who you are. Users >>>>>> reuse passwords, and they're entitled to the best protection you can >>>>>> reasonably provide them. >>>>>> >>>>>> 2) Bcrypt is not there to protect against online attacks, it's there >>>>>> to protect against offline attacks, where an attacker obtains your hashed >>>>>> and salted passwords. >>>>>> >>>>>> 3) Doing "something weird" is security through obscurity. Do not base >>>>>> your security on your attacker not knowing what you did. Really, really >>>>>> don't just concatenate salts to the beginning or end of the password. >>>>>> >>>>>> 4) Both MD5 and SHA1 are merkle-damgard construction hashes >>>>>> (http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction). >>>>>> As >>>>>> a result, the concatenation of several hashes is no more secure than the >>>>>> most secure of the individual hashes. >>>>>> >>>>>> >>>>>> >>>>>> -Nick Johnson >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Nov 13, 2011 at 2:58 PM, Brandon Wirtz <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Unless you are protecting Medical records bcrypt is overkill if you do >>>>>> some >>>>>> reasonably smart things like "Failed logins from IP >9" >>>>>> >>>>>> Or, if you just do something weird to the password BEFORE you SHA it. >>>>>> Like >>>>>> interleave the user name in the password, Salt1 + UpSaEsRsNwAoMrEd + >>>>>> Salt2 >>>>>> >>>>>> Or Pick 2 Hash's SHA(pass) + Md5(pass) >>>>>> >>>>>> Don't want to store all that string length? Odd Characters from >>>>>> Sha(Pass+salt) + Even Characters from MD5(Pass+Salt) >>>>>> >>>>>> Uniqueness of the method is more important than the method. >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: [email protected] >>>>>> [mailto:[email protected]] On Behalf Of Brian Quinlan >>>>>> Sent: Saturday, November 12, 2011 6:58 PM >>>>>> To: [email protected] >>>>>> Subject: Re: [google-appengine] Help resolve massive performance >>>>>> regression >>>>>> in 2.7 vs 2.5 runtime >>>>>> >>>>>> Hi Pol, >>>>>> >>>>>> On Sun, Nov 13, 2011 at 1:48 PM, Pol <[email protected]> wrote: >>>>>> > Hi, >>>>>> > >>>>>> > Since switching to 2.7 runtime, logging in to http://www.everpix.com >>>>>> > went from about a second to anywhere from 15s to 60s. I tracked it >>>>>> > down to this single password checking line: >>>>>> > >>>>>> > from bcrypt import bcrypt >>>>>> > bcrypt.hashpw(password, self.password_hash) == self.password_hash >>>>>> >>>>>> What value are you using for "threadsafe" in your app.yaml? >>>>>> >>>>>> How large is self.password_hash? >>>>>> >>>>>> Cheers, >>>>>> Brian >>>>>> >>>>>> > This comes from "a native Python implementation of the py-bcrypt >>>>>> > package from http://www.mindrot.org/projects/py-bcrypt/"; grabbed >>>>>> > from >>>>>> > here: https://github.com/erlichmen/py-bcrypt. >>>>>> > >>>>>> > So what's happening here and how can we fix this? >>>>>> > >>>>>> > Thanks, >>>>>> > >>>>>> > - Pol >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> > Groups >>>>>> "Google App Engine" group. >>>>>> > To post to this group, send email to >>>>>> > [email protected]. >>>>>> > To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> > For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> > >>>>>> > >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups >>>>>> "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nick Johnson, Developer Programs Engineer, App Engine >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nick Johnson, Developer Programs Engineer, App Engine >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to >>>>>> [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >>> >>> -- >>> Nick Johnson, Developer Programs Engineer, App Engine >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
