Hi Waleed, It's impossible to say what the cause of the latency might be without more details. You say you spent a lot of time investigating it; can you show us some of the data you gathered? Did you use appstats?
-Nick Johnson On Wed, Nov 16, 2011 at 11:37 AM, Waleed Abdulla <[email protected]> wrote: > I just switched one of my apps to Python 2.7 and the average latency went > up from 125ms to about 1.3 seconds (about 10x). My app is 'not' CPU-bound. > It has a mix of different use patterns (some requests use urlfetch, others > load data from datastore or memcache). That variety should make it a good > candidate to benefit from multi-threading, but that doesn't seem to be > working as expected. While the number of instances went down from 9 > instances to 6, the 10X increase in latency is not acceptable. > > I spent a lot of time investigating it, and finally changed threadsafe to > false and that fixed the problem (and, of course, raised my instance count > to it's original level). Like the original poster, I was hoping that > multithreading will reduce my instance cost, but that plan failed. > > Am I doing something wrong? Or is multi-threading not ready for live > traffic yet? > > Waleed > app id: bitpixelshr > > > > > > On Tue, Nov 15, 2011 at 3:10 AM, Jeff Schnitzer <[email protected]>wrote: > >> Dunno about the Python library, but the standard Java jBCrypt library >> uses a random salt per-user by default. >> >> I'm also very suspicious of this idea that the attacker doesn't have the >> source code. Python is trivially easy to decompile. Also, where do you >> keep your source code? In Github? Opens up a whole new set of attack >> vectors, including disgruntled Github employees. >> >> Jeff >> >> On Mon, Nov 14, 2011 at 10:52 PM, Brandon Wirtz <[email protected]>wrote: >> >>> Nick,**** >>> >>> ** ** >>> >>> I agree, that my threat model assumes they didn’t get my source code. >>> That “Somebody else’s problem” works under the assumption people are going >>> to get my data, not my source code because I don’t ever write my own DB >>> server code I am stuck using someone else’s which means the vulnerability >>> that I am most likely to face is that somebody else’s screw up will be >>> where my problem lies.**** >>> >>> ** ** >>> >>> Granted this is a better strategy if you are running compiled code, >>> since my code lives on the Google Server I’m at the mercy of Google’s >>> Security, where as if I were running compiled code it would be less likely >>> someone would get the code.**** >>> >>> ** ** >>> >>> I would say that unique salt per user, is a good thing. The most common >>> way to attack a large password database is to look at the most common >>> entries and compare against the most common passwords from other sources. >>> If you know the 15 most used passwords and the 15 most often occurring >>> database results you are a long ways towards knowing what those 15 values >>> are and calculating the salt. You aren’t crunching millions of >>> combinations you are crunching 1000’s and once you have the salt, you take >>> your already deciphered list of the most common passwords and you calculate >>> the top 5k using bcrypt and you now have about 50% of the data in fewer >>> than 10k operations.**** >>> >>> ** ** >>> >>> Compare that with my scenario. You have data. You don’t have the source >>> code. The UserID or other “spoiler” is in every salt so the reoccurrence of >>> a hash doesn’t correspond to a duplicate password, and now the computation >>> is nearly impossible even if you have the source code, because you have to >>> calculate every value for every user anyway.**** >>> >>> ** ** >>> >>> Would Brcypt(Pass+UserID+Salt) be the best? Yes. But >>> MD5(Pass+UserID+Salt) is going to still going to be orders of magnitude >>> more difficult than Bcrypt(Pass+salt), because I can’t use knowledge of >>> frequency tables to predict likely outcomes or detect duplicate passwords. >>> **** >>> >>> ** ** >>> >>> -Brandon**** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> *From:* [email protected] [mailto: >>> [email protected]] *On Behalf Of *Nick Johnson >>> *Sent:* Monday, November 14, 2011 6:21 PM >>> >>> *To:* [email protected] >>> *Subject:* Re: [google-appengine] Help resolve massive performance >>> regression in 2.7 vs 2.5 runtime**** >>> >>> ** ** >>> >>> Hi Brandon,**** >>> >>> ** ** >>> >>> What you say is fine if your threat model only includes "script kiddies" >>> who don't have your source code. If either of those is not true - you have >>> an adversary with some level of independent skill, or your source code is >>> compromised - any method that relies on obscurity for its security will >>> fare very poorly.**** >>> >>> ** ** >>> >>> One thing to bear in mind is that if your app is ever compromised, your >>> password database and/or source may be posted publicly; at that point, you >>> no longer have to worry about just the initial attacker, but anyone with >>> sufficient motivation.**** >>> >>> ** ** >>> >>> Of course, using federated login like OpenID or the Users API obviates >>> the need to store passwords at all, making it Someone Else's Problem. :) >>> **** >>> >>> ** ** >>> >>> -Nick**** >>> >>> On Tue, Nov 15, 2011 at 1:07 PM, Brandon Wirtz <[email protected]> >>> wrote:**** >>> >>> If I know your salt, I can “De-Hash” bcrypts faster than I can any of >>> the “weird” combinations. Because there are libraries for doing so on ATI >>> cards.**** >>> >>> **** >>> >>> If you do something weird a script kiddie can’t just pull code off the >>> web and attack it. **** >>> >>> **** >>> >>> You want to see who can offline crack a set of 1M users? Your bcrypt >>> list vs my “Weird” You don’t even have to give me the salt I’ll have 10k >>> of those cracked in the first 72 hours. 10 to 1 odds you won’t get through >>> mine without my source code in my life time.**** >>> >>> **** >>> >>> -Brandon Wirtz**** >>> >>> **** >>> >>> PS >>> I don’t usually do the “trust me I’m far more evil” but FBI, Homeland >>> Security, and the CIA have been to my doorstep for things I have defeated, >>> documented, or built to keep from being defeated. The first time I was in 3 >>> rd grade.**** >>> >>> **** >>> >>> *From:* [email protected] [mailto: >>> [email protected]] *On Behalf Of *Nick Johnson >>> *Sent:* Monday, November 14, 2011 3:56 PM**** >>> >>> >>> *To:* [email protected] >>> *Subject:* Re: [google-appengine] Help resolve massive performance >>> regression in 2.7 vs 2.5 runtime**** >>> >>> **** >>> >>> No! Please, please don't do this. Obscurity is no substitute for >>> security.**** >>> >>> **** >>> >>> 1) Bcrypt or similar is not 'overkill' no matter who you are. Users >>> reuse passwords, and they're entitled to the best protection you can >>> reasonably provide them.**** >>> >>> 2) Bcrypt is not there to protect against online attacks, it's there to >>> protect against offline attacks, where an attacker obtains your hashed and >>> salted passwords.**** >>> >>> 3) Doing "something weird" is security through obscurity. Do not base >>> your security on your attacker not knowing what you did. Really, really >>> don't just concatenate salts to the beginning or end of the password.*** >>> * >>> >>> 4) Both MD5 and SHA1 are merkle-damgard construction hashes ( >>> http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction). >>> As a result, the concatenation of several hashes is no more secure than the >>> most secure of the individual hashes.**** >>> >>> **** >>> >>> -Nick Johnson**** >>> >>> **** >>> >>> On Sun, Nov 13, 2011 at 2:58 PM, Brandon Wirtz <[email protected]> >>> wrote:**** >>> >>> Unless you are protecting Medical records bcrypt is overkill if you do >>> some >>> reasonably smart things like "Failed logins from IP >9" >>> >>> Or, if you just do something weird to the password BEFORE you SHA it. >>> Like >>> interleave the user name in the password, Salt1 + UpSaEsRsNwAoMrEd + >>> Salt2 >>> >>> Or Pick 2 Hash's SHA(pass) + Md5(pass) >>> >>> Don't want to store all that string length? Odd Characters from >>> Sha(Pass+salt) + Even Characters from MD5(Pass+Salt) >>> >>> Uniqueness of the method is more important than the method. >>> >>> >>> >>> -----Original Message----- >>> From: [email protected] >>> [mailto:[email protected]] On Behalf Of Brian Quinlan >>> Sent: Saturday, November 12, 2011 6:58 PM >>> To: [email protected] >>> Subject: Re: [google-appengine] Help resolve massive performance >>> regression >>> in 2.7 vs 2.5 runtime**** >>> >>> >>> Hi Pol, >>> >>> On Sun, Nov 13, 2011 at 1:48 PM, Pol <[email protected]> wrote: >>> > Hi, >>> > >>> > Since switching to 2.7 runtime, logging in to http://www.everpix.com >>> > went from about a second to anywhere from 15s to 60s. I tracked it >>> > down to this single password checking line: >>> > >>> > from bcrypt import bcrypt >>> > bcrypt.hashpw(password, self.password_hash) == self.password_hash >>> >>> What value are you using for "threadsafe" in your app.yaml? >>> >>> How large is self.password_hash? >>> >>> Cheers, >>> Brian >>> >>> > This comes from "a native Python implementation of the py-bcrypt >>> > package from http://www.mindrot.org/projects/py-bcrypt/"; grabbed from >>> > here: https://github.com/erlichmen/py-bcrypt. >>> > >>> > So what's happening here and how can we fix this? >>> > >>> > Thanks, >>> > >>> > - Pol >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> "Google App Engine" group. >>> > To post to this group, send email to [email protected] >>> . >>> > To unsubscribe from this group, send email to >>> [email protected]. >>> > For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> > >>> > >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.**** >>> >>> >>> >>> **** >>> >>> **** >>> >>> -- >>> Nick Johnson, Developer Programs Engineer, App Engine**** >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.**** >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.**** >>> >>> >>> >>> **** >>> >>> ** ** >>> >>> -- >>> Nick Johnson, Developer Programs Engineer, App Engine >>> >>> **** >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en.**** >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
