Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-07 Thread Jon Stevens
Chris, I've tried to setup billing on my appid three times now. I set it up and then it seems to turn off on its own without even sending me an email or any notification. Every few days, I check and it keeps saying that I have a new past due bill. When I go to turn it on again, it asks me for a

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-07 Thread Jon Stevens
Sorry, I apologize for the amateur remark, that was a bit much. jon On Tuesday, August 7, 2012 12:51:56 AM UTC-7, Jon Stevens wrote: Chris, I've tried to setup billing on my appid three times now. I set it up and then it seems to turn off on its own without even sending me an email or

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-07 Thread Cesium
Chris, I am using GAE for my startup and it's just freakin' awesome. I don't use frameworks. I don't use CDNs (whatever those are!?). I don't use SSL. Just MVP and Objectify, and it flat out rocks. Daily, I pee in my pants with excitement. My customers are thrilled. (Both of them). David

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-07 Thread Chris Ramsdale
Would you mind sending me (via direct email) your app ID(s) and we'll look into this? We're aware of Checkout/Wallet issues and are improving the user experience...I promise. -- Chris On Tue, Aug 7, 2012 at 12:51 AM, Jon Stevens latch...@gmail.com wrote: Chris, I've tried to setup billing

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-06 Thread Rick Mangi
You could start by being a little more transparent about what you're doing. Publishing release roadmaps are the norm for almost every software company in the world. You guys seem to love to leave us (the users) in the dark about everything you do until it's released. This is the 2nd time our

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-06 Thread Chris Ramsdale
Is GAE incompatible with other CDNs? No. As mentioned earlier there was an issue on our side that that resulted in Google incorrectly blocking traffic from services such as CloudFlare. For security reasons, we cannot provide greater detail. We did identify the root cause and have rolled back

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-06 Thread Francois MASUREL
Thanx Chris for your answer. François On Tue, Aug 7, 2012 at 1:00 AM, Chris Ramsdale cramsd...@google.com wrote: Is GAE incompatible with other CDNs? No. As mentioned earlier there was an issue on our side that that resulted in Google incorrectly blocking traffic from services such as

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-06 Thread Chris Ramsdale
Rick, We publish upcoming features and functionality within the App Engine Features section of our developer site: https://developers.google.com/appengine/docs/features In regards to bailing on the platform, it is unfortunate to hear. If you have time, I would like to understand what we

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-03 Thread Jon Stevens
If you are upset by Google's pricing for SSL, please star this issue: http://code.google.com/p/googleappengine/issues/detail?id=7932 jon On Thursday, August 2, 2012 10:46:14 AM UTC-7, Chris Ramsdale wrote: Jeff, et al.-- We have verified that a configuration change on our side led to

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-03 Thread Jon Stevens
Here is another issue to star... SSL is too difficult to setup. http://code.google.com/p/googleappengine/issues/detail?id=7933 jon On Friday, August 3, 2012 10:16:36 AM UTC-7, Jon Stevens wrote: If you are upset by Google's pricing for SSL, please star this issue:

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-03 Thread Francois Masurel
Will we have an explanation from Google or CloudFlare about what went wrong? Everything seems to be back to normal at the moment but for how long? Is GAE CDN incompatible? Thanx for your answers. Francois On Wednesday, August 1, 2012 9:06:05 PM UTC+2, Takashi Matsuo (Google) wrote: Hi

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-03 Thread Damon Billian
Hi Francois, A Google employee already posted about the issue. Jeff, et al.-- We have verified that a configuration change on our side led to certain requests being denied / redirected. The rollback of this change started earlier this morning and should be completed shortly. We are actively

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-02 Thread Rick Mangi
That's what we did as well. To answer the question of why did we switch to cloudflare ssl, and was it the price. The answer is that google's ssl is a brand new product, we were on cloudflare before it was an option with google. And yes, it's a lot cheaper. Plus they have all the other features

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-02 Thread Kaan Soral
Indeed CF looks great on paper, but no one should expect it working properly with GAE anymore That's how big companies operate, since CF and GAE/Google are potential competitors, logically speaking they would disrupt their service as much as they can with minimal responsibility I would guess

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-02 Thread Chris Ramsdale
Jeff, et al.-- We have verified that a configuration change on our side led to certain requests being denied / redirected. The rollback of this change started earlier this morning and should be completed shortly. We are actively looking into measures that we can take to ensure that issues like

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-02 Thread Cesium
I hate it when Mommy and Daddy fight. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4QHhaqkMOPUJ. To post to this group, send email to

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-02 Thread Michael Hermus
+1 Internets for Cesium posts. They usually make me laugh. On Thursday, August 2, 2012 5:06:35 PM UTC-4, Cesium wrote: I hate it when Mommy and Daddy fight. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Takashi Matsuo
Hi Jeff, I've escalated this issue. However, I don't think we can completely turn this off. What was the main reason that you're using CloudFlare? -- Takashi On Thu, Aug 2, 2012 at 3:13 AM, Jeff Schnitzer j...@infohazard.org wrote: Ok, this is fucked up. Visit http://www.voo.st/, and get

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
1) SSL at a reasonable price 2) Documented edge cache behavior 3) Real (as opposed to javascript-driven GA-type) traffic statistics 4) They do a number of convenient bits of processing (stripping whitespace from html, etc) Jeff On Wed, Aug 1, 2012 at 12:06 PM, Takashi Matsuo tmat...@google.com

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
Hi Takashi, CloudFlare is a reverse proxy/CDN that provides services to hundreds of thousands of websites. It would be nice to know what is triggering this and you can contact me at my email address (damon@). My hunch is that one domain on the service may be causing this trigger, so we would

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Francois Masurel
Hi Takashi, Do you mean that we can't use any kind of CDN with GAE? I've been successfully using CloudFlare for a few months to make my websites reachable from China. I don't have any other solution at the moment, at least in my budget. It would be great to have a clear answer about this as

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Rick Mangi
Same here! I'm on a thread with enterprise support right now and they are telling me that the reverse proxy is to blame. WTF?? You guys have to stop changing things on us with no warning. This is NOT OK! We are paying customers (at least my company is). You're really giving us every reason to

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread masterblaster
ditto. CloudFlare support recommended Pausing cloudfalre until google and CF work this out. Please help, Google. On Wednesday, August 1, 2012 5:48:03 PM UTC-4, Rick Mangi wrote: Same here! I'm on a thread with enterprise support right now and they are telling me that the reverse proxy is

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
Our CEO literally just shot one of our contacts over there some additional information moments ago. I'll try to update you as soon as I know more. On Wed, Aug 1, 2012 at 3:05 PM, masterblaster dan...@gmail.com wrote: ditto. CloudFlare support recommended Pausing cloudfalre until google and CF

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Francois MASUREL
Thanx Damon for keeping us informed. Francois On Thu, Aug 2, 2012 at 12:19 AM, Damon Billian da...@cloudflare.com wrote: Our CEO literally just shot one of our contacts over there some additional information moments ago. I'll try to update you as soon as I know more. On Wed, Aug 1, 2012

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
I'll do my best. Like I recommended to a few other folks that contacted us already, pausing CloudFlare *might* help in the interim (settings-pause CloudFlare). The only problem is that we will be going direct at that point no SSL:( On Wed, Aug 1, 2012 at 3:23 PM, Francois MASUREL

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
CloudFlare biting you in the ass. It happens. Bonus... Cloudflare doesn't form its proxy request headers correctly. Check the archive this isn't undocumented I've mentioned a lot when helping other people with the same error. -- You received this message because you are subscribed to the

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
On Wed, Aug 1, 2012 at 3:47 PM, Drake drak...@digerat.com wrote: CloudFlare biting you in the ass. What stops me from crafting requests that Google will recognize as an attack and running them through your CDN in the middle of the Survivor season finale, shutting you down? I mean, other than

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Joshua Smith
Who had 3:47pm PDT in the pool for when Brandon would pipe up with his I told you so? Anyone? :) On Aug 1, 2012, at 6:47 PM, Drake drak...@digerat.com wrote: CloudFlare biting you in the ass. It happens. Bonus... Cloudflare doesn't form its proxy request headers correctly. Check the

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
Whatever, Drake (Brandon). Hi Jeff, We're in contact with our Google contact to find out what the issue is. I'll post as I have any additional details to share. On Wed, Aug 1, 2012 at 3:47 PM, Drake drak...@digerat.com wrote: CloudFlare biting you in the ass. It happens. Bonus...

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Not if they form their request headers properly. CF sends IPs that are malformed, nonsensical, and non-responsive. (and often blank) My CDN solution says Request on behalf of X the right way. Oh, and CF has already tried doing that to me. More than once. -- You received this message because

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
Hi, We have heard back from our contact at Google and this is being worked on (don't have an eta for a full fix yet will let you know...but hoping within the next day or less). It also looks like our IPs are going to get whitelisted to prevent these issues in the future. On Wed, Aug 1, 2012

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
That's what y'all say every 9 months... https://groups.google.com/forum/?fromgroups#!topic/google-appengine/om1DF_61 Xl4 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
What evidence do you have to suspect that Google is treating CF as an attack because of the way they form headers? That doesn't even make sense - if CF is mangling headers, presumably they've been doing it for a while, and it wasn't being blocked before. More likely there's some sort of real

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
Hi Drake, We have already had the confirmation from the Google Security Engineer that is our contact that he will look into this. It is probably best not to comment on discussions that you are not involved in can't read minds. On Wed, Aug 1, 2012 at 4:07 PM, Drake drak...@digerat.com wrote:

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Instead of whitelisting your IPs Fix your Proxy. Google is not the only place with these issues. Don't lie about the IP the request is on behalf of, don't create IPs that don't exist or are in unassigned ranges. Don't try to piggy back session headers that aren't from the same session. Google

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
On Wed, Aug 1, 2012 at 4:07 PM, Drake drak...@digerat.com wrote: That's what y'all say every 9 months... https://groups.google.com/forum/?fromgroups#!topic/google-appengine/om1DF_61 Xl4 Brandon, this is why I can't trust anything you say anymore. That thread is unrelated to this issue, and

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
you are being deliberately disingenuous. I vote for this, personally. Hi Drake. I'm pretty sure our engineering team, given their background at major internet companies and the like, are probably pretty familiar with creating properly formed headers. On Wed, Aug 1, 2012 at 4:15 PM, Jeff

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
This is a mirror of the conversation from 9 months ago and the one from 9 months before that, and the conversation on the Amazon forum, and the Rackspace forum and the Google Webmasters forum... https://groups.google.com/forum/#!msg/google-appengine/om1DF_61Xl4/cRtKFojKc BMJ

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Damon Billian
The headers would be coming from nginx. Unless nginx is passing along malformed headers, it wouldn't be caused by us. CF does headers wrong Please provide something specific that is malformed. We're more than happy to check on it both internally and with nginx. To date, you just keep making claim

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
On Wed, Aug 1, 2012 at 5:29 PM, Drake drak...@digerat.com wrote: This is a mirror of the conversation from 9 months ago and the one from 9 months before that, and the conversation on the Amazon forum, and the Rackspace forum and the Google Webmasters forum... I'll pretend for a moment that

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Here from when I was working with the guys at Google about proxying requests when I was consulting for Akamai. Your requests will be blocked if you cross a thresh hold of requests that meet the following criteria. PROXY reports request on behalf of Google Internal IP address. (or any IP

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Are you so certain of this that you're willing to grant me free license to DOS your CDN at a time of my choosing? I have some reasonable suspicions as to how to tickle Google's defenses, and shutting down your entire business in the middle of prime-time would be a really spectacular way to prove

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jon Stevens
On Wed, Aug 1, 2012 at 6:03 PM, Drake drak...@digerat.com wrote: I only have one CDN client at the moment. I think this says it all. jon -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
If I read this right, you're telling me that anyone can DOS your CDN by making a lot of requests with bogus User-Agent strings (which, incidentally, was one of my guesses). Forget CloudFlare for a moment. You aren't the least bit concerned about that? Or are you saying that you filter

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
If I read this right, you're telling me that anyone can DOS your CDN by making a lot of requests with bogus User-Agent strings (which, incidentally, was one of my guesses). Forget CloudFlare for a moment. You aren't the least bit concerned about that? If your CDN is worth their salt they

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jon Stevens
On Wed, Aug 1, 2012 at 6:37 PM, Drake drak...@digerat.com wrote: If I read this right, you're telling me that anyone can DOS your CDN by making a lot of requests with bogus User-Agent strings (which, incidentally, was one of my guesses). Forget CloudFlare for a moment. You aren't the least

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Message- From: google-appengine@googlegroups.com [mailto:google- appeng...@googlegroups.com] On Behalf Of Jon Stevens Sent: Wednesday, August 01, 2012 6:16 PM To: google-appengine@googlegroups.com Subject: Re: [google-appengine] APP DOWN due to some sort of undocumented Google security

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
whitelist / headers Playing with the headers sucks. CF plays with theirs a lot, so if you are in the business already, be in the business. If you Cache you kind of have to, because Referrer often isn't going to be what is expected. CF runs on other people's IPs so they have less control over

Re: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Jeff Schnitzer
There are a variety of reasons people run reverse proxies against GAE. Someone in this thread has already mentioned they do this to support access from China. That poor developer getting DDOS'd would probably like to use CloudFlare right now. Folks using wwwizer and custom solutions for SSL

RE: [google-appengine] APP DOWN due to some sort of undocumented Google security system

2012-08-01 Thread Drake
Jeff, Your CDN should have the conversation, it should never hit the list. Google is not the only place that CF bumps heads on the network edge. There have been similar discussions on other cloud providers. Google is doing some things with security through obscurity. Which sucks. They are also