I've noticed for when Twitterbot crawls my app on GAE, the response does not appear to be gzipped (as seen by the response bytes size in GAE logs). I've tested this with other apps deployed on the *.appspot.com, for example https://ga-dev-tools.appspot.com/.
To illustrate, I'm using a test user agent "Twitterbot/9.0", although the actual Twitter user agent is "Twitterbot/1.0". # Test case 1: With a generic Mozilla useragent Mozilla/9.0 + gzip headers, response returned is gzipped $ curl 'https://ga-dev-tools.appspot.com/' -H 'Accept-Encoding: gzip, deflate, sdch' --compressed -A 'Mozilla/9.0' -i HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: no-cache Content-Encoding: gzip Vary: Accept-Encoding Date: Mon, 29 Jun 2015 10:11:35 GMT Server: Google Frontend Alternate-Protocol: 443:quic,p=1 Transfer-Encoding: chunked # Test case 2: With a Twitterbot useragent Twitterbot/9.0 + gzip headers, response returned is not gzipped $ curl 'https://ga-dev-tools.appspot.com/' -H 'Accept-Encoding: gzip, deflate, sdch' --compressed -A 'Twitterbot/9.0' -i HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: no-cache Date: Mon, 29 Jun 2015 10:12:06 GMT Server: Google Frontend Content-Length: 7956 Alternate-Protocol: 443:quic,p=1 # Test case 3: With a Twitterbot useragent Twitterbot/9.0 + no other headers, response returned is not gzipped $ curl 'https://ga-dev-tools.appspot.com/' -A 'Mozilla/9.0' -i HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: no-cache Date: Mon, 29 Jun 2015 10:13:17 GMT Server: Google Frontend Content-Length: 7956 Alternate-Protocol: 443:quic,p=1 You will noticed that GAE is returning identical responses for test #2 (Twitterbot) and #3 (uncompressed request). This is unexpected and rather puzzling. Any idea why? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d3119e67-db89-4e25-92c3-2b087a00ec6b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
