[twitter-dev] Re: Search API cURL strangeness
So it must be based on the IP Address and the UserAgent...
I have changed the UserAgent, so it works now, but I don't
particularly like this solution. It would be nice to know what
happened, and what caused it, so I can try to prevent it from
happening in the future.
On Dec 14, 11:24 am, Tom van der Woerdt i...@tvdw.eu wrote:
Tested it myself with :
tom-mbp:~ tom$ curl --user-agent PivotalVeracity/0.4
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...;
Result :
{results:[],max_id:14696108638863361,since_id:9431322892177408,refresh_url:?since_id=14696108638863361q=batteryoperatedcandles.net,results_per_page:100,page:1,completed_in:0.006856,since_id_str:9431322892177408,max_id_str:14696108638863361,query:batteryoperatedcandles.net}
Seems to work fine... Getting exactly the same results when using the
default User Agent.
Tom
On 12/14/10 5:15 PM, Brian Medendorp wrote:
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdti...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Search API cURL strangeness
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdt i...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Search API cURL strangeness
Tested it myself with :
tom-mbp:~ tom$ curl --user-agent PivotalVeracity/0.4
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrpp=100since_id=9431322892177408;
Result :
{results:[],max_id:14696108638863361,since_id:9431322892177408,refresh_url:?since_id=14696108638863361q=batteryoperatedcandles.net,results_per_page:100,page:1,completed_in:0.006856,since_id_str:9431322892177408,max_id_str:14696108638863361,query:batteryoperatedcandles.net}
Seems to work fine... Getting exactly the same results when using the
default User Agent.
Tom
On 12/14/10 5:15 PM, Brian Medendorp wrote:
UserAgent is 'PivotalVeracity/0.4'
Here's the test script that helped me track down the problem:
[code]
?php
$timeout = 30;
$useragent = 'PivotalVeracity/0.4';
#$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:
1.9.2.13) Gecko/20101203 Firefox/3.6.13';
$url = 'http://search.twitter.com/search.json?
q=batteryoperatedcandles.netrpp=100since_id=9431322892177408since=until=';
#$url = 'http://search.twitter.com/search.json?
q=carnationbreakfastessentials.comrpp=100since_id=since=2010-12-14until=';
#$url = 'http://search.twitter.com/search.json?
q=apple.comrpp=100since_id=since=2010-12-14until=';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
$content = curl_exec($ch);
if(curl_errno($ch))
{
print curl error: .curl_error($ch).\n;
print_r(curl_getinfo($ch));
}
print_r($content);
[/code]
On Dec 14, 11:03 am, Tom van der Woerdti...@tvdw.eu wrote:
And your UserAgent is?
Tom
On 12/14/10 5:02 PM, Brian Medendorp wrote:
I'm building an application that uses the search API to check for data
related to particular domains, and suddenly (within the last week or
so), I have started to experience a strange problem. Some of my
requests are coming back with a cURL error Empty reply from server,
but only when I am searching for a specific set of domains (all of the
other domains work fine).
I wrote a small test script to try and track down the problem, and it
seems that the UserAgent I am setting with cURL seems to be causing
the problem (or part of the problem). If I change the UserAgent to
anything else, I get a normal response.
I remember reading in the documentation that Twitter expects a unique
UserAgent for the application, so that's what I did, but that seems to
be causing problems. This seems like it's likely some sort of
blacklist problem, but I can't figure out why it would work in this
manner (only blocking a small subset of my queries, and not IP-based).
Here are some sample queries I am trying to cURL:
http://search.twitter.com/search.json?q=batteryoperatedcandles.netrp...
http://search.twitter.com/search.json?q=carnationbreakfastessentials
http://search.twitter.com/search.json?q=apple.comrpp=100since_id=s...
The first two don't work unless I change my UserAgent to something
else, but the last one works no matter what.
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
