[twitter-dev] Re: URL encode problem
Both of the solutions work! Seems the link was broken by the "&" before... Thanks very much for all of your kind help even on these holidays! :-) Best, epomqo On Dec 26, 4:48 pm, Matt Harris wrote: > It looks like you're running curl from the command line. Try wrapping the > entire URL in quotes - this would make your request: > curl > "http://search.twitter.com/search.json?geocode=37.781157,-122.398720,1..."; > > file.txt > > Alternatively leave out the quotes and escape the &. e.g. lang=en&rpp=100 > becomes lang=en\&rpp=100 > > Best, > @themattharris > > On Dec 26, 2010, at 5:57, epomqo wrote: > > > Thanks Adam. I will carefully look at the results from the "lang" > > parameter. > > > From the screen display above: > > > [1] Done > > curlhttp://search.twitter.com/search.json?geocode=37.781157,-122.398720,15km > > [2]- Done lang=en > > [3]+ Done rpp=100 > > > It seems that the last two parameters "lang" and "rpp" are not treated > > together with the main cURL command, and they don't really affect the > > search result. I also don't get 100 results after adding the "rpp" > > command. Have you ever succeed in running such a multi-parameter query > > with cURL and getting an output file? > > > Many thanks, > > epomqo > > > On Dec 26, 11:28 am, Adam Green <140...@gmail.com> wrote: > >> On Sun, Dec 26, 2010 at 5:11 AM, epomqo wrote: > > >>> And apparently the "lang" parameter sometimes doesn't work: I still > >>> get tweets in other languages. > > >> From my experience the lang parameter, is not a language detection > >> algorithm. It just pays attention to the language the user has set in their > >> profile settings. If they haven't changed it, the default of English is > >> left > >> set, and you get their tweets no matter what the lang parameter is sent to. > >> If you don't use the lang parameter, you get all tweets that match the > >> search in any language. If you set lang=en, you get far fewer tweets in > >> languages other than English, but at least 10-20% are not in English. Of > >> course that depends on whether your search words match another language. > > > -- > > Twitter developer documentation and resources:http://dev.twitter.com/doc > > API updates via Twitter:http://twitter.com/twitterapi > > Issues/Enhancements Tracker:http://code.google.com/p/twitter-api/issues/list > > Change your membership to this > > group:http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: URL encode problem
Thanks Adam. I will carefully look at the results from the "lang" parameter. >From the screen display above: [1] Donecurl http://search.twitter.com/search.json?geocode=37.781157,-122.398720,15km [2]- Donelang=en [3]+ Donerpp=100 It seems that the last two parameters "lang" and "rpp" are not treated together with the main cURL command, and they don't really affect the search result. I also don't get 100 results after adding the "rpp" command. Have you ever succeed in running such a multi-parameter query with cURL and getting an output file? Many thanks, epomqo On Dec 26, 11:28 am, Adam Green <140...@gmail.com> wrote: > On Sun, Dec 26, 2010 at 5:11 AM, epomqo wrote: > > > And apparently the "lang" parameter sometimes doesn't work: I still > > get tweets in other languages. > > From my experience the lang parameter, is not a language detection > algorithm. It just pays attention to the language the user has set in their > profile settings. If they haven't changed it, the default of English is left > set, and you get their tweets no matter what the lang parameter is sent to. > If you don't use the lang parameter, you get all tweets that match the > search in any language. If you set lang=en, you get far fewer tweets in > languages other than English, but at least 10-20% are not in English. Of > course that depends on whether your search words match another language. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: URL encode problem
Thanks for the reply David, happy holidays too! I tried with your suggestion, however the following cURL command in shell script still cannot export search result to a .txt file: curl http://search.twitter.com/search.json?geocode=37.781157,-122.398720,15km&lang=en&rpp=100&page=1 >> searchrecord.txt The screen display says something like: [1] Donecurl http://search.twitter.com/search.json?geocode=37.781157,-122.398720,15km [2]- Donelang=en [3]+ Donerpp=100 And apparently the "lang" parameter sometimes doesn't work: I still get tweets in other languages. What I don't understand is that a simple parameter query works fine, and the output is like: curl http://search.twitter.com/search.json?geocode=37.781157,-122.398720,15km >> searchrecord.txt % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total Spent Left Speed 100 10488 100 104880 0 14541 0 --:--:-- --:--:-- --:--:-- 52806 Any suggestions? Thanks, epomqo On Dec 26, 7:34 am, David wrote: > You don't want to url encode the ampersands in this case since you want to > use as part of their reserved function for separating parameters. > > This URL should > work:http://search.twitter.com/search.json?geocode=37.781157,-122.398720,1... > > If you wanted to specify an ampersand inside a query parameter, then you > would url encode it, but in this case, the &'s and the ? are special > characters used for defining and separating parameters so you don't want to > encode them. > > Happy Hollidays, > David -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: URL encode problem
Sorry the links seem to be truncated:
this works:
curl ... search.json?geocode=37.781157%2C-122.398720%2C15km > test.txt
this didn't work:
curl ... search.json?geocode=37.781157%2C-122.398720%2C15km%26lang=en
%26rpp=100%26page=1 > test.txt
On Dec 25, 5:08 pm, epomqo wrote:
> Dear all,
>
> Merry Christmas!
>
> I just have a quick question about the URL encode when writing a cURL
> command in shell. The following command works:
>
> curlhttp://search.twitter.com/search.json?geocode=37.781157%2C-122.398720...
>
> > test.txt
>
> However, if I want to put more parameters like
> "lang=en&rpp=100&page=1" in http, I encoded "&" as "%26" in the
> following way but it didn't work:
>
> curlhttp://search.twitter.com/search.json?geocode=37.781157%2C-122.398720...
>
> > test.txt
>
> It complained that: {"error":"The within: operator requires distance
> and units of 'mi' or 'km'"} So this "%26" immediately after "15km"
> seems to cause problems. Any ideas how to fix it?
>
> Many thanks for the help!
> epomqo
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] URL encode problem
Dear all,
Merry Christmas!
I just have a quick question about the URL encode when writing a cURL
command in shell. The following command works:
curl
http://search.twitter.com/search.json?geocode=37.781157%2C-122.398720%2C15km
> test.txt
However, if I want to put more parameters like
"lang=en&rpp=100&page=1" in http, I encoded "&" as "%26" in the
following way but it didn't work:
curl
http://search.twitter.com/search.json?geocode=37.781157%2C-122.398720%2C15km%26lang=en%26rpp=100%26page=1
> test.txt
It complained that: {"error":"The within: operator requires distance
and units of 'mi' or 'km'"} So this "%26" immediately after "15km"
seems to cause problems. Any ideas how to fix it?
Many thanks for the help!
epomqo
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] Reconnection to Streaming API
Hi All, I was trying to write some shell scripts to do the following job: when the Streaming API stops working for whatever reason, make a reconnection immediately. Here is how I did it: I wrote a script called "check.sh": c=$(ps aux | grep curl|wc -l) if [[ "$c" -eq 1 ]] % which means there is no active curl process, the only live one is "grep" then curl -d @locations http://stream.twitter.com/1/statuses/filter.json - u username:password >> result.txt fi And I created a cron job which runs this "check.sh" every minute: 0-59 0-23 * * * check.sh >> monitor.txt It does the job partially: it can reconnect to Streaming API in the case of disconnection. My problem is that, by running this script as a cron job, the Streaming API got disconnected at the end of every minute, and then reconnect again at the beginning of the next minute, and I have no clue why it got disconnected every minute. The problem seems coming from the cron job I created, but I couldn't find mistakes there. Does anyone have suggestions on this? Many thanks for the kind help! epomqo -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: How to access to friendship data of twitter users
Have a look here: http://an.kaist.ac.kr/traces/WWW2010.html The authors described how you can get a social graph from Twitter, and there is even a ready-to-use social graph of many Twitter users available for you to download. On Dec 22, 3:31 pm, Xintao Liu wrote: > Hello! > > If I want to access all of the friendship data of twitter users, i.e. > how twitter users connect with each other, is it possible to do it? I > am not interested in other information of users such as name, gender > and so on. In other words, I just want to get the network of twitter > users, no other information. I am just the beginner of twitter > developer, and I am sorry if the message would bother you. > > Any of your reply is highly appreciated! Thank you. > > Xintao > -- > PhD Candidate > Dept. of Urban Planning and Environment > Royal Institute of Technology, 100 44 Stockholm, Sweden > Address: Drottning Kristinas väg 30 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: Streaming API - Multiple connections on 1 IP?
Many thanks! On Dec 16, 12:18 pm, Augusto Santos wrote: > Yes you can. > > > > On Thu, Dec 16, 2010 at 8:23 AM, epomqo wrote: > > Hello everyone, > > > Just a simple question: can I launch multiple connections to Streaming > > API using different accounts, but on the same machine? The official > > document seems don't say this clearly: > > > "Each account may create only one standing connection to the Streaming > > API. Subsequent connections from the same account may cause previously > > established connections to be disconnected. Excessive connection > > attempts, regardless of success, will result in an automatic ban of > > the client's IP address. Continually failing connections will result > > in your IP address being blacklisted from all Twitter access." > > > Many thanks for the help! > > epomqo > > > -- > > Twitter developer documentation and resources:http://dev.twitter.com/doc > > API updates via Twitter:http://twitter.com/twitterapi > > Issues/Enhancements Tracker: > >http://code.google.com/p/twitter-api/issues/list > > Change your membership to this group: > >http://groups.google.com/group/twitter-development-talk > > -- > 氣 -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Streaming API - Multiple connections on 1 IP?
Hello everyone, Just a simple question: can I launch multiple connections to Streaming API using different accounts, but on the same machine? The official document seems don't say this clearly: "Each account may create only one standing connection to the Streaming API. Subsequent connections from the same account may cause previously established connections to be disconnected. Excessive connection attempts, regardless of success, will result in an automatic ban of the client's IP address. Continually failing connections will result in your IP address being blacklisted from all Twitter access." Many thanks for the help! epomqo -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Some questions about tracking
Dear all, Hello, this is Xiaowen from Switzerland. I have been using streaming api for a while to track and filter public tweets as I want, and it works great, thank you! However, I am posting to ask a few questions that I encountered recently but have no idea how to solve after reading the streaming api documents: 1) The "location" query parameter enables us to track all geo-tagged tweets in a bounding box, but is it possible to just get all tweets in the box (an area), no matter they are geo-tagged or not? 2) From what I understand, the "track" query parameter doesn't care about where the tweet is posted. When tracking tweets with a set of keywords, is it possible to also limit the location of these tweets, for example, to get all tweets containing "iphone" in a box of San Francisco? It seems difficult, firstly Twitter api are logical ORs rather than ANDs, secondly even I add the "location" query parameter this will only return geo-tagged tweets in the SF area. Any suggestions? Or I have to get all the tweets with "iphone" all over the world at first, and then check the relevant info in each of them, say, time_zone? That would be extremely time consuming however... I mean if we have a easy solution for question 1), then I could do some simple post-processing to achieve the goal in 2), just by filtering all the tweets with the word "iphone". What do you say? Many thanks for the help! Xiaowen -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Re: API Tracking inaccuracy
I see. Thanks for the prompt replies! I will do post-processing then. I already noticed that the streaming results were not well-structured and displayed :( epomqo On Apr 27, 4:06 pm, John Kalucki wrote: > You can't do logical ANDs between predicate types. The assumption is > that you can do post processing on your end to further filter your > results. We over deliver, you de-duplication, reorder and filter. > Streaming results are not display-ready. > > -John Kaluckihttp://twitter.com/jkalucki > Infrastructure, Twitter Inc. > > On Tue, Apr 27, 2010 at 6:52 AM, epomqo wrote: > > Thanks for the advice. I checked some of the results, this is indeed > > the case. Then is there a way to make it logical ANDs? If we cannot do > > it with the same command, maybe I could try to put these two kinds of > > parameters in a single file and separate them with "&"? > > > Example file (locandfollow): > > locations=-122.75,36.8,-121.75,37.8 & > > follow=...,...,... > > > command: > > "curl -d @locandfollowhttp://stream.twitter.com/1/statuses/filter.json > > -uAnyTwitterUser:Password" > > > Best Regards, > > epomqo > > > On Apr 27, 3:21 pm, John Kalucki wrote: > >> There isn't quite enough information in this email to provide a > >> definitive answer. Providing a predicate list and an example of an > >> unexpected match would be helpful in describing the Streaming API > >> behavior. > > >> Note that the predicates are logical ORs, so that any match will cause > >> a delivery. These are not logical ANDs. So, if a user in your > >> following file tweets without geodata, you'll get a tweet without > >> geodata. > > >> The only known issues with track are around non-space-separated > >> languages, such as Chinese, Japanese and, I think, Arabic, language > >> representations. > > >> -John Kaluckihttp://twitter.com/jkalucki > >> Infrastructure, Twitter Inc. > > >> On Tue, Apr 27, 2010 at 6:17 AM, epomqo wrote: > >> > Dear all, > > >> > Good morning! I have been experiencing some problems when I was trying > >> > to use streaming api to get filtered tweets, and I have no idea why > >> > they happened :( > > >> > Specifically I wanted to get tweets from specific users within > >> > specified locations. I got some advices here and I proceeded > >> > successfully with this command: > > >> > curl -d @following -d > >> > @locationshttp://stream.twitter.com/1/statuses/filter.json > >> > -u > > >> > The problem is that when I was checking the filtered results, there > >> > are still quite a lot tweets appeared without geographical > >> > information, and tweets from users that I didn't include in the > >> > following list. So there seems some inaccuracy in this filtering > >> > function. > > >> > Any ideas or suggestions? Thanks! > > >> > Best, > >> > epomqo > > >> > -- > >> > Subscription > >> > settings:http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] Re: API Tracking inaccuracy
Thanks for the advice. I checked some of the results, this is indeed the case. Then is there a way to make it logical ANDs? If we cannot do it with the same command, maybe I could try to put these two kinds of parameters in a single file and separate them with "&"? Example file (locandfollow): locations=-122.75,36.8,-121.75,37.8 & follow=...,...,... command: "curl -d @locandfollow http://stream.twitter.com/1/statuses/filter.json -uAnyTwitterUser:Password" Best Regards, epomqo On Apr 27, 3:21 pm, John Kalucki wrote: > There isn't quite enough information in this email to provide a > definitive answer. Providing a predicate list and an example of an > unexpected match would be helpful in describing the Streaming API > behavior. > > Note that the predicates are logical ORs, so that any match will cause > a delivery. These are not logical ANDs. So, if a user in your > following file tweets without geodata, you'll get a tweet without > geodata. > > The only known issues with track are around non-space-separated > languages, such as Chinese, Japanese and, I think, Arabic, language > representations. > > -John Kaluckihttp://twitter.com/jkalucki > Infrastructure, Twitter Inc. > > On Tue, Apr 27, 2010 at 6:17 AM, epomqo wrote: > > Dear all, > > > Good morning! I have been experiencing some problems when I was trying > > to use streaming api to get filtered tweets, and I have no idea why > > they happened :( > > > Specifically I wanted to get tweets from specific users within > > specified locations. I got some advices here and I proceeded > > successfully with this command: > > > curl -d @following -d > > @locationshttp://stream.twitter.com/1/statuses/filter.json > > -u > > > The problem is that when I was checking the filtered results, there > > are still quite a lot tweets appeared without geographical > > information, and tweets from users that I didn't include in the > > following list. So there seems some inaccuracy in this filtering > > function. > > > Any ideas or suggestions? Thanks! > > > Best, > > epomqo > > > -- > > Subscription > > settings:http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] API Tracking inaccuracy
Dear all, Good morning! I have been experiencing some problems when I was trying to use streaming api to get filtered tweets, and I have no idea why they happened :( Specifically I wanted to get tweets from specific users within specified locations. I got some advices here and I proceeded successfully with this command: curl -d @following -d @locations http://stream.twitter.com/1/statuses/filter.json -u The problem is that when I was checking the filtered results, there are still quite a lot tweets appeared without geographical information, and tweets from users that I didn't include in the following list. So there seems some inaccuracy in this filtering function. Any ideas or suggestions? Thanks! Best, epomqo -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] Re: Streaming API
It works! Now I know how to put multiple parameters. Thanks very much! Best, epomqo On Apr 24, 4:39 pm, John Kalucki wrote: > The curl man page explains how to send multiple parameters. If you > just put them in one file, curl runs then all together. You have to > separate them with &, or have curl do it for you by specifying > multiple -d @ pairs: > > adrift.local:/tmp> head foo bar > ==> foo <== > follow=1234 > > ==> bar <== > track=north > adrift.local:/tmp> curl -v -x "" -d @foo -d @bar > stream.twitter.com/1/statuses/filter.xml > > -John Kaluckihttp://twitter.com/jkalucki > Infrastructure, Twitter Inc. > > On Sat, Apr 24, 2010 at 5:27 AM, epomqo wrote: > > Dear Sirs, > > > Hello! My name is Xiaowen, I have been playing with Twitter for some > > time and I am writing to ask a small question regarding Streaming > > API :) > > > From the documentation I know we can use the command "curl -d > > @locationshttp://stream.twitter.com/1/statuses/filter.json- > > uAnyTwitterUser:Password" to filter geo-tagged tweets in a bounded box > > area, we could also use similar method to follow tweets of specific > > users. These are two parameters which can be implemented by statuses/ > > filter method. My question is, if I want to incorporate both of these > > two parameters in one command, how to put them together? I have tried > > something like > > > "curl -d @locations > > @followinghttp://stream.twitter.com/1/statuses/filter.json > > -uAnyTwitterUser:Password" > > > but it doesn't work :( > > > Later I got response from Twitter support, suggesting that: > > > "One Streaming API connection can filter both geo-tagged tweets and > > tweets from specific users. You may want to combine both of your > > parameters into one file so that you do not need to reference two in > > your command (for example @locandfollow instead of @locations > > @following)." > > > But then what is the correct way to put the parameters in one file? I > > tried simply put them together, like this: > > > locations=-122.75,36.8,-121.75,37.8; > > follow=...,...,... > > > but still it doesn't work. > > > Many thanks for your advices! > > > -- > > Subscription > > settings:http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] Streaming API
Dear Sirs, Hello! My name is Xiaowen, I have been playing with Twitter for some time and I am writing to ask a small question regarding Streaming API :) >From the documentation I know we can use the command "curl -d @locations http://stream.twitter.com/1/statuses/filter.json - uAnyTwitterUser:Password" to filter geo-tagged tweets in a bounded box area, we could also use similar method to follow tweets of specific users. These are two parameters which can be implemented by statuses/ filter method. My question is, if I want to incorporate both of these two parameters in one command, how to put them together? I have tried something like "curl -d @locations @following http://stream.twitter.com/1/statuses/filter.json -uAnyTwitterUser:Password" but it doesn't work :( Later I got response from Twitter support, suggesting that: "One Streaming API connection can filter both geo-tagged tweets and tweets from specific users. You may want to combine both of your parameters into one file so that you do not need to reference two in your command (for example @locandfollow instead of @locations @following)." But then what is the correct way to put the parameters in one file? I tried simply put them together, like this: locations=-122.75,36.8,-121.75,37.8; follow=...,...,... but still it doesn't work. Many thanks for your advices! -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
