[twitter-dev] Re: Best way to implement caching of searches (with multithreading)?
Really a database is the way to go. Any modern database should allow you to check if a value is in there before inserting, so the same tweet won't go in there twice. Additionally, not every user search has to use the up to the minute results. They can go back just a little in time (30 seconds or so) to have batches in case there are multiple people searching for the same thing. On Oct 2, 7:36 am, Bjoern bjoer...@googlemail.com wrote: Hi, just wondering about a best practice thing. Suppose I show results of specific Twitter searches on a web site. How would I go about caching the searches? The naive approach seems to be to first check in my own database, then do a twitter search with the since_id parameter to only get results I don't already have. Then store the results from twitter in the database, too, and return the merged results to the web site. The problem I see is that if multiple user run the same search on my web site, threading issues might occur (as each user starts a separate thread on my server). Not only could multiple twitter searches with the same since_id be executed (maybe forgivable), but trouble starts when said results are to be inserted in my local database. Different threads could attempt to insert the same messages into my database. One simple solution I could imagine: just use the message ids from twitter as the primary key in my local database. That way, multiple threads saving the same message would just overwrite the message with itself. I actually wonder if that is a common solution - to use the twitter ids as primary keys (also for users, direct messages...). I have kind of arrived at the opinion that this would be the way of least resistance, although I feel a bit uneasy about it. An alternative that came to my mind might be to have single threaded background jobs do the copying of the search results from twitter to my database, and only show the results from my cache to the web site. This would cause some lag in the time the search results would appear, but it would not be too bad. However, if I have a lot of different searches, it would become infeasible to update all of them periodically. It would become necessary to only trigger an update when a user does the search. At that point things might get overly complicated: presumably I would need some kind of Ajax solution to trigger the caching with the first request, show a spinner while the updating of my local db is going on, and then show the results from my local cache/db. The trickiest part being to prevent the starting of multiple update tasks for the same search. All in all the simple solution might be the better way to go? Would be interested in hearing your opinions, experiences and solutions! Thanks! Björn
[twitter-dev] Re: how are the ten trends born?
It's pretty simple, but with a few twists. First of all, remember that everything that Twitter does is done with simplicity and efficiency in mind. For the most part its just a frequency count of words over a short time period, minus stop words, filtering out usernames (notice @foo is never a trend) and URLs. How it combines Wave OR Google Wave I'm unsure of, and then there's some basic spam filtering in there additionally. I was theorizing that some of it is based on accelleration of words above their standard volume. Apple for example is always talked about a great deal, but isn't always trending. Sometimes it has a greater volume than other trending words, but it doesn't trend. Yet some things stick around for a long time like IranElection. I need to dig into this more. I almost had it fully modeled at one point, then lost the code (damn you version control mistakes) dave On Oct 2, 10:54 am, Nigel Cannings nigelcanni...@googlemail.com wrote: @secretbear did it first in the halcyon days of the PubSub Firehose... I'd ask him == Why not encrypt the mail you send me? You never know who's looking. If you use Firefox, why not use the FireGPG plugin to make it easy (http://getfiregpg.org) Get my key from:http://keyserver.pgp.com/ == On Thu, Oct 1, 2009 at 4:20 PM, Martin Dudek goosegoesgro...@gmail.com wrote: Good morning wonder if somebody knows how twitter determines the ten trends it declares every five minutes? Is this a pure word/phrase frequency algorithm or some more complexity behind. Thanks martin
[twitter-dev] Re: About the oneforty application directory
I'll vouch for Laura as being an upright person, and all of the other oneforty people I've met are as well. They aren't going to try to screw anyone over and I have faith that whatever they come up with will be fair for developers and the community both. david
[twitter-dev] Re: master thesis related to Twitter
A small group of us (mainly Harvard students, but others as well) had similar questions a few months back and we've started digging into the research pretty heavily (also doing contracting/consulting). Our group is called the Web Ecology Project and we've released a handful of academic (but accessible and market relevant) papers about events and influence on Twitter. We also released some source code for identifying the language of Tweets using the Google API and python. We have a research relations arm and I've forwarded your contact information to our research coordinator Dharmishta who will probably be in touch soon. It might be worth looking over our papers. When you do, feel free to email me (or any of us) and ask questions. We've done some semantic analysis of tweets for our paper on the death of michael jackson and the response on Twitter using the ANEW method/dataset and NLTK. Our work can be found at http://webecologyproject.org and all of our stuff is released under a Creative Commons license so feel free to quote us and use bits where you need to. One of the hardest initial things that we encountered with Twitter was gathering and storing of useful and meaningful data but now we've gotten mostly past those issues and we are now starting to mine and analyze other social networks as well. Thanks, David Fisher Web Ecology Project On Sep 27, 8:11 pm, Stefna mstefa...@gmail.com wrote: Thank you all for the feedback. My main motivation for posting this thread was to gather some loose ideas. One man’s trash is another man’s treasure, posting a link costs nothing but saved a lot of mine time. strict formed data - 140 chars, #tag, @username, RT etc. - that's why there are so many sites presenting graphs, charts, trends, tendencies etc. first pick was vaguely to analyse the semantic meaning of tweets - like take a tag, take all the words with tweets with this tag and on that base find tweets that might me related to the topic. Or finding and measuring tags that are in tweets with aforementioned one. I should keep in mind that my task is to present a satisfactory dissertation, not cure for cancer. And probably by the time I finish my work someone else will ship similar software independently. Nevertheless by creating it I will learn a lot of stuff about Twitter itself not to mention improving programming skills. On 27 Wrz, 20:38, Mitchel Berberich mitch...@mbsw.com wrote: Careful! Stefna was talking about semantic meaning. Not syntactic ... But I think you're right - Stefna, please us tell a bit more about the context. And - what do you think of when you say strict formed data? What exactly do you want to achieve? Maybe your promoter should tell you in more detail, what he expects from you? On Sep 27, 6:44 am, Nalin Savara nsn...@gmail.com wrote: Good luck buddy.. Btw I'm curious, What exactly are you referring to or focussing on when you say 'syntactic meaning of tweets' ? I mean I'd appreciate a clarification on 'syntactic meaning to whom ?' and 'syntactic meaning of tweets in what context ?' Just wondering.. Best Regards, Nalin On 9/25/09, Stefna mstefa...@gmail.com wrote: I've submitted a ticket with following content: *** *** *** I am a 23 years old student of informatics at AGH Universtity of Science and Technology in Cracow (Poland). Due to a rapid development, strict formed data and accessible API I would like to designate my master thesis to the Twitter related topic. My promoter is the PhD at the Department of Computer Linguistics and our first pick was vaguely to analyse the semantic meaning of tweets. Do you have suggestions about the dissertation topic? Do you have any pending requests or prospect features you want to develop? I will browse known issues, I will think thoroughly about the topic but still - your suggestion might be very helpful. Even the shortest one (like good luck) will encourage me to more intensive research. *** *** *** Does anyone have any suggestions? My ticket has a six-digit number so I'm afraid I won't get any answer :) I'll probably ask for help during my work so I subscribe to this group anyway. Thanks in advance! -- Sent from Gmail for mobile | mobile.google.com- Hide quoted text - - Show quoted text -
[twitter-dev] Re: Search API Query Limitation (140 characters?)
If you need to search specific users why don't you use the Shadow API and grab all of their tweets and then search them locally? On Sep 28, 3:14 pm, Chad Etzel c...@twitter.com wrote: Hello, The limit is indeed 140 and most likely won't be going up any time soon. The reason for the limit is for performance reasons. In order to do timely queries we don't allow for longer/arbitrary queries which could be very complex. -Chad On Mon, Sep 28, 2009 at 3:08 PM, zapnap npla...@gmail.com wrote: (this could be overcome, I suppose, by performing multiple queries, but that isn't much of a solution if you want to use the stock twitter js search widget, etc) On Sep 27, 11:37 am, zapnap npla...@gmail.com wrote: Search API queries appear to be limited to 140 characters. I mean, that's cute and all, but it's also rather limiting. In my particular case I was hoping to construct a search for a particular term from a group of N Twitter users only (instead of searching everyone). Is there another way to achieve this? Are there any plans to allow longer / more complex search queries? Thanks! ..nap
[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids
I'm sorry, but the problem isn't Twitter- its your language and JSON parser. Outputting everything as a string, when it clearly should be a number, is inefficient and crazy. Saying that startups can't afford 64 bit processors in systems is crazy. Most startups I know are running on EC2 or have fairly new hardware. I bought a killer 64-bit quad xeon server for less than $1,500 for our startup and its rocking. If your startup doesn't have $1,500 for a primary capital computing expense that's another problem you have there. You either need to run on a newer system or use a language that can properly handle 64-bit numbers. C, Python, Ruby, Scala, Erlang, C#, etc none of them have problems with 64-bit ints. Not a Twitter problem. It's a programming issue on your end and unfortunately I can't help as I don't know PHP in depth. -dave On Sep 24, 3:29 pm, Dewald Pretorius dpr...@gmail.com wrote: That magical maximum number appears to be 1 (1.0E+12). So, for tweet ids we still have a bit of breathing space. Dewald On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote: Clearly PHP_INT_MAX plays no role in json_decode. There must be some other mystical maximum number above which it represents the number as float in the decoded data. Dewald
[twitter-dev] Re: Getting screen_name from id without gazillion API calls?
I've done this this way: Every time I get a user's data- I store it in my database. Doesn't matter if its from gardenhose or a REST method. I track various versions of it and keep all changes (good data warehousing practice). Then, when I crawl a user's list of friends I ask the database if we already have the user (by id). If we've already got them, then I don't waste an API call in asking for it again (as I only have so many). If we don't have them, then I ask Twitter for it. If the next person I go through has all the same friends, then I never make an additional REST API requests. I also do this when accessing the Search API, to make sure I have all user profiles (at least in some stage). Unfortunately I have to do this one by screen_name/from_user because the Search API returns different user id numbers (Which I also store so I can match them if I must). I've got nearly 8M twitter user profiles this way so far and its always growing. -dave On Sep 8, 12:41 pm, dizid glasw...@gmail.com wrote: Owkaye, Well, Twitter would not need to change the system (with id's an screenname's) because there is a user advantage to being able to change your screenname. But, maybe the Twitter API could return both ID and screen_name with only 1 call to the API. Programmers can then choose to work with ID and/or screen_name and the API calls would still be limited. Only the data returned would be (a little) more... But, as it stands, there seems to be no other way to get screen_names (for friends and or followers) without doing 1 API call for each person (ID) on the returndata, which amounts to alot of API calls. On Sep 5, 5:09 pm, owkaye owk...@gmail.com wrote: The ideal solution is for Twitter to change the system and allow each account to have only one screen name, all the time, forever, with no changes. Then a separate id value is not required because all account identification will be done by the original screen name. REST and SEARCH would finally be consistent. No extra calls to figure out who the user really is. Users would complain until they got used to the fact that they cannot change their screen names on a whim anymore, but they will learn to deal with it soon enough. Email doesn't just let you change your address whenever you feel like it, and I see no reason why Twitter should allow screen name changes either ... except that it takes more work to standardize the system in this way than to continue with what already exists. But with only the screen name as each unique account identifier things would certainly be much simpler. Many fewer requests to the server. Less data storage. And being that Twitter is supposed to be simple this seems like a goal worth pursuing, at least from my point of view. Owkaye When i request friends (or followers) from the Twitter API i want to get the screen_name's based on the id's. I use users/show for this, inputting the id and getting back de screen_name. This costs ALOT of API calls and i run into the API rate limit fast, especially with many friends. Is there a better way of getting screen_names for friends / followers? ( Better, meaning in fewer API calls.)
[twitter-dev] Re: Search API limits
The rpp defaults to 15 or something if you don't specify it. Sounds like you need to mess around and play with things a bit more. The key to max search results isn't in paging or rpp, but in max_id. Be careful what you ask for. Retrieval of everything available can take a long time (hours) dave fisher http://webecologyproject.org On Sep 4, 10:16 am, owkaye owk...@gmail.com wrote: How can I retrieve the maximum number of tweets in a search? Can rpp be set to more than 100? What if I do not send a rpp value, does twitter default to returning more than 100 per page? Owkaye
[twitter-dev] Re: appended to API works in progress
lul. That's like asking the Duke Nukem Forever team when they'll release. When its done While I love the twitter dev team, I'm not sure how hard they are able to stick to deadlines in their development, so its probably just better to say, the future On Aug 24, 10:15 am, METROmilwaukee ... metromilwau...@gmail.com wrote: It’s great to see new API items are getting marked up. I would find it very helpful if [COMING SOON] was replaced by [MM-DD-] when the new item is officially released presuming why to do so is obvious marking the milestone.
[twitter-dev] Re: legal issues - is tweet an official verb in the US language?
Unless someone here is a lawyer, we should probably avoid legal debate- consult with each our own counsels, and move on to doing what we do best (coding). I find these debates are often filled with FUD, misinformation, speculation, a misunderstanding of law, etc The easiest way to get around it is to not use Twitter based words in your company/product name. Otherwise, just do what Biz said and use Tweet. It seems he gave the thumbs up on Tweet. dave fisher On Aug 18, 5:27 pm, Bill Kocik bko...@gmail.com wrote: On Aug 17, 8:06 am, Nicole Simon nee...@gmail.com wrote: Question: is to tweet an official word in the english language both american and english? as in widely used? does the US and UK trademark system reject such applications? Microsoft has a registered trademark on Windows. Apple Computer has a registered trademark on Apple.
[twitter-dev] Re: Do My Customers Have a Twitter Account?
Sounds like something you should be able to do in an email to them or with a message on your website. On Aug 19, 1:29 pm, arawajy araw...@gmail.com wrote: I want to invite them to follow the company on Twitter. On Aug 19, 8:25 pm, Andrew Badera and...@badera.us wrote: On Wed, Aug 19, 2009 at 11:07 AM, arawajyaraw...@gmail.com wrote: Dear Developers, I have a list of 400,000 e-mail addresses of my clients. I want to know Is it possible to develop a script to check if they have a twitter account or not?. I will then want to generate 2 separate lists based upon the result; one for the twitter users and one for the non-twitter users. I want to only invite the users and create a custom invitation message. Is it possible to check if the e-mail address's owner is a twitter user or not? provide details please. Thanks and Regards, Mahmoud If they're already your clients then what are you inviting them to? ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)
[twitter-dev] Re: Twitter's Tweet Trademark Torpedoed
This is good news. No one will be getting sued over Tweet then. Yet, keep in mind that Twitter probably *could * shut off access to the API to any company they choose, as its their playground and their rules. Not that they'll start doing that at all however. On Aug 19, 1:57 pm, Sam Johnston s...@samj.net wrote: [refer to the article itself for the inline links - @samj] Twitter's Tweet Trademark Torpedoedhttp://samj.net/2009/08/twitters-tweet-trademark-torpedoed.html Last month Twitter founder Biz Stone announced in a blog post (May The Tweets Be With You) that they have applied to trademark Tweet because it is clearly attached to Twitter from a brand perspective. This understandably caused widespread upset as the word tweet has been used generically by users for some time as well as in any number of product names by independent software vendors. Here's some samples from the resulting media storm: * CNET News: Is Twitter freaking out over 'tweet' trademark? * TechExpert: Twitter Trying to Trademark Tweet * LA Times: Will Twitter trademark 'tweet' before it’s genericized? * PC Magazine: Twitter Trying to Trademark 'Tweet' * TechCrunch: Twitter Grows “Uncomfortable” With The Use Of The Word Tweet In Applications * TechCrunch: Twitter To Developers: “Tweet” Your Heart Out, But Don’t “Twitter” It * Bloomberg: Twitter Lays Claim to ‘Tweet’ Trademark in Bid to Protect Brand What they failed to mention though was that according to USPTO records (#77715815) not only had they actually applied some months before (on 16 April 2009) but that their application had been refused that very same day (1 July 2009). According to documents from the Trademark Document Retrieval system, their lawyers (Fenwick West LLP) were notified of the rejection by email to tradema...@fenwick.com that day. The USPTO had explained that marks in prior-filed pending applications may present a bar to registration of applicant’s mark. [...] If the marks in the referenced applications register, applicant’s mark may be refused registration under Trademark Act Section 2(d) because of a likelihood of confusion between the two marks, referencing and attaching not one, not two but three separate trademark applications: * #77695071 for TWEETMARKS (pending receipt of Statement of Use) * #77697186 for COTWEET (pending clarification) * #77701645 for TWEETPHOTO (pending transfer to Supplemental Register) Now I may not be a lawyer (I did play a role in overturning Dell's cloud computing and Psion's Netbook trademarks) but given all three of the marks identified look like proceeding to registration (it only takes one to rain on their parade), it's my non-expert opinion that Twitter has a snowflake's chance in hell of securing a monopoly over the word Tweet. That's too bad for Twitter but it's great news for the rest of the community as it's one less tool for locking in Twitter's rapidly growing microblogging monopoly. People do use the word tweet generically (including with non-Twitter services) and if Twitter, Inc. were successful in removing it from the public lexicon then we could all suffer in the long run. In any case it is neither serious nor safe for one company to become the pulse of the planet and that is why I will be following up with a series of posts as to how distributed social networking can be made a reality through open standards (if that stuff is of interest to you then subscribe and/or follow me for updates). I've also got some interesting things in the pipeline in relation to standards and trademarks in general so watch this space. Anyway it just goes to show that with trademarks you need to use it or lose it. The propagation delay of the media has dropped from months at the outset to near real-time today so companies need to move fast to protect their marks or lose them forever. As for whether the 1 July post was a scramble to protect the mark on receipt of the USPTO's denial, whether the USPTO was acting in response to it, or whether it was just a coincidence and particularly bad timing I don't know. I don't really care either as the result is the same, but I would like to believe that the USPTO is becoming more responsive to the needs of the community (after all, they revoked Dell's cloud computing trademark in the days following the uproar, despite having already issued a Notice of Allowance offering it to them).
[twitter-dev] Re: Cease Desist from Twitter
I am wondering if this is a case of their legal department getting a bit heavyhanded and running loose. What they asked of you seemed fairly reasonable however, and the name of your application doesn't seem to be the issue. I'm glad you didn't think you were being sued :) It seems that Twitter should develop better UI guidelines for applications using their API, since it seems that there are many potential trademark concerns that their legal department is now bringing up. Happens with every big company at some point I guess- and Twitter is now getting 'big' and probably is just around the corner from hiring their first full time attorney or something at this rate. Hope it resolves well. Sounds like you're handling it well. -dave
[twitter-dev] Re: FW: Twitter is Suing me!!!
The OP should have first gotten a lawyer to look over it, instead of freaking out hysterically here. He's not being sued. Twitter does own the name Twitter and can selectively choose to sue/CD (or not sue) anyone they like who infringes on it. You defense being other people have registered the domain names too! isn't any better one than you'll get with saying, But other people use Bittorrent too! Someone up in the thread said, No one can now use the Twitter API because it says you can't use Twitter's name in your application, but they want you to put Powered by Twitter on the site. That's just FUD. Come on guys, put on your big boy pants and lets show a bit of business acumen and maturity here. Twitter isn't draconian, shutting down the developer community, etc. That's just silly.
[twitter-dev] Re: Trademark infringement
My guess is that Twitter will hold both Tweet and Twitter as IP, but allow general use of Tweet. It seems to really make sense to defend the name of their company always, and Tweet only when it is misused. -dave On Aug 12, 11:43 am, Dewald Pretorius dpr...@gmail.com wrote: Nick, TweetLater.com has been using tweet as a verb since April 2008. Dewald On Aug 12, 12:21 pm, Nick Arnett nick.arn...@gmail.com wrote: In case anybody wants some decent facts on this issue, Wikipedia has a pretty good article. http://en.wikipedia.org/wiki/Trademark_infringement I'm not a lawyer, but I published a book on IP for developers a number of years ago and learned a lot in the process. In the end, it is extraordinarily unlikely that anybody could possibly fail to infringe when using Twitter and/or their logo in a product or service name that is based on Twitter. That inevitably could cause confusion about whether or not it is from the company Twitter or not. Infringement has nothing to do with whether or not the infringement is connected with a good or bad use, such as spamming, etc. It will be interesting to see what they do with tweet. Many attorneys advise that if you want to preserve rights in a mark, you have to use it only as a proper adjective. Are they going to grant everybody permission to use tweet but only that way? I'll send you a Tweet(tm) message? Seems unlikely. Tweet has become a noun and a verb, which I suspect means it is fated to become genericized, if it hasn't already. Nick
[twitter-dev] Re: FW: Twitter is Suing me!!!
Let's not get into a semantic argument of spam. Spam, noise, trash followers, whatever you want to call them. They are all annoying and not what Twitter or most users want in their service. I'm unsure if a company HAS to pursue every trademark infringement to hold their trademark. Otherwise you could in theory make a company go bankrupt quickly just by setting up a bunch of shill companies using similar names and force them to pursue each fully in court. Also there is no obligation for Twitter to constantly search, identify and sue/ CD/defend against every company worldwide that uses the name Twitter. Just seems silly. IANAL however and could be wrong. dave
[twitter-dev] Re: Following a large number of users
Streaming methods are your friend. Start with the follow method, and email the API team about getting Shadow. If your use is legit (likely) then they'll give you Shadow. If/when you outstrip Shadow they will likely upgrade you then. The only caveat to streaming is that if you miss some time (downtime, etc) then you'll miss updates. You can also use the REST api to grab updates from non-protected users. Get your account whitelisted and you'll be able to pull 20K queries per hour. Not a bad way to catch up if you miss some of the streaming. Sounds definitely possible. dave webecologyproject.org On Aug 12, 10:10 am, murphy murphyb...@gmail.com wrote: We are building a suite of applications for Facebook Pages and one of the features is integration of user's twitter status updates into their facebook page feed. Thus the need to get a lot of users' twitter statuses at reasonable intervals. Looking at the API documentation there seems to be 3 ways to do that: 1. Use the streaming API. The 'follow' method looks good but allows only 200 users to be followed. How hard is it to get approved for the 'shadow' method? 2. Have a bot follow the users' accounts and get their updates with 'statuses/friends_timeline'. That allows up to 200 statuses to be retrieved every 30 seconds (to stay within the 150 calls per hour limit). This will probably do until we get a number of users ( I'd say more than 20 000). After that there will be updates cut off because of the 200 statuses per call limit. 3. Use the search API. e.g.http://search.twitter.com/search.json?q=from:user1OR from:user2... Now the query can be only 140 characters long and with the rate limit for search API calls, this approach will probably not work well (if at all). Any suggestions and comments how to approach this are appreciated. Slav Ivanov
[twitter-dev] Re: FW: Twitter is Suing me!!!
@Vincent No. Do you not understand that Trademark infringements occur between things that could be mistaken for each other or in the same industry, diluting a brand? A Disney film from the 1940's has what to do with a 3rd part application for a 2006-present social network? There is a CLEAR connection between MyTwitterButler and Twitter. MyTwitterButler is using Twitter as the basis for their business and clearly in the same brand space. This isn't uncomfortable for Twitter or MyTwitterButler. Were you asleep for the whole Apple Computers v Apple Music thing? As long as they weren't in similar spaces (computers weren't supposed to make sounds/music initially) everything was ok, but then once Apple got into music, the stuff got nuts and went to court. No Apple (fruit) companies were sued however or sent notices. You guys are so stupid this makes my head hurt. david On Aug 12, 3:18 pm, Vincent Wright mylinkedinpowerfo...@gmail.com wrote: None of us actually know how this might turn out but, even recognizing that this could become an uncomfortable matter for MyTwitterButler and/or Twitter, I nonetheless, decided to ask the question regarding the 1942 movie Bambi: Would Twitter Sue Bambi For Being Twitterpated? :-) :http://mylinkingpowerforum.ning.com/group/twitterpated/forum/topics/w... Thanks, and Keep STRONG!! Vincent Wright Director Of Community MyLinkingPowerForum.ning.com |
[twitter-dev] Re: 404 for statuses/user_timeline of non-existing users?
Yep. In my experience calling a user that doesn't exist does yield a
404. You'll have to handle the errors on this.
I guess 404 does make sense here, as the user is not found.
dave
On Aug 11, 5:40 am, Carlo Zottmann czottm...@gmail.com wrote:
Hi there,
I'm doing statuses/user_timeline requests via JS, and sometimes due to
faulty user input, I query statuses/user_timeline for nonexisting
users. The statuses/user_timeline API endpoint returns correct data
(for example, `callbackFunction({request:\/statuses\/user_timeline\/
nonexistingexample.json?count=200callback=
callbackFunction,error:Not found})`), but unfortunately it sends
a 404 status code as well.
Neither Firefox nor Safari consider these documents as valid and
refuse to execute the code within.
Is this the correct behaviour or an API bug?
Thanks,
Carlohttp://TwerpScan.com
[twitter-dev] Re: Following Churn: Specific guidance needed
IDOLPeeps, I feel you're being overly alarmist and haven't painted the situation properly. You can unfollow anyone you want. The issue is a quick follow and then unfollowing if not reciprocated. You're *supposed* to follow someone because you want to hear what they are saying, not because you want them to hear what you say. People were churning in attempts to gather mass numbers of followers. A lot of the MLM types did a good job at this and soon had 20, 40, or even 60K followers. Now they are clamping down. Comparing the situation/rules to communist regimes or dictatorships is a straw man argument and uncalled for. Surely given another post or two and this would be a comparison to Hitler, fulfilling Godwin's law of the internetz. I'm speculating, but I have to wonder more about your friends that were banned. Did they do exactly as Scoble (were they active members for over two years and offering massive community contribution?)? Were they following people prior to unfollowing? I know Scoble isn't going on massive following kicks. Scoble's intentions were also made very clear. Did your friends make their intentions clear? Had they been following and unfollowing people a lot? The other day I unfollowed about 30% of the people I was following. No ban. They aren't playing favorites. They are looking at multiple factors. If you're acting like a spammer, then you'll be treated like one. Churning and spam following is being cracked down on. Instead, grow slowly, naturally and contribute to the community. -David http://WebEcologyProject.org On Aug 11, 8:34 am, Dewald Pretorius dpr...@gmail.com wrote: I follow a simple principle in TweetLater. Where Twitter rules are clearly spelled out, such as for spam, I give the users a hammer and caution them, Carefully read the rules because you can drive a nail into wood, but you can also smash your thumb to a pulp with this thing. Where Twitter rules are nebulous, such as for unfollow, I do not give them a tool, because they wouldn't know (and neither would I) how to use it properly and safely. Dewald On Aug 11, 8:42 am, Dewald Pretorius dpr...@gmail.com wrote: On Aug 11, 3:11 am, TFT Media tftmedia1...@gmail.com wrote: For its auto-follow, tweetlater.com specifically states: [w]e have limits in place to ensure that your daily following remains well within the limits imposed by Twitter. So you are presumably touching the rate limit then going back -1, -2, -3, or whatever. You pulled that snippet from my feature that finds potential new friends based on user-selected keywords. That snippet simply means that the user can pause that feature when her account reaches the point where she would not be able to follow more people anyway. And no, my system absolutely does NOT then unfollow to make room for more. Here is what I very publicly said about bulk unfollow: http://bit.ly/JM3as Dewald
[twitter-dev] Search API Limits lowered?
While i haven't done scientific testing of this, I was able to run up to 3-4 instances of my search script prior at a time before it told me to enhance my calm. Now I'm barely able to run one without hitting the limit. I can put delays in my code to slow it down, but I'm wondering if this is just a symptom of the aftermath of the DDoS attack or something else? My server has a dedicated IP and no one else runs code from it, so it isn't other people on my IP hitting the Search API. Maybe I need to talk about Search API whitelisting... dave http://webecologyproject.org
[twitter-dev] Re: Search API Limits lowered?
The user agent for each search request is the same. I'm using the Ruby Twitter API wrapper, so sending anything else with search requests isn't possible unless that is now deprecated. dave On Aug 11, 10:36 am, Andrew Badera and...@badera.us wrote: On Tue, Aug 11, 2009 at 10:30 AM, David Fishertib...@gmail.com wrote: While i haven't done scientific testing of this, I was able to run up to 3-4 instances of my search script prior at a time before it told me to enhance my calm. Now I'm barely able to run one without hitting the limit. I can put delays in my code to slow it down, but I'm wondering if this is just a symptom of the aftermath of the DDoS attack or something else? My server has a dedicated IP and no one else runs code from it, so it isn't other people on my IP hitting the Search API. Maybe I need to talk about Search API whitelisting... dave http://webecologyproject.org Are you sending a unique client ID header with each request, per previous Search API throttling conversations? (Not sure that it matters, that seemed pretty fuzzy when discussed ...) ∞ Andy Badera ∞ This email is: [ ] bloggable [x] ask first [ ] private ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)
[twitter-dev] Re: Following Churn: Specific guidance needed
Follower churn wouldn't exist, but getting hundreds of spam emails (about being followed) would still exist. I've got over 12,000 emails in my inbox about being followed on Twitter. Dozens of those are from the same users. Some weeks the same users unfollow and refollow me nonstop to try to get me to fall for it. I don't. I've got only ~2000 followers because most of them were just spammers churning through trying to follow me. Its annoying. Yes, many users fall for it. Many big users auto-follow back because they want the numbers (don't tell me everyone that Scoble followed back initially was a real user). I personally don't think you should be able to follow more than 25 people per day, but that's just me. -david On Aug 11, 1:55 pm, Kevin Mesiab ke...@mesiablabs.com wrote: This entire debate focuses on the wrong side of the coin. Follow churn exists as a side effect of the improper Twitter culture of reciprocating follows blindly. If users paid due diligence to those they follow and only followed those people who demonstrate some value to them, follower churn would not exist. Period. On Tue, Aug 11, 2009 at 7:51 AM, owkaye owk...@gmail.com wrote: Would be very helpful to know the definition of quick as relates to following churn suspensions. As Cameron pointed out earlier, as soon as they do that, the following churners will adjust their methods to be just inside that definition of OK. This seems like a really short-sighted reason for NOT clarifying what's acceptable and what's not. If it's acceptable then who cares if the churners adjust their methods? At least everyone will know how to avoid problems for a change, right? -- Kevin Mesiab CEO, Mesiab Labs L.L.C.http://twitter.com/kmesiabhttp://mesiablabs.comhttp://retweet.com
[twitter-dev] Re: Search API Limits lowered?
I don't have a domain to point back to. I'm doing data-mining and analysis on a server that isn't public. I have set the User-Agent to something unique (I thought you were saying to change it for every request?). Yet I'm still getting rate limited and told to back off a lot. Ryan S said it might still have to do something with the DDoS attack, which is still going on apparently. dave On Aug 11, 4:13 pm, Chad Etzel c...@twitter.com wrote: The referrer is not as important as the user-agent. You can also put your URL in the user-agent instead. -Chad On Tue, Aug 11, 2009 at 4:09 PM, Larry Wrightlarrywri...@gmail.com wrote: In addition to setting a unique user-agent, I believe it was requested that we set a referrer header that pointed back to a domain. On Tue, Aug 11, 2009 at 9:30 AM, David Fisher tib...@gmail.com wrote: While i haven't done scientific testing of this, I was able to run up to 3-4 instances of my search script prior at a time before it told me to enhance my calm. Now I'm barely able to run one without hitting the limit. I can put delays in my code to slow it down, but I'm wondering if this is just a symptom of the aftermath of the DDoS attack or something else? My server has a dedicated IP and no one else runs code from it, so it isn't other people on my IP hitting the Search API. Maybe I need to talk about Search API whitelisting... dave http://webecologyproject.org -- Larry Wright
[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED
They haven't overreacted. If you think you can do better, then apply for a job with them. Have you tried your account from multiple IP blocks? They have faced a crushing attack. They are working on it and they have worked hard to communicate with us the status of things. Anyone that was around in 2006/2007 knows that Twitter was excessively unstable then, and people didn't whine/complain as much. They've gotten better but they don't have the capacity to serve traffic and handle a DDoS attack at the same time. Your ticket will be dealt with and they are working to resolve overall issues before they address specific people. dave fisher/@tibbon webecologyproject.org On Aug 9, 10:30 am, chuckblakeman cfblake...@gmail.com wrote: For what it's worth. I am a Twitter use who does not use TweetDeck, TweetLater, or any other 3rd party software doing API calls to Twitter. I use Twitter directly without any connection to other API calls. Since Wednesday I have not been able to use any Twitter functions and as of Friday evening I was no longer even able to sign in and watch the Tweet stream helplessly. Login function has been disabled since then. So they are not just blocking majore APIs - for some reason my direct Twitter account has been blocked by Twitter as well. I've sent two tickets and one email to support over the last three days with no response. They have clearly over-reacted to this issue.
[twitter-dev] Re: The silence is deafening....
I'm disgusted by you guys. Were any of you around in 2007 when Twitter was down nonstop? Did you bitch and moan as much then? You're acting like children. The reason they are still facing issues is because of the DDoS. Oh, and they are still also coping with scaling overall and astronomical growth. A DDoS that is shifting and trying to thwart them isn't easy to stand up to- moreso if you were just able to handle the load of your standard business traffic anyway. Its very clear that many of you have NEVER worked in a startup or technology environment. Its no like everyone from Biz down to the receptionist can focus all their magical energy on Twitter at the same time to make it work. I've worked for a startup before when our service/site has gone down. There were one, maybe two people in the company who had the ability, authorization or knowledge to do it. The rest of us were as frustrated as hell, but even if one of us was the cause (a bad query to the database or something) that didn't mean that we could all instantly fix it. Companies have different roles for people and it doesn't help at all for EVERYONE to SSH to the same box and try to muck with the settings to fix things. All hands are on deck that need to be. Everyone else at Twitter can't do anything about it, but is stressing out too. Chad not working today isn't a sign of them not working on the stuff. He just joined the team, and isn't in the engineering/operations department. Him flying out to California to stand and stare at the others working on the issue won't help. Different people have different roles and his role isn't to fix this stuff. If something happens at your office, does everyone from the CEO to the receptionist jump on the issue all at once? No, they can't. It doesn't make sense. Stop bitching and complaining. You're making yourselves look unprofessional and bad. -David Fisher Web Ecology Project http://webecologyproject.org On Aug 9, 2:51 am, chinaski007 chinaski...@gmail.com wrote: And, by the way, if you're a deckhand on a submarine going down, you think you would go to a movie because it's your time off, or do whatever you can to help out? On Aug 8, 11:47 pm, chinaski007 chinaski...@gmail.com wrote: Part of it is DDOS response, part of it is API issues... as one of the updates most recently noted with the known issues list. On Aug 8, 11:42 pm, Bill Kocik bko...@gmail.com wrote: On Aug 9, 2:28 am, chinaski007 chinaski...@gmail.com wrote: You're wrong. If you check the tweets of the other main Twitter developers, you will see that they are doing sushi, rock concerts, weddings, watching movies on Saturday afternoon, etc. And while getting married is certainly a legitimate excuse, some of the other activities, during this major upheaval, are certainly questionable! Oh...I get it, you think that the developers are the right people to handle a DDoS attack. I'm afraid I have some bad news for you: you're wrong. And because not *all* the developers are sweating it out at Twitter HQ, that means no one's working on it. I can't argue with that logic. Seriously, I can't.
[twitter-dev] Re: OK Seriously People
A few of you are acting like real children and a few of you still have your heads screwed on right. I'm confident they are doing everything they can. Chill and enjoy your weekend. They'll get it sorted out. What did you guys do in 2007? Twitter was down all the time then. Your blood pressure must have been through the roof with weekly visits to a shrink if you responded this way every time it went down. dave On Aug 9, 1:48 pm, Neil Ellis neilellis1...@googlemail.com wrote: Nice story Adam, however the band are actually trying to run a business, not doing this for love/free. I can assure you the investors in Twitter will be looking to turn profit. Of course if the band are laid up then the danger is the hotdog man (and all his customers) will go to another band that are still playing and have fans. That's why I'm 100% confident all that can be done is being done, cos plenty of people at Twitter will know how fickle a user base can be. Good luck guys, I know what these situations are like and it's hard on you all - I actually hope you guys are getting some rest because it doesn't sound like this is a 100 yard sprint. I also hope someone is making sure the ops/devs aren't reading this list (or getting emails etc) - stress doesn't help productivity in my experience. Knowing what is at stake does. Again good luck chaps, I know how the trenches feel :-) And of course it does suck for the rest of us too, alas that is business. ATB Neil On 9 Aug 2009, at 18:24, Adam Cloud wrote: ***Scenario*** A band broadcasts their music on a radio station all the time, and people are able to freely tune into it, or go buy their music. They go and play in a city park for free every day just because it's a much nicer experience for the listener then to be just sitting at home listening on their radio. You as an up and coming entrepreneur go buy a hotdog drink stand and setup camp in that park to make some cash off of the flow of people who come to see this free event every day. You being there, giving the ability for people to eat drink without leaving the park allows for more of this bands songs to be heard, in effect increasing the chance that their music might be purchased. So you're essentially helping them, by taking advantage of them for your business. The band gets in a car crash, and alot of equipment is damaged to the point of not being able to be used, along with their main source of transportation. The band starts working to find and replace all that is damaged in their equipment and for their car. Now you can imagine that little hotdog stand guy standing on their doorstep while they recover yelling profanities and how they should be skipping the shipping company who's delivering their parts and get their parts themselves to save time. Yelling that they shouldn't be sleeping, they should be working on their band van right now to make sure it can take them back to the park so he can make some money. People aren't coming to my stand anymore!!! They're going to fast food restaurants and going home. WTF i sold my wife for this stand!!! Now of course, this little hotdog stand man may not have really sold his wife, depending upon which one of you people who are still up in arms about this was put in his place, but i think you get my points. The band could easily move to a venue that has their own hotdog/ drink stand making your services not necessary, but instead of doing that and capitalizing on the profit they could get from that, they're still planning on going back to the same park they do their free shows at, and allowing you to continue earning your money. And this concludes storytime. :) Happy sunday! (Relax!) On Sun, Aug 9, 2009 at 10:58 AM, Terry Jones te...@jon.es wrote: Stuart == Stuart stut...@gmail.com writes: Stuart * I can't believe you lot don't realise that constantly demanding Stuart status updates, while certainly important to you, is little more Stuart than a distraction for those who are actually fighting the good Stuart fight. I woke up this morning with the thought that the Twitter mailing list has now become part of the DDoS. What percentage of the people complaining loudly and increasing the general stress/pressure level are actually bots? :-) Terry
[twitter-dev] Re: Can Twitter team Respond?
It will be fixed when it is. Stop complaining. We're all in the same boat. The ETA is when its done. dave On Aug 9, 1:08 pm, freefall tehgame...@googlemail.com wrote: Decentralisation. On Aug 9, 5:47 pm, Dewald Pretorius dpr...@gmail.com wrote: I wonder how many times this weekend has Chad heard, FO, we're busy, when he tried to get a status update for us. On Aug 9, 1:42 pm, Jesse Stay jesses...@gmail.com wrote: Good luck. On Sun, Aug 9, 2009 at 6:01 AM, Jacob yac...@gmail.com wrote: I have thousands of people coming to my application only to get Twitter Api is not responding! can Twitter team help me? I'm aware of the API difficulties, but, I need some answers or maybe a date when the problem will be fixed. Thanks in advance Jacob http://twitteranalyzer.com-Hide quoted text - - Show quoted text -
[twitter-dev] Re: OK Seriously People
It is like a surreal tech soap opera on this list though :-) Hey maybe I could pitch this to Fox . All the best Neil DO IT! Maybe Twitter would have made a good reality show. Lots of super-dramatic angles and queues pointing at sysadmins typing at a console. :) The mailing list would be like the random room they walk into to talk to the camera. dave
[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED
Yea, and we all threatened to go to Pownce. That didn't go so well (seeing that Pownce is now dead) dave On Aug 9, 3:05 pm, Dossy Shiobara do...@panoptic.com wrote: On 8/9/09 1:53 PM, David Fisher wrote: Anyone that was around in 2006/2007 knows that Twitter was excessively unstable then, and people didn't whine/complain as much. Sure we did, there were just fewer of us at the time. Weren't you around in 2006/2007? Perhaps you have a short memory retention. -- Dossy Shiobara | do...@panoptic.com |http://dossy.org/ Panoptic Computer Network |http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70)
[twitter-dev] Re: You have been rate limited. Enhance your calm.
If you're getting that then you're getting more than me.
I'm just doing:
require 'rubygems'
gem 'twitter'
require 'twitter'
Twitter::Search.new('foo').each do |r|
puts r.inspect
end
And I get only this back now:
/var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in
`setup_raw_request': undefined method `request_uri' for #URI::Generic:
0x7f99f9516ea0 (NoMethodError)
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
39:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
99:in `handle_response'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
40:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:156:in
`perform_request'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:122:in
`get'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
100:in `fetch'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
108:in `each'
from searchtest.rb:5
My search is totally borked. Its just the API though I think, give it
some time
On Aug 7, 11:20 am, diddy david.barrowcl...@gmail.com wrote:
Hi,
I use the Twitter search api, e.g: -
http://search.twitter.com/search.rss?q=iphone
and I now get: -
You have been rate limited. Enhance your calm.
I rely on this for my application. Anything I can do to stop it?
Thanks!
[twitter-dev] Re: You have been rate limited. Enhance your calm.
I can't be sure if my client is following redirects. Probably not. I'm just using the Ruby Twitter Gem which haven't been updated for a month or so I think dave On Aug 7, 1:15 pm, lucasnicolato eternitya...@gmail.com wrote: im having the same problem. im just lucky my app is still in test. RT @twitter Due to defense measures some Twitter clients are unable to communicate with our API, and many users are unable to tweet via SMS. I think we can only wait for twitter to normalize de api. On 7 ago, 12:20, diddy david.barrowcl...@gmail.com wrote: Hi, I use the Twitter search api, e.g: - http://search.twitter.com/search.rss?q=iphone and I now get: - You have been rate limited. Enhance your calm. I rely on this for my application. Anything I can do to stop it? Thanks!
[twitter-dev] Re: You have been rate limited. Enhance your calm.
Yea I'm using John's. Have to pull out the packet sniffer soon On Aug 7, 3:32 pm, Larry Wright larrywri...@gmail.com wrote: You can use Wireshark or any other packet sniffer to determine whether your client is following redirects. I'm not sure what ruby twitter client you're using, but if it's John Nunemaker's, I believe it does follow redirects. Larry Wright/@larrywright On Fri, Aug 7, 2009 at 2:03 PM, David Fisher tib...@gmail.com wrote: I can't be sure if my client is following redirects. Probably not. I'm just using the Ruby Twitter Gem which haven't been updated for a month or so I think dave On Aug 7, 1:15 pm, lucasnicolato eternitya...@gmail.com wrote: im having the same problem. im just lucky my app is still in test. RT @twitter Due to defense measures some Twitter clients are unable to communicate with our API, and many users are unable to tweet via SMS. I think we can only wait for twitter to normalize de api. On 7 ago, 12:20, diddy david.barrowcl...@gmail.com wrote: Hi, I use the Twitter search api, e.g: - http://search.twitter.com/search.rss?q=iphone and I now get: - You have been rate limited. Enhance your calm. I rely on this for my application. Anything I can do to stop it? Thanks! -- Larry Wright
[twitter-dev] Re: Current Twitter site status
Is the Search API being effected? I thought at first that I had messed up my code, but I rolled back pretty far and I'm still getting really odd errors /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in `setup_raw_request': undefined method `request_uri' for #URI::Generic: 0x7ff36a8295f0 (NoMethodError) from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb: 39:in `perform' from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb: 99:in `handle_response' from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb: 40:in `perform' from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:156:in `perform_request' from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:122:in `get' from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb: 100:in `fetch' from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb: 108:in `each' On Aug 6, 11:45 am, Stuart stut...@gmail.com wrote: 2009/8/6 Chad Etzel c...@twitter.com: Some of you may already be aware that the main Twitter site is under a DDoS attack. Please keep a close eye onhttp://status.twitter.com/ and this list for details and updates. Encountered seemingly neverending redirects - that can't be helping!! http://titsup.net/http://twitter.com/ -Stuart --http://stut.net/
[twitter-dev] Re: API only shows messages from last 7 days
I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave On Jul 26, 4:11 am, Flashing Moose flashingmo...@gmail.com wrote: Hmm, then i can't use the API for this project, thx for replies guys. On 26 jul, 04:10, Dewald Pretorius dpr...@gmail.com wrote: I believe the tweet retention in Twitter Search has always been 7 days. On Jul 25, 1:18 pm, Flashing Moose flashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose
[twitter-dev] Re: API only shows messages from last 7 days
I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data. I'd sign agreements and pay money for such. dave On Jul 28, 12:03 pm, owkaye owk...@gmail.com wrote: I agree with you Dave. I have several thought about new services based on searching Twitter's historical data. Unfortunately my ideas appear to be getting less and less practical. Twitter claims to have all its data stored in disk-based databases from what I understand ... yet without access to this data it is worthless. It seems to me they could allow searches of this historical data via a new History API then let us cache the results on our own servers. Most of the services I've conceived would do this infrequently -- never in real time -- and would not impact their existing cached server data because this historical data would exist on separate data storage servers ... theoretically anyways. Owkaye I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave
[twitter-dev] Re: API only shows messages from last 7 days
I don't think that adding more people to the staff at Twitter is the solution. In one startup I saw a thing posted on the refrigerator that had the adage, Adding more people to a project already behind schedule will only slow it down more. Surely for support and customer service issues having more people on the team to deal with growth is good, but I doubt throwing more programmers at it will help fix most issues. It just never seems to work that way. While many startups do tend toward younger employees (I personally think because being younger normally means that you can work a lot with minimal life impact), I'm sure that someone with a strong background would be able to get a job at Twitter if they were local to the company (or willing to move). A lot of this surely comes down to priorities inside the company. While Doug and Team want to support us developers as much as possible, much of our initial 'value' that we've offered in helping push twitter to the masses has already happened. We aren't the core business strategy, and with a fixed amount of resources and focus they aren't working to push mainly for developer access, but for standard user access. This 100% makes sense. Users are what is going to make twitter happen, not 3rd party developers. They want to provide a stable experience on both fronts, but users come first. In my private discussions with some team members, I've gotten the sense that they have good stuff in the pipeline for us and that they are working hard to make it happen. However we're only a small part of the overall strategy of a quickly growing company that is still dealing with massive growing pains which is no fault of theirs and something they are dealing with as best they can. david On Jul 28, 1:46 pm, owkaye owk...@gmail.com wrote: I'm sure others feel the same way Dave, but it looks and feels like Twitter is moving in the opposite direction. The load on a server to extract a big dataset once a month would be minimal, and both you and I can see the value in this approach. But I'm not sure the folks at Twitter do, or if they do maybe they just don't have the people who can (and will) get things like this implemented. Is a shortage of competent staff the cause of this type of problem? Even though I have the capabilities I do not have the 'resume' to get a job there and help them deal with some of this stuff, nor do I have the contacts within the Twitter organization to put a good word in for me and help me get hired so I could do good things for them. I'm 52 years old too, and my age seems to be a negative to most of the Web 2.x companies hiring these days. This is kind of a shame considering that people like me frequently have broader-based experience and insights that are sometimes lacking in younger people, and because of this we can add a lot more value in the areas of planning and structural development than people half our age. Our coding skills are honed after so many years of experience too, not to mention the thousands of code snippets we have collected over the years to contribute to making us even faster. But since jobs like this are basically not open to me and many other folks my age, my alternative is to remain self- employed and try to build something on top of their existing available source data and API's ... and then deal with the issues and frustrations created when building a service on top of a 'moving target' that sometimes seems to be moving in funny directions. I hear about Twitter having lots of money to work with, and I'm probably wrong here but it almost seems like there's too little of this money being dedicated to paying new talent with long term views of some of these issues, and who will implement wise policies to help support and encourage rapid growth in the areas that are lacking. But once again this might just be due to a shortage of the right staff. Obviously we cannot do anything from the outside except point out these issues and ask questions, or beg and plead for changes, but it sure would be great if a few of us could actually get in there as employees and implement a couple of the new features we really need -- such as a new Historical Search API for example. Then developers like you and I could proceed with some of our plans now, instead of months or years from now ... or possibly never. I would love to lead a team on a project like this, or even be one of its members, but until it happens I'll focus on building my own little space in the Twitter universe and continue to hope for the best. :) Owkaye I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data.
[twitter-dev] Re: New Twist To Follow Terms Violations
What gives Twitter the right to dictate who you want to follow or not? Its their service. They can dictate what they want. Their playground, their rules. The ToS clearly says they can alter their terms at any time and if you don't want to comply you can leave. That being said, this is to prevent people who are massively following and unfollowing (as I see happen dozens of times daily on my account). You aren't going to get banned for not following someone back, or for unfollowing someone that bothers you. dave On Jul 24, 2:15 pm, Joshua Perry j...@6bit.com wrote: Think about a bot who just bulk follows random people, it then would kept track of users who didn't blindly or automatically follow back and dump them quickly and try following another batch of users so that it wouldn't bust it's follow ratio limit. Using this strategy a bot could eventually build a very large following/followers list for someone while still keeping it's ratio within the boundaries set. I believe that the second part of that term is to protect against this scenario. Vision Jinx wrote: What? Re: as well as following and unfollowing those who don't follow back, are both violations of our terms of service. What gives Twitter the right to dictate who you want to follow or not? That is like Gmail saying you can't remove contacts from your contacts list. When I signed up it suggested a list of people to follow but I didn't find the tweets interesting so I un-followed them (they didn't follow me back, but that was not the reason I un-followed them). I should have the right to decide who I want to follow or not unless Twitter is under a communist regime? Is there also a term that if someone posts a link I have to click it also? I also followed iGoogle for a while but didn't find the tweets that interesting so I un-followed them, they never followed me back, so if Twitter wants to delete my account (for TOS violations) then fine go a head, do so right now then, but I feel it is my right to decide who I do and do not want to follow and that will not change. They need to post a message when you sign up that you are not allowed to un-follow people. Why is there even that option then? Regards, Vision Jinx @visionjinx (In case Twitter wants to delete my account for feeling I have the right to decide who I follow, fine then do it now.) I also, un- followed someone because they kept posting the same tweets over again so who's the bigger offender there then? On Jul 24, 10:22 am, Dewald Pretorius dpr...@gmail.com wrote: On Twitter's new site,http://business.twitter.com, under the heading Best Practices, the following is listed as a spamming practice: Following churn: Following and unfollowing the same people repeatedly, as well as following and unfollowing those who don't follow back, are both violations of our terms of service. Take note devs, the ...unfollowing those who don't follow back... statement is posing a risk for any of your apps that do bulk unfollow. On that point, I would like to get clear guidance from Twitter whether unfollowing someone who has stopped following you, i.e., unfollowed you first, would also constitute a violation of Twitter terms.
[twitter-dev] Re: Twitter is not making money
Show me these killer companies doing great NLP with social networks. I find the ones that are doing stuff right now themselves are far behind the curve and not really pushing stuff to the edge. They are often marketing companies that have hired one NLP guy (and underpaid them) and are just pushing the marketing side. I have yet to see anything truly revolutionary come from most of these monitoring companies yet and they are all too narrow focused. Plus, none of them have the VC funding to really expand and grow (and not many people are getting new funding these days) -David On Jul 18, 3:29 am, M. Edward (Ed) Borasky zzn...@gmail.com wrote: On Jul 16, 1:14 pm, Stuart stut...@gmail.com wrote: Twitter have a business plan, we're just not worthy enough to know all the details. What we know so far is that they're planning to launch a premium account type with a bunch of tools to aid brand and engagement tracking. I've got news for you ... Twitter itself is woefully behind the curve on monitoring / marketing / analytics technologies. Third parties are springing up daily with offerings in this area, many of them involving cutting-edge natural language processing. Twitter could obviously invest in these areas, but I'm not sure why they would, rather than focusing on stability, scalability and security of the underlying platform and messaging systems. Perhaps one way to monetize Twitter would be to implement a per- follower charge, say, free up to 2000 followers, then a small monthly fee up to 10,000, a larger fee up to 100,000 and so on. I haven't seen follower count distribution data recently, but I'd say that people with more than 2000 followers are rare and are probably using Twitter as a push marketing / sales platform in some sense. Of course, I'm at around 3500 followers at the moment, so I would be paying a monthly fee and would need to justify it as a business expense (or block about 1500 people, which isn't out of the question) ;-)
[twitter-dev] LOLCODE API Wrapper
I can't seem to find a LOLCODE (http://en.wikipedia.org/wiki/LOLCODE) API wrapper for Twitter. I was hoping to write my next Twitter application in LOLCODE, but its very hard to do so without one. Can anyone help me get started with a basic program? My code so far isn't working HAI CAN HAS STDIO? MAKE CONNECT TWITTER NAO I HAS A VAR TWEET IM IN YR LOOP UP VAR!!1 GET UR TWEETS VISIBLE TWEETS IM OUTTA YR LOOP KTHXBYE Thoughts? An API wrapper would make all this much easier. David
[twitter-dev] Re: Filter Profanity
Why don't you just do the filtering on your end? Twitter's API's job is to give you data- what you do with filtering it on your end should be up to you... On Jul 17, 12:43 pm, Steve Brunton sbrun...@gmail.com wrote: On Fri, Jul 17, 2009 at 12:14 PM, Cameron Kaiserspec...@floodgap.com wrote: (IDNSOWFT) The problem with this is that it puts Twitter in the position of having to a be a de facto content censor. Besides people having varying ideas of what constitutes offensive, it also possibly subjects them to legal consequences. As a funny example of the varying ideas topic. Here where I work (hi Doug) we had the Big List of Bad Terms from one of our other business units that was supposed to be the master list of words that we were to filter on. The one that always gave us humour in meetings was downblouse .. Yes downblouse was on the restricted list and supposed to be filtered out. YMMV as to what should or should not be filtered and should really be something that the consumer side deals with. Personally I'm attempting to increase my ranking on Cursebird, but that's just me. -steve
[twitter-dev] Re: Twitter is not making money
Don't feed the trolls. Name a Venture backed startup that DOESNT run on VC for the first few years. Ok? Right. Stupid topic. Lets move on and talk about things that matter (ie. development) On Jul 16, 4:34 pm, Andrew Badera and...@badera.us wrote: On Thu, Jul 16, 2009 at 4:26 PM, Nick Arnett nick.arn...@gmail.com wrote: On Thu, Jul 16, 2009 at 1:14 PM, Stuart stut...@gmail.com wrote: Twitter have a business plan, we're just not worthy enough to know all the details. What we know so far is that they're planning to launch a premium account type with a bunch of tools to aid brand and engagement tracking. If Twitter can maintain their popularity big business will pay a small fortune to be able to measure the effectiveness of the way they're using their accounts. At the risk of really deviating from developer talk... We know this? Who's we and how do we know this? I have a hard time seeing how analysis of Twitter alone would compete with existing services that monitor brands in conversations across many platforms. I started one of the first companies to do that, ten years ago, which is quite a head start... and it is now owned by one of the biggest brand monitoring companies on the planet. Lots of competition has come along since then. Anyway, this was fun, but it's not about developing code as such, so I'll shut up. Maybe this is a conversation for that non-platform-specific social media developer community I was wondering about... ;-) Nick We is anyone who have paid attention to the twitter dev list and blogs and scuttlebutt. Biz has talked about the premium accounts. Contextual advertising has been brought up several times. Search-driven monetization has also been brought up. Who's we? Anyone with their eyes and ears open. Thanks- - Andy Badera - and...@badera.us - Google me:http://www.google.com/search?q=andrew+badera - This email is: [ ] bloggable [x] ask first [ ] private
[twitter-dev] Re: How to insure that all tweets are retrieved in a search?
Err, the Search API isn't limited to 150 requests per hour. It's much higher than that. Much, but not unlimited. As John said, read into the Search API more, and check into the Streaming API as well. It is certainly possible to get more than 1,500 results for a term, but not by using simple paging. I've been able to pull 2M+ results for a query before. It took 8 hours or so, but it worked. Read up more on the Search API and you should be able to figure it out. -David Fisher http://WebecologyProject.org On Jul 9, 8:16 pm, John Kalucki jkalu...@gmail.com wrote: First, I wouldn't expect that thousands are going to post your promo code per minute. That doesn't seem realistic. Second, in addition to the Search API, which is quite liberal, you can use the /track method on the Streaming API, which will return all keyword matches up to a certain limit with no other rate limiting. Contact us if the default limits are an issue. -John Kalucki Services, Twitter Inc. On Jul 9, 3:51 pm, owkaye owk...@gmail.com wrote: You are correct, you have to do 15 requests. However, you can cache the results in your end, so when you come back, you are only getting the new stuff. Thanks Scott. I'm storing the results in a database on my server but that doesn't stop the search from retrieving the same results repetitively, because the search string/terms are still the same. My problem is going to occur when thousands of people start tweeting my promo codes every minute and I'm not able to retrieve all those tweets because of the search API limitations. If I'm limited to retrieving 1500 tweets every 6 minutes and people post 1000 tweets every minute I need some way of retrieving the missing 4500 tweets -- but apparently Twitter doesn't offer anything even remotely close to this capability -- so I can see where it has a long way to go before it's ready to support the kind of search capabilities I need. Twitter has pretty good date handling, so you specify your last date, and pull forward from there. You may even be able to get the last id of the last tweet you pulled, and just tell it to get you all the new ones. Yep, that's what I'm doing ... pulling from the records I haven't already retrieved based on the since_id value. But when the new tweets total more than 1500 in a short time, the excess tweets will get lost and there's no way to retrieve them -- unless I run my searches from multiple servers to avoid Twitter's ip address limits -- and doing this would be a real kludge that I'm not tempted to bother with. I'm building an app that uses the atom search API to retrieve recent posts which contain a specific keyword. The API docs say: Clients may request up to 1,500 statuses via the page and rpp parameters for the search method. But this 1500 hits per search cannot be done in a single request because of the rpp limit. Instead I have to perform 15 sequential requests in order to get only 100 items returned on each page ... for a total of 1500 items. This is certainly a good way to increase the server load, since 15 connections at 100 results each takes far more server resources than 1 connection returring all 1500 results. Therefore I'm wondering if I'm misunderstanding something here, or if this is really the only way I can get the maximum of 1500 items via atom search? -- Scott * If you contact me off list replace talklists@ with scott@ *
[twitter-dev] Re: Twitter User List
Thanks for the replies Doug.
btw, as far as I can tell the ~ 52M integer values for user ids aren't
fully sequential, with some amount missing in between, which I'm
assuming is intentionally done to make it slightly more difficult to
know the exact user growth trends/patterns (seems valid to me) or for
some other system reason. There's definitely some gaps in there FYI.
dave
On Jun 29, 2:04 pm, Doug Williams d...@twitter.com wrote:
Please do not scrape our site. We have processes in place that will
automatically block your spiders.
If you feel that you have a compelling need for vast amounts of data please
email the API team [1] with a detailed description of your needs and the
value you hope to create and let's have a conversation.
1.http://apiwiki.twitter.com/Support
Thanks,
Doug
On Mon, Jun 29, 2009 at 10:30 AM, Scott Haneda talkli...@newgeo.com wrote:
I dint think this is a function of a workaround. This is a function of
Twitter having a good policy in place to prevent abuse.
You can do what you want by incrementally querrying the API. The API limits
will make it take too long. Even with multiple accounts it will be months
before you get a final list. Even then, I'm not sure you could keep on top
of new user registrations.
Having acces to this data could only be used for nefarious efforts. What
you want would be a spammers dream.
I think you would be better and faster to build a crawl farm and crawl all
links on Twitter.com and parse the users out, bypassing the API.
Even with the API, as you add new records, those records you just added
will expire, delete, get banned, blocked etc. There is no way you could ever
have a reconciled system.
Consider if each username is an average 10 bytes. You have 520,000,000
bytes to download of just username data. Let's double that for http overhead
and other misc data that will come over the wire. 1 billion bytes.
That's a strongly conservative terrabyte of data that you would have to
download once a day and reconcile against the previous day. A terrabyte of
just usernames.
Then you have all the CPU that you will need, network lag, time to insert
into your data source.
This is not something that can be worked around. This is simply a
limitation of scale, one that can not be overcome. You need a direct link to
twitters data sources, ideally from within their data center to reduce
network lag. This probably will not be approved :)
--
Scott
Iphone says hello.
On Jun 29, 2009, at 9:06 AM, Arunachalam arunachala...@gmail.com wrote:
Even if i have my account whitelisted, which have 20,000 request per hour,
i need to run for many days which is not feasible. Any other workaround.
Any other way to get rid of these request limit.
Cheers,
Arunachalam
On Mon, Jun 29, 2009 at 7:01 PM, Abraham Williams 4bra...@gmail.com
4bra...@gmail.com wrote:
There has been over 5200 profiles created. You could just start at
1 and count up. Might take you a while though.
Abraham
On Mon, Jun 29, 2009 at 07:55, Arunachalam arunachala...@gmail.com
arunachala...@gmail.com wrote:
Any idea how to implement the same using php / any other language.
Im confused abt the implementation.
Cheers,
Arunachalam
On Mon, Jun 29, 2009 at 5:57 PM, Cameron Kaiser spec...@floodgap.com
spec...@floodgap.com
wrote:
I am looking to find the entire twitter user list ids.
Social graph method provides the way to fetch the friends and
followers
id,
thorough which we can access the profile of the person using user
method
-
show. But this requires a code to be written to recursively crawl the
list
from any starting id and appending the followers and friends id of
the
person without duplicating.
Do we have any other API to get entire list. If not, any other ways
apart
from crawling to get the entire list.
No, and no, there are no other ways.
--
personal:
http://www.cameronkaiser.com/http://www.cameronkaiser.com/--
Cameron Kaiser * Floodgap Systems * http://www.floodgap.com
www.floodgap.com*
ckai...@floodgap.comckai...@floodgap.com
-- Careful with that Axe, Eugene. -- Pink Floyd
---
--
Abraham Williams | Community Evangelist | http://web608.org
http://web608.org
Hacker | http://abrah.amhttp://abrah.am| http://twitter.com/abraham
http://twitter.com/abraham
Project | http://fireeagle.labs.poseurtech.com
http://fireeagle.labs.poseurtech.com
This email is: [ ] blogable [x] ask first [ ] private.
[twitter-dev] Re: airline accident case study
Topics don't just trend because its something 'important'. Now if it was of significantly larger volume than another topics (like the iphone's launch today), then that is rather interesting, but from what I can tell its mostly the most popular things floating to the top generally, plus some spam-filtering. I haven't figured out the exacting mechanisim for when something hits trending, but its not rocket science either. Maybe I'm missing your point -...@tibbon On Jun 19, 2:35 am, Bjoern bjoer...@googlemail.com wrote: On Jun 19, 7:00 am, AJ cano...@gmail.com wrote: This case study shows the difference between various trending applications. A good real time semantic analysis is the key that makes the difference, I think. Maybe I misunderstood, but isn't the more likely explanation that the topic simply wasn't trending? Björn
[twitter-dev] Re: how are people collecting spritzer/gardenhose?
I'm just using a realtime json parser in Ruby written as a native C extension (http://github.com/brianmario/yajl-ruby/tree/master) It's really simple to use and well documented. I'm just storing everything in a Postgres database, and then using other scripts to query it. Note: using gardenhose at least you get a LOT of data fast. In just a few days, I have a 4GB+ database now or so On Jun 11, 5:10 pm, M. Edward (Ed) Borasky zzn...@gmail.com wrote: Right now, I'm collecting spritzer data with a simple shell script curl magic incantations | bzip2 -c mmddhhmmss.bz2. A cron job checks every minute and restarts the script if it crashes. The rest is simple ETL. :)
[twitter-dev] Re: Is the spritzer supposed to be as fast as the datamining feed?
Gardenhose is around what... 1.5M tweets/day? I think that's about right. Yes, that is a LOT of data and going through it then takes a lot. My server started crying a bit when I switched up to gardenhose. I'm going to have to get a bigger one if I ever can consume the firehose On Jun 11, 5:04 pm, M. Edward (Ed) Borasky zzn...@gmail.com wrote: I clocked spritzer (JSON feed) at something like 29,000 tweets per hour between 10 and 11 AM PDT the other day. That may in fact be more data than I can practically use. On May 25, 11:12 pm, John Kalucki jkalu...@gmail.com wrote: The sampling rates of the Streaming API and the REST API public methods are independent, and any similarity in proportions is basically happenstance. Currently, Gardenhose will average three timesSpritzer, with some deviation. The rate of all statuses varies significantly throughout the day and week, so be sure you are comparing the exact same time periods. -John Kalucki Services, Twitter Inc. On May 25, 7:20 pm, Twittledee webs...@twittledee.com wrote: Hello: In an answer to a different question in this group, Doug said that thespritzerwas supposed to be as fast as the datamining feed. Is that right? We only get 5.3 msg/sec on avg. for thespritzer. If this is right, then we are only getting half the speed we should be ... and I will have to debug our code. We get 18.7 msg/sec avg. on the gardenhose, so I thought everything was right/okay. Can others state their rates for these two streams? Thanks
[twitter-dev] Re: Twitter conference
They had one the other week: http://twtrcon.com/ -David Fisher @tibbon On Jun 8, 2:26 pm, Emrah e...@ekanet.net wrote: Hi everybody, Is there any planned Twitter Meetup in the near future? Is there any official schedule available online? I heard about #140conf but I was imagining something more developers oriented. Something to discuss possible improvements, add-ons or similar stuffs about Twitter and Twitter apps. I was actually imagining planning a Web or Phone conference between Twitter admins, developers and perhaps some interesting users? In both cases, the conference could be recorded as a podcast to be redistributed if necessary. Mixing both telephone and Web conferencing for users who are not able to join the Web meeting is possible. About the Web meeting, it would be possible to support 2 or 3 concurrent video streams but the number of participants would be unlimited. Regarding the telephone conferencing, it could be possible to integrate Skype, VoIP, regular phone lines and a callback function for users who can't make international phone calls. What do you think? Emrah
[twitter-dev] Re: What is current search time range limit?
Can't you get more than 1500 by using Max_id and ID together and creatively? -David @tibbon On Jun 6, 2:41 pm, Doug Williams d...@twitter.com wrote: See [1] for information on replication. Sometimes there is a lag in the replication of the database across multiple machines. 1.http://en.wikipedia.org/wiki/Replication_(computer_science) Thanks, Doug On Sat, Jun 6, 2009 at 8:24 AM, davidz...@yahoo.com wrote: What do you mean by replication lag? Is this related to repeated tweets in the search results?
[twitter-dev] Re: Forgive Forgive Forgive - XML 2 CSV XSL
http://snippets.dzone.com/posts/show/3701 Gets it done in Ruby. I haven't tested it, but it looks right and simple On Jun 4, 4:04 pm, Mark marktheac...@gmail.com wrote: I would like to take the follower .xml output and convert to csv. From googling I believe I need to create an XSL or XSD and then use MSXML example: MSXSL C:\twitterout\adventuregirl69.txt C:\Users\mark\Documents \twitteratom2txt.xsl –o out.xml I don't know xsl trying to learn from examples, does anyone have any .XSLs they created for Twitter's API responses? Would like to create flat file output from xml response. Goal is to tie a zip code to a twitter profile like @geofollow
[twitter-dev] Re: random sampling of users....do we know anything about user id range?
I'm hoping that Twitter counts users when reporting their numbers, and not accounts. The reason being that I've signed up probably... 5 accounts myself (main, API testing, business, etc, etc). I'm not sure how many the average user signs up, but it's definitely on average more than 1. I'm wondering if IDs are sequential for users afterall. Not using system generated primary keys if I remember right puts more strain on the system as it has to check uniqueness and generate a number (not hard, but still), and most of Twitter is all about scaling and speed as I see it. Otherwise it seems on creating a new user, they are taking the last ID, and adding an artibrary number to it (1d20?) for the next user ID. 45M seems like a lot of users, but I could see there being that many accounts. perhaps On Jun 4, 11:44 am, Nick Arnett nick.arn...@gmail.com wrote: On Wed, Jun 3, 2009 at 7:13 PM, TechRavingMad techraving...@gmail.comwrote: There are a little over 44.5 million twitter IDs as of right now (10:10pm cst 6/3/9) with what seems to be about 10 being added every second. However, Twitter has been quite clear about not saying if status IDs correspond to the actual number of statuses, so I'd guess that they're equally circumspect about whether or not the number of user IDs corresponds to the number of users. In other words, we can be sure there are not more than 44.5 million users, but we don't know how much lower the actual number is. We don't know if all IDs have been used... and even Twitter doesn't know how many of those IDs belong to the same users. I would think that if one wants a random sample of users, one would have to propose a selection method and ask Twitter if there's any reason that it would introduce a selection bias... and hope that they are willing to reply. Seems to me that the biggest problem would be to include quiet users, since only those who post in public become visible. NIck
