[twitter-dev] Re: Forgive Forgive Forgive - XML 2 CSV XSL

2009-06-04 Thread David Fisher
http://snippets.dzone.com/posts/show/3701 Gets it done in Ruby. I haven't tested it, but it looks right and simple On Jun 4, 4:04 pm, Mark marktheac...@gmail.com wrote: I would like to take the follower .xml output and convert to csv. From googling I believe I need to create an XSL or XSD

[twitter-dev] Re: random sampling of users....do we know anything about user id range?

2009-06-04 Thread David Fisher
I'm hoping that Twitter counts users when reporting their numbers, and not accounts. The reason being that I've signed up probably... 5 accounts myself (main, API testing, business, etc, etc). I'm not sure how many the average user signs up, but it's definitely on average more than 1. I'm

[twitter-dev] Re: What is current search time range limit?

2009-06-07 Thread David Fisher
Can't you get more than 1500 by using Max_id and ID together and creatively? -David @tibbon On Jun 6, 2:41 pm, Doug Williams d...@twitter.com wrote: See [1] for information on replication. Sometimes there is a lag in the replication of the database across multiple machines.

[twitter-dev] Re: Twitter conference

2009-06-08 Thread David Fisher
They had one the other week: http://twtrcon.com/ -David Fisher @tibbon On Jun 8, 2:26 pm, Emrah e...@ekanet.net wrote: Hi everybody, Is there any planned Twitter Meetup in the near future? Is there any official schedule available online? I heard about #140conf but I was imagining something

[twitter-dev] Re: how are people collecting spritzer/gardenhose?

2009-06-11 Thread David Fisher
I'm just using a realtime json parser in Ruby written as a native C extension (http://github.com/brianmario/yajl-ruby/tree/master) It's really simple to use and well documented. I'm just storing everything in a Postgres database, and then using other scripts to query it. Note: using gardenhose

[twitter-dev] Re: Is the spritzer supposed to be as fast as the datamining feed?

2009-06-11 Thread David Fisher
Gardenhose is around what... 1.5M tweets/day? I think that's about right. Yes, that is a LOT of data and going through it then takes a lot. My server started crying a bit when I switched up to gardenhose. I'm going to have to get a bigger one if I ever can consume the firehose On Jun 11, 5:04 

[twitter-dev] Re: airline accident case study

2009-06-19 Thread David Fisher
Topics don't just trend because its something 'important'. Now if it was of significantly larger volume than another topics (like the iphone's launch today), then that is rather interesting, but from what I can tell its mostly the most popular things floating to the top generally, plus some

[twitter-dev] Re: Twitter User List

2009-06-29 Thread David Fisher
Thanks for the replies Doug. btw, as far as I can tell the ~ 52M integer values for user ids aren't fully sequential, with some amount missing in between, which I'm assuming is intentionally done to make it slightly more difficult to know the exact user growth trends/patterns (seems valid to me)

[twitter-dev] Re: How to insure that all tweets are retrieved in a search?

2009-07-09 Thread David Fisher
paging. I've been able to pull 2M+ results for a query before. It took 8 hours or so, but it worked. Read up more on the Search API and you should be able to figure it out. -David Fisher http://WebecologyProject.org On Jul 9, 8:16 pm, John Kalucki jkalu...@gmail.com wrote: First, I wouldn't expect

[twitter-dev] Re: Twitter is not making money

2009-07-16 Thread David Fisher
Don't feed the trolls. Name a Venture backed startup that DOESNT run on VC for the first few years. Ok? Right. Stupid topic. Lets move on and talk about things that matter (ie. development) On Jul 16, 4:34 pm, Andrew Badera and...@badera.us wrote: On Thu, Jul 16, 2009 at 4:26 PM, Nick Arnett

[twitter-dev] Re: Filter Profanity

2009-07-17 Thread David Fisher
Why don't you just do the filtering on your end? Twitter's API's job is to give you data- what you do with filtering it on your end should be up to you... On Jul 17, 12:43 pm, Steve Brunton sbrun...@gmail.com wrote: On Fri, Jul 17, 2009 at 12:14 PM, Cameron Kaiserspec...@floodgap.com wrote:

[twitter-dev] Re: Twitter is not making money

2009-07-18 Thread David Fisher
Show me these killer companies doing great NLP with social networks. I find the ones that are doing stuff right now themselves are far behind the curve and not really pushing stuff to the edge. They are often marketing companies that have hired one NLP guy (and underpaid them) and are just

[twitter-dev] LOLCODE API Wrapper

2009-07-18 Thread David Fisher
I can't seem to find a LOLCODE (http://en.wikipedia.org/wiki/LOLCODE) API wrapper for Twitter. I was hoping to write my next Twitter application in LOLCODE, but its very hard to do so without one. Can anyone help me get started with a basic program? My code so far isn't working HAI CAN HAS

[twitter-dev] Re: New Twist To Follow Terms Violations

2009-07-24 Thread David Fisher
What gives Twitter the right to dictate who you want to follow or not? Its their service. They can dictate what they want. Their playground, their rules. The ToS clearly says they can alter their terms at any time and if you don't want to comply you can leave. That being said, this is to

[twitter-dev] Re: API only shows messages from last 7 days

2009-07-28 Thread David Fisher
I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only

[twitter-dev] Re: API only shows messages from last 7 days

2009-07-28 Thread David Fisher
I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to

[twitter-dev] Re: API only shows messages from last 7 days

2009-07-28 Thread David Fisher
I don't think that adding more people to the staff at Twitter is the solution. In one startup I saw a thing posted on the refrigerator that had the adage, Adding more people to a project already behind schedule will only slow it down more. Surely for support and customer service issues having

[twitter-dev] Re: Current Twitter site status

2009-08-06 Thread David Fisher
Is the Search API being effected? I thought at first that I had messed up my code, but I rolled back pretty far and I'm still getting really odd errors /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in `setup_raw_request': undefined method `request_uri' for #URI::Generic:

[twitter-dev] Re: You have been rate limited. Enhance your calm.

2009-08-07 Thread David Fisher
If you're getting that then you're getting more than me. I'm just doing: require 'rubygems' gem 'twitter' require 'twitter' Twitter::Search.new('foo').each do |r| puts r.inspect end And I get only this back now: /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in

[twitter-dev] Re: You have been rate limited. Enhance your calm.

2009-08-07 Thread David Fisher
I can't be sure if my client is following redirects. Probably not. I'm just using the Ruby Twitter Gem which haven't been updated for a month or so I think dave On Aug 7, 1:15 pm, lucasnicolato eternitya...@gmail.com wrote: im having the same problem. im just lucky my app is still in test.

[twitter-dev] Re: You have been rate limited. Enhance your calm.

2009-08-07 Thread David Fisher
, but if it's John Nunemaker's, I believe it does follow redirects. Larry Wright/@larrywright On Fri, Aug 7, 2009 at 2:03 PM, David Fisher tib...@gmail.com wrote: I can't be sure if my client is following redirects. Probably not. I'm just using the Ruby Twitter Gem which haven't been updated

[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED

2009-08-09 Thread David Fisher
They haven't overreacted. If you think you can do better, then apply for a job with them. Have you tried your account from multiple IP blocks? They have faced a crushing attack. They are working on it and they have worked hard to communicate with us the status of things. Anyone that was around

[twitter-dev] Re: The silence is deafening....

2009-08-09 Thread David Fisher
bitching and complaining. You're making yourselves look unprofessional and bad. -David Fisher Web Ecology Project http://webecologyproject.org On Aug 9, 2:51 am, chinaski007 chinaski...@gmail.com wrote: And, by the way, if you're a deckhand on a submarine going down, you think you would go

[twitter-dev] Re: OK Seriously People

2009-08-09 Thread David Fisher
A few of you are acting like real children and a few of you still have your heads screwed on right. I'm confident they are doing everything they can. Chill and enjoy your weekend. They'll get it sorted out. What did you guys do in 2007? Twitter was down all the time then. Your blood pressure

[twitter-dev] Re: Can Twitter team Respond?

2009-08-09 Thread David Fisher
It will be fixed when it is. Stop complaining. We're all in the same boat. The ETA is when its done. dave On Aug 9, 1:08 pm, freefall tehgame...@googlemail.com wrote: Decentralisation. On Aug 9, 5:47 pm, Dewald Pretorius dpr...@gmail.com wrote: I wonder how many times this weekend has

[twitter-dev] Re: OK Seriously People

2009-08-09 Thread David Fisher
It is like a surreal tech soap opera on this list though :-) Hey maybe I could pitch this to Fox . All the best Neil DO IT! Maybe Twitter would have made a good reality show. Lots of super-dramatic angles and queues pointing at sysadmins typing at a console. :) The mailing list would

[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED

2009-08-09 Thread David Fisher
Yea, and we all threatened to go to Pownce. That didn't go so well (seeing that Pownce is now dead) dave On Aug 9, 3:05 pm, Dossy Shiobara do...@panoptic.com wrote: On 8/9/09 1:53 PM, David Fisher wrote: Anyone that was around in 2006/2007 knows that Twitter was excessively unstable

[twitter-dev] Re: 404 for statuses/user_timeline of non-existing users?

2009-08-11 Thread David Fisher
Yep. In my experience calling a user that doesn't exist does yield a 404. You'll have to handle the errors on this. I guess 404 does make sense here, as the user is not found. dave On Aug 11, 5:40 am, Carlo Zottmann czottm...@gmail.com wrote: Hi there, I'm doing statuses/user_timeline

[twitter-dev] Re: Following Churn: Specific guidance needed

2009-08-11 Thread David Fisher
IDOLPeeps, I feel you're being overly alarmist and haven't painted the situation properly. You can unfollow anyone you want. The issue is a quick follow and then unfollowing if not reciprocated. You're *supposed* to follow someone because you want to hear what they are saying, not because you

[twitter-dev] Search API Limits lowered?

2009-08-11 Thread David Fisher
While i haven't done scientific testing of this, I was able to run up to 3-4 instances of my search script prior at a time before it told me to enhance my calm. Now I'm barely able to run one without hitting the limit. I can put delays in my code to slow it down, but I'm wondering if this is just

[twitter-dev] Re: Search API Limits lowered?

2009-08-11 Thread David Fisher
The user agent for each search request is the same. I'm using the Ruby Twitter API wrapper, so sending anything else with search requests isn't possible unless that is now deprecated. dave On Aug 11, 10:36 am, Andrew Badera and...@badera.us wrote: On Tue, Aug 11, 2009 at 10:30 AM, David

[twitter-dev] Re: Following Churn: Specific guidance needed

2009-08-11 Thread David Fisher
Follower churn wouldn't exist, but getting hundreds of spam emails (about being followed) would still exist. I've got over 12,000 emails in my inbox about being followed on Twitter. Dozens of those are from the same users. Some weeks the same users unfollow and refollow me nonstop to try to get

[twitter-dev] Re: Search API Limits lowered?

2009-08-11 Thread David Fisher
, Larry Wrightlarrywri...@gmail.com wrote: In addition to setting a unique user-agent, I believe it was requested that we set a referrer header that pointed back to a domain. On Tue, Aug 11, 2009 at 9:30 AM, David Fisher tib...@gmail.com wrote: While i haven't done scientific testing of this, I

[twitter-dev] Re: FW: Twitter is Suing me!!!

2009-08-12 Thread David Fisher
The OP should have first gotten a lawyer to look over it, instead of freaking out hysterically here. He's not being sued. Twitter does own the name Twitter and can selectively choose to sue/CD (or not sue) anyone they like who infringes on it. You defense being other people have registered the

[twitter-dev] Re: Trademark infringement

2009-08-12 Thread David Fisher
My guess is that Twitter will hold both Tweet and Twitter as IP, but allow general use of Tweet. It seems to really make sense to defend the name of their company always, and Tweet only when it is misused. -dave On Aug 12, 11:43 am, Dewald Pretorius dpr...@gmail.com wrote: Nick,

[twitter-dev] Re: FW: Twitter is Suing me!!!

2009-08-12 Thread David Fisher
Let's not get into a semantic argument of spam. Spam, noise, trash followers, whatever you want to call them. They are all annoying and not what Twitter or most users want in their service. I'm unsure if a company HAS to pursue every trademark infringement to hold their trademark. Otherwise you

[twitter-dev] Re: Following a large number of users

2009-08-12 Thread David Fisher
Streaming methods are your friend. Start with the follow method, and email the API team about getting Shadow. If your use is legit (likely) then they'll give you Shadow. If/when you outstrip Shadow they will likely upgrade you then. The only caveat to streaming is that if you miss some time

[twitter-dev] Re: FW: Twitter is Suing me!!!

2009-08-12 Thread David Fisher
@Vincent No. Do you not understand that Trademark infringements occur between things that could be mistaken for each other or in the same industry, diluting a brand? A Disney film from the 1940's has what to do with a 3rd part application for a 2006-present social network? There is a CLEAR

[twitter-dev] Re: Cease Desist from Twitter

2009-08-13 Thread David Fisher
I am wondering if this is a case of their legal department getting a bit heavyhanded and running loose. What they asked of you seemed fairly reasonable however, and the name of your application doesn't seem to be the issue. I'm glad you didn't think you were being sued :) It seems that Twitter

[twitter-dev] Re: legal issues - is tweet an official verb in the US language?

2009-08-19 Thread David Fisher
Unless someone here is a lawyer, we should probably avoid legal debate- consult with each our own counsels, and move on to doing what we do best (coding). I find these debates are often filled with FUD, misinformation, speculation, a misunderstanding of law, etc The easiest way to get around it

[twitter-dev] Re: Do My Customers Have a Twitter Account?

2009-08-19 Thread David Fisher
Sounds like something you should be able to do in an email to them or with a message on your website. On Aug 19, 1:29 pm, arawajy araw...@gmail.com wrote: I want to invite them to follow the company on Twitter. On Aug 19, 8:25 pm, Andrew Badera and...@badera.us wrote: On Wed, Aug 19, 2009

[twitter-dev] Re: Twitter's Tweet Trademark Torpedoed

2009-08-19 Thread David Fisher
This is good news. No one will be getting sued over Tweet then. Yet, keep in mind that Twitter probably *could * shut off access to the API to any company they choose, as its their playground and their rules. Not that they'll start doing that at all however. On Aug 19, 1:57 pm, Sam Johnston

[twitter-dev] Re: appended to API works in progress

2009-08-24 Thread David Fisher
lul. That's like asking the Duke Nukem Forever team when they'll release. When its done While I love the twitter dev team, I'm not sure how hard they are able to stick to deadlines in their development, so its probably just better to say, the future On Aug 24, 10:15 am, METROmilwaukee ...

[twitter-dev] Re: Search API limits

2009-09-04 Thread David Fisher
The rpp defaults to 15 or something if you don't specify it. Sounds like you need to mess around and play with things a bit more. The key to max search results isn't in paging or rpp, but in max_id. Be careful what you ask for. Retrieval of everything available can take a long time (hours)

[twitter-dev] Re: Getting screen_name from id without gazillion API calls?

2009-09-08 Thread David Fisher
I've done this this way: Every time I get a user's data- I store it in my database. Doesn't matter if its from gardenhose or a REST method. I track various versions of it and keep all changes (good data warehousing practice). Then, when I crawl a user's list of friends I ask the database if we

[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread David Fisher
I'm sorry, but the problem isn't Twitter- its your language and JSON parser. Outputting everything as a string, when it clearly should be a number, is inefficient and crazy. Saying that startups can't afford 64 bit processors in systems is crazy. Most startups I know are running on EC2 or have

[twitter-dev] Re: master thesis related to Twitter

2009-09-28 Thread David Fisher
data but now we've gotten mostly past those issues and we are now starting to mine and analyze other social networks as well. Thanks, David Fisher Web Ecology Project On Sep 27, 8:11 pm, Stefna mstefa...@gmail.com wrote: Thank you all for the feedback. My main motivation for posting

[twitter-dev] Re: Search API Query Limitation (140 characters?)

2009-09-28 Thread David Fisher
If you need to search specific users why don't you use the Shadow API and grab all of their tweets and then search them locally? On Sep 28, 3:14 pm, Chad Etzel c...@twitter.com wrote: Hello, The limit is indeed 140 and most likely won't be going up any time soon. The reason for the limit is

[twitter-dev] Re: About the oneforty application directory

2009-09-29 Thread David Fisher
I'll vouch for Laura as being an upright person, and all of the other oneforty people I've met are as well. They aren't going to try to screw anyone over and I have faith that whatever they come up with will be fair for developers and the community both. david

[twitter-dev] Additional Favorites methods?

2009-09-30 Thread David Fisher
even, that would be somewhat awesome and very useful for me. Thoughts? -David Fisher Web Ecology Project http://webecologyproject.org

[twitter-dev] Re: Best way to implement caching of searches (with multithreading)?

2009-10-02 Thread David Fisher
Really a database is the way to go. Any modern database should allow you to check if a value is in there before inserting, so the same tweet won't go in there twice. Additionally, not every user search has to use the up to the minute results. They can go back just a little in time (30 seconds or

[twitter-dev] Re: how are the ten trends born?

2009-10-02 Thread David Fisher
It's pretty simple, but with a few twists. First of all, remember that everything that Twitter does is done with simplicity and efficiency in mind. For the most part its just a frequency count of words over a short time period, minus stop words, filtering out usernames (notice @foo is never a