Re: [twitter-dev] t.co issue -- querying for original url in streaming & search apis
Fantastic, thank you! On Wed, Jun 9, 2010 at 2:48 PM, Mark McBride wrote: > We will have this support in the streaming API. Track terms will work > against tweet text as well as entity text. Currently streaming does > *not* work as Abraham describes below. We only match against tweet > text, and don't do any link expansion/contraction. > > ---Mark > > http://twitter.com/mccv > > > > On Wed, Jun 9, 2010 at 12:13 PM, Jim Gilliam wrote: > > I'm creating a new thread for this because a few others have mentioned > it, > > and we haven't gotten a response yet. My hunch is that changing those > APIs > > involve other teams within Twitter, so figuring out a solution could be > > challenging. > > Here is the issue. We need to be able to get matches on the original URL > > through the streaming and search APIs. For me, I'm tracking "act" so > I can > > match tweets that link to 'http://act.ly'. This is not a link shortener > > service, the actual pages live at act.ly, and it was all designed > > specifically for Twitter so there would be no need for url shorteners. > > As far as I'm concerned, it's fine if that link changes to t.co, as long > as > > I can still get matches on act.ly (or act) through the streaming API > (the > > search API is going to be important for people too, but less of an issue > for > > me personally). > > The most elegant way to fix this would be to allow tracking of the > original > > URL. So I can put in a domain name, or URL substring, and match > everything > > that way. Same with search. This would be useful to a lot of people, and > > virtually all link oriented web apps with APIs provide a way to get all > the > > matches for a particular domain. (digg, google, yahoo, etc) > > I'm sure there are other workaround ways of doing this, and I'm all ears. > > It would be SUPER NICE (wink wink) to hear some kind of assurance that > > there will be a way for us to query this type of information before the > t.co > > changes go live. > > Thanks guys... > > Jim Gilliam > > http://act.ly/ > > http://twitter.com/jgilliam > > On Tue, Jun 8, 2010 at 4:43 PM, Jim Gilliam wrote: > >> > >> Will we be able to get matches on the original URL through the streaming > >> API? > >> For example, I'm tracking "act" so I can match tweets that link to > >> 'http://act.ly'. Will I still be able to do that? > >> Jim Gilliam > >> http://act.ly/ > >> http://twitter.com/jgilliam > >> > >> On Tue, Jun 8, 2010 at 4:33 PM, Dewald Pretorius > wrote: > >>> > >>> Raffi, > >>> > >>> I'm fine with everything up to the new 140 character count. > >>> > >>> If you count the characters *after* link wrapping, you are seriously > >>> going to mess up my system. My short URLs are currently 18 characters > >>> long, and they will be 18 long for quite some time to come. After that > >>> they will be 19 for a very long time to come. > >>> > >>> If you implement this change, a ton, and I mean a *huge* number of my > >>> system's updates are going to be rejected for being over 140 > >>> characters. > >>> > >>> On Jun 8, 7:57 pm, Raffi Krikorian wrote: > >>> > hi all. > >>> > > >>> > twitter has been wrapping links in e-mailed DMs for a couple months > >>> > now<http://bit.ly/twttldmemail>. > >>> > with that feature, we're trying to protect users against phishing and > >>> > other > >>> > malicious attacks. the way that we're doing this is that any URL that > >>> > comes > >>> > through in a DM gets currently wrapped with a twt.tl URL -- if the > URL > >>> > turns > >>> > out to be malicious, Twitter can simply shut it down, and whoever > >>> > follows > >>> > that link will be presented with a page that warns them of > potentially > >>> > malicious content. in a few weeks, we're going to start slowly > enabling > >>> > this > >>> > throughout the API for all statuses as well, but instead of twt.tl, > we > >>> > will > >>> > be using t.co. > >>> > > >>> > practically, any tweet that is sent through statuses/update that has > a > >>> > link > >>> > on it will have the link
[twitter-dev] t.co issue -- querying for original url in streaming & search apis
I'm creating a new thread for this because a few others have mentioned it, and we haven't gotten a response yet. My hunch is that changing those APIs involve other teams within Twitter, so figuring out a solution could be challenging. Here is the issue. We need to be able to get matches on the original URL through the streaming and search APIs. For me, I'm tracking "act" so I can match tweets that link to 'http://act.ly'. This is not a link shortener service, the actual pages live at act.ly, and it was all designed specifically for Twitter so there would be no need for url shorteners. As far as I'm concerned, it's fine if that link changes to t.co, as long as I can still get matches on act.ly (or act) through the streaming API (the search API is going to be important for people too, but less of an issue for me personally). The most elegant way to fix this would be to allow tracking of the original URL. So I can put in a domain name, or URL substring, and match everything that way. Same with search. This would be useful to a lot of people, and virtually all link oriented web apps with APIs provide a way to get all the matches for a particular domain. (digg, google, yahoo, etc) I'm sure there are other workaround ways of doing this, and I'm all ears. It would be SUPER NICE (wink wink) to hear some kind of assurance that there will be a way for us to query this type of information before the t.cochanges go live. Thanks guys... Jim Gilliam http://act.ly/ http://twitter.com/jgilliam On Tue, Jun 8, 2010 at 4:43 PM, Jim Gilliam wrote: > Will we be able to get matches on the original URL through the streaming > API? > > For example, I'm tracking "act" so I can match tweets that link to ' > http://act.ly'. Will I still be able to do that? > > Jim Gilliam > http://act.ly/ > http://twitter.com/jgilliam > > > On Tue, Jun 8, 2010 at 4:33 PM, Dewald Pretorius wrote: > >> Raffi, >> >> I'm fine with everything up to the new 140 character count. >> >> If you count the characters *after* link wrapping, you are seriously >> going to mess up my system. My short URLs are currently 18 characters >> long, and they will be 18 long for quite some time to come. After that >> they will be 19 for a very long time to come. >> >> If you implement this change, a ton, and I mean a *huge* number of my >> system's updates are going to be rejected for being over 140 >> characters. >> >> On Jun 8, 7:57 pm, Raffi Krikorian wrote: >> > hi all. >> > >> > twitter has been wrapping links in e-mailed DMs for a couple months >> > now<http://bit.ly/twttldmemail>. >> > with that feature, we're trying to protect users against phishing and >> other >> > malicious attacks. the way that we're doing this is that any URL that >> comes >> > through in a DM gets currently wrapped with a twt.tl URL -- if the URL >> turns >> > out to be malicious, Twitter can simply shut it down, and whoever >> follows >> > that link will be presented with a page that warns them of potentially >> > malicious content. in a few weeks, we're going to start slowly enabling >> this >> > throughout the API for all statuses as well, but instead of twt.tl, we >> will >> > be using t.co. >> > >> > practically, any tweet that is sent through statuses/update that has a >> link >> > on it will have the link automatically converted to a t.co link on its >> way >> > through the Twitter platform. if you fetch any tweet created after this >> > change goes live, then its text field will have all its links >> automatically >> > wrapped with t.co links. when a user clicks on that link, Twitter will >> > redirect them to the original URL after first confirming with our >> database >> > that that URL is not malicious. on top of the end-user benefit, we hope >> to >> > eventually provide all developers with aggregate usage data around your >> > applications such as the number of clicks people make on URLs you >> display >> > (it will, of course, be in aggregate and not identifiable manner). >> > additionally, we want to be able to build services and APIs that can >> make >> > algorithmic recommendations to users based on the content they are >> > consuming. gathering the data from t.co will help make these possible. >> > >> > our current plan is that no user will see a t.co URL on twitter.com but >> we >> > still have some details to work through. the links wi
Re: [twitter-dev] Re: link wrapping on the API
Will we be able to get matches on the original URL through the streaming
API?
For example, I'm tracking "act" so I can match tweets that link to '
http://act.ly'. Will I still be able to do that?
Jim Gilliam
http://act.ly/
http://twitter.com/jgilliam
On Tue, Jun 8, 2010 at 4:33 PM, Dewald Pretorius wrote:
> Raffi,
>
> I'm fine with everything up to the new 140 character count.
>
> If you count the characters *after* link wrapping, you are seriously
> going to mess up my system. My short URLs are currently 18 characters
> long, and they will be 18 long for quite some time to come. After that
> they will be 19 for a very long time to come.
>
> If you implement this change, a ton, and I mean a *huge* number of my
> system's updates are going to be rejected for being over 140
> characters.
>
> On Jun 8, 7:57 pm, Raffi Krikorian wrote:
> > hi all.
> >
> > twitter has been wrapping links in e-mailed DMs for a couple months
> > now<http://bit.ly/twttldmemail>.
> > with that feature, we're trying to protect users against phishing and
> other
> > malicious attacks. the way that we're doing this is that any URL that
> comes
> > through in a DM gets currently wrapped with a twt.tl URL -- if the URL
> turns
> > out to be malicious, Twitter can simply shut it down, and whoever follows
> > that link will be presented with a page that warns them of potentially
> > malicious content. in a few weeks, we're going to start slowly enabling
> this
> > throughout the API for all statuses as well, but instead of twt.tl, we
> will
> > be using t.co.
> >
> > practically, any tweet that is sent through statuses/update that has a
> link
> > on it will have the link automatically converted to a t.co link on its
> way
> > through the Twitter platform. if you fetch any tweet created after this
> > change goes live, then its text field will have all its links
> automatically
> > wrapped with t.co links. when a user clicks on that link, Twitter will
> > redirect them to the original URL after first confirming with our
> database
> > that that URL is not malicious. on top of the end-user benefit, we hope
> to
> > eventually provide all developers with aggregate usage data around your
> > applications such as the number of clicks people make on URLs you display
> > (it will, of course, be in aggregate and not identifiable manner).
> > additionally, we want to be able to build services and APIs that can make
> > algorithmic recommendations to users based on the content they are
> > consuming. gathering the data from t.co will help make these possible.
> >
> > our current plan is that no user will see a t.co URL on twitter.com but
> we
> > still have some details to work through. the links will still be
> displayed
> > as they were sent in, but the target of the link will be the t.co link
> > instead. and, we want to provide the same ability to display original
> links
> > to developers. we're going to use the entities attribute to make this
> > possible.
> >
> > let's say i send out the following tweet: "you have to check outhttp://
> dev.twitter.com!"
> >
> > a returned (and truncated) status object may look like:
> >
> > {
> > "text" : "you have to check outhttp://t.co/s9gfk2d4!";,
> > ...
> > "user" : {
> > "screen_name" : "raffi",
> > ...
> > },
> > ...
> > "entities" : {
> > "urls" : [
> > {
> > "url" : "http://t.co/s9gfk2d4";,
> > "display_url" : "http://dev.twitter.com";,
> > "indices" : [23, 43]
> > }
> > ],
> > ...
> > },
> > ...
> >
> > }
> >
> > two things to note: the text of the returned status object doesn't have
> the
> > original URL and instead it has a t.co URL, and the entities block now
> has a
> > display_url attribute associated with it. what we're hoping is that with
> > this data, it should be relatively easy to create a UI where you replace
> thehttp://t.co/s9gfk2d4in the text with the equivalent of
> >
> > http://t.co/s9gfk2d4";>http://dev.twitter.com
> >
> > this means the user would not see the t.co link, but we all can still
> > provide the protection and gather data from the wrapped link. for the
> > applications that don't choose to update, the t.co link will be shown
> (and
> > the goal to protect user
Re: [twitter-dev] Streaming API - Partial word match
I've been able to track act.ly urls by using "act". So try "bit" and just throw out anything that isn't a bit.ly url. On Mon, Jan 18, 2010 at 1:05 PM, vivekpuri wrote: > Search API team is recommending developers to migrate over to > Streaming API. To get started with this, i was looking at the > Streaming API docs and they state that if using Track for query > parameter, "Terms are exact-matched, and also exact-matched ignoring > punctuation". From what i can figure out from that statement and > running couple of tests, Streaming API is not returning partial word > matches, which Searce API does. For example - keyword bit.ly returns > all results on Search API with *bit.ly*, while Streaming API returns > only results with exact bit.ly. Are there any plans to support partial > word matches in the Streaming API? >
Re: [twitter-dev] Re: retweets aren't appearing in track streaming api
I added this retweet truncation bug to the issue tracker. http://code.google.com/p/twitter-api/issues/detail?id=1219 On Thu, Nov 19, 2009 at 2:39 PM, Jim Gilliam wrote: > It's great that most retweets are coming through, but the fact that it's > cutting off the end of tweets, making the track API not work if what you're > tracking happens to be in the last 15-20 characters of the tweet, is a > problem. As it stands now, the only way for me to get at the retweets is > to query each individual tweet (like /statuses/retweets/5865706501.json), > over and over and over again to check. Yuck! > > Jim > > > On Thu, Nov 19, 2009 at 2:08 PM, Mark McBride wrote: > >> Great! I checked through the code paths, and they look solid on our end. >> >> ---Mark >> >> On Thu, Nov 19, 2009 at 1:24 PM, Jim Gilliam wrote: >> > I figured out what's happening. When "RT @thekabira" gets added to the >> > front of the tweet, it makes the tweet text longer than 140 characters, >> so >> > it then cuts off the rest of the tweet, which is what had "act" in it, >> so it >> > never shows up through the streaming api tracking for anything with >> "act" in >> > it. >> > >> > So this tweet: "What’s the deal with the Climate Bill? RT new >> > #climategraphic, win a prize! http://bit.ly/12H1X7 via @PhaedraEL >> > http://act.ly/Rxt"; >> > >> > turns into this, when it's retweeted: >> > >> > "RT @thekabira: What’s the deal with the Climate Bill? RT new >> > #climategraphic, win a prize! http://bit.ly/12H1X7 via @PhaedraEL >> http://ac >> > ..." >> > >> > Jim >> > >> > On Thu, Nov 19, 2009 at 12:05 PM, Jim Gilliam wrote: >> >> >> >> My understanding is that the only way to get all the new retweets is >> >> through the streaming API because they don't show up in search. So >> I'm >> >> using the track method, but I'm not seeing retweets come through. >> >> >> >> Specifically, this: http://twitter.com/thekabira/status/5865706501 was >> >> retweeted twice, but I didn't see it come through. Old style retweets >> >> continue to come through normally. >> >> >> >> Jim Gilliam >> >> http://act.ly/ >> >> http://twitter.com/jgilliam >> > >> > >> > >
Re: [twitter-dev] Re: retweets aren't appearing in track streaming api
It's great that most retweets are coming through, but the fact that it's cutting off the end of tweets, making the track API not work if what you're tracking happens to be in the last 15-20 characters of the tweet, is a problem. As it stands now, the only way for me to get at the retweets is to query each individual tweet (like /statuses/retweets/5865706501.json), over and over and over again to check. Yuck! Jim On Thu, Nov 19, 2009 at 2:08 PM, Mark McBride wrote: > Great! I checked through the code paths, and they look solid on our end. > > ---Mark > > On Thu, Nov 19, 2009 at 1:24 PM, Jim Gilliam wrote: > > I figured out what's happening. When "RT @thekabira" gets added to the > > front of the tweet, it makes the tweet text longer than 140 characters, > so > > it then cuts off the rest of the tweet, which is what had "act" in it, so > it > > never shows up through the streaming api tracking for anything with "act" > in > > it. > > > > So this tweet: "What’s the deal with the Climate Bill? RT new > > #climategraphic, win a prize! http://bit.ly/12H1X7 via @PhaedraEL > > http://act.ly/Rxt"; > > > > turns into this, when it's retweeted: > > > > "RT @thekabira: What’s the deal with the Climate Bill? RT new > > #climategraphic, win a prize! http://bit.ly/12H1X7 via @PhaedraEL > http://ac > > ..." > > > > Jim > > > > On Thu, Nov 19, 2009 at 12:05 PM, Jim Gilliam wrote: > >> > >> My understanding is that the only way to get all the new retweets is > >> through the streaming API because they don't show up in search. So I'm > >> using the track method, but I'm not seeing retweets come through. > >> > >> Specifically, this: http://twitter.com/thekabira/status/5865706501 was > >> retweeted twice, but I didn't see it come through. Old style retweets > >> continue to come through normally. > >> > >> Jim Gilliam > >> http://act.ly/ > >> http://twitter.com/jgilliam > > > > >
[twitter-dev] Re: retweets aren't appearing in track streaming api
I figured out what's happening. When "RT @thekabira" gets added to the front of the tweet, it makes the tweet text longer than 140 characters, so it then cuts off the rest of the tweet, which is what had "act" in it, so it never shows up through the streaming api tracking for anything with "act" in it. So this tweet: "What’s the deal with the Climate Bill? RT new #climategraphic <http://twitter.com/search?q=%23climategraphic>, win a prize! http://bit.ly/12H1X7 via @PhaedraEL <http://twitter.com/PhaedraEL> http://act.ly/Rxt"; turns into this, when it's retweeted: "RT @thekabira: What’s the deal with the Climate Bill? RT new #climategraphic, win a prize! http://bit.ly/12H1X7 via @PhaedraEL http://ac..."; Jim On Thu, Nov 19, 2009 at 12:05 PM, Jim Gilliam wrote: > My understanding is that the only way to get all the new retweets is > through the streaming API because they don't show up in search. So I'm > using the track method, but I'm not seeing retweets come through. > > Specifically, this: http://twitter.com/thekabira/status/5865706501 was > retweeted twice, but I didn't see it come through. Old style retweets > continue to come through normally. > > Jim Gilliam > http://act.ly/ > http://twitter.com/jgilliam >
[twitter-dev] retweets aren't appearing in track streaming api
My understanding is that the only way to get all the new retweets is through the streaming API because they don't show up in search. So I'm using the track method, but I'm not seeing retweets come through. Specifically, this: http://twitter.com/thekabira/status/5865706501 was retweeted twice, but I didn't see it come through. Old style retweets continue to come through normally. Jim Gilliam http://act.ly/ http://twitter.com/jgilliam
[twitter-dev] will user payload include lists_count soon?
I assume lists_count must be coming to the user payload, but haven't heard anyone mention it. Jim
[twitter-dev] Re: Profile search based on Bio field
Topsy provides an API for searching user bios. http://code.google.com/p/otterapi/wiki/Resources#/profilesearch On Wed, Oct 28, 2009 at 2:38 PM, MuratMetu wrote: > > Hello, we are new to twitter development, we want to know is there any > open source API that provides us to send String query (regular > expression) and return the list of the users whose bio text match with > the query in their profile. Thank you. >
[twitter-dev] need ability to search retweets
Right now, retweets show up in the search API, but with the new retweet system, it appears all retweets will be yanked from the search API, and there will be no way to get all retweets that match a particular keyword. This is a problem for my http://act.ly application. It's Twitter petitions, and one core piece of functionality is that people can sign the petition simply by retweeting. There's no need to go to the act.ly site, so it works easily from mobile, etc. To get these tweets, I hit the search API every minute or so and pull back anything that matches "act.ly" and then add them to the appropriate petition. I need some API method to find out all the retweets that match "act.ly" Pretty please! Don't make me start a petition. :) Jim Gilliam http://twitter.com/jgilliam
[twitter-dev] Re: Streaming API Help
I ran into the same problem trying to do this in Ruby last night using this
technique: http://www.robares.com/2009/07/14/streaming-twitter-api.html
I think it's because /statuses/filter.json requires a POST, which isn't
supported by yajl-ruby. So I'm stuck. It works fine if you use one of the
other GET methods like /statuses/sample.json.
Jim Gilliam
http://act.ly/
http://twitter.com/jgilliam
On Sat, Sep 19, 2009 at 7:31 AM, Greg wrote:
>
> I'm trying to implement Streaming API on my Twitter application to
> enhance the stabilty instead of using the Search API.
>
> Basically - I'm trying to use the track paramter to track a keyword.
> However, when I run this code - it just times out - nothing occurs.
> Perhaps I'm not using the track parameter correctly?
>
> I know I'm just dumping the code right now - but it was merely for
> test.
>
> My ad-hoc code is below:
>
> // twitter_stream.php
> $fp = fopen("http://username:passw...@stream.twitter.com/1/statuses/
> filter.json?track=ttl,wine<http://username:passw...@stream.twitter.com/1/statuses/%0Afilter.json?track=ttl,wine>",
> "r");
> while($data = fgets($fp))
> {
>$new_data = json_decode($data);
>
>foreach($status as $new_data)
>{
>var_dump($status);
>echo "";
>}
> }
> ?>
>
> Thanks for the help!
>
> Greg
>
[twitter-dev] more elegant retweet solution, that also fixes replies
In the same way that Twitter ditched "replies" for "mentions", they should ditch the concept of a reply to a tweet, and instead let people "attach" or "refer" to a specific tweet, which would then be included along with the new tweet. This fixes a number of different problems. 1. It handles all the situations that RT is currently being used 2. It gives people a full 140 characters to respond or add their own commentary to another tweet. 3. Conversation threading, which is currently a total mess, is now much more explicit and will probably even work. Jim Gilliam http://act.ly/ http://twitter.com/jgilliam
