Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[M. Edward (Ed) Borasky]

Quoting DeWitt Clinton :


Awesome, thanks for the quick response!

Those are the right answers, too.  : )

Though there's an inconsistency with returning 301's and also requiring
every click to go through the t.co link (as required by the ToS).  A 301
means that the redirect is cacheable by any intermediary (because it is
permanent and will never change).

The 301 also implies that you actually *can* replace only the malicious
links, (not every link), because clients will already have resolved and
cached the 301 redirects (which again, can never change), so you won't be
able to change the redirect down the road anyway.

So, I think you might actually have meant to use 302's, not 301's, if
redirecting every click is the goal.

But then again, 301's really are the (philosophically? morally?) right
answer, so maybe I *don't* want you to fix that. : )

Or better still, resolving *all* URLs upfront and returning the full URL
inline, making tweets longer than 140 characters, and stopping this whole
URL shortening nonsense to begin with.  (But you knew I'd say that...!)

-DeWitt


Well ... while we're on the subject ... since we're talking *tweets*  
and not arbitrary text from just anywhere, why do we have to waste  
seven characters with the "http://";? Can't it just be "t.co/abcdef"?  
After all, the receiver got the message from Twitter, not from thin  
air, and can easily supply the "http://";. You're never going to have  
an "ftp://t.co";, "file://t.co", etc., right?

Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[John Barratt]

Hi Raffi,

On 9/06/10 9:57 AM, Raffi Krikorian wrote:

that would be an awesome service!
Currently we use one our own services (http://metauri.com/) to do this 
for http://trendsmap.com/.  In addition to the title, it also gives the 
content type, which can be useful in determining how, or if to use the 
link in your display.


I was just wondering if Twitter were going to possibly supply this extra 
data, as it would remove a time & resource intensive step in the tweet 
analysis process :).


Another question, will you wrap links that have no protocol in tweets, eg :

"Hey check out www.mysite.com/evil_page"

and :

"Hey check out mysite.com/evil_page"

I imagine many clients will currently link these out as urls and link 
them up automatically?


Thanks again,

JB.


On Tue, Jun 8, 2010 at 4:50 PM, John Barratt mailto:djo...@gmail.com>> wrote:

Hi Raffi,


On 9/06/10 8:57 AM, Raffi Krikorian wrote:

"url" : "http://t.co/s9gfk2d4";,
"display_url" : "http://dev.twitter.com";,
"indices" : [23, 43]

Any chance of getting the title of the resolved URL added in here
too if available?

Then we could display a link like :

http://t.co/s9gfk2d4";>

http://dev.twitter.com


or :

http://dev.twitter.com"; href="http://t.co/s9gfk2d4";>
   Twitter Dev


This would give even more context to users, without having to follow
the redirect path, & load and parse the page to extract it as well.

Thanks,

JB.




--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

[twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[Dewald Pretorius]

Raffi: Never mind. I just saw the Twitter blog post. The motivation
for this is to get metrics for Promoted Tweets and Resonance. Hence,
the answer is: Suck it up.

DeWitt: Yikes, discarding all shortens between t.co and the final link
will seriously mess with the click stats of a few million people.

Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[DeWitt Clinton]

Awesome, thanks for the quick response!

Those are the right answers, too.  : )

Though there's an inconsistency with returning 301's and also requiring
every click to go through the t.co link (as required by the ToS).  A 301
means that the redirect is cacheable by any intermediary (because it is
permanent and will never change).

The 301 also implies that you actually *can* replace only the malicious
links, (not every link), because clients will already have resolved and
cached the 301 redirects (which again, can never change), so you won't be
able to change the redirect down the road anyway.

So, I think you might actually have meant to use 302's, not 301's, if
redirecting every click is the goal.

But then again, 301's really are the (philosophically? morally?) right
answer, so maybe I *don't* want you to fix that. : )

Or better still, resolving *all* URLs upfront and returning the full URL
inline, making tweets longer than 140 characters, and stopping this whole
URL shortening nonsense to begin with.  (But you knew I'd say that...!)

-DeWitt


On Tue, Jun 8, 2010 at 4:54 PM, Raffi Krikorian  wrote:

> *1)* Will the redirect from t.co -> domain.com be a 301 Moved Permanently
>> or a 302 Found response?
>>
>
> 301!
>
>
>> *2)* Will the t.co URL redirect point to the URL in the original tweet,
>> or will it point to the ultimate resolved URL?
>>
>> I.e., if I post "Check out my site at http://bit.ly/abcd"; where
>> bit.ly/abcd redirects to domain.com, and the resultant tweet becomes
>> "Check out my site at http://t.co/abcd";, will the t.co URL redirect like
>> this:
>>
>>   a) t.co/abcd -> domain.com
>>
>> Or like this:
>>
>>   b) t.co/abcd -> bit.ly/abcd -> domain.com
>>
>
> we're not modifying or tampering with URLs - if you send us a bit.ly link,
> we will wrap that bit.ly link.  analytics will still work, etc.
>
>
>> *3)* In the above scenario, will the 'display_url' contain '
>> http://bit.ly/abcd' or 'http://domain.com'?
>>
>
> bit.ly!
>
>
>> *4)* Why redirect all URLs, btw?  Why not just redirect the malicious
>> ones?
>>
>
> in the case of malicious URLs, you sometimes don't know it at the time of
> tweet creation.  or the URL may eventually become malicious.  this allows us
> to do shutdown after tweet creation.
>
>
>> Thanks!
>>
>
> that's what i'm here for :P
>
>
>>
>> -DeWitt
>>
>>
>>
>> On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian wrote:
>>
>>>  hi all.
>>>
>>> twitter has been wrapping links in e-mailed DMs for a couple months 
>>> now.
>>> with that feature, we're trying to protect users against phishing and other
>>> malicious attacks. the way that we're doing this is that any URL that comes
>>> through in a DM gets currently wrapped with a twt.tl URL -- if the URL
>>> turns out to be malicious, Twitter can simply shut it down, and whoever
>>> follows that link will be presented with a page that warns them of
>>> potentially malicious content. in a few weeks, we're going to start slowly
>>> enabling this throughout the API for all statuses as well, but instead of
>>> twt.tl, we will be using t.co.
>>>
>>> practically, any tweet that is sent through statuses/update that has a
>>> link on it will have the link automatically converted to a t.co link on
>>> its way through the Twitter platform. if you fetch any tweet created after
>>> this change goes live, then its text field will have all its links
>>> automatically wrapped with t.co links. when a user clicks on that link,
>>> Twitter will redirect them to the original URL after first confirming with
>>> our database that that URL is not malicious.  on top of the end-user
>>> benefit, we hope to eventually provide all developers with aggregate usage
>>> data around your applications such as the number of clicks people make on
>>> URLs you display (it will, of course, be in aggregate and not identifiable
>>> manner). additionally, we want to be able to build services and APIs that
>>> can make algorithmic recommendations to users based on the content they are
>>> consuming. gathering the data from t.co will help make these possible.
>>>
>>> our current plan is that no user will see a t.co URL on twitter.com but
>>> we still have some details to work through. the links will still be
>>> displayed as they were sent in, but the target of the link will be the
>>> t.co link instead. and, we want to provide the same ability to display
>>> original links to developers. we're going to use the entities attribute to
>>> make this possible.
>>>
>>> let's say i send out the following tweet: "you have to check out
>>> http://dev.twitter.com!";
>>>
>>> a returned (and truncated) status object may look like:
>>>
>>> {
>>>   "text" : "you have to check out http://t.co/s9gfk2d4!";,
>>>   ...
>>>   "user" : {
>>> "screen_name" : "raffi",
>>> ...
>>>   },
>>>   ...
>>>   "entities" : {
>>> "urls" : [
>>>   {
>>> "url" : "http://t.co/s9gfk2d4";,
>>> "display_url" : "http://dev.twitter.com";,
>>>   

Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[Raffi Krikorian]

that would be an awesome service!

On Tue, Jun 8, 2010 at 4:50 PM, John Barratt  wrote:

> Hi Raffi,
>
>
> On 9/06/10 8:57 AM, Raffi Krikorian wrote:
>
>> "url" : "http://t.co/s9gfk2d4";,
>> "display_url" : "http://dev.twitter.com";,
>> "indices" : [23, 43]
>>
> Any chance of getting the title of the resolved URL added in here too if
> available?
>
> Then we could display a link like :
>
> http://t.co/s9gfk2d4";>
>
>   http://dev.twitter.com
> 
>
> or :
>
> http://dev.twitter.com"; href="http://t.co/s9gfk2d4";>
>   Twitter Dev
> 
>
> This would give even more context to users, without having to follow the
> redirect path, & load and parse the page to extract it as well.
>
> Thanks,
>
> JB.
>



-- 
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

Re: [twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[Raffi Krikorian]

>
> *1)* Will the redirect from t.co -> domain.com be a 301 Moved Permanently
> or a 302 Found response?
>

301!


> *2)* Will the t.co URL redirect point to the URL in the original tweet, or
> will it point to the ultimate resolved URL?
>
> I.e., if I post "Check out my site at http://bit.ly/abcd"; where
> bit.ly/abcd redirects to domain.com, and the resultant tweet becomes
> "Check out my site at http://t.co/abcd";, will the t.co URL redirect like
> this:
>
>   a) t.co/abcd -> domain.com
>
> Or like this:
>
>   b) t.co/abcd -> bit.ly/abcd -> domain.com
>

we're not modifying or tampering with URLs - if you send us a bit.ly link,
we will wrap that bit.ly link.  analytics will still work, etc.


> *3)* In the above scenario, will the 'display_url' contain '
> http://bit.ly/abcd' or 'http://domain.com'?
>

bit.ly!


> *4)* Why redirect all URLs, btw?  Why not just redirect the malicious
> ones?
>

in the case of malicious URLs, you sometimes don't know it at the time of
tweet creation.  or the URL may eventually become malicious.  this allows us
to do shutdown after tweet creation.


> Thanks!
>

that's what i'm here for :P


>
> -DeWitt
>
>
>
> On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian  wrote:
>
>>  hi all.
>>
>> twitter has been wrapping links in e-mailed DMs for a couple months 
>> now.
>> with that feature, we're trying to protect users against phishing and other
>> malicious attacks. the way that we're doing this is that any URL that comes
>> through in a DM gets currently wrapped with a twt.tl URL -- if the URL
>> turns out to be malicious, Twitter can simply shut it down, and whoever
>> follows that link will be presented with a page that warns them of
>> potentially malicious content. in a few weeks, we're going to start slowly
>> enabling this throughout the API for all statuses as well, but instead of
>> twt.tl, we will be using t.co.
>>
>> practically, any tweet that is sent through statuses/update that has a
>> link on it will have the link automatically converted to a t.co link on
>> its way through the Twitter platform. if you fetch any tweet created after
>> this change goes live, then its text field will have all its links
>> automatically wrapped with t.co links. when a user clicks on that link,
>> Twitter will redirect them to the original URL after first confirming with
>> our database that that URL is not malicious.  on top of the end-user
>> benefit, we hope to eventually provide all developers with aggregate usage
>> data around your applications such as the number of clicks people make on
>> URLs you display (it will, of course, be in aggregate and not identifiable
>> manner). additionally, we want to be able to build services and APIs that
>> can make algorithmic recommendations to users based on the content they are
>> consuming. gathering the data from t.co will help make these possible.
>>
>> our current plan is that no user will see a t.co URL on twitter.com but
>> we still have some details to work through. the links will still be
>> displayed as they were sent in, but the target of the link will be the
>> t.co link instead. and, we want to provide the same ability to display
>> original links to developers. we're going to use the entities attribute to
>> make this possible.
>>
>> let's say i send out the following tweet: "you have to check out
>> http://dev.twitter.com!";
>>
>> a returned (and truncated) status object may look like:
>>
>> {
>>   "text" : "you have to check out http://t.co/s9gfk2d4!";,
>>   ...
>>   "user" : {
>> "screen_name" : "raffi",
>> ...
>>   },
>>   ...
>>   "entities" : {
>> "urls" : [
>>   {
>> "url" : "http://t.co/s9gfk2d4";,
>> "display_url" : "http://dev.twitter.com";,
>> "indices" : [23, 43]
>>   }
>> ],
>> ...
>>   },
>>   ...
>> }
>>
>> two things to note: the text of the returned status object doesn't have
>> the original URL and instead it has a t.co URL, and the entities block
>> now has a display_url attribute associated with it. what we're hoping is
>> that with this data, it should be relatively easy to create a UI where you
>> replace the http://t.co/s9gfk2d4 in the text with the equivalent of
>>
>> http://t.co/s9gfk2d4";>http://dev.twitter.com
>>
>> this means the user would not see the t.co link, but we all can still
>> provide the protection and gather data from the wrapped link. for the
>> applications that don't choose to update, the t.co link will be shown
>> (and the goal to protect users will be met). i just want to emphasize -- we
>> really do hope that you all render the original URL, but please send the
>> user through the t.co link.   if you do choose to prefetch all the URLs
>> on a timeline, then, when a user actually clicks on one of the links, please
>> still send him or her through t.co. We will be updating the TOS to
>> require you to check t.co and register the click.
>>
>> related to this: the way the Twitter API counts characters is g

[twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[John Barratt]

Hi Raffi,

On 9/06/10 8:57 AM, Raffi Krikorian wrote:

"url" : "http://t.co/s9gfk2d4";,
"display_url" : "http://dev.twitter.com";,
"indices" : [23, 43]
Any chance of getting the title of the resolved URL added in here too if 
available?


Then we could display a link like :

http://t.co/s9gfk2d4";>
   http://dev.twitter.com


or :

http://dev.twitter.com"; href="http://t.co/s9gfk2d4";>
   Twitter Dev


This would give even more context to users, without having to follow the 
redirect path, & load and parse the page to extract it as well.


Thanks,

JB.

[twitter-dev] Re: [twitter-api-announce] link wrapping on the API

[DeWitt Clinton]

Hi Raffi,

Interesting...  A couple of quick questions:

*1)* Will the redirect from t.co -> domain.com be a 301 Moved Permanently or
a 302 Found response?

*2)* Will the t.co URL redirect point to the URL in the original tweet, or
will it point to the ultimate resolved URL?

I.e., if I post "Check out my site at http://bit.ly/abcd"; where
bit.ly/abcdredirects to
domain.com, and the resultant tweet becomes "Check out my site at
http://t.co/abcd";, will the t.co URL redirect like this:

  a) t.co/abcd -> domain.com

Or like this:

  b) t.co/abcd -> bit.ly/abcd -> domain.com

*3)* In the above scenario, will the 'display_url' contain '
http://bit.ly/abcd' or 'http://domain.com'?

*4)* Why redirect all URLs, btw?  Why not just redirect the malicious ones?

Thanks!

-DeWitt



On Tue, Jun 8, 2010 at 3:57 PM, Raffi Krikorian  wrote:

> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months 
> now.
> with that feature, we're trying to protect users against phishing and other
> malicious attacks. the way that we're doing this is that any URL that comes
> through in a DM gets currently wrapped with a twt.tl URL -- if the URL
> turns out to be malicious, Twitter can simply shut it down, and whoever
> follows that link will be presented with a page that warns them of
> potentially malicious content. in a few weeks, we're going to start slowly
> enabling this throughout the API for all statuses as well, but instead of
> twt.tl, we will be using t.co.
>
> practically, any tweet that is sent through statuses/update that has a
> link on it will have the link automatically converted to a t.co link on
> its way through the Twitter platform. if you fetch any tweet created after
> this change goes live, then its text field will have all its links
> automatically wrapped with t.co links. when a user clicks on that link,
> Twitter will redirect them to the original URL after first confirming with
> our database that that URL is not malicious.  on top of the end-user
> benefit, we hope to eventually provide all developers with aggregate usage
> data around your applications such as the number of clicks people make on
> URLs you display (it will, of course, be in aggregate and not identifiable
> manner). additionally, we want to be able to build services and APIs that
> can make algorithmic recommendations to users based on the content they are
> consuming. gathering the data from t.co will help make these possible.
>
> our current plan is that no user will see a t.co URL on twitter.com but we
> still have some details to work through. the links will still be displayed
> as they were sent in, but the target of the link will be the t.co link
> instead. and, we want to provide the same ability to display original links
> to developers. we're going to use the entities attribute to make this
> possible.
>
> let's say i send out the following tweet: "you have to check out
> http://dev.twitter.com!";
>
> a returned (and truncated) status object may look like:
>
> {
>   "text" : "you have to check out http://t.co/s9gfk2d4!";,
>   ...
>   "user" : {
> "screen_name" : "raffi",
> ...
>   },
>   ...
>   "entities" : {
> "urls" : [
>   {
> "url" : "http://t.co/s9gfk2d4";,
> "display_url" : "http://dev.twitter.com";,
> "indices" : [23, 43]
>   }
> ],
> ...
>   },
>   ...
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has
> a display_url attribute associated with it. what we're hoping is that with
> this data, it should be relatively easy to create a UI where you replace the
> http://t.co/s9gfk2d4 in the text with the equivalent of
>
> http://t.co/s9gfk2d4";>http://dev.twitter.com
>
> this means the user would not see the t.co link, but we all can still
> provide the protection and gather data from the wrapped link. for the
> applications that don't choose to update, the t.co link will be shown (and
> the goal to protect users will be met). i just want to emphasize -- we
> really do hope that you all render the original URL, but please send the
> user through the t.co link.   if you do choose to prefetch all the URLs on
> a timeline, then, when a user actually clicks on one of the links, please
> still send him or her through t.co. We will be updating the TOS to require
> you to check t.co and register the click.
>
> related to this: the way the Twitter API counts characters is going to
> change ever so slightly. our 140 characters is now going to be defined as
> 140 characters after link wrapping. t.co links are of a predictable length
> -- they will always be 20 characters. after we make this live, it will be
> feasible to send in the text for a status that is greater than 140
> characters. the rule is after the link wrapping, the text transforms to 140
> characters or fewer. we'll be using the same logic that is in
> twitter-te