[twitter-dev] Twitter List App Needs a New Home
We've built http://twlisted.com and the app is up and working and the database is in very good shape, but we've been sidetracked onto other (paying) projects and don't have the bandwidth to give this the attention that it needs. This is a machine driven directory of Twitter lists. Sort of like Listorious, except that there is no manual curation -- we don't edit the lists, we rank based upon user specified criteria. We've also implemented an API to query the list. There's a capability to purchase sponsored listings, but we haven't done anything with this. It's in PHP/MySql and is based upon qlWebDS Premium 5.1.0d_1DL, which we have *heavily* customized. There are several crawlers that go looking for lists via the Twitter API. We've done zero marketing of this and yet Google has indexed 1080+ pages. You might just be interested in the database, or you may actually know what you're doing and would be able to monetize something like this. Fred Wilson said that search and content would be one of the growth areas, so you'll be all set to be a player. If you're comfortable with PHP and MySQL you'll be able to take this over. Just follow the For Sale link under the ad banner. Make us an offer. Right now, $1 buys it. We'll even consider a paltry amount up front and a percentage of your earnings in the future. -- To unsubscribe, reply using remove me as the subject.
[twitter-dev] OAuth authentication challenge sent for some API but not others
I'm working on an iPhone app using xAuth (which is great!). I'm
exchanging user credentials for an access token, storing it and using
the token to make subsequent calls to the API's just fine. I'm having
trouble however with the account update_profile_image API (http://
api.twitter.com/1/account/update_profile_image.xml). When calling that
API I'm receiving a 401 [unauthroized] response with a return of:
hash
request/1/account/update_profile_image.xml/request
errorInvalid / used nonce/error
/hash
Full headers:
(401) [unauthorized]:
{
Cache-Control = no-cache, max-age=1800;
Connection = close;
Content-Encoding = gzip;
Content-Length = 142;
Content-Type = application/xml; charset=utf-8;
Date = Thu, 08 Apr 2010 15:45:12 GMT;
Expires = Thu, 08 Apr 2010 16:15:09 GMT;
Server = hi;
Set-Cookie = guest_id=1270741512433; path=/; expires=Sat, 08
May 2010 15:45:12 GMT,
_twitter_sess=BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo
%250ASGFzaHsABjoKQHVzZWR7AA%253D
%253D--1164b91ac812d853b877e93ddb612b7471bebc74; domain=.twitter.com;
path=/;
Status = 401 Unauthorized;
Vary = Accept-Encoding;
Www-Authenticate = Basic realm=\Twitter API\;
}
When I send a request to the statuses/update API (http://
api.twitter.com/1/statuses/update.xml) using the same access token it
works fine. I've also tested the same update_profile_image call in my
code with basic auth to rule out any malformed multpart form-data etc.
and that works OK.
I've noticed that when sending to the update_profile_image API I'm
receiving an authentication challenge. I don't receive the challenge
with other API calls. It's my understanding that when sending an OAuth
authentication header, a challenge should not be sent.
I'm using Ben Gottlieb's twitter OAuth library for the OAuth portion.
This library (and the other iPhone OAuth libraries) essentially ignore
any challenges by responding with
continueWithoutCredentialForAuthenticationChallenge in the challenge
delegate method. Ben's library is integrated with the MGTwitter engie
which unfortunately does not include many of the account API calls
(including update_profile_image) which is why I'm writing my own.
Should the authentication challenge be expected? If so, any
recommendation on how to respond to the challenge?
--
To unsubscribe, reply using remove me as the subject.
[twitter-dev] Attending Chirp? Let's get to know each other better, Before the Conference!
Dear Developers, I'm sure we’re a lot - in this group - to have our ticket for Chirp, and I wanted to launch a thread so we could know each other a little better, before the conference starts. The conference looks very promising on a technical side: we’re going to learn from the expert, practice during the hack day... Now, I don't know for you, but as great as it looks like, another reason that pushed me to do this 6,000 miles trip (coming from Paris), is actually to meet YOU guys! So I would love to learn more about my fellow Chirp attendees the official website is mentioning! Where are you coming from? Which projects are you working on? What are you interested in (concerning the conference and/or in general)? We’re going to be a lot there (something like 800 people if I remember well), but for a short time and in a very busy schedule. That’s why I thought it would be interesting to start the conversation, presentation and “who’s who” game before. On my side, I’m a 26 years old French Engineer and I created my own company to focus on IT consulting and (mainly on) Twitoaster (http:// twitoaster.com). It will be my first time in San Francisco, and I’d love to meet other developers to chat around Twitter projects, talk about our experiences, exchange ideas... I’ll arrive the 13th (will do my best to be at pre-chirp conf) and I’ll stay at the Orchard Hotel. Concerning the conference, I’m particularly curious about the Monetization part; and the Media Ecosystem development (but this part recently disappeared from the schedule). I’m also wondering how I’ll manage to stay awake for 48h+ :) Arnaud Meunier - http://twitter.com/twitoaster Twitoaster - http://twitoaster.com PS: For those who would have missed them, here are a couple of links about the conference: - Pre Chirp Conference: http://bit.ly/akOk0R - Hack day collaboration: http://bit.ly/cD3KAJ - Attendees list: http://bit.ly/9IeEmo http://bit.ly/byKOjS - Attendees conversations: http://bit.ly/98nvAr - Questions for the speakers: http://bit.ly/aakPUN -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] OAuth Revoke Token?
There is no API endpoint that I know of and don't think one should exist. Users should not trust thirdparties to self-revoke access to their accounts. Users should know how to do it from twitter.com via the connections page. It might be nice if we could generate a redirect link to a page on twitter.com where the user can then revoke the access (sort of like the authorization page). Josh On Wed, Apr 7, 2010 at 11:59 PM, Ryan Amos amos.r...@gmail.com wrote: Is there anyway to send a request to revoke a token completely without requiring the user goto their connections page on twitter? We allow our users to revoke access via our application, but that only revokes it on our side. The application would still show up on their twitter.com connections page. Google has one by sending a request to: https://www.google.com/accounts/accounts/AuthSubRevokeToken -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] Attending Chirp? Let's get to know each other better, Before the Conference!
Hi Arnaud All I'm also making the trip across the Atlantic with a group of Brits under the heading of the Developer Mission: http://developermission.com Please get in touch with me off list if any of you are also travelling a long way to get to San Francisco. We have a bunch of activities planned including meetups, campus tours and meals between 12th and 19th and we have room for a few more to join us. We just released the first wave of ticket's to Europe's Unofficial Twitter Developer Unconference in London on the 8th and 9th of May. They were all allocated in under an hour but we'll be releasing more soon. Details at http://warblecamp.eventbrite.com Also coming up on the evening 20th of April is the 8th Twitter Developer Nest in London. More details and free tickets available at http://twitterdevelopernest.com I'm particularly looking forward to the hackday part of Chirp and the opportunity to hear about what other people in the ecosystem are working on. I'm looking at building a one page app that we can use to get a quick daily snap shot of all the things going on in the Twitter developer ecosystem from this Google Group, to GitHub, StackOverflow and Twitter. If I get the chance I'll also be adding some refinements to a soon to be launched app I'm woking on - http://smidgn.com See you next week! Jon http://twitter.com/jot On Thu, Apr 8, 2010 at 7:13 PM, Arnaud Meunier arnaud.meun...@twitoaster.com wrote: Dear Developers, I'm sure we’re a lot - in this group - to have our ticket for Chirp, and I wanted to launch a thread so we could know each other a little better, before the conference starts. The conference looks very promising on a technical side: we’re going to learn from the expert, practice during the hack day... Now, I don't know for you, but as great as it looks like, another reason that pushed me to do this 6,000 miles trip (coming from Paris), is actually to meet YOU guys! So I would love to learn more about my fellow Chirp attendees the official website is mentioning! Where are you coming from? Which projects are you working on? What are you interested in (concerning the conference and/or in general)? We’re going to be a lot there (something like 800 people if I remember well), but for a short time and in a very busy schedule. That’s why I thought it would be interesting to start the conversation, presentation and “who’s who” game before. On my side, I’m a 26 years old French Engineer and I created my own company to focus on IT consulting and (mainly on) Twitoaster (http:// twitoaster.com). It will be my first time in San Francisco, and I’d love to meet other developers to chat around Twitter projects, talk about our experiences, exchange ideas... I’ll arrive the 13th (will do my best to be at pre-chirp conf) and I’ll stay at the Orchard Hotel. Concerning the conference, I’m particularly curious about the Monetization part; and the Media Ecosystem development (but this part recently disappeared from the schedule). I’m also wondering how I’ll manage to stay awake for 48h+ :) Arnaud Meunier - http://twitter.com/twitoaster Twitoaster - http://twitoaster.com PS: For those who would have missed them, here are a couple of links about the conference: - Pre Chirp Conference: http://bit.ly/akOk0R - Hack day collaboration: http://bit.ly/cD3KAJ - Attendees list: http://bit.ly/9IeEmo http://bit.ly/byKOjS - Attendees conversations: http://bit.ly/98nvAr - Questions for the speakers: http://bit.ly/aakPUN -- To unsubscribe, reply using remove me as the subject. -- Jonathan Markwell Engineer | Founder | Connector Inuda Innovations Ltd, Brighton, UK Web application development support Twitter Facebook integration specialists http://inuda.com Organising the world's first events for the Twitter developer Community http://TwitterDeveloperNest.com Providing a nice little place to work in the middle of Brighton - http://theskiff.org Measuring your brand's visibility on the social web - http://HowSociable.com mob: 07766 021 485 | tel: 01273 704 549 | fax: 01273 376 953 skype: jlmarkwell | twitter: http://twitter.com/jot
Re: [twitter-dev] OAuth Revoke Token?
A scenario for justifying invalidateToken: - User visits AwesomeApp and wants to connect his Twitter account - AwesomeApp redirects to Twitter's OAuth flow - User fails to notice that someone else, UserX, is already logged in to Twitter in the current browser and clicks through - AwesomeApp detects (somehow, perhaps later) that the wrong Twitter user is connected. They can be a good citizen and revoke the token completely, then send the user back through a full OAuth flow that asks for username/password regardless of sign-in state. Just my $0.02, Mike On Thu, Apr 8, 2010 at 12:06 PM, Josh Roesslein jroessl...@gmail.comwrote: There is no API endpoint that I know of and don't think one should exist. Users should not trust thirdparties to self-revoke access to their accounts. Users should know how to do it from twitter.com via the connections page. It might be nice if we could generate a redirect link to a page on twitter.com where the user can then revoke the access (sort of like the authorization page). Josh On Wed, Apr 7, 2010 at 11:59 PM, Ryan Amos amos.r...@gmail.com wrote: Is there anyway to send a request to revoke a token completely without requiring the user goto their connections page on twitter? We allow our users to revoke access via our application, but that only revokes it on our side. The application would still show up on their twitter.com connections page. Google has one by sending a request to: https://www.google.com/accounts/accounts/AuthSubRevokeToken -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
Thank you for the feedback. It's great to hear about the variety of use cases people have for the API, and in particular all the different ways people are using IDs. To alleviate some of the concerns raised in this thread we thought it would be useful to give more details about how we plan to generate IDs 1) IDs are still 64-bit integers. This should minimize any migration pains. 2) You can still sort on ID. Within a few millieconds you may get out of order results, but for most use cases this shouldn't be an issue. 3) since_id will still work (within the caveats given above). 4) We will provide a way to backfill from the streaming API. 5) You cannot use the generated ID to reverse engineer tweet velocity. Note that you can still use the streaming API to determine the rate of public statuses. Additional items of interest 1) At some point we will likely start using this as an ID for direct messages too 2) We will almost certainly open source the ID generation code, probably before we actually cut over to using it. 3) We STRONGLY suggest that you treat IDs as roughly sorted (roughly being within a few ms buckets), opaque 64-bit integers. We may need to change the scheme again at some point in the future, and want to minimize migration pains should we need to do this. Hopefully this puts you more at ease with the changes we're making. If it raises new concerns, please let us know! ---Mark http://twitter.com/mccv On Mon, Apr 5, 2010 at 4:18 PM, M. Edward (Ed) Borasky zn...@comcast.netwrote: On 04/05/2010 12:55 AM, Tim Haines wrote: This made me laugh. Hard. On Fri, Apr 2, 2010 at 6:47 AM, Dewald Pretorius dpr...@gmail.com wrote: Mark, It's extremely important where you have two bots that reply to each others' tweets. With incorrectly sorted tweets, you get conversations that look completely unnatural. On Apr 1, 1:39 pm, Mark McBride mmcbr...@twitter.com wrote: Just out of curiosity, what applications are you building that require sub-second sorting resolution for tweets? Yeah - my bot laughed too ;-) -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky A mathematician is a device for turning coffee into theorems. ~ Paul Erdős -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
On Thu, Apr 1, 2010 at 10:47 AM, Dewald Pretorius dpr...@gmail.com wrote: Mark, It's extremely important where you have two bots that reply to each others' tweets. With incorrectly sorted tweets, you get conversations that look completely unnatural. I'd love to see an example of two bots replying to each other and looking entirely natural! We all knew this sort of thing was going on, removing the pesky humans from the loop, but I always thought it was unintentional. There's a science fiction story in there somewhere. Nick -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
Re: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
On Thu, Apr 8, 2010 at 5:39 PM, Nick Arnett nick.arn...@gmail.com wrote: I'd love to see an example of two bots replying to each other and looking entirely natural! We all knew this sort of thing was going on, removing the pesky humans from the loop, but I always thought it was unintentional. There's a science fiction story in there somewhere. Do Twitterbots dream of electric sheep? -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
RE: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
What does “within the caveats given above” mean? Either since_id will work or
it won’t. It seems to me that if IDs are only in a “rough” order, since_id
won’t work—in particular, there is a possibility that paging through tweets
using since_id will completely skip over some tweets.
My concern is that, since tweets will not be serialized at the time they are
written, there will be a race condition between me making a request and users
posting new statuses. That is, I could get a response with the largest id in
the response being X that gets evaluated just before a tweet (X-1) has been
saved in the database; If so, when I issue a request with since_id=X, my
program will never see the newer tweet (X-1).
Are you going to change the implementation of the timeline methods so that they
never return a tweet with ID X until all nodes in the cluster guarantee that
they won’t create a new tweet with an ID less than X?
I implement the following logic:
1. Let LATEST start out as the earliest tweet available in the user’s
timeline.
2. Make a request with since_id={LATEST}, which returns a set of tweets T.
3. If T is empty then stop.
4. Let LATEST= max({ id(t), for all t in T}).
5. Goto 2.
Will I be guaranteed not to skip over any tweets in the timeline using this
logic? If not, what do I need to do to ensure I get them all?
Thanks,
Brian
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Mark McBride
Sent: Thursday, April 08, 2010 5:10 PM
To: twitter-development-talk@googlegroups.com
Subject: Re: [twitter-dev] Re: Upcoming changes to the way status IDs are
sequenced
Thank you for the feedback. It's great to hear about the variety of use cases
people have for the API, and in particular all the different ways people are
using IDs. To alleviate some of the concerns raised in this thread we thought
it would be useful to give more details about how we plan to generate IDs
1) IDs are still 64-bit integers. This should minimize any migration pains.
2) You can still sort on ID. Within a few millieconds you may get out of order
results, but for most use cases this shouldn't be an issue.
3) since_id will still work (within the caveats given above).
4) We will provide a way to backfill from the streaming API.
5) You cannot use the generated ID to reverse engineer tweet velocity. Note
that you can still use the streaming API to determine the rate of public
statuses.
Additional items of interest
1) At some point we will likely start using this as an ID for direct messages
too
2) We will almost certainly open source the ID generation code, probably before
we actually cut over to using it.
3) We STRONGLY suggest that you treat IDs as roughly sorted (roughly being
within a few ms buckets), opaque 64-bit integers. We may need to change the
scheme again at some point in the future, and want to minimize migration pains
should we need to do this.
Hopefully this puts you more at ease with the changes we're making. If it
raises new concerns, please let us know!
---Mark
http://twitter.com/mccv http://twitter.com/mccv
On Mon, Apr 5, 2010 at 4:18 PM, M. Edward (Ed) Borasky zn...@comcast.net
wrote:
On 04/05/2010 12:55 AM, Tim Haines wrote:
This made me laugh. Hard.
On Fri, Apr 2, 2010 at 6:47 AM, Dewald Pretorius dpr...@gmail.com wrote:
Mark,
It's extremely important where you have two bots that reply to each
others' tweets. With incorrectly sorted tweets, you get conversations
that look completely unnatural.
On Apr 1, 1:39 pm, Mark McBride mmcbr...@twitter.com wrote:
Just out of curiosity, what applications are you building that require
sub-second sorting resolution for tweets?
Yeah - my bot laughed too ;-)
--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky
A mathematician is a device for turning coffee into theorems. ~ Paul Erdős
--
To unsubscribe, reply using remove me as the subject.
[twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
This was my initial concern with the randomly generated ids that I
brought up, though I think Brian described it better than I.
It simply seems very likely that when using since_id to populate newer
tweets for the user, that some tweets will never be seen, because the
since_id of the last message received will be larger than one
generated 1ms later.
With the random generation of ids, I can see two way guarantee
delivery of all tweets in a users timeline
1. Page forwards and backwards to ensure no tweets generated at or
near the same time as the newest one did not receive a lower id. This
will be very expensive for a mobile client not to mention complicate
any refresh algorithms significantly.
2. Given that we know how IDs are generated (i.e. which bits represent
the time) we can simply over request by decrementing the since_id time
bits, by a second or two and filter out duplicates. (again, not really
ideal for mobile clients where battery life is an issue, plus it then
makes the implementation very dependent on twitters id format
remaining stable)
Please anyone explain if Brian and I are misinterpreting this as a
very real possibility of never displaying some tweets in a time line,
without changing how we request data from twitter (i.e. since_id
doesn't break)
--Naveen Ayyagari
@knight9
@SocialScope
On Apr 8, 7:01 pm, Brian Smith br...@briansmith.org wrote:
What does “within the caveats given above” mean? Either since_id will work or
it won’t. It seems to me that if IDs are only in a “rough” order, since_id
won’t work—in particular, there is a possibility that paging through tweets
using since_id will completely skip over some tweets.
My concern is that, since tweets will not be serialized at the time they are
written, there will be a race condition between me making a request and users
posting new statuses. That is, I could get a response with the largest id in
the response being X that gets evaluated just before a tweet (X-1) has been
saved in the database; If so, when I issue a request with since_id=X, my
program will never see the newer tweet (X-1).
Are you going to change the implementation of the timeline methods so that
they never return a tweet with ID X until all nodes in the cluster guarantee
that they won’t create a new tweet with an ID less than X?
I implement the following logic:
1. Let LATEST start out as the earliest tweet available in the user’s
timeline.
2. Make a request with since_id={LATEST}, which returns a set of tweets
T.
3. If T is empty then stop.
4. Let LATEST= max({ id(t), for all t in T}).
5. Goto 2.
Will I be guaranteed not to skip over any tweets in the timeline using this
logic? If not, what do I need to do to ensure I get them all?
Thanks,
Brian
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Mark McBride
Sent: Thursday, April 08, 2010 5:10 PM
To: twitter-development-talk@googlegroups.com
Subject: Re: [twitter-dev] Re: Upcoming changes to the way status IDs are
sequenced
Thank you for the feedback. It's great to hear about the variety of use
cases people have for the API, and in particular all the different ways
people are using IDs. To alleviate some of the concerns raised in this thread
we thought it would be useful to give more details about how we plan to
generate IDs
1) IDs are still 64-bit integers. This should minimize any migration pains.
2) You can still sort on ID. Within a few millieconds you may get out of
order results, but for most use cases this shouldn't be an issue.
3) since_id will still work (within the caveats given above).
4) We will provide a way to backfill from the streaming API.
5) You cannot use the generated ID to reverse engineer tweet velocity. Note
that you can still use the streaming API to determine the rate of public
statuses.
Additional items of interest
1) At some point we will likely start using this as an ID for direct messages
too
2) We will almost certainly open source the ID generation code, probably
before we actually cut over to using it.
3) We STRONGLY suggest that you treat IDs as roughly sorted (roughly being
within a few ms buckets), opaque 64-bit integers. We may need to change the
scheme again at some point in the future, and want to minimize migration
pains should we need to do this.
Hopefully this puts you more at ease with the changes we're making. If it
raises new concerns, please let us know!
---Mark
http://twitter.com/mccvhttp://twitter.com/mccv
On Mon, Apr 5, 2010 at 4:18 PM, M. Edward (Ed) Borasky zn...@comcast.net
wrote:
On 04/05/2010 12:55 AM, Tim Haines wrote:
This made me laugh. Hard.
On Fri, Apr 2, 2010 at 6:47 AM, Dewald Pretorius dpr...@gmail.com wrote:
Mark,
It's extremely important where you have two bots that reply to each
others' tweets. With incorrectly sorted tweets, you
Re: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
It's a possibility, but by no means a probability. Note that you can
mitigate this by using the newest tweet that is outside your danger zone.
For example in a sequence of tweets t1, t2 ... ti ... tn with creation
times c1, c2 ... ci ... cn and a comfort threshold e you could use since_id
from the latest ti such that c1 - ci e.
---Mark
http://twitter.com/mccv
On Thu, Apr 8, 2010 at 4:27 PM, Naveen knig...@gmail.com wrote:
This was my initial concern with the randomly generated ids that I
brought up, though I think Brian described it better than I.
It simply seems very likely that when using since_id to populate newer
tweets for the user, that some tweets will never be seen, because the
since_id of the last message received will be larger than one
generated 1ms later.
With the random generation of ids, I can see two way guarantee
delivery of all tweets in a users timeline
1. Page forwards and backwards to ensure no tweets generated at or
near the same time as the newest one did not receive a lower id. This
will be very expensive for a mobile client not to mention complicate
any refresh algorithms significantly.
2. Given that we know how IDs are generated (i.e. which bits represent
the time) we can simply over request by decrementing the since_id time
bits, by a second or two and filter out duplicates. (again, not really
ideal for mobile clients where battery life is an issue, plus it then
makes the implementation very dependent on twitters id format
remaining stable)
Please anyone explain if Brian and I are misinterpreting this as a
very real possibility of never displaying some tweets in a time line,
without changing how we request data from twitter (i.e. since_id
doesn't break)
--Naveen Ayyagari
@knight9
@SocialScope
On Apr 8, 7:01 pm, Brian Smith br...@briansmith.org wrote:
What does “within the caveats given above” mean? Either since_id will
work or it won’t. It seems to me that if IDs are only in a “rough” order,
since_id won’t work—in particular, there is a possibility that paging
through tweets using since_id will completely skip over some tweets.
My concern is that, since tweets will not be serialized at the time they
are written, there will be a race condition between me making a request and
users posting new statuses. That is, I could get a response with the largest
id in the response being X that gets evaluated just before a tweet (X-1) has
been saved in the database; If so, when I issue a request with since_id=X,
my program will never see the newer tweet (X-1).
Are you going to change the implementation of the timeline methods so
that they never return a tweet with ID X until all nodes in the cluster
guarantee that they won’t create a new tweet with an ID less than X?
I implement the following logic:
1. Let LATEST start out as the earliest tweet available in the
user’s timeline.
2. Make a request with since_id={LATEST}, which returns a set of
tweets T.
3. If T is empty then stop.
4. Let LATEST= max({ id(t), for all t in T}).
5. Goto 2.
Will I be guaranteed not to skip over any tweets in the timeline using
this logic? If not, what do I need to do to ensure I get them all?
Thanks,
Brian
From: twitter-development-talk@googlegroups.com [mailto:
twitter-development-t...@googlegroups.com] On Behalf Of Mark McBride
Sent: Thursday, April 08, 2010 5:10 PM
To: twitter-development-talk@googlegroups.com
Subject: Re: [twitter-dev] Re: Upcoming changes to the way status IDs are
sequenced
Thank you for the feedback. It's great to hear about the variety of use
cases people have for the API, and in particular all the different ways
people are using IDs. To alleviate some of the concerns raised in this
thread we thought it would be useful to give more details about how we plan
to generate IDs
1) IDs are still 64-bit integers. This should minimize any migration
pains.
2) You can still sort on ID. Within a few millieconds you may get out of
order results, but for most use cases this shouldn't be an issue.
3) since_id will still work (within the caveats given above).
4) We will provide a way to backfill from the streaming API.
5) You cannot use the generated ID to reverse engineer tweet velocity.
Note that you can still use the streaming API to determine the rate of
public statuses.
Additional items of interest
1) At some point we will likely start using this as an ID for direct
messages too
2) We will almost certainly open source the ID generation code, probably
before we actually cut over to using it.
3) We STRONGLY suggest that you treat IDs as roughly sorted (roughly
being within a few ms buckets), opaque 64-bit integers. We may need to
change the scheme again at some point in the future, and want to minimize
migration pains should we need to do this.
Hopefully this puts you more at ease with the changes we're
Re: [twitter-dev] I think I know why I'm missing from Streaming locations filter
This should be fixed now. Tweets tagged with a place should be correctly dispatched to geohose clients. ---Mark http://twitter.com/mccv On Fri, Apr 2, 2010 at 10:51 PM, M. Edward (Ed) Borasky zn...@comcast.netwrote: On 04/02/2010 09:37 PM, Mark McBride wrote: You can file an issue, but this is very nearly fixed. The update does collision detection on the bounding box for a place. This means that we may overdeliver (you may get a tweet that isn't in the place, but is within its bounding box). If you want absolute collision detection with a place you can run an additional check on the client. We think this is a reasonable balance between processing time consumed on our side and on the client side. I want as many tweets in or near the bounding box as possible, including my own. ;-) The issue is that my tweets are coming from inside the bounding box and aren't showing up in Streaming. So if it's very nearly fixed, I'll just wait. ;-) -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky A mathematician is a device for turning coffee into theorems. ~ Paul Erdős -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
[twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
Ahh, yes, your workaround is a little better than mine, but it is
still a work around and requires changes to how since_id is currently
used by what I have assume is most applications. I understand the need
for change and am willing to work around it, I can imagine the
scalability issues of trying to use a synchronized id for all tweets.
However, I wanted to be clear and feel it should be made obvious that
with this change, there is a possibility that a tweet may not be
delivered to client if the implementation of how since_id is currently
used is not updated to cover the case. I still envision the situation
as more likely than you seem to believe and figure as tweet velocity
increases, the likelihood will also increase; But I am assuming have
better data to support your viewpoint than I and shall defer.
--Naveen Ayyagari
@knight9
@SocialScope
On Apr 8, 7:37 pm, Mark McBride mmcbr...@twitter.com wrote:
It's a possibility, but by no means a probability. Note that you can
mitigate this by using the newest tweet that is outside your danger zone.
For example in a sequence of tweets t1, t2 ... ti ... tn with creation
times c1, c2 ... ci ... cn and a comfort threshold e you could use since_id
from the latest ti such that c1 - ci e.
---Mark
http://twitter.com/mccv
On Thu, Apr 8, 2010 at 4:27 PM, Naveen knig...@gmail.com wrote:
This was my initial concern with the randomly generated ids that I
brought up, though I think Brian described it better than I.
It simply seems very likely that when using since_id to populate newer
tweets for the user, that some tweets will never be seen, because the
since_id of the last message received will be larger than one
generated 1ms later.
With the random generation of ids, I can see two way guarantee
delivery of all tweets in a users timeline
1. Page forwards and backwards to ensure no tweets generated at or
near the same time as the newest one did not receive a lower id. This
will be very expensive for a mobile client not to mention complicate
any refresh algorithms significantly.
2. Given that we know how IDs are generated (i.e. which bits represent
the time) we can simply over request by decrementing the since_id time
bits, by a second or two and filter out duplicates. (again, not really
ideal for mobile clients where battery life is an issue, plus it then
makes the implementation very dependent on twitters id format
remaining stable)
Please anyone explain if Brian and I are misinterpreting this as a
very real possibility of never displaying some tweets in a time line,
without changing how we request data from twitter (i.e. since_id
doesn't break)
--Naveen Ayyagari
@knight9
@SocialScope
On Apr 8, 7:01 pm, Brian Smith br...@briansmith.org wrote:
What does “within the caveats given above” mean? Either since_id will
work or it won’t. It seems to me that if IDs are only in a “rough” order,
since_id won’t work—in particular, there is a possibility that paging
through tweets using since_id will completely skip over some tweets.
My concern is that, since tweets will not be serialized at the time they
are written, there will be a race condition between me making a request and
users posting new statuses. That is, I could get a response with the largest
id in the response being X that gets evaluated just before a tweet (X-1) has
been saved in the database; If so, when I issue a request with since_id=X,
my program will never see the newer tweet (X-1).
Are you going to change the implementation of the timeline methods so
that they never return a tweet with ID X until all nodes in the cluster
guarantee that they won’t create a new tweet with an ID less than X?
I implement the following logic:
1. Let LATEST start out as the earliest tweet available in the
user’s timeline.
2. Make a request with since_id={LATEST}, which returns a set of
tweets T.
3. If T is empty then stop.
4. Let LATEST= max({ id(t), for all t in T}).
5. Goto 2.
Will I be guaranteed not to skip over any tweets in the timeline using
this logic? If not, what do I need to do to ensure I get them all?
Thanks,
Brian
From: twitter-development-talk@googlegroups.com [mailto:
twitter-development-t...@googlegroups.com] On Behalf Of Mark McBride
Sent: Thursday, April 08, 2010 5:10 PM
To: twitter-development-talk@googlegroups.com
Subject: Re: [twitter-dev] Re: Upcoming changes to the way status IDs are
sequenced
Thank you for the feedback. It's great to hear about the variety of use
cases people have for the API, and in particular all the different ways
people are using IDs. To alleviate some of the concerns raised in this
thread we thought it would be useful to give more details about how we plan
to generate IDs
1) IDs are still 64-bit integers. This should minimize any migration
pains.
2) You can still sort on ID.
Re: [twitter-dev] I think I know why I'm missing from Streaming locations filter
On 04/08/2010 04:38 PM, Mark McBride wrote: This should be fixed now. Tweets tagged with a place should be correctly dispatched to geohose clients. ---Mark http://twitter.com/mccv Thanks!! Now all I need is for twitter.com to stop saying, Unable to locate you - Try Again ;-) Probably a Firefox glitch - I just updated to 3.6.3 -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky A mathematician is a device for turning coffee into theorems. ~ Paul Erdős -- To unsubscribe, reply using remove me as the subject.
[twitter-dev] help with xAuth
I've done exactly what the docs say to do for xAuth (http:// apiwiki.twitter.com/Twitter-REST-API-Method%3A-oauth-access_token-for- xAuth). Yet I keep getting Failed to validate oauth signature and token. The only thing the docs doesn't note is the secret key that is used to sign. Am I suppose to use the consumer secret or do I need to get a token secret as well then combine them like in oauth requests? -- To unsubscribe, reply using remove me as the subject.
RE: [twitter-dev] Re: Upcoming changes to the way status IDs are sequenced
Mark, thank you for taking the time to respond.
What is the smallest “comfort threshold” that will guarantee that we will see
all the tweets, with none skipped over and the fewest tweets returned multiple
times?
Let’s say the comfort threshold was 2 seconds. It seems to me like there could
realistically be dozens or hundreds of tweets within those two seconds in a
single timeline, and a request that used the logic you mentioned would return
an entire page (200 tweets) consisting of tweets that the application already
has; the application would be making a relatively large download, receiving
nothing useful for it, and not be able to make any progress because its
since_id would get “stuck”. This is at odds with many (most?) applications goal
in using since_id, which is to transfer as little data as possible.
It seems like a better alternative would a new parameter that says “don’t give
me any tweets that are less than X seconds old,” where X seconds is the
comfort threshold. That way, the application may lag behind by a few of
seconds, but at least it would be able to confidently page through the timeline
without excessive data transfer. Without such a mechanism, it looks like this
change will be a significant degradation of service that result in
applications’ “refresh” features becoming either unreliable or very wasteful.
But, is it realistic for applications to expect the Twitter cluster to be in
sync within 2 seconds? 10 seconds? 30 seconds? That is the part that is unclear
to me.
Thanks again,
Brian
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Mark McBride
Sent: Thursday, April 08, 2010 6:38 PM
To: twitter-development-talk@googlegroups.com
Subject: Re: [twitter-dev] Re: Upcoming changes to the way status IDs are
sequenced
It's a possibility, but by no means a probability. Note that you can mitigate
this by using the newest tweet that is outside your danger zone. For example
in a sequence of tweets t1, t2 ... ti ... tn with creation times c1, c2 ... ci
... cn and a comfort threshold e you could use since_id from the latest ti such
that c1 - ci e.
---Mark
http://twitter.com/mccv
On Thu, Apr 8, 2010 at 4:27 PM, Naveen knig...@gmail.com wrote:
This was my initial concern with the randomly generated ids that I
brought up, though I think Brian described it better than I.
It simply seems very likely that when using since_id to populate newer
tweets for the user, that some tweets will never be seen, because the
since_id of the last message received will be larger than one
generated 1ms later.
With the random generation of ids, I can see two way guarantee
delivery of all tweets in a users timeline
1. Page forwards and backwards to ensure no tweets generated at or
near the same time as the newest one did not receive a lower id. This
will be very expensive for a mobile client not to mention complicate
any refresh algorithms significantly.
2. Given that we know how IDs are generated (i.e. which bits represent
the time) we can simply over request by decrementing the since_id time
bits, by a second or two and filter out duplicates. (again, not really
ideal for mobile clients where battery life is an issue, plus it then
makes the implementation very dependent on twitters id format
remaining stable)
Please anyone explain if Brian and I are misinterpreting this as a
very real possibility of never displaying some tweets in a time line,
without changing how we request data from twitter (i.e. since_id
doesn't break)
--Naveen Ayyagari
@knight9
@SocialScope
On Apr 8, 7:01 pm, Brian Smith br...@briansmith.org wrote:
What does “within the caveats given above” mean? Either since_id will work or
it won’t. It seems to me that if IDs are only in a “rough” order, since_id
won’t work—in particular, there is a possibility that paging through tweets
using since_id will completely skip over some tweets.
My concern is that, since tweets will not be serialized at the time they are
written, there will be a race condition between me making a request and users
posting new statuses. That is, I could get a response with the largest id in
the response being X that gets evaluated just before a tweet (X-1) has been
saved in the database; If so, when I issue a request with since_id=X, my
program will never see the newer tweet (X-1).
Are you going to change the implementation of the timeline methods so that
they never return a tweet with ID X until all nodes in the cluster guarantee
that they won’t create a new tweet with an ID less than X?
I implement the following logic:
1. Let LATEST start out as the earliest tweet available in the user’s
timeline.
2. Make a request with since_id={LATEST}, which returns a set of tweets
T.
3. If T is empty then stop.
4. Let LATEST= max({ id(t), for all t in T}).
5. Goto 2.
Will I be guaranteed not to
[twitter-dev] Storing information from the API
Hi, I'm trying to a pull in a user's profile picture and use it as their profile picture on my site. Am I allowed to store the URL in my database (until the user deletes the account/removes the image)? Or are there terms in the Twitter API which suggests that I'm not allowed to store information obtained from the API? Thanks
[twitter-dev] Re: help with xAuth
On Apr 9, 2:24 am, John munz...@gmail.com wrote: The only thing the docs doesn't note is the secret key that is used to sign. Am I suppose to use the consumer secret or do I need to get a token secret as well then combine them like in oauth requests? You need to use the consumer secret at this point. There aren't any previous requests with xAuth - there's just the one to obtain the access token. Later on you'll need the token secret from the access token, of course. Let me see your request details, please. -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] help with xAuth
I've done exactly what the docs say to do for xAuth (http:// apiwiki.twitter.com/Twitter-REST-API-Method%3A-oauth-access_token-for- xAuth). Yet I keep getting Failed to validate oauth signature and token. The only thing the docs doesn't note is the secret key that is used to sign. Am I suppose to use the consumer secret or do I need to get a token secret as well then combine them like in oauth requests? To sign the xAuth request, the secret key is your consumer secret, followed by (i.e., a null token). -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- Seen on hand dryer: Push button for a message from your congressman. - -- To unsubscribe, reply using remove me as the subject.
