Re: [twitter-dev] Delete messages from the filter stream with location parameters??
Couldn't you add a cleaning process to statuses just before they are sent to clients but after they have been filtered into streams? The cleaning process could pick up delete flags and remove extraneous metadata from the status. Abraham On Fri, Feb 26, 2010 at 23:31, M. Edward (Ed) Borasky zzn...@gmail.comwrote: -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting Mark McBride mmcbr...@twitter.com: Yes, that's correct. We've considered adding more metadata to delete messages to make routing easier, but the privacy issues involved get tricky (if I delete something, do I *really* want the full text re-sent to a bunch of people?) Yeah - definitely tricky. The delete messages coming from sample only give the user_id and status_id, and I have to assume that the publish process doesn't send me a delete for a status that it didn't send to me. ;-) I suppose you could do the same for filter, but that would mean keeping track of all the tweets sent to *each* filter connection, not just one set of tweets like sample. That could get ugly since you can't predict / control how many filter connections you're going to get or how many tweets are going to be passed by the filter criteria. -- Abraham Williams | Community Advocate | http://abrah.am Project | Out Loud | http://outloud.labs.poseurtech.com This email is: [ ] shareable [x] ask first [ ] private.
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
A user has essentially an infinite amount of time to delete a tweet he has sent. The sequence of events is a. User sends a tweet. b. Twitter forwards it to Streaming and inserts it into the Search indexing queue if the user is not blocked. c. User says at some point later, Oh crap - did I really say that? and deletes the tweet. d. Twitter deletes the tweet from Search and its other public-facing facilities, and sends a delete message with the deleted tweet's user and status IDs out to the *unfiltered* Streaming connections. I'm assuming it knows enough not to send all the firehose deletes to lower-frequency streams - it's none of sample's business what tweets got deleted from firehose or gardenhose. ;-) -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting Abraham Williams 4bra...@gmail.com: Couldn't you add a cleaning process to statuses just before they are sent to clients but after they have been filtered into streams? The cleaning process could pick up delete flags and remove extraneous metadata from the status. Abraham On Fri, Feb 26, 2010 at 23:31, M. Edward (Ed) Borasky zzn...@gmail.comwrote: -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting Mark McBride mmcbr...@twitter.com: Yes, that's correct. We've considered adding more metadata to delete messages to make routing easier, but the privacy issues involved get tricky (if I delete something, do I *really* want the full text re-sent to a bunch of people?) Yeah - definitely tricky. The delete messages coming from sample only give the user_id and status_id, and I have to assume that the publish process doesn't send me a delete for a status that it didn't send to me. ;-) I suppose you could do the same for filter, but that would mean keeping track of all the tweets sent to *each* filter connection, not just one set of tweets like sample. That could get ugly since you can't predict / control how many filter connections you're going to get or how many tweets are going to be passed by the filter criteria. -- Abraham Williams | Community Advocate | http://abrah.am Project | Out Loud | http://outloud.labs.poseurtech.com This email is: [ ] shareable [x] ask first [ ] private.
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
We could do so, but we have a dogmatic belief that Hosebird should remain middleware and that it should not render content. Well, except for limit messages. But, other than that, no rendering. Recently, we committed a major apostasy for an experiment, and re-rendered Tweets inside Hosebird on a non-public cluster. And then, Mark and I spent a whole day chasing down intermittent character encoding bugs. And we sucked half of the Infrastructure team down with us for a while too. On a deadline. And it still doesn't work quite right. This was not a good deal. That being said, I think I know how we could work this out. (Files a bug against himself.) We'll see. -John On Sat, Feb 27, 2010 at 4:53 PM, Abraham Williams 4bra...@gmail.com wrote: Couldn't you add a cleaning process to statuses just before they are sent to clients but after they have been filtered into streams? The cleaning process could pick up delete flags and remove extraneous metadata from the status. Abraham On Fri, Feb 26, 2010 at 23:31, M. Edward (Ed) Borasky zzn...@gmail.comwrote: -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting Mark McBride mmcbr...@twitter.com: Yes, that's correct. We've considered adding more metadata to delete messages to make routing easier, but the privacy issues involved get tricky (if I delete something, do I *really* want the full text re-sent to a bunch of people?) Yeah - definitely tricky. The delete messages coming from sample only give the user_id and status_id, and I have to assume that the publish process doesn't send me a delete for a status that it didn't send to me. ;-) I suppose you could do the same for filter, but that would mean keeping track of all the tweets sent to *each* filter connection, not just one set of tweets like sample. That could get ugly since you can't predict / control how many filter connections you're going to get or how many tweets are going to be passed by the filter criteria. -- Abraham Williams | Community Advocate | http://abrah.am Project | Out Loud | http://outloud.labs.poseurtech.com This email is: [ ] shareable [x] ask first [ ] private.
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
P.S.: A refresh of the Search page for my tweets shows that the deleted tweet has been removed from Search results, as desired. -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting M. Edward (Ed) Borasky zzn...@gmail.com: I'm doing some testing with the filter stream using location bounding boxes around PDX. I can send a geocoded tweet, and it shows up in the output of filter not too long after that. But when I deleted the tweet from the web app, I never saw a delete message come down the filter pipe. Should I see one? How long should I have to wait for it?
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
Mark might have to comment on this too, but there's a metadata issue that makes this currently impossible. The deletion notice does not contain all of the Tweet metadata, just the primary key, and maybe another field or two. So, there's nothing to match to forward on to your stream. All firehose consumers will receive the delete, however, so your Tweet should be gone from search.twitter.com, Bing, Google, Yahoo, etc. etc. -John On Fri, Feb 26, 2010 at 1:18 AM, M. Edward (Ed) Borasky zzn...@gmail.comwrote: P.S.: A refresh of the Search page for my tweets shows that the deleted tweet has been removed from Search results, as desired. -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting M. Edward (Ed) Borasky zzn...@gmail.com: I'm doing some testing with the filter stream using location bounding boxes around PDX. I can send a geocoded tweet, and it shows up in the output of filter not too long after that. But when I deleted the tweet from the web app, I never saw a delete message come down the filter pipe. Should I see one? How long should I have to wait for it?
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
Yes, that's correct. We've considered adding more metadata to delete messages to make routing easier, but the privacy issues involved get tricky (if I delete something, do I *really* want the full text re-sent to a bunch of people?) ---Mark http://twitter.com/mccv On Fri, Feb 26, 2010 at 6:14 AM, John Kalucki j...@twitter.com wrote: Mark might have to comment on this too, but there's a metadata issue that makes this currently impossible. The deletion notice does not contain all of the Tweet metadata, just the primary key, and maybe another field or two. So, there's nothing to match to forward on to your stream. All firehose consumers will receive the delete, however, so your Tweet should be gone from search.twitter.com, Bing, Google, Yahoo, etc. etc. -John On Fri, Feb 26, 2010 at 1:18 AM, M. Edward (Ed) Borasky zzn...@gmail.comwrote: P.S.: A refresh of the Search page for my tweets shows that the deleted tweet has been removed from Search results, as desired. -- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting M. Edward (Ed) Borasky zzn...@gmail.com: I'm doing some testing with the filter stream using location bounding boxes around PDX. I can send a geocoded tweet, and it shows up in the output of filter not too long after that. But when I deleted the tweet from the web app, I never saw a delete message come down the filter pipe. Should I see one? How long should I have to wait for it?
Re: [twitter-dev] Delete messages from the filter stream with location parameters??
-- M. Edward (Ed) Borasky borasky-research.net/m-edward-ed-borasky/ A mathematician is a device for turning coffee into theorems. ~ Paul Erdos Quoting Mark McBride mmcbr...@twitter.com: Yes, that's correct. We've considered adding more metadata to delete messages to make routing easier, but the privacy issues involved get tricky (if I delete something, do I *really* want the full text re-sent to a bunch of people?) Yeah - definitely tricky. The delete messages coming from sample only give the user_id and status_id, and I have to assume that the publish process doesn't send me a delete for a status that it didn't send to me. ;-) I suppose you could do the same for filter, but that would mean keeping track of all the tweets sent to *each* filter connection, not just one set of tweets like sample. That could get ugly since you can't predict / control how many filter connections you're going to get or how many tweets are going to be passed by the filter criteria.
