[twitter-dev] Re: API only shows messages from last 7 days
You're probably correct when you say that throwing more programmers at the problem is not the solution. That's not what I was suggesting ... My thought is that there may be no one at Twitter actually planning or developing a plan for historical data access, and if this is true then hiring someone with the skills and the desire to implement this in a practical manner would go a very long way towards providing people like us with a workable solution now. Having said this, I agree that in the absence of enough people in the company who can be trusted to make wise decisions and accomplish a wide variety of projects all at the same time, it ends up becoming a priority issue. When there are too few people available to actually take charge and make progress on projects like the one we've been discussing in this thread, it all comes down to priorities -- and when those priorities focus on things we do not need, the things we really want are set aside and ignored, with no progress being made. In other companies money is a significant limiting factor, but I tend to question this at Twitter given all the reports of their financial condition, so I really think it's a priority issue in Twitter's case. Now, if only someone at Twitter could see how important historical data access can be to real businesses, and how these businesses might be willing to pay for this data, then all it would take is to hire the right person to implement it. Twitter simply needs the money, the current ability to recognize the future value of such a project, and the commitment to make it happen ... and then they hire a leader who gets it done. Easier said than done of course, but there are excellent people available who can accomplish such goals when given the chance -- and the support they need from within the company of course. Then again, if these people are already working on it (as you may have suggested) then it's going to happen one of these days anyways ... :) Owkaye I don't think that adding more people to the staff at Twitter is the solution. In one startup I saw a thing posted on the refrigerator that had the adage, Adding more people to a project already behind schedule will only slow it down more. Surely for support and customer service issues having more people on the team to deal with growth is good, but I doubt throwing more programmers at it will help fix most issues. It just never seems to work that way. While many startups do tend toward younger employees (I personally think because being younger normally means that you can work a lot with minimal life impact), I'm sure that someone with a strong background would be able to get a job at Twitter if they were local to the company (or willing to move). A lot of this surely comes down to priorities inside the company. While Doug and Team want to support us developers as much as possible, much of our initial 'value' that we've offered in helping push twitter to the masses has already happened. We aren't the core business strategy, and with a fixed amount of resources and focus they aren't working to push mainly for developer access, but for standard user access. This 100% makes sense. Users are what is going to make twitter happen, not 3rd party developers. They want to provide a stable experience on both fronts, but users come first. In my private discussions with some team members, I've gotten the sense that they have good stuff in the pipeline for us and that they are working hard to make it happen. However we're only a small part of the overall strategy of a quickly growing company that is still dealing with massive growing pains which is no fault of theirs and something they are dealing with as best they can. david On Jul 28, 1:46 pm, owkaye owk...@gmail.com wrote: I'm sure others feel the same way Dave, but it looks and feels like Twitter is moving in the opposite direction. The load on a server to extract a big dataset once a month would be minimal, and both you and I can see the value in this approach. But I'm not sure the folks at Twitter do, or if they do maybe they just don't have the people who can (and will) get things like this implemented. Is a shortage of competent staff the cause of this type of problem? Even though I have the capabilities I do not have the 'resume' to get a job there and help them deal with some of this stuff, nor do I have the contacts within the Twitter organization to put a good word in for me and help me get hired so I could do good things for them. I'm 52 years old too, and my age seems to be a negative to most of the Web 2.x companies hiring these days. This is kind of a shame considering that people like me frequently have broader-based experience and insights that are sometimes lacking in younger people, and because of this we can add a lot more value in the areas of planning and structural
[twitter-dev] Re: API only shows messages from last 7 days
I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave On Jul 26, 4:11 am, Flashing Moose flashingmo...@gmail.com wrote: Hmm, then i can't use the API for this project, thx for replies guys. On 26 jul, 04:10, Dewald Pretorius dpr...@gmail.com wrote: I believe the tweet retention in Twitter Search has always been 7 days. On Jul 25, 1:18 pm, Flashing Moose flashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose
[twitter-dev] Re: API only shows messages from last 7 days
I agree with you Dave. I have several thought about new services based on searching Twitter's historical data. Unfortunately my ideas appear to be getting less and less practical. Twitter claims to have all its data stored in disk-based databases from what I understand ... yet without access to this data it is worthless. It seems to me they could allow searches of this historical data via a new History API then let us cache the results on our own servers. Most of the services I've conceived would do this infrequently -- never in real time -- and would not impact their existing cached server data because this historical data would exist on separate data storage servers ... theoretically anyways. Owkaye I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave
[twitter-dev] Re: API only shows messages from last 7 days
I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data. I'd sign agreements and pay money for such. dave On Jul 28, 12:03 pm, owkaye owk...@gmail.com wrote: I agree with you Dave. I have several thought about new services based on searching Twitter's historical data. Unfortunately my ideas appear to be getting less and less practical. Twitter claims to have all its data stored in disk-based databases from what I understand ... yet without access to this data it is worthless. It seems to me they could allow searches of this historical data via a new History API then let us cache the results on our own servers. Most of the services I've conceived would do this infrequently -- never in real time -- and would not impact their existing cached server data because this historical data would exist on separate data storage servers ... theoretically anyways. Owkaye I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave
[twitter-dev] Re: API only shows messages from last 7 days
I'm sure others feel the same way Dave, but it looks and feels like Twitter is moving in the opposite direction. The load on a server to extract a big dataset once a month would be minimal, and both you and I can see the value in this approach. But I'm not sure the folks at Twitter do, or if they do maybe they just don't have the people who can (and will) get things like this implemented. Is a shortage of competent staff the cause of this type of problem? Even though I have the capabilities I do not have the 'resume' to get a job there and help them deal with some of this stuff, nor do I have the contacts within the Twitter organization to put a good word in for me and help me get hired so I could do good things for them. I'm 52 years old too, and my age seems to be a negative to most of the Web 2.x companies hiring these days. This is kind of a shame considering that people like me frequently have broader-based experience and insights that are sometimes lacking in younger people, and because of this we can add a lot more value in the areas of planning and structural development than people half our age. Our coding skills are honed after so many years of experience too, not to mention the thousands of code snippets we have collected over the years to contribute to making us even faster. But since jobs like this are basically not open to me and many other folks my age, my alternative is to remain self- employed and try to build something on top of their existing available source data and API's ... and then deal with the issues and frustrations created when building a service on top of a 'moving target' that sometimes seems to be moving in funny directions. I hear about Twitter having lots of money to work with, and I'm probably wrong here but it almost seems like there's too little of this money being dedicated to paying new talent with long term views of some of these issues, and who will implement wise policies to help support and encourage rapid growth in the areas that are lacking. But once again this might just be due to a shortage of the right staff. Obviously we cannot do anything from the outside except point out these issues and ask questions, or beg and plead for changes, but it sure would be great if a few of us could actually get in there as employees and implement a couple of the new features we really need -- such as a new Historical Search API for example. Then developers like you and I could proceed with some of our plans now, instead of months or years from now ... or possibly never. I would love to lead a team on a project like this, or even be one of its members, but until it happens I'll focus on building my own little space in the Twitter universe and continue to hope for the best. :) Owkaye I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data. I'd sign agreements and pay money for such. dave On Jul 28, 12:03 pm, owkaye owk...@gmail.com wrote: I agree with you Dave. I have several thought about new services based on searching Twitter's historical data. Unfortunately my ideas appear to be getting less and less practical. Twitter claims to have all its data stored in disk-based databases from what I understand ... yet without access to this data it is worthless. It seems to me they could allow searches of this historical data via a new History API then let us cache the results on our own servers. Most of the services I've conceived would do this infrequently -- never in real time -- and would not impact their existing cached server data because this historical data would exist on separate data storage servers ... theoretically anyways. Owkaye I am a bit concerned. I remember at one point it being between 30-45 days. Now it seems to be getting smaller by about 1-day per month. Last month it was closer to 10 days. Is it basically going to keep getting smaller and smaller until we get V2 of the API, or will we be forced to all use only streaming services and then locally cache everything that we'd want to search for any time period? I know there are a LOT of problems inherent in the massive scaling out of Twitter, and this is just a symptom of them- but at the same time I can only imagine how unusable Google would be if you only had a 7-day window to Search in, and couldn't get any content made prior to that. Very worried about this soon being a 2-3 day window. dave
[twitter-dev] Re: API only shows messages from last 7 days
I don't think that adding more people to the staff at Twitter is the solution. In one startup I saw a thing posted on the refrigerator that had the adage, Adding more people to a project already behind schedule will only slow it down more. Surely for support and customer service issues having more people on the team to deal with growth is good, but I doubt throwing more programmers at it will help fix most issues. It just never seems to work that way. While many startups do tend toward younger employees (I personally think because being younger normally means that you can work a lot with minimal life impact), I'm sure that someone with a strong background would be able to get a job at Twitter if they were local to the company (or willing to move). A lot of this surely comes down to priorities inside the company. While Doug and Team want to support us developers as much as possible, much of our initial 'value' that we've offered in helping push twitter to the masses has already happened. We aren't the core business strategy, and with a fixed amount of resources and focus they aren't working to push mainly for developer access, but for standard user access. This 100% makes sense. Users are what is going to make twitter happen, not 3rd party developers. They want to provide a stable experience on both fronts, but users come first. In my private discussions with some team members, I've gotten the sense that they have good stuff in the pipeline for us and that they are working hard to make it happen. However we're only a small part of the overall strategy of a quickly growing company that is still dealing with massive growing pains which is no fault of theirs and something they are dealing with as best they can. david On Jul 28, 1:46 pm, owkaye owk...@gmail.com wrote: I'm sure others feel the same way Dave, but it looks and feels like Twitter is moving in the opposite direction. The load on a server to extract a big dataset once a month would be minimal, and both you and I can see the value in this approach. But I'm not sure the folks at Twitter do, or if they do maybe they just don't have the people who can (and will) get things like this implemented. Is a shortage of competent staff the cause of this type of problem? Even though I have the capabilities I do not have the 'resume' to get a job there and help them deal with some of this stuff, nor do I have the contacts within the Twitter organization to put a good word in for me and help me get hired so I could do good things for them. I'm 52 years old too, and my age seems to be a negative to most of the Web 2.x companies hiring these days. This is kind of a shame considering that people like me frequently have broader-based experience and insights that are sometimes lacking in younger people, and because of this we can add a lot more value in the areas of planning and structural development than people half our age. Our coding skills are honed after so many years of experience too, not to mention the thousands of code snippets we have collected over the years to contribute to making us even faster. But since jobs like this are basically not open to me and many other folks my age, my alternative is to remain self- employed and try to build something on top of their existing available source data and API's ... and then deal with the issues and frustrations created when building a service on top of a 'moving target' that sometimes seems to be moving in funny directions. I hear about Twitter having lots of money to work with, and I'm probably wrong here but it almost seems like there's too little of this money being dedicated to paying new talent with long term views of some of these issues, and who will implement wise policies to help support and encourage rapid growth in the areas that are lacking. But once again this might just be due to a shortage of the right staff. Obviously we cannot do anything from the outside except point out these issues and ask questions, or beg and plead for changes, but it sure would be great if a few of us could actually get in there as employees and implement a couple of the new features we really need -- such as a new Historical Search API for example. Then developers like you and I could proceed with some of our plans now, instead of months or years from now ... or possibly never. I would love to lead a team on a project like this, or even be one of its members, but until it happens I'll focus on building my own little space in the Twitter universe and continue to hope for the best. :) Owkaye I would do anything (including paying good amounts of money) to be able to purchase access to older datasets that I could transfer to my database through non-rest-api methods. I'm envisioning being able to download a CSV or SQL file that I could merge with my database easily, but only have to make a single request to the server to get a month of data.
[twitter-dev] Re: API only shows messages from last 7 days
It was at least a month when Twitter acquired Summize. Abraham 2009/7/25 Dewald Pretorius dpr...@gmail.com I believe the tweet retention in Twitter Search has always been 7 days. On Jul 25, 1:18 pm, Flashing Moose flashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose -- Abraham Williams | Community Evangelist | http://web608.org Hacker | http://abrah.am | http://twitter.com/abraham Project | http://fireeagle.labs.poseurtech.com This email is: [ ] blogable [x] ask first [ ] private.
[twitter-dev] Re: API only shows messages from last 7 days
Hmm, then i can't use the API for this project, thx for replies guys. On 26 jul, 04:10, Dewald Pretorius dpr...@gmail.com wrote: I believe the tweet retention in Twitter Search has always been 7 days. On Jul 25, 1:18 pm, Flashing Moose flashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose
[twitter-dev] Re: API only shows messages from last 7 days
Hi Moose, The documentation may be a bit out-dated. Right now the limit for all searches is pretty much 7 days b/c of performance/storage reasons. They are working on extending that window, but there is not a specific date for having that accomplished. -Chad On Sat, Jul 25, 2009 at 12:18 PM, Flashing Mooseflashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose
[twitter-dev] Re: API only shows messages from last 7 days
I believe the tweet retention in Twitter Search has always been 7 days. On Jul 25, 1:18 pm, Flashing Moose flashingmo...@gmail.com wrote: Hello, having some trouble with the API because only the messages from the last 7 days show up: example: feed://search.twitter.com/search.atom?q=from%3AstimulusHome Yes, there must be older posts in there... but how do i get to them? I read about the Operator Limits: filter:links operator: results are limited to 7 days source: operator: results are limited to 7 days queries must contain at least one keyword parameter with the source: operator lang= operator: results are limited to 7 days location operator: results are limited to 7 days but i'm not using filter, source, language or location do i? regards, Moose
