I'm sure others feel the same way Dave, but it looks and 
feels like Twitter is moving in the opposite direction.  

The load on a server to extract a big dataset once a month 
would be minimal, and both you and I can see the value in 
this approach. But I'm not sure the folks at Twitter do, or 
if they do maybe they just don't have the people who can 
(and will) get things like this implemented.  Is a shortage 
of competent staff the cause of this type of problem?

Even though I have the capabilities I do not have the 
'resume' to get a job there and help them deal with some of 
this stuff, nor do I have the contacts within the Twitter 
organization to put a good word in for me and help me get 
hired so I could do good things for them. 

I'm 52 years old too, and my age seems to be a negative to 
most of the Web 2.x companies hiring these days.  This is 
kind of a shame considering that people like me frequently 
have broader-based experience and insights that are 
sometimes lacking in younger people, and because of this we 
can add a lot more value in the areas of planning and 
structural development than people half our age.  Our coding 
skills are honed after so many years of experience too, not 
to mention the thousands of code snippets we have collected 
over the years to contribute to making us even faster.

But since jobs like this are basically not open to me and 
many other folks my age, my alternative is to remain self-
employed and try to build something on top of their existing 
available source data and API's ... and then deal with the 
issues and frustrations created when building a service on 
top of a 'moving target' that sometimes seems to be moving 
in funny directions.

I hear about Twitter having lots of money to work with, and 
I'm probably wrong here but it almost seems like there's too 
little of this money being dedicated to paying new talent 
with long term views of some of these issues, and who will 
implement wise policies to help support and encourage rapid 
growth in the areas that are lacking.  But once again this 
might just be due to a shortage of the right staff.

Obviously we cannot do anything from the outside except 
point out these issues and ask questions, or beg and plead 
for changes, but it sure would be great if a few of us could 
actually get in there as employees and implement a couple of 
the new features we really need -- such as a new Historical 
Search API for example.  Then developers like you and I 
could proceed with some of our plans now, instead of months 
or years from now ... or possibly never.  I would love to 
lead a team on a project like this, or even be one of its 
members, but until it happens I'll focus on building my own 
little space in the Twitter universe and continue to hope 
for the best.



> I would do anything (including paying good amounts of
> money) to be able to purchase access to older datasets
> that I could transfer to my database through non-rest-api
> methods. I'm envisioning being able to download a CSV or
> SQL file that I could merge with my database easily, but
> only have to make a single request to the server to get a
> month of data. I'd sign agreements and pay money for
> such.
> dave
> On Jul 28, 12:03 pm, owkaye <owk...@gmail.com> wrote:
> > I agree with you Dave.  I have several thought about
> > new services based on searching Twitter's historical
> > data. Unfortunately my ideas appear to be getting less
> > and less practical.
> >
> > Twitter claims to have all its data stored in
> > disk-based databases from what I understand ... yet
> > without access to this data it is worthless.
> >
> > It seems to me they could allow searches of this
> > historical data via a new "History API" then let us
> > cache the results on our own servers.  Most of the
> > services I've conceived would do this infrequently --
> > never in real time -- and would not impact their
> > existing "cached" server data because this historical
> > data would exist on separate data storage servers ...
> > theoretically anyways.
> >
> > Owkaye
> >
> > > I am a bit concerned. I remember at one point it
> > > being between 30-45 days. Now it seems to be getting
> > > smaller by about 1-day per month. Last month it was
> > > closer to 10 days.
> > >
> > > Is it basically going to keep getting smaller and
> > > smaller until we get V2 of the API, or will we be
> > > forced to all use only streaming services and then
> > > locally cache everything that we'd want to search for
> > > any time period?
> > >
> > > I know there are a LOT of problems inherent in the
> > > massive scaling out of Twitter, and this is just a
> > > symptom of them- but at the same time I can only
> > > imagine how unusable Google would be if you only had
> > > a 7-day window to Search in, and couldn't get any
> > > content made prior to that. Very worried about this
> > > soon being a 2-3 day window.
> > >
> > > dave

Reply via email to