[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread jmathai

I like Abraham's idea:
Twitter could add: next_cursor_string:1314614526448841129

Or a general return_as_string parameter which would return all values
as strings.  This would solve the problem for future values that might
exceed the max int.  There are going to be a lot of devs using PHP
that don't have the option to switch to 64 bit or make any php.ini
changes.


On Sep 24, 12:29 pm, Dewald Pretorius dpr...@gmail.com wrote:
 That magical maximum number appears to be 1 (1.0E+12).

 So, for tweet ids we still have a bit of breathing space.

 Dewald

 On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:

  Clearly PHP_INT_MAX plays no role in json_decode.

  There must be some other mystical maximum number above which it
  represents the number as float in the decoded data.

  Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Scott Haneda


If you are on 32 bit, what php.ini changes are you referring to that  
would be beneficial?


--
Scott
Iphone says hello.

On Sep 25, 2009, at 10:35 AM, jmathai jmat...@gmail.com wrote:



I like Abraham's idea:

Twitter could add: next_cursor_string:1314614526448841129


Or a general return_as_string parameter which would return all values
as strings.  This would solve the problem for future values that might
exceed the max int.  There are going to be a lot of devs using PHP
that don't have the option to switch to 64 bit or make any php.ini
changes.


On Sep 24, 12:29 pm, Dewald Pretorius dpr...@gmail.com wrote:

That magical maximum number appears to be 1 (1.0E+12).

So, for tweet ids we still have a bit of breathing space.

Dewald

On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:


Clearly PHP_INT_MAX plays no role in json_decode.



There must be some other mystical maximum number above which it
represents the number as float in the decoded data.



Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread David Fisher

I'm sorry, but the problem isn't Twitter- its your language and JSON
parser. Outputting everything as a string, when it clearly should be a
number, is inefficient and crazy.

Saying that startups can't afford 64 bit processors in systems is
crazy. Most startups I know are running on EC2 or have fairly new
hardware. I bought a killer 64-bit quad xeon server for less than
$1,500 for our startup and its rocking. If your startup doesn't have
$1,500 for a primary capital computing expense that's another problem
you have there.

You either need to run on a newer system or use a language that can
properly handle 64-bit numbers. C, Python, Ruby, Scala, Erlang, C#,
etc none of them have problems with 64-bit ints.

Not a Twitter problem. It's a programming issue on your end and
unfortunately I can't help as I don't know PHP in depth.

-dave

On Sep 24, 3:29 pm, Dewald Pretorius dpr...@gmail.com wrote:
 That magical maximum number appears to be 1 (1.0E+12).

 So, for tweet ids we still have a bit of breathing space.

 Dewald

 On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:



  Clearly PHP_INT_MAX plays no role in json_decode.

  There must be some other mystical maximum number above which it
  represents the number as float in the decoded data.

  Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread jmathai

I don't think anyone's saying it's Twitter's problem (at least I hope
not).  It's the same issue as supporting Internet Explorer.  You can
bitch and complain all you want about IE not supporting standards but
if it's the most popular (or one of the most popular) browsers that
are in use --- then practical beats principle.

My suggestion was to return everything as a string if that's how it's
being requested.  It's a safe backwards compatible addition that
doesn't affect anyone that doesn't run into this problem.

If your scope is limited to what startups can or can't afford, then
that's really narrow.  This applies to anyone consuming the API be
than on a laptop for development or anything else.

On Sep 25, 12:23 pm, David Fisher tib...@gmail.com wrote:
 I'm sorry, but the problem isn't Twitter- its your language and JSON
 parser. Outputting everything as a string, when it clearly should be a
 number, is inefficient and crazy.

 Saying that startups can't afford 64 bit processors in systems is
 crazy. Most startups I know are running on EC2 or have fairly new
 hardware. I bought a killer 64-bit quad xeon server for less than
 $1,500 for our startup and its rocking. If your startup doesn't have
 $1,500 for a primary capital computing expense that's another problem
 you have there.

 You either need to run on a newer system or use a language that can
 properly handle 64-bit numbers. C, Python, Ruby, Scala, Erlang, C#,
 etc none of them have problems with 64-bit ints.

 Not a Twitter problem. It's a programming issue on your end and
 unfortunately I can't help as I don't know PHP in depth.

 -dave

 On Sep 24, 3:29 pm, Dewald Pretorius dpr...@gmail.com wrote:

  That magical maximum number appears to be 1 (1.0E+12).

  So, for tweet ids we still have a bit of breathing space.

  Dewald

  On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:

   Clearly PHP_INT_MAX plays no role in json_decode.

   There must be some other mystical maximum number above which it
   represents the number as float in the decoded data.

   Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Scott Haneda


I tend to agree, a MacMini, around 600.00, would build php out as 64  
bit no issue.  If you are a baby startup, that should really cover you  
and be able to handle a lot more load than you would think.  You may  
have to pop your database on something more robust, but it works quite  
well as a php front end.


Take it to 1000.00 or so, and you can built out your own 1U server  
that is going to hold it's own very very well.  Or, rent a 1U, they  
are a few hundred bucks a month, you get to pick the OS, pick the  
bit'ness, etc.


If you are a startup, on a shoestring, part of the fun is being  
creative, and making due with what you have.


And don't laugh at that MacMini, you can easily get to the SATA port,  
and plug it right into a SATA array elsewhere, getting SATA II on full  
size dives.  I just pegged the drives to a cork board, its not pretty,  
but it works.  I have a second on standby in case that one craps.  I  
just had them laying around, and made them up from parts.


While I am not going to mention the site, I can tell you, it is php on  
the front, apache2 with a good deal of mod_rewriting going on, and  
substantial database work, all happening on the MacMini, handling  
500,000+ average unique visitors, which is a lot more requests when  
you add in images and the rest. (this site is not twitter related)


If I was sloppy, that database would fall on it's face under that  
load.  I have added in some caching where needed, so pages are  
rendered to disk at the expense of a php include of the data.


I am in this thread to learn about the issue, but so far, getting to  
64 bits should not be a barrier.  Isn't this really a php issue  
anyway?  Why can't php handle a 64 bit int in a 32 bit system?  Maybe  
they need a bug report or a patch.  MySql seems to handle 64 bit on my  
32 bit test system here just fine.  Maybe someone can write a wrapper  
for the json encode/decode that fixes this in that part of php?

--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 12:23 PM, David Fisher wrote:


Saying that startups can't afford 64 bit processors in systems is
crazy. Most startups I know are running on EC2 or have fairly new
hardware. I bought a killer 64-bit quad xeon server for less than
$1,500 for our startup and its rocking. If your startup doesn't have
$1,500 for a primary capital computing expense that's another problem
you have there.




[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread jmathai

It's ridiculous to suggest a change in hardware (64 bit) or software
(switch from PHP) to use Twitter's API.  It's not like either of these
are archaic.  It sucks, sure, but it's silly to suggest such a
solution.

BTW, I don't have this problem. I'm just trying to be the voice of
reason.

On Sep 25, 1:55 pm, Scott Haneda talkli...@newgeo.com wrote:
 I tend to agree, a MacMini, around 600.00, would build php out as 64  
 bit no issue.  If you are a baby startup, that should really cover you  
 and be able to handle a lot more load than you would think.  You may  
 have to pop your database on something more robust, but it works quite  
 well as a php front end.

 Take it to 1000.00 or so, and you can built out your own 1U server  
 that is going to hold it's own very very well.  Or, rent a 1U, they  
 are a few hundred bucks a month, you get to pick the OS, pick the  
 bit'ness, etc.

 If you are a startup, on a shoestring, part of the fun is being  
 creative, and making due with what you have.

 And don't laugh at that MacMini, you can easily get to the SATA port,  
 and plug it right into a SATA array elsewhere, getting SATA II on full  
 size dives.  I just pegged the drives to a cork board, its not pretty,  
 but it works.  I have a second on standby in case that one craps.  I  
 just had them laying around, and made them up from parts.

 While I am not going to mention the site, I can tell you, it is php on  
 the front, apache2 with a good deal of mod_rewriting going on, and  
 substantial database work, all happening on the MacMini, handling  
 500,000+ average unique visitors, which is a lot more requests when  
 you add in images and the rest. (this site is not twitter related)

 If I was sloppy, that database would fall on it's face under that  
 load.  I have added in some caching where needed, so pages are  
 rendered to disk at the expense of a php include of the data.

 I am in this thread to learn about the issue, but so far, getting to  
 64 bits should not be a barrier.  Isn't this really a php issue  
 anyway?  Why can't php handle a 64 bit int in a 32 bit system?  Maybe  
 they need a bug report or a patch.  MySql seems to handle 64 bit on my  
 32 bit test system here just fine.  Maybe someone can write a wrapper  
 for the json encode/decode that fixes this in that part of php?
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 12:23 PM, David Fisher wrote:

  Saying that startups can't afford 64 bit processors in systems is
  crazy. Most startups I know are running on EC2 or have fairly new
  hardware. I bought a killer 64-bit quad xeon server for less than
  $1,500 for our startup and its rocking. If your startup doesn't have
  $1,500 for a primary capital computing expense that's another problem
  you have there.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Cameron Kaiser

 You either need to run on a newer system or use a language that can
 properly handle 64-bit numbers. C, Python, Ruby, Scala, Erlang, C#,
 etc none of them have problems with 64-bit ints.

Heck, Perl does it fine. I run TTYtter on a 32-bit system that runs a Perl
that's over 10 years old, and it's perfectly happy with bigints.

-- 
 personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com
-- The less we know, the better we feel. -- David Bowie, Miracle Goodnight --


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Scott Haneda


I would not change either.  But there are those here that are stating  
they need new hardware to work around this issue, and that they can  
not afford that.  I was trying to be that voice of reason if that is  
the road/excuse they are choosing to go.


There seem to be acceptable workarounds, solid proposed workarounds,  
etc.  I guess I am not getting it, JSON is just a string returned,  
yes, it can represent type of data, but it is still just a string.  I  
can not see it being that huge a performance hit to massage that  
string a bit once you get ahold of it.

--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 2:02 PM, jmathai wrote:


It's ridiculous to suggest a change in hardware (64 bit) or software
(switch from PHP) to use Twitter's API.  It's not like either of these
are archaic.  It sucks, sure, but it's silly to suggest such a
solution.

BTW, I don't have this problem. I'm just trying to be the voice of
reason.




[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Chad Etzel

On Fri, Sep 25, 2009 at 5:16 PM, Scott Haneda talkli...@newgeo.com wrote:

 I can not see it
 being that huge a performance hit to massage that string a bit once you get
 ahold of it.

Precisely. If you really want to turn the numbers into strings in PHP,
here are 2 workaround examples. Please note, I use PHP personally, so
I am not apathetic to this issue. There are programmatic ways of
dealing with it, and that's half the fun of programming, right?

Example 1 (cursor specific example);
?
/* normally use curl, but using file_get_contents for simplicity of example */
$json = 
file_get_contents(http://twitter.com/followers/ids/barackobama.json?cursor=-1;);

echo $json;

$pattern = /\next_cursor\:([0-9]+),/;
$replace = \next_cursor\:\$1\,;

$new_json = preg_replace($pattern, $replace, $json);

$pattern = /\previous_cursor\:([0-9]+),/;
$replace = \previous_cursor\:\$1\,;

/* note $new_json in third parameter */
$new_json = preg_replace($pattern, $replace, $new_json);

echo $new_json;


var_dump(json_decode($new_json));

?


Example 2 (generalized json naked numbers example):
?

/* generic helper function to turn all number types in json into strings */
function stringify_json_numbers($json)
{
  $pattern = /([:[]?)([-]?[0-9]+)([,}\]])/;
  $replace = $1\$2\$3;

  $new_json = preg_replace($pattern, $replace, $json);

  return $new_json;

}

/* normally use curl, but using file_get_contents for simplicity of example */
$json = 
file_get_contents(http://twitter.com/followers/ids/barackobama.json?cursor=-1;);

echo $json;

$new_json = stringify_json_numbers($json);


echo $new_json;

echo pre;
var_dump(json_decode($new_json));
echo /pre;
?

Can't we all just get along?

-Chad


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Richard

Can this not be returned as hex or base64?
It would save bandwidth for Twitter (and us) and make it a string
people could convert it to 64bit int if they still want to.

On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:
 I would not change either.  But there are those here that are stating  
 they need new hardware to work around this issue, and that they can  
 not afford that.  I was trying to be that voice of reason if that is  
 the road/excuse they are choosing to go.

 There seem to be acceptable workarounds, solid proposed workarounds,  
 etc.  I guess I am not getting it, JSON is just a string returned,  
 yes, it can represent type of data, but it is still just a string.  I  
 can not see it being that huge a performance hit to massage that  
 string a bit once you get ahold of it.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 2:02 PM, jmathai wrote:

  It's ridiculous to suggest a change in hardware (64 bit) or software
  (switch from PHP) to use Twitter's API.  It's not like either of these
  are archaic.  It sucks, sure, but it's silly to suggest such a
  solution.

  BTW, I don't have this problem. I'm just trying to be the voice of
  reason.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread JDG
and it would also break everyone who CAN handle 64 bit ints and expects
results in decimal numeric format.

On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:


 Can this not be returned as hex or base64?
 It would save bandwidth for Twitter (and us) and make it a string
 people could convert it to 64bit int if they still want to.

 On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:
  I would not change either.  But there are those here that are stating
  they need new hardware to work around this issue, and that they can
  not afford that.  I was trying to be that voice of reason if that is
  the road/excuse they are choosing to go.
 
  There seem to be acceptable workarounds, solid proposed workarounds,
  etc.  I guess I am not getting it, JSON is just a string returned,
  yes, it can represent type of data, but it is still just a string.  I
  can not see it being that huge a performance hit to massage that
  string a bit once you get ahold of it.
  --
  Scott * If you contact me off list replace talklists@ with scott@ *
 
  On Sep 25, 2009, at 2:02 PM, jmathai wrote:
 
   It's ridiculous to suggest a change in hardware (64 bit) or software
   (switch from PHP) to use Twitter's API.  It's not like either of these
   are archaic.  It sucks, sure, but it's silly to suggest such a
   solution.
 
   BTW, I don't have this problem. I'm just trying to be the voice of
   reason.




-- 
Internets. Serious business.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Dewald Pretorius

I think the point that some folks are trying to make is that forcing
consumers of the API to use 64-bit machines is stifling innovation.

Many Twitter apps start as pet projects hosted on a shared HostGator
or other provider's server for nine bucks a month. That's how mine
started way back. You don't have the option of editing php.ini or
running on a 64-bit machine, unless you want to reduce your groceries
per month and fork out more for a server.

Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread jmathai

Dewald sums it up great.  I'm not affected by this issue but keeping
the barrier of entry low is of high value.  It sucks that some PHP
installations have this issue, but if there's a good way to accomodate
such a popular language then what's the harm?

This brings up an issue though.  I'm not in favor of changing the
default int to a string.  It's the default that's important.  Adding
in an override parameter isn't really anymore helpful than suggesting
to parse the json once you get it.  Except, the thought of pre-parsing
json (no matter how simple) makes my stomach churn.

Think about it from the perspective of a developer that's just
starting with the API.  What would trip them up?  If there's a way to
resolve that, then that should be a high priority.

For me it's not a big deal since I've used it enough and know people
in the API community and this group where I can get an answer.  That's
a luxury I didn't have when I started though.

So far, two proposed solutions:
 * Abraham suggested an additional key (this is great because you see
it in the response without having to know of an additional paramter)
 * I suggested an additional parameter which would return all values
as a string (drawback is that this is a bit more hidden - not
everyone RTM :( )
 * Your suggestion here!

Think about the children! :)

On Sep 25, 6:48 pm, Dewald Pretorius dpr...@gmail.com wrote:
 I think the point that some folks are trying to make is that forcing
 consumers of the API to use 64-bit machines is stifling innovation.

 Many Twitter apps start as pet projects hosted on a shared HostGator
 or other provider's server for nine bucks a month. That's how mine
 started way back. You don't have the option of editing php.ini or
 running on a 64-bit machine, unless you want to reduce your groceries
 per month and fork out more for a server.

 Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Scott Haneda


Why is the API not versioned then? api.twitter.com/?v=1,  
api.twitter.com/?v=1.1, api.twitter.com/?v=1.2 etc


Or, if that is too much maintenance, how about
api.twitter.com/?bitfix=32 or whatever.
--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 6:40 PM, JDG wrote:

and it would also break everyone who CAN handle 64 bit ints and  
expects

results in decimal numeric format.

On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:



Can this not be returned as hex or base64?
It would save bandwidth for Twitter (and us) and make it a string
people could convert it to 64bit int if they still want to.

On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:
I would not change either.  But there are those here that are  
stating

they need new hardware to work around this issue, and that they can
not afford that.  I was trying to be that voice of reason if that is
the road/excuse they are choosing to go.

There seem to be acceptable workarounds, solid proposed workarounds,
etc.  I guess I am not getting it, JSON is just a string returned,
yes, it can represent type of data, but it is still just a  
string.  I

can not see it being that huge a performance hit to massage that
string a bit once you get ahold of it.
--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 2:02 PM, jmathai wrote:

It's ridiculous to suggest a change in hardware (64 bit) or  
software
(switch from PHP) to use Twitter's API.  It's not like either of  
these

are archaic.  It sucks, sure, but it's silly to suggest such a
solution.



BTW, I don't have this problem. I'm just trying to be the voice of
reason.






[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Dewald Pretorius

I actually like the idea of versioning the API. Having an API version
that returns all values as strings can be as simple on the Twitter
side as an IF statement in the final stages of constructing the JSON
output that just converts all numerical values to strings before the
JSON encode.

Dewald

On Sep 25, 10:57 pm, jmathai jmat...@gmail.com wrote:
 Dewald sums it up great.  I'm not affected by this issue but keeping
 the barrier of entry low is of high value.  It sucks that some PHP
 installations have this issue, but if there's a good way to accomodate
 such a popular language then what's the harm?

 This brings up an issue though.  I'm not in favor of changing the
 default int to a string.  It's the default that's important.  Adding
 in an override parameter isn't really anymore helpful than suggesting
 to parse the json once you get it.  Except, the thought of pre-parsing
 json (no matter how simple) makes my stomach churn.

 Think about it from the perspective of a developer that's just
 starting with the API.  What would trip them up?  If there's a way to
 resolve that, then that should be a high priority.

 For me it's not a big deal since I've used it enough and know people
 in the API community and this group where I can get an answer.  That's
 a luxury I didn't have when I started though.

 So far, two proposed solutions:
  * Abraham suggested an additional key (this is great because you see
 it in the response without having to know of an additional paramter)
  * I suggested an additional parameter which would return all values
 as a string (drawback is that this is a bit more hidden - not
 everyone RTM :( )
  * Your suggestion here!

 Think about the children! :)


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Jim Renkel

+1. For this and other reasons the API should be versioned. 

Jim Renkel

-Original Message-
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Scott
Haneda
Sent: Friday, September 25, 2009 21:28
To: twitter-development-talk@googlegroups.com
Subject: [twitter-dev] Re: SERIOUS Problem With Cursors In JSON
Followers/Friends Ids


Why is the API not versioned then? api.twitter.com/?v=1,  
api.twitter.com/?v=1.1, api.twitter.com/?v=1.2 etc

Or, if that is too much maintenance, how about
api.twitter.com/?bitfix=32 or whatever.
-- 
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 6:40 PM, JDG wrote:

 and it would also break everyone who CAN handle 64 bit ints and  
 expects
 results in decimal numeric format.

 On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:


 Can this not be returned as hex or base64?
 It would save bandwidth for Twitter (and us) and make it a string
 people could convert it to 64bit int if they still want to.

 On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:
 I would not change either.  But there are those here that are  
 stating
 they need new hardware to work around this issue, and that they can
 not afford that.  I was trying to be that voice of reason if that is
 the road/excuse they are choosing to go.

 There seem to be acceptable workarounds, solid proposed workarounds,
 etc.  I guess I am not getting it, JSON is just a string returned,
 yes, it can represent type of data, but it is still just a  
 string.  I
 can not see it being that huge a performance hit to massage that
 string a bit once you get ahold of it.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 2:02 PM, jmathai wrote:

 It's ridiculous to suggest a change in hardware (64 bit) or  
 software
 (switch from PHP) to use Twitter's API.  It's not like either of  
 these
 are archaic.  It sucks, sure, but it's silly to suggest such a
 solution.

 BTW, I don't have this problem. I'm just trying to be the voice of
 reason.





[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Michael Steuer


Twitter actually announced versioning along those lines during the  
API 2.0 developer track session at #140tc.




On Sep 25, 2009, at 8:08 PM, Jim Renkel james.ren...@gmail.com  
wrote:




+1. For this and other reasons the API should be versioned.

Jim Renkel

-Original Message-
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Scott
Haneda
Sent: Friday, September 25, 2009 21:28
To: twitter-development-talk@googlegroups.com
Subject: [twitter-dev] Re: SERIOUS Problem With Cursors In JSON
Followers/Friends Ids


Why is the API not versioned then? api.twitter.com/?v=1,
api.twitter.com/?v=1.1, api.twitter.com/?v=1.2 etc

Or, if that is too much maintenance, how about
api.twitter.com/?bitfix=32 or whatever.
--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 6:40 PM, JDG wrote:


and it would also break everyone who CAN handle 64 bit ints and
expects
results in decimal numeric format.

On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:



Can this not be returned as hex or base64?
It would save bandwidth for Twitter (and us) and make it a string
people could convert it to 64bit int if they still want to.

On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:

I would not change either.  But there are those here that are
stating
they need new hardware to work around this issue, and that they can
not afford that.  I was trying to be that voice of reason if that  
is

the road/excuse they are choosing to go.

There seem to be acceptable workarounds, solid proposed  
workarounds,

etc.  I guess I am not getting it, JSON is just a string returned,
yes, it can represent type of data, but it is still just a
string.  I
can not see it being that huge a performance hit to massage that
string a bit once you get ahold of it.
--
Scott * If you contact me off list replace talklists@ with scott@ *

On Sep 25, 2009, at 2:02 PM, jmathai wrote:


It's ridiculous to suggest a change in hardware (64 bit) or
software
(switch from PHP) to use Twitter's API.  It's not like either of
these
are archaic.  It sucks, sure, but it's silly to suggest such a
solution.



BTW, I don't have this problem. I'm just trying to be the voice of
reason.







[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread Chad Etzel

Yes, versioning the API is on the roadmap, I just don't know of any
official timeline.
-Chad

On Fri, Sep 25, 2009 at 11:47 PM, Michael Steuer mste...@gmail.com wrote:

 Twitter actually announced versioning along those lines during the API 2.0
 developer track session at #140tc.



 On Sep 25, 2009, at 8:08 PM, Jim Renkel james.ren...@gmail.com wrote:


 +1. For this and other reasons the API should be versioned.

 Jim Renkel

 -Original Message-
 From: twitter-development-talk@googlegroups.com
 [mailto:twitter-development-t...@googlegroups.com] On Behalf Of Scott
 Haneda
 Sent: Friday, September 25, 2009 21:28
 To: twitter-development-talk@googlegroups.com
 Subject: [twitter-dev] Re: SERIOUS Problem With Cursors In JSON
 Followers/Friends Ids


 Why is the API not versioned then? api.twitter.com/?v=1,
 api.twitter.com/?v=1.1, api.twitter.com/?v=1.2 etc

 Or, if that is too much maintenance, how about
 api.twitter.com/?bitfix=32 or whatever.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 6:40 PM, JDG wrote:

 and it would also break everyone who CAN handle 64 bit ints and
 expects
 results in decimal numeric format.

 On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:


 Can this not be returned as hex or base64?
 It would save bandwidth for Twitter (and us) and make it a string
 people could convert it to 64bit int if they still want to.

 On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:

 I would not change either.  But there are those here that are
 stating
 they need new hardware to work around this issue, and that they can
 not afford that.  I was trying to be that voice of reason if that is
 the road/excuse they are choosing to go.

 There seem to be acceptable workarounds, solid proposed workarounds,
 etc.  I guess I am not getting it, JSON is just a string returned,
 yes, it can represent type of data, but it is still just a
 string.  I
 can not see it being that huge a performance hit to massage that
 string a bit once you get ahold of it.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 2:02 PM, jmathai wrote:

 It's ridiculous to suggest a change in hardware (64 bit) or
 software
 (switch from PHP) to use Twitter's API.  It's not like either of
 these
 are archaic.  It sucks, sure, but it's silly to suggest such a
 solution.

 BTW, I don't have this problem. I'm just trying to be the voice of
 reason.






[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-25 Thread dennisu rozenbaum

???
were is my information

2009/9/26, Michael Steuer mste...@gmail.com:

 Twitter actually announced versioning along those lines during the
 API 2.0 developer track session at #140tc.



 On Sep 25, 2009, at 8:08 PM, Jim Renkel james.ren...@gmail.com
 wrote:


 +1. For this and other reasons the API should be versioned.

 Jim Renkel

 -Original Message-
 From: twitter-development-talk@googlegroups.com
 [mailto:twitter-development-t...@googlegroups.com] On Behalf Of Scott
 Haneda
 Sent: Friday, September 25, 2009 21:28
 To: twitter-development-talk@googlegroups.com
 Subject: [twitter-dev] Re: SERIOUS Problem With Cursors In JSON
 Followers/Friends Ids


 Why is the API not versioned then? api.twitter.com/?v=1,
 api.twitter.com/?v=1.1, api.twitter.com/?v=1.2 etc

 Or, if that is too much maintenance, how about
 api.twitter.com/?bitfix=32 or whatever.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 6:40 PM, JDG wrote:

 and it would also break everyone who CAN handle 64 bit ints and
 expects
 results in decimal numeric format.

 On Fri, Sep 25, 2009 at 16:01, Richard ryt...@gmail.com wrote:


 Can this not be returned as hex or base64?
 It would save bandwidth for Twitter (and us) and make it a string
 people could convert it to 64bit int if they still want to.

 On Sep 25, 10:16 pm, Scott Haneda talkli...@newgeo.com wrote:
 I would not change either.  But there are those here that are
 stating
 they need new hardware to work around this issue, and that they can
 not afford that.  I was trying to be that voice of reason if that
 is
 the road/excuse they are choosing to go.

 There seem to be acceptable workarounds, solid proposed
 workarounds,
 etc.  I guess I am not getting it, JSON is just a string returned,
 yes, it can represent type of data, but it is still just a
 string.  I
 can not see it being that huge a performance hit to massage that
 string a bit once you get ahold of it.
 --
 Scott * If you contact me off list replace talklists@ with scott@ *

 On Sep 25, 2009, at 2:02 PM, jmathai wrote:

 It's ridiculous to suggest a change in hardware (64 bit) or
 software
 (switch from PHP) to use Twitter's API.  It's not like either of
 these
 are archaic.  It sucks, sure, but it's silly to suggest such a
 solution.

 BTW, I don't have this problem. I'm just trying to be the voice of
 reason.






[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread jmathai

Agreed that the problem isn't Twitter's fault. But a basic feature
like cursor should work in a language as popular as PHP.  Not so much
in principle but in practice.

Has anyone tried PEAR's Services_JSON?  I haven't tried nor looked at
the source.  It's also slower than the native json_* functions but
that may or may not be an issue.

On Sep 24, 6:52 am, Joseph Cheek jos...@cheek.com wrote:
 To be fair to Twitter, the problem lies in PHP's json_decode() function,
 not the twitter API.

 Joseph Cheek
 jos...@cheek.com,www.cheek.com
 twitter:http://twitter.com/cheekdotcom

 Dewald Pretorius wrote:
  Chad,

  Shouldn't Twitter be providing an API that works for everyone?

  From what you said it sounds as if you're saying, Tough. If you want
  to consume the API with PHP, either run your stuff on a 64-bit
  machine, or scrape the raw JSON output and make it so that it works
  for you.

  That doesn't sound right.

  Dewald

  On Sep 24, 1:02 am, Chad Etzel c...@twitter.com wrote:

  Hello,

  As Joseph points out, PHP on a 64-bit system can handle these numbers.

  If you really want this data as a string, you could write a regex in
  PHP to alter the json string to wrap the digits in quotes before
  sending it through json_decode(), but that would be a pretty gnarly
  kludge.

  -Chad

  On Wed, Sep 23, 2009 at 10:28 PM, Dewald Pretorius dpr...@gmail.com 
  wrote:

  All that Twitter needs to do to solve this problem is to build the
  JSON out with next_cursor and previous_cursor as string values.

  I.e., the JSON data should contain:

  next_cursor:12398712981212987,previous_cursor:-12398712981212987

  I don't know what it will do to Java apps, but for PHP apps it will
  solve the problem.

  Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Provided that it will not break the JSON for other language apps, the
solution is extremely simple for Twitter to implement, and it will
save a lot of us a lot of work and expense.

Just create any large numeric values in the JSON output as strings,
instead of numbers.

This goes for any large numbers, including tweet ids. As far as I am
concerned they can output everything in JSON as strings.

Thoughts, Twitter?

Dewald

On Sep 24, 12:08 pm, jmathai jmat...@gmail.com wrote:
 Agreed that the problem isn't Twitter's fault. But a basic feature
 like cursor should work in a language as popular as PHP.  Not so much
 in principle but in practice.

 Has anyone tried PEAR's Services_JSON?  I haven't tried nor looked at
 the source.  It's also slower than the native json_* functions but
 that may or may not be an issue.

 On Sep 24, 6:52 am, Joseph Cheek jos...@cheek.com wrote:

  To be fair to Twitter, the problem lies in PHP's json_decode() function,
  not the twitter API.

  Joseph Cheek
  jos...@cheek.com,www.cheek.com
  twitter:http://twitter.com/cheekdotcom

  Dewald Pretorius wrote:
   Chad,

   Shouldn't Twitter be providing an API that works for everyone?

   From what you said it sounds as if you're saying, Tough. If you want
   to consume the API with PHP, either run your stuff on a 64-bit
   machine, or scrape the raw JSON output and make it so that it works
   for you.

   That doesn't sound right.

   Dewald

   On Sep 24, 1:02 am, Chad Etzel c...@twitter.com wrote:

   Hello,

   As Joseph points out, PHP on a 64-bit system can handle these numbers.

   If you really want this data as a string, you could write a regex in
   PHP to alter the json string to wrap the digits in quotes before
   sending it through json_decode(), but that would be a pretty gnarly
   kludge.

   -Chad

   On Wed, Sep 23, 2009 at 10:28 PM, Dewald Pretorius dpr...@gmail.com 
   wrote:

   All that Twitter needs to do to solve this problem is to build the
   JSON out with next_cursor and previous_cursor as string values.

   I.e., the JSON data should contain:

   next_cursor:12398712981212987,previous_cursor:-12398712981212987

   I don't know what it will do to Java apps, but for PHP apps it will
   solve the problem.

   Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Jeffrey Greenberg
My 2 cents are:1. we're using the xml form of the api on 32bit development
machines and it works fine.
2. not supporting 32bit machines seems like a bad idea for twitter and
developers, no matter who(twitter or php) you want to blame as the problem;
  a) PHP is perhaps the most popular web development language out there, so
why make this difficult.
  b) the cost of a 64 bit machine is more than a 32 bit one and this cost
matters to startups
3. conversion to strings is a good idea... it's a red herring to talk
about space, since once you get the data you can convert it as you wish, and
it goes over the wire as a string in any case (doesn't it)

jeffrey greenberg
http://www.tweettronics.com
http://www.inventivity.com

On Thu, Sep 24, 2009 at 8:34 AM, Jesse Stay jesses...@gmail.com wrote:

 On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.comwrote:

 This goes for any large numbers, including tweet ids. As far as I am
 concerned they can output everything in JSON as strings.


 That would create quite a memory footprint! I prefer to use ints where
 possible and strings only where necessary. I think it would be to your
 benefit to just convert to 64-bit PHP. While PHP is type-less, other
 languages aren't, and converting back to int is much more a pain in C than
 it is in PHP. I suggest Twitter leave it the way it is - it should be up to
 the end recipient to convert it in a way that works.  Maybe write some new
 JSON libraries that parse it correctly? That's what open source is for.

 Jesse



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Jesse,

It will add exactly two quote characters () per numeric field in the
JSON payload.

In any event, I am now hacking the raw JSON output to convert the ids
and cursors to string. It's not an ideal solution but it works.

Dewald

On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
 On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com wrote:
  This goes for any large numbers, including tweet ids. As far as I am
  concerned they can output everything in JSON as strings.

 That would create quite a memory footprint! I prefer to use ints where
 possible and strings only where necessary. I think it would be to your
 benefit to just convert to 64-bit PHP. While PHP is type-less, other
 languages aren't, and converting back to int is much more a pain in C than
 it is in PHP. I suggest Twitter leave it the way it is - it should be up to
 the end recipient to convert it in a way that works.  Maybe write some new
 JSON libraries that parse it correctly? That's what open source is for.

 Jesse


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Abraham Williams
Twitter could add:
next_cursor_string:1314614526448841129

Minimal cost and it would be backwards compatible.

Abraham

On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:


 Jesse,

 It will add exactly two quote characters () per numeric field in the
 JSON payload.

 In any event, I am now hacking the raw JSON output to convert the ids
 and cursors to string. It's not an ideal solution but it works.

 Dewald

 On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
  On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
 wrote:
   This goes for any large numbers, including tweet ids. As far as I am
   concerned they can output everything in JSON as strings.
 
  That would create quite a memory footprint! I prefer to use ints where
  possible and strings only where necessary. I think it would be to your
  benefit to just convert to 64-bit PHP. While PHP is type-less, other
  languages aren't, and converting back to int is much more a pain in C
 than
  it is in PHP. I suggest Twitter leave it the way it is - it should be up
 to
  the end recipient to convert it in a way that works.  Maybe write some
 new
  JSON libraries that parse it correctly? That's what open source is for.
 
  Jesse




-- 
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
Project | http://fireeagle.labs.poseurtech.com
This email is: [ ] blogable [x] ask first [ ] private.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Jim Renkel

creat[ing] any large numeric values in the JSON output as strings
could potentially break existing JSON-using applications, depending on
the library they use and how they have typed the variable to receive
the large numeric value.

For http://twxlate.com, I use Google's GSON library
(http://code.google.com/p/google-gson/), and, if I had typed my
variables (actually, Java class fields) as int, Integer, long, or Long,
then the change you suggest would break my code.

Of course the code would also have been broken when the values exceeded
2**31 or 2**63. Anticipating this and also possibly non-numeric values,
I typed the fields as Strings. So I have lucked out on the impact of
Twitpocalylses.

But others may not have been or be so lucky.

On a higher plane, to my mind this is an issue of versioning in the API
syntax. Twitter does not support explicit versioning (i.e., the
inclusion of an API version parameter in API requests) as other RESTful
interfaces do. For example, most Google API RESTful interface methods
include a v= parameter to indicate which version of the method is being
used.

If twitter included this, then to receive large numeric values encoded
as strings the requester could simply include, e.g., v=1.1, on the
request. Given that twitter specified in advance that if you request
version 1.1 responses then large numeric values would be encoded as
strings, then any request made with v=1.1 would have known and had to be
prepared to accept string encoded values. Without v=1.1, you would still
get integer encoded values, and nothing would break.

There have been, and I suspect will continue to be, changes to the API
specification that were or will be not backward compatible without a
version parameter, and for this reason I strongly urge twitter to adopt
this mechanism for all non-backward compatible API changes.

Version negotiation could also be done via HTTP headers, but I prefer
method parameters.

Comments expected and welcome.

Jim Renkel

-Original Message-
From: twitter-development-talk@googlegroups.com
[mailto:twitter-development-t...@googlegroups.com] On Behalf Of Dewald
Pretorius
Sent: Thursday, September 24, 2009 10:26
To: Twitter Development Talk
Subject: [twitter-dev] Re: SERIOUS Problem With Cursors In JSON
Followers/Friends Ids


Provided that it will not break the JSON for other language apps, the
solution is extremely simple for Twitter to implement, and it will
save a lot of us a lot of work and expense.

Just create any large numeric values in the JSON output as strings,
instead of numbers.

This goes for any large numbers, including tweet ids. As far as I am
concerned they can output everything in JSON as strings.

Thoughts, Twitter?

Dewald

On Sep 24, 12:08 pm, jmathai jmat...@gmail.com wrote:
 Agreed that the problem isn't Twitter's fault. But a basic feature
 like cursor should work in a language as popular as PHP.  Not so much
 in principle but in practice.

 Has anyone tried PEAR's Services_JSON?  I haven't tried nor looked at
 the source.  It's also slower than the native json_* functions but
 that may or may not be an issue.

 On Sep 24, 6:52 am, Joseph Cheek jos...@cheek.com wrote:

  To be fair to Twitter, the problem lies in PHP's json_decode()
function,
  not the twitter API.

  Joseph Cheek
  jos...@cheek.com,www.cheek.com
  twitter:http://twitter.com/cheekdotcom

  Dewald Pretorius wrote:
   Chad,

   Shouldn't Twitter be providing an API that works for everyone?

   From what you said it sounds as if you're saying, Tough. If you
want
   to consume the API with PHP, either run your stuff on a 64-bit
   machine, or scrape the raw JSON output and make it so that it
works
   for you.

   That doesn't sound right.

   Dewald

   On Sep 24, 1:02 am, Chad Etzel c...@twitter.com wrote:

   Hello,

   As Joseph points out, PHP on a 64-bit system can handle these
numbers.

   If you really want this data as a string, you could write a regex
in
   PHP to alter the json string to wrap the digits in quotes before
   sending it through json_decode(), but that would be a pretty
gnarly
   kludge.

   -Chad

   On Wed, Sep 23, 2009 at 10:28 PM, Dewald Pretorius
dpr...@gmail.com wrote:

   All that Twitter needs to do to solve this problem is to build
the
   JSON out with next_cursor and previous_cursor as string values.

   I.e., the JSON data should contain:

  
next_cursor:12398712981212987,previous_cursor:-12398712981212987

   I don't know what it will do to Java apps, but for PHP apps it
will
   solve the problem.

   Dewald



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Abraham,

That is true, but we are going to run into exactly the same problem
with 64-bit status ids.

And that is going to break a LOT of PHP applications in one fell
swoop.

Dewald

On Sep 24, 2:27 pm, Abraham Williams 4bra...@gmail.com wrote:
 Twitter could add:
 next_cursor_string:1314614526448841129

 Minimal cost and it would be backwards compatible.

 Abraham



 On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:

  Jesse,

  It will add exactly two quote characters () per numeric field in the
  JSON payload.

  In any event, I am now hacking the raw JSON output to convert the ids
  and cursors to string. It's not an ideal solution but it works.

  Dewald

  On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
   On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
  wrote:
This goes for any large numbers, including tweet ids. As far as I am
concerned they can output everything in JSON as strings.

   That would create quite a memory footprint! I prefer to use ints where
   possible and strings only where necessary. I think it would be to your
   benefit to just convert to 64-bit PHP. While PHP is type-less, other
   languages aren't, and converting back to int is much more a pain in C
  than
   it is in PHP. I suggest Twitter leave it the way it is - it should be up
  to
   the end recipient to convert it in a way that works.  Maybe write some
  new
   JSON libraries that parse it correctly? That's what open source is for.

   Jesse

 --
 Abraham Williams | Community Evangelist |http://web608.org
 Hacker |http://abrah.am|http://twitter.com/abraham
 Project |http://fireeagle.labs.poseurtech.com
 This email is: [ ] blogable [x] ask first [ ] private.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Chad Etzel

On Thu, Sep 24, 2009 at 1:50 PM, Dewald Pretorius dpr...@gmail.com wrote:

 Abraham,

 That is true, but we are going to run into exactly the same problem
 with 64-bit status ids.

This has already happened. Tweet ids are now bigger than a 32 bit int
can store. That's what Twitpocalypse 2 was all about.

-Chad



 And that is going to break a LOT of PHP applications in one fell
 swoop.

 Dewald

 On Sep 24, 2:27 pm, Abraham Williams 4bra...@gmail.com wrote:
 Twitter could add:
 next_cursor_string:1314614526448841129

 Minimal cost and it would be backwards compatible.

 Abraham



 On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:

  Jesse,

  It will add exactly two quote characters () per numeric field in the
  JSON payload.

  In any event, I am now hacking the raw JSON output to convert the ids
  and cursors to string. It's not an ideal solution but it works.

  Dewald

  On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
   On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
  wrote:
This goes for any large numbers, including tweet ids. As far as I am
concerned they can output everything in JSON as strings.

   That would create quite a memory footprint! I prefer to use ints where
   possible and strings only where necessary. I think it would be to your
   benefit to just convert to 64-bit PHP. While PHP is type-less, other
   languages aren't, and converting back to int is much more a pain in C
  than
   it is in PHP. I suggest Twitter leave it the way it is - it should be up
  to
   the end recipient to convert it in a way that works.  Maybe write some
  new
   JSON libraries that parse it correctly? That's what open source is for.

   Jesse

 --
 Abraham Williams | Community Evangelist |http://web608.org
 Hacker |http://abrah.am|http://twitter.com/abraham
 Project |http://fireeagle.labs.poseurtech.com
 This email is: [ ] blogable [x] ask first [ ] private.



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Chad,

Thanks for setting me straight on that one.

My app is not reliant on Twitter tweet ids except for one specific
informational use. Hence, I have not been paying much attention to the
Twitpocalypse 2.

Dewald

On Sep 24, 3:14 pm, Chad Etzel c...@twitter.com wrote:
 On Thu, Sep 24, 2009 at 1:50 PM, Dewald Pretorius dpr...@gmail.com wrote:

  Abraham,

  That is true, but we are going to run into exactly the same problem
  with 64-bit status ids.

 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Joseph Cheek

until 65+ bit computing becomes standard, a lot of things will break
(twitter included) when id's go past 64 bits.  i hate to be a naysayer
though, but the difference between 2^32 and 2^64 is, well, huge.  i
don't expect twitter id's to break 64 bits anytime soon, until at least
far past twitter's usefulness (sorry, twitter...)

i know this is akin to COBOL programmers in the fifties saying hmm, i
wonder if we should concern ourselves with what will happen to our apps
in the year 2000? and giving it no further thought, and i apologize,
but i'm ok with that.

Joseph

Dewald Pretorius wrote:
 Abraham,

 That is true, but we are going to run into exactly the same problem
 with 64-bit status ids.

 And that is going to break a LOT of PHP applications in one fell
 swoop.

 Dewald

 On Sep 24, 2:27 pm, Abraham Williams 4bra...@gmail.com wrote:
   
 Twitter could add:
 next_cursor_string:1314614526448841129

 Minimal cost and it would be backwards compatible.

 Abraham



 On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:

 
 Jesse,
   
 It will add exactly two quote characters () per numeric field in the
 JSON payload.
   
 In any event, I am now hacking the raw JSON output to convert the ids
 and cursors to string. It's not an ideal solution but it works.
   
 Dewald
   
 On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
   
 On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
 
 wrote:
   
 This goes for any large numbers, including tweet ids. As far as I am
 concerned they can output everything in JSON as strings.
   
 That would create quite a memory footprint! I prefer to use ints where
 possible and strings only where necessary. I think it would be to your
 benefit to just convert to 64-bit PHP. While PHP is type-less, other
 languages aren't, and converting back to int is much more a pain in C
 
 than
   
 it is in PHP. I suggest Twitter leave it the way it is - it should be up
 
 to
   
 the end recipient to convert it in a way that works.  Maybe write some
 
 new
   
 JSON libraries that parse it correctly? That's what open source is for.
 
 Jesse
 
 --
 Abraham Williams | Community Evangelist |http://web608.org
 Hacker |http://abrah.am|http://twitter.com/abraham
 Project |http://fireeagle.labs.poseurtech.com
 This email is: [ ] blogable [x] ask first [ ] private.
 

   

-- 
Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Joseph Cheek

already past 32-bit signed  but not 32-bit unsigned, right? 
twitpocalypse moves the max to 2^32-1, right?  or did i misread it?

Chad Etzel wrote:
 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad
   

-- 
Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Although, looking at a few tweets in the public timeline, I'm seeing
tweet ids of 4348298962, 4348298957, etc.

At least on my server, json_decode is still correctly decoding those
numbers.

So, it's probably safe to say that Twitpocalypse 2 has not arrived yet
for many PHP apps in terms of JSON decodes.

Dewald

On Sep 24, 3:14 pm, Chad Etzel c...@twitter.com wrote:
 On Thu, Sep 24, 2009 at 1:50 PM, Dewald Pretorius dpr...@gmail.com wrote:

  Abraham,

  That is true, but we are going to run into exactly the same problem
  with 64-bit status ids.

 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad



  And that is going to break a LOT of PHP applications in one fell
  swoop.

  Dewald

  On Sep 24, 2:27 pm, Abraham Williams 4bra...@gmail.com wrote:
  Twitter could add:
  next_cursor_string:1314614526448841129

  Minimal cost and it would be backwards compatible.

  Abraham

  On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:

   Jesse,

   It will add exactly two quote characters () per numeric field in the
   JSON payload.

   In any event, I am now hacking the raw JSON output to convert the ids
   and cursors to string. It's not an ideal solution but it works.

   Dewald

   On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
   wrote:
 This goes for any large numbers, including tweet ids. As far as I am
 concerned they can output everything in JSON as strings.

That would create quite a memory footprint! I prefer to use ints where
possible and strings only where necessary. I think it would be to your
benefit to just convert to 64-bit PHP. While PHP is type-less, other
languages aren't, and converting back to int is much more a pain in C
   than
it is in PHP. I suggest Twitter leave it the way it is - it should be 
up
   to
the end recipient to convert it in a way that works.  Maybe write some
   new
JSON libraries that parse it correctly? That's what open source is for.

Jesse

  --
  Abraham Williams | Community Evangelist |http://web608.org
  Hacker |http://abrah.am|http://twitter.com/abraham
  Project |http://fireeagle.labs.poseurtech.com
  This email is: [ ] blogable [x] ask first [ ] private.


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Chad Etzel

On Thu, Sep 24, 2009 at 2:29 PM, Joseph Cheek jos...@cheek.com wrote:

 already past 32-bit signed  but not 32-bit unsigned, right?
 twitpocalypse moves the max to 2^32-1, right?  or did i misread it?

They have passed 32-bit *un*signed. My latest tweet id is 4345383616
(which is  2^32 - 1)

Twitpocalypse 2 was about overflowing the 32 bit int space.

-Chad



 Chad Etzel wrote:
 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad


 --
 Joseph Cheek
 jos...@cheek.com, www.cheek.com
 twitter: http://twitter.com/cheekdotcom




[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Chad Etzel

Dewald,

What do you get on your server if you do:

echo PHP_INT_MAX;

also, what version of PHP are you using?

-Chad

On Thu, Sep 24, 2009 at 2:48 PM, Dewald Pretorius dpr...@gmail.com wrote:

 Although, looking at a few tweets in the public timeline, I'm seeing
 tweet ids of 4348298962, 4348298957, etc.

 At least on my server, json_decode is still correctly decoding those
 numbers.

 So, it's probably safe to say that Twitpocalypse 2 has not arrived yet
 for many PHP apps in terms of JSON decodes.

 Dewald

 On Sep 24, 3:14 pm, Chad Etzel c...@twitter.com wrote:
 On Thu, Sep 24, 2009 at 1:50 PM, Dewald Pretorius dpr...@gmail.com wrote:

  Abraham,

  That is true, but we are going to run into exactly the same problem
  with 64-bit status ids.

 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad



  And that is going to break a LOT of PHP applications in one fell
  swoop.

  Dewald

  On Sep 24, 2:27 pm, Abraham Williams 4bra...@gmail.com wrote:
  Twitter could add:
  next_cursor_string:1314614526448841129

  Minimal cost and it would be backwards compatible.

  Abraham

  On Thu, Sep 24, 2009 at 12:06, Dewald Pretorius dpr...@gmail.com wrote:

   Jesse,

   It will add exactly two quote characters () per numeric field in the
   JSON payload.

   In any event, I am now hacking the raw JSON output to convert the ids
   and cursors to string. It's not an ideal solution but it works.

   Dewald

   On Sep 24, 12:34 pm, Jesse Stay jesses...@gmail.com wrote:
On Thu, Sep 24, 2009 at 9:26 AM, Dewald Pretorius dpr...@gmail.com
   wrote:
 This goes for any large numbers, including tweet ids. As far as I am
 concerned they can output everything in JSON as strings.

That would create quite a memory footprint! I prefer to use ints where
possible and strings only where necessary. I think it would be to your
benefit to just convert to 64-bit PHP. While PHP is type-less, other
languages aren't, and converting back to int is much more a pain in C
   than
it is in PHP. I suggest Twitter leave it the way it is - it should be 
up
   to
the end recipient to convert it in a way that works.  Maybe write some
   new
JSON libraries that parse it correctly? That's what open source is 
for.

Jesse

  --
  Abraham Williams | Community Evangelist |http://web608.org
  Hacker |http://abrah.am|http://twitter.com/abraham
  Project |http://fireeagle.labs.poseurtech.com
  This email is: [ ] blogable [x] ask first [ ] private.



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Chad,

PHP_INT_MAX is 2147483647.

And yet, json_decode is still correctly decoding id:4348298957.

O-o

Dewald

On Sep 24, 3:58 pm, Chad Etzel c...@twitter.com wrote:
 Dewald,

 What do you get on your server if you do:

 echo PHP_INT_MAX;

 also, what version of PHP are you using?

 -Chad


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Joseph Cheek

um...

As mentioned previously, the Twitter operations team will artificially
increase the maximum status ID to 4294967296 this coming Friday,
September 11th. This action is part of routine database upgrades and
maintenance. 

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/8fde4804c5a75fb2

it appears to me that tw2k9 (-- my new abbreviation, sorry, have typed
twitpocalypse 2 too many times) is about using the full 32-bit int
space.  agreed, it didn't make much sense to announce that you were
going to now utilize the full address space, but that *is* how it was
announced - increase the new *maximum* status ID [emphasis mine] to
4294967296.

$ echo $[ 65536 * 65536 ]
4294967296

thanks!

Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom



Chad Etzel wrote:
 On Thu, Sep 24, 2009 at 2:29 PM, Joseph Cheek jos...@cheek.com wrote:
   
 already past 32-bit signed  but not 32-bit unsigned, right?
 twitpocalypse moves the max to 2^32-1, right?  or did i misread it?
 

 They have passed 32-bit *un*signed. My latest tweet id is 4345383616
 (which is  2^32 - 1)

 Twitpocalypse 2 was about overflowing the 32 bit int space.

 -Chad


   
 Chad Etzel wrote:
 
 This has already happened. Tweet ids are now bigger than a 32 bit int
 can store. That's what Twitpocalypse 2 was all about.

 -Chad

   
 --
 Joseph Cheek
 jos...@cheek.com, www.cheek.com
 twitter: http://twitter.com/cheekdotcom
 


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

Clearly PHP_INT_MAX plays no role in json_decode.

There must be some other mystical maximum number above which it
represents the number as float in the decoded data.

Dewald

On Sep 24, 4:05 pm, Dewald Pretorius dpr...@gmail.com wrote:
 Chad,

 PHP_INT_MAX is 2147483647.

 And yet, json_decode is still correctly decoding id:4348298957.

 O-o

 Dewald

 On Sep 24, 3:58 pm, Chad Etzel c...@twitter.com wrote:

  Dewald,

  What do you get on your server if you do:

  echo PHP_INT_MAX;

  also, what version of PHP are you using?

  -Chad


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Dewald Pretorius

That magical maximum number appears to be 1 (1.0E+12).

So, for tweet ids we still have a bit of breathing space.

Dewald

On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:
 Clearly PHP_INT_MAX plays no role in json_decode.

 There must be some other mystical maximum number above which it
 represents the number as float in the decoded data.

 Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Chad Etzel

For reference, the PHP_INT_MAX on my 64-bit linux VM is:
9223372036854775807

and using json_decode on the next_cursor field of an ids list works just fine.

I'm using PHP 5.2.10-0.dotdeb.1 (though PHP 5.2.11-0.dotdeb.1 is available)

fwiw,
-Chad

On Thu, Sep 24, 2009 at 3:29 PM, Dewald Pretorius dpr...@gmail.com wrote:

 That magical maximum number appears to be 1 (1.0E+12).

 So, for tweet ids we still have a bit of breathing space.

 Dewald

 On Sep 24, 4:18 pm, Dewald Pretorius dpr...@gmail.com wrote:
 Clearly PHP_INT_MAX plays no role in json_decode.

 There must be some other mystical maximum number above which it
 represents the number as float in the decoded data.

 Dewald



[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-24 Thread Scott Haneda


Can someone explain this issue to me in more detail?

How do I know my php can handle it? I'm running on an older PPC  
machine, which is 64 bit, but I have no idea if I built it as 64, I  
just supplied standard configure arguments.


If php sees a 64 bit integer come in as JSON what happens on 32 bit?   
Does it just truncate it or completly ruin it's representation?


Why can't it just be treated as a string?don't these ID's end up in a  
database, or maybe just passed as a URL argument? As a URL arg you  
pass to Twitter, so it will work fine.


Any database can store a 64 bit int as a string, which gives your  
ability to get the string back to post to a URL.


Can a 32 bit build of a database store 64 bit ints?  That to me seems  
the bigger issue.


I'm not seeing where there is a need for math on a 64 bit int ditectly  
in php. You get the value from JSON, treat it as a string, store that  
string, and use it for your lookups and sorts.


Inneficient on lookups and storage, is that the core of this?

What key aspect of this problem am I missing?

--
Scott
Iphone says hello.

On Sep 24, 2009, at 6:07 AM, Dewald Pretorius dpr...@gmail.com wrote:



Chad,

Shouldn't Twitter be providing an API that works for everyone?

From what you said it sounds as if you're saying, Tough. If you want
to consume the API with PHP, either run your stuff on a 64-bit
machine, or scrape the raw JSON output and make it so that it works
for you.

That doesn't sound right.

Dewald

On Sep 24, 1:02 am, Chad Etzel c...@twitter.com wrote:

Hello,

As Joseph points out, PHP on a 64-bit system can handle these  
numbers.


If you really want this data as a string, you could write a regex in
PHP to alter the json string to wrap the digits in quotes before
sending it through json_decode(), but that would be a pretty gnarly
kludge.

-Chad

On Wed, Sep 23, 2009 at 10:28 PM, Dewald Pretorius  
dpr...@gmail.com wrote:



All that Twitter needs to do to solve this problem is to build the
JSON out with next_cursor and previous_cursor as string values.



I.e., the JSON data should contain:


next_cursor 
:12398712981212987,previous_cursor:-12398712981212987



I don't know what it will do to Java apps, but for PHP apps it will
solve the problem.



Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-23 Thread Joseph Cheek

from http://us2.php.net/json_decode:

|In PHP = 5.1.6 trying to decode an integer value that's  PHP_INT_MAX
will result in an intger of PHP_INT_MAX.

In PHP 5.2+ decoding an integer  PHP_INT_MAX will cause a conversion to
a float.

Neither behaviour is perfect, capping at PHP_INT_MAX is marginally
worse, but the float conversion loses precision.

If you expect to deal with large numbers at all, let alone in JSON,
ensure you're using a 64-bit system.

--

my thoughts:

if moving to a 64-bit system is not feasible and you aren't interesting
in hacking the (php itself) source (num_as_string json_decode() patch at
http://bugs.php.net/bug.php?id=46363), you could try xml or writing a
quick-n-dirty twitter-specific decoder.  other alternatives are looking
at a hosting provider or paying someone (ahem) to patch your PHP
instance plug(I do Linux only, not BSD or 'doze, but I've been doing
Linux for a long time and I'm pretty good at it)/plug.

thanks!

Joseph Cheek
jos...@cheek.com, www.cheek.com
www.twitter.com/cheekdotcom
|
Dewald Pretorius wrote:
 I've run into a serious issue and I don't know if I am overlooking
 something.

 When retrieving ids with cursoring, and then doing a PHP json_decode
 on the data, the cursor value is converted to an exponentional number
 expression.

 For example:

 http://twitter.com/followers/ids/barackobama.json?cursor=-1

 yields next_cursor:1314614526448841129 in the raw JSON data.

 After json_decode the value is converted to 1.31461452645E+18.

 I've tried both json_decode to an object and to an associative array
 and both give the same result.

 As you will notice, even if one were to typecast that number to float,
 there is still 8 digits missing, which would yield an invalid cursor.

 So, am I missing something glaringly obvious, or does someone have a
 solution to this issue?

 Dewald


   


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-23 Thread Dewald Pretorius

All that Twitter needs to do to solve this problem is to build the
JSON out with next_cursor and previous_cursor as string values.

I.e., the JSON data should contain:

next_cursor:12398712981212987,previous_cursor:-12398712981212987

I don't know what it will do to Java apps, but for PHP apps it will
solve the problem.

Dewald


[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

2009-09-23 Thread Chad Etzel

Hello,

As Joseph points out, PHP on a 64-bit system can handle these numbers.

If you really want this data as a string, you could write a regex in
PHP to alter the json string to wrap the digits in quotes before
sending it through json_decode(), but that would be a pretty gnarly
kludge.

-Chad

On Wed, Sep 23, 2009 at 10:28 PM, Dewald Pretorius dpr...@gmail.com wrote:

 All that Twitter needs to do to solve this problem is to build the
 JSON out with next_cursor and previous_cursor as string values.

 I.e., the JSON data should contain:

 next_cursor:12398712981212987,previous_cursor:-12398712981212987

 I don't know what it will do to Java apps, but for PHP apps it will
 solve the problem.

 Dewald