Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
Thanks Tom. I am not doing any real data analysis, so timestamps (even
if they are the same) is just fine for me.
Jim
On 11/2/10 10:35 AM, Tom van der Woerdt wrote:
The first 41 bits of the ID are the timestamp, so yes, they can be used
to sort. However, be aware that they won't always have a perfect
accuracy: only the first 41 bits can be used to sort, the other 23 bits
are to make sure that the ID is unique. If you have two tweets where the
first 41 bits are the same, there is no way to know which one was first.
Tom
On 11/2/10 6:25 PM, Jim Cortez wrote:
Can anyone help out on this? My question is: If I compare Direct Message
ID's to each other to determine send/receive order, will it work under
Snowflake?
Thanks,
Jim
On 11/1/10 12:27 PM, Jim Cortez wrote:
When 2 numeric strings are compared, they are done by magnitude. Some
tests:
"10765432100123456789"> "10765432100123456790"
=> false
"1076543210012345678901010101">"10765432100123456789"
=> true
"1076543210012345678901010101"< "10765432100123456789"
=> false
"10765432100123456789" === "10765432100123456789"
=> true
"10765432100123456789" == "10765432100123456789"
=> true
"10765432100123456789" === "10765432100123456789010101011001"
=> false
The only problem with this approach occurs when the numeric strings
have a different length:
"10"< "2"
=> true
I get around this by checking length first. See this sort comparison
function:
function(b,a){
var aLen = a.id_str.length;
var bLen = b.id_str.length;
if(aLen == bLen){
return a.id_str> b.id_str ? 1 : a.id_str == b.id_str ? 0 : -1;
}else{
return aLen> bLen ? 1 : 0-1;
}
}
As for the JSON parser in our environment, I might be OK if what you
say is true:
var j = JSON.parse('{"id": 10765432100123456789, "id_str":
"10765432100123456789"}');
_dump(j);
id: 10765432100123458000
id_str: 10765432100123456789
So my question about theoretical safety is still outstanding. If I
compare Direct Message Snowflake ID's, will it blend?
Jim
On 11/1/10 12:00 PM, Tom van der Woerdt wrote:
Yes, and no. Javascript has problems using numbers larger than 53 bits.
Comparing them anyway (after conversion from string to int) may cause
loss of accuracy. This itself is not very much an issue: only the first
41 bits of the ID can be used for comparison, the rest is simply to make
sure it is actually an unique ID.
The question remains where the loss of accuracy is: in the first 11 bits
or the last 11 bits. If it's the last 11 bits that get ignored, there is
no problem with comparing the numbers. If it's the first 11 bits that
get ignored, you will have to shift the numbers 11 bits before comparing
them.
Tom
On 11/1/10 7:47 PM, Jim Cortez wrote:
I have learned that I can safely compare 2 long integer strings without
any problem. Is comparing Snowflake direct message id's in the manner
described safe?
Jim
On 11/1/10 11:29 AM, Jim Cortez wrote:
Hello all,
I have an non-browser xAuth client written in Javascript. I am in
the process of changing up the codebase to use id_str in anticipation
of Snowflake. One popular design decision we have made is to
interleave sent and received direct messages into one unified list.
Currently, we sort by message ID to figure out which message comes
before the other. Since JS cannot parse the new 64bit integers, we can
no longer do this. The only way I see to do this under Snowflake is to
sort based on created_at, is that a valid approach? Is there a better
way to approach this problem?
Thank you,
Jim Cortez
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
The first 41 bits of the ID are the timestamp, so yes, they can be used
to sort. However, be aware that they won't always have a perfect
accuracy: only the first 41 bits can be used to sort, the other 23 bits
are to make sure that the ID is unique. If you have two tweets where the
first 41 bits are the same, there is no way to know which one was first.
Tom
On 11/2/10 6:25 PM, Jim Cortez wrote:
> Can anyone help out on this? My question is: If I compare Direct Message
> ID's to each other to determine send/receive order, will it work under
> Snowflake?
> Thanks,
> Jim
>
> On 11/1/10 12:27 PM, Jim Cortez wrote:
>> When 2 numeric strings are compared, they are done by magnitude. Some
>> tests:
>>
>> "10765432100123456789" > "10765432100123456790"
>> => false
>> "1076543210012345678901010101" >"10765432100123456789"
>> => true
>> "1076543210012345678901010101" < "10765432100123456789"
>> => false
>> "10765432100123456789" === "10765432100123456789"
>> => true
>> "10765432100123456789" == "10765432100123456789"
>> => true
>> "10765432100123456789" === "10765432100123456789010101011001"
>> => false
>>
>> The only problem with this approach occurs when the numeric strings
>> have a different length:
>> "10" < "2"
>> => true
>>
>> I get around this by checking length first. See this sort comparison
>> function:
>>
>> function(b,a){
>>var aLen = a.id_str.length;
>>var bLen = b.id_str.length;
>>if(aLen == bLen){
>> return a.id_str > b.id_str ? 1 : a.id_str == b.id_str ? 0 : -1;
>>}else{
>> return aLen > bLen ? 1 : 0-1;
>>}
>> }
>>
>> As for the JSON parser in our environment, I might be OK if what you
>> say is true:
>>
>> var j = JSON.parse('{"id": 10765432100123456789, "id_str":
>> "10765432100123456789"}');
>> _dump(j);
>> id: 10765432100123458000
>> id_str: 10765432100123456789
>>
>> So my question about theoretical safety is still outstanding. If I
>> compare Direct Message Snowflake ID's, will it blend?
>> Jim
>>
>> On 11/1/10 12:00 PM, Tom van der Woerdt wrote:
>>> Yes, and no. Javascript has problems using numbers larger than 53 bits.
>>> Comparing them anyway (after conversion from string to int) may cause
>>> loss of accuracy. This itself is not very much an issue: only the first
>>> 41 bits of the ID can be used for comparison, the rest is simply to make
>>> sure it is actually an unique ID.
>>>
>>> The question remains where the loss of accuracy is: in the first 11 bits
>>> or the last 11 bits. If it's the last 11 bits that get ignored, there is
>>> no problem with comparing the numbers. If it's the first 11 bits that
>>> get ignored, you will have to shift the numbers 11 bits before comparing
>>> them.
>>>
>>> Tom
>>>
>>>
>>> On 11/1/10 7:47 PM, Jim Cortez wrote:
I have learned that I can safely compare 2 long integer strings without
any problem. Is comparing Snowflake direct message id's in the manner
described safe?
Jim
On 11/1/10 11:29 AM, Jim Cortez wrote:
> Hello all,
> I have an non-browser xAuth client written in Javascript. I am in
> the process of changing up the codebase to use id_str in anticipation
> of Snowflake. One popular design decision we have made is to
> interleave sent and received direct messages into one unified list.
> Currently, we sort by message ID to figure out which message comes
> before the other. Since JS cannot parse the new 64bit integers, we can
> no longer do this. The only way I see to do this under Snowflake is to
> sort based on created_at, is that a valid approach? Is there a better
> way to approach this problem?
> Thank you,
> Jim Cortez
>
>>
>
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
Can anyone help out on this? My question is: If I compare Direct Message
ID's to each other to determine send/receive order, will it work under
Snowflake?
Thanks,
Jim
On 11/1/10 12:27 PM, Jim Cortez wrote:
When 2 numeric strings are compared, they are done by magnitude. Some
tests:
"10765432100123456789" > "10765432100123456790"
=> false
"1076543210012345678901010101" >"10765432100123456789"
=> true
"1076543210012345678901010101" < "10765432100123456789"
=> false
"10765432100123456789" === "10765432100123456789"
=> true
"10765432100123456789" == "10765432100123456789"
=> true
"10765432100123456789" === "10765432100123456789010101011001"
=> false
The only problem with this approach occurs when the numeric strings
have a different length:
"10" < "2"
=> true
I get around this by checking length first. See this sort comparison
function:
function(b,a){
var aLen = a.id_str.length;
var bLen = b.id_str.length;
if(aLen == bLen){
return a.id_str > b.id_str ? 1 : a.id_str == b.id_str ? 0 : -1;
}else{
return aLen > bLen ? 1 : 0-1;
}
}
As for the JSON parser in our environment, I might be OK if what you
say is true:
var j = JSON.parse('{"id": 10765432100123456789, "id_str":
"10765432100123456789"}');
_dump(j);
id: 10765432100123458000
id_str: 10765432100123456789
So my question about theoretical safety is still outstanding. If I
compare Direct Message Snowflake ID's, will it blend?
Jim
On 11/1/10 12:00 PM, Tom van der Woerdt wrote:
Yes, and no. Javascript has problems using numbers larger than 53 bits.
Comparing them anyway (after conversion from string to int) may cause
loss of accuracy. This itself is not very much an issue: only the first
41 bits of the ID can be used for comparison, the rest is simply to make
sure it is actually an unique ID.
The question remains where the loss of accuracy is: in the first 11 bits
or the last 11 bits. If it's the last 11 bits that get ignored, there is
no problem with comparing the numbers. If it's the first 11 bits that
get ignored, you will have to shift the numbers 11 bits before comparing
them.
Tom
On 11/1/10 7:47 PM, Jim Cortez wrote:
I have learned that I can safely compare 2 long integer strings without
any problem. Is comparing Snowflake direct message id's in the manner
described safe?
Jim
On 11/1/10 11:29 AM, Jim Cortez wrote:
Hello all,
I have an non-browser xAuth client written in Javascript. I am in
the process of changing up the codebase to use id_str in anticipation
of Snowflake. One popular design decision we have made is to
interleave sent and received direct messages into one unified list.
Currently, we sort by message ID to figure out which message comes
before the other. Since JS cannot parse the new 64bit integers, we can
no longer do this. The only way I see to do this under Snowflake is to
sort based on created_at, is that a valid approach? Is there a better
way to approach this problem?
Thank you,
Jim Cortez
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
When 2 numeric strings are compared, they are done by magnitude. Some tests:
"10765432100123456789" > "10765432100123456790"
=> false
"1076543210012345678901010101" >"10765432100123456789"
=> true
"1076543210012345678901010101" < "10765432100123456789"
=> false
"10765432100123456789" === "10765432100123456789"
=> true
"10765432100123456789" == "10765432100123456789"
=> true
"10765432100123456789" === "10765432100123456789010101011001"
=> false
The only problem with this approach occurs when the numeric strings have
a different length:
"10" < "2"
=> true
I get around this by checking length first. See this sort comparison
function:
function(b,a){
var aLen = a.id_str.length;
var bLen = b.id_str.length;
if(aLen == bLen){
return a.id_str > b.id_str ? 1 : a.id_str == b.id_str ? 0 : -1;
}else{
return aLen > bLen ? 1 : 0-1;
}
}
As for the JSON parser in our environment, I might be OK if what you say
is true:
var j = JSON.parse('{"id": 10765432100123456789, "id_str":
"10765432100123456789"}');
_dump(j);
id: 10765432100123458000
id_str: 10765432100123456789
So my question about theoretical safety is still outstanding. If I
compare Direct Message Snowflake ID's, will it blend?
Jim
On 11/1/10 12:00 PM, Tom van der Woerdt wrote:
Yes, and no. Javascript has problems using numbers larger than 53 bits.
Comparing them anyway (after conversion from string to int) may cause
loss of accuracy. This itself is not very much an issue: only the first
41 bits of the ID can be used for comparison, the rest is simply to make
sure it is actually an unique ID.
The question remains where the loss of accuracy is: in the first 11 bits
or the last 11 bits. If it's the last 11 bits that get ignored, there is
no problem with comparing the numbers. If it's the first 11 bits that
get ignored, you will have to shift the numbers 11 bits before comparing
them.
Tom
On 11/1/10 7:47 PM, Jim Cortez wrote:
I have learned that I can safely compare 2 long integer strings without
any problem. Is comparing Snowflake direct message id's in the manner
described safe?
Jim
On 11/1/10 11:29 AM, Jim Cortez wrote:
Hello all,
I have an non-browser xAuth client written in Javascript. I am in
the process of changing up the codebase to use id_str in anticipation
of Snowflake. One popular design decision we have made is to
interleave sent and received direct messages into one unified list.
Currently, we sort by message ID to figure out which message comes
before the other. Since JS cannot parse the new 64bit integers, we can
no longer do this. The only way I see to do this under Snowflake is to
sort based on created_at, is that a valid approach? Is there a better
way to approach this problem?
Thank you,
Jim Cortez
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
Yes, and no. Javascript has problems using numbers larger than 53 bits. Comparing them anyway (after conversion from string to int) may cause loss of accuracy. This itself is not very much an issue: only the first 41 bits of the ID can be used for comparison, the rest is simply to make sure it is actually an unique ID. The question remains where the loss of accuracy is: in the first 11 bits or the last 11 bits. If it's the last 11 bits that get ignored, there is no problem with comparing the numbers. If it's the first 11 bits that get ignored, you will have to shift the numbers 11 bits before comparing them. Tom On 11/1/10 7:47 PM, Jim Cortez wrote: > I have learned that I can safely compare 2 long integer strings without > any problem. Is comparing Snowflake direct message id's in the manner > described safe? > Jim > > On 11/1/10 11:29 AM, Jim Cortez wrote: >> Hello all, >> I have an non-browser xAuth client written in Javascript. I am in >> the process of changing up the codebase to use id_str in anticipation >> of Snowflake. One popular design decision we have made is to >> interleave sent and received direct messages into one unified list. >> Currently, we sort by message ID to figure out which message comes >> before the other. Since JS cannot parse the new 64bit integers, we can >> no longer do this. The only way I see to do this under Snowflake is to >> sort based on created_at, is that a valid approach? Is there a better >> way to approach this problem? >> Thank you, >> Jim Cortez >> > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Javascript, Snowflake, and Direct Message id's
I have learned that I can safely compare 2 long integer strings without any problem. Is comparing Snowflake direct message id's in the manner described safe? Jim On 11/1/10 11:29 AM, Jim Cortez wrote: Hello all, I have an non-browser xAuth client written in Javascript. I am in the process of changing up the codebase to use id_str in anticipation of Snowflake. One popular design decision we have made is to interleave sent and received direct messages into one unified list. Currently, we sort by message ID to figure out which message comes before the other. Since JS cannot parse the new 64bit integers, we can no longer do this. The only way I see to do this under Snowflake is to sort based on created_at, is that a valid approach? Is there a better way to approach this problem? Thank you, Jim Cortez -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
[twitter-dev] Javascript, Snowflake, and Direct Message id's
Hello all, I have an non-browser xAuth client written in Javascript. I am in the process of changing up the codebase to use id_str in anticipation of Snowflake. One popular design decision we have made is to interleave sent and received direct messages into one unified list. Currently, we sort by message ID to figure out which message comes before the other. Since JS cannot parse the new 64bit integers, we can no longer do this. The only way I see to do this under Snowflake is to sort based on created_at, is that a valid approach? Is there a better way to approach this problem? Thank you, Jim Cortez -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
