Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 7 February 2013 20:43, Brett Henderson  wrote:

>
> On 7 February 2013 20:21, Ilya Zverev  wrote:
>
>> I'd like idTrackerType removed, since after tomorrow using it will
>> certainly cause osmosis to fail. I doubt this options was useful, and it
>> only puzzled those who were learning osmosis, sometimes being the only
>> option besides inPipe and outPipe.
>>
>
> I've always avoided removing options unless necessary to avoid breaking
> existing scripts, but in this case I agree that it would be better
> removed.  It might be nice if Osmosis had a way of deprecating options so
> that warnings would be displayed if they were used.  I'm a bit time
> constrained at the moment but happy to accept patches/pull requests ;-)
>

I've removed the option.  It is still available in 0.42 to allow a smooth
transition from 0.41 which requires it to be specified for --used-xxx
tasks, but it will not be available in 0.43.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 13 February 2013 19:10, Jukka Laaksola  wrote:

> 13.02.2013 01:11, Brett Henderson wrote:
> > On 12 February 2013 23:57, Jukka Laaksola  > > wrote:
> >> There is workaround to use option "--used-node idTrackerType=Dynamic",
> >> but it is not documented. As I can see it is working also with version
> >> 0.41. And my tests also looks like Dynamic will work with the
> >> version 0.41.
> >
> >
> > The lack of documentation is easy to fix :-)  Would you mind updating
> > the 0.41 wiki page?
>
> Have not had real knowledge of the Dynamic feature in 0.41. Just tried
> it and it looked like it's working. Now I got the answer from you that
> it's really undocumented feature. Perhaps I can create an account to
> wiki and update the page...
>

Great, it looks like somebody has beaten me to it.


>
> > The other option is to use a nightly build ...
>
> Is there any place to download nightly builds? Osmosis wiki page
> Development Build links are broken.
>
> http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build
>
> Zip Files:
>
> http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/zips/
>
> Tar Balls:
>
> http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/tgzs/
>

Oh, I didn't realise they were broken.  I think it has been caused by the
recent change to a gradle build system.  I've fixed the wiki.  The link for
both zip and tgz files is:
http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/build/distribution/

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Jukka Laaksola]

13.02.2013 01:11, Brett Henderson wrote:
> On 12 February 2013 23:57, Jukka Laaksola  > wrote:
>> There is workaround to use option "--used-node idTrackerType=Dynamic",
>> but it is not documented. As I can see it is working also with version
>> 0.41. And my tests also looks like Dynamic will work with the
>> version 0.41.
> 
> 
> The lack of documentation is easy to fix :-)  Would you mind updating
> the 0.41 wiki page?

Have not had real knowledge of the Dynamic feature in 0.41. Just tried
it and it looked like it's working. Now I got the answer from you that
it's really undocumented feature. Perhaps I can create an account to
wiki and update the page...

> The other option is to use a nightly build ...

Is there any place to download nightly builds? Osmosis wiki page
Development Build links are broken.

http://wiki.openstreetmap.org/wiki/Osmosis#Development_Build

Zip Files:
http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/zips/

Tar Balls:
http://dev.openstreetmap.de:23457/hudson/job/osmosis/lastSuccessfulBuild/artifact/package/distrib/tgzs/

Jukka
-- 
Jukka Laaksola
laaks...@iki.fi

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

Hi Jukka,

On 12 February 2013 23:57, Jukka Laaksola  wrote:

> Hi Brett!
>
> Is there any plan to release osmosis 0.42 soon?
>

It's just a matter of finding time.  I'll struggle to do it any time soon.


>
> Now the 0.41 is broken with --used-node option.
>

Yes, the default value was changed to Dynamic after 0.41 was released.


>
> Wiki documentation for 0.41 tells only about idTrackerType values IdList
> and BitSet. But none of them works...
>

Yes, both of them use 32-bit ints internally so they will never work.


>
> There is workaround to use option "--used-node idTrackerType=Dynamic",
> but it is not documented. As I can see it is working also with version
> 0.41. And my tests also looks like Dynamic will work with the version 0.41.
>

The lack of documentation is easy to fix :-)  Would you mind updating the
0.41 wiki page?

The other option is to use a nightly build ...

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Jukka Laaksola]

07.02.2013 11:43, Brett Henderson wrote:
> On 7 February 2013 20:21, Ilya Zverev  wrote:
>> Thanks for clarification. I've checked and agree with you. But as the
>> documentation states, ListIdTracker is still the default options for
>> --used-node and --used-way: http://wiki.openstreetmap.org/**
>> wiki/Osmosis/Detailed_Usage_0.**41#--used-node_.28--un.29
>> Will those break?
>>
> 
> That has been fixed in Git (ie. after 0.41 was released).  I probably
> should release a 0.42 at some point, but haven't had time.  The nightly
> builds should be okay.
> http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.42#--used-node_.28--un.29

Hi Brett!

Is there any plan to release osmosis 0.42 soon?

Now the 0.41 is broken with --used-node option.

Wiki documentation for 0.41 tells only about idTrackerType values IdList
and BitSet. But none of them works...

There is workaround to use option "--used-node idTrackerType=Dynamic",
but it is not documented. As I can see it is working also with version
0.41. And my tests also looks like Dynamic will work with the version 0.41.

Jukka
-- 
Jukka Laaksola
laaks...@iki.fi

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

On 7 February 2013 20:21, Ilya Zverev  wrote:

> Brett Henderson писал 2013-02-07 13:05:
>
>>
>>  Hi! As some of you have read (http://lists.openstreetmap.**
>>> org/pipermail/dev/2013-**February/026495.html[1]),
>>>  in three days node ids are expected to surpass
>>> 2147483647 [2], and this method https://github.com/**
>>> openstreetmap/osmosis/blob/**master/core/src/main/java/org/**
>>> openstreetmap/osmosis/core/**util/LongAsInt.java#L30[3]
>>>  will throw an exception "Cannot represent " + value + " as an integer."
>>> It is used in every IdTracker implementation, so id trackers will become
>>> unusable.
>>>
>>>
>>> This will affect tag and area filters. Regional extracts that are made
>>> with osmosis will break. There is a comment at the start of each IdTracker
>>> class: "The current implementation only supports 31 bit numbers, but will
>>> be enhanced if and when required." I guess, now is the time. Can anybody
>>> fix that? There must be a reason why this hasn't done sooner.
>>>
>>
>> Thanks for the heads up.  I could be wrong but I don't think this is an
>> issue.
>>
>> It is used by ListIdTracker and BitSetIdTracker so those
>> implementations will soon fail if you try to use them.  However, the
>> default implementation is now DynamicIdTracker which doesn't suffer
>> from this issue (I hope ;-).
>>
>> The idTrackerType arguments could probably be removed from the
>> --bounding-box and --bounding-polygon tasks now because the default
>> implementation should be better than specifying one in the vast
>> majority of cases.
>>
>
> Thanks for clarification. I've checked and agree with you. But as the
> documentation states, ListIdTracker is still the default options for
> --used-node and --used-way: http://wiki.openstreetmap.org/**
> wiki/Osmosis/Detailed_Usage_0.**41#--used-node_.28--un.29
> Will those break?
>

That has been fixed in Git (ie. after 0.41 was released).  I probably
should release a 0.42 at some point, but haven't had time.  The nightly
builds should be okay.
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.42#--used-node_.28--un.29


>
> I'd like idTrackerType removed, since after tomorrow using it will
> certainly cause osmosis to fail. I doubt this options was useful, and it
> only puzzled those who were learning osmosis, sometimes being the only
> option besides inPipe and outPipe.
>

I've always avoided removing options unless necessary to avoid breaking
existing scripts, but in this case I agree that it would be better
removed.  It might be nice if Osmosis had a way of deprecating options so
that warnings would be displayed if they were used.  I'm a bit time
constrained at the moment but happy to accept patches/pull requests ;-)

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Ilya Zverev]

Brett Henderson писал 2013-02-07 13:05:


Hi! As some of you have read 
(http://lists.openstreetmap.org/pipermail/dev/2013-February/026495.html 
[1]), in three days node ids are expected to surpass 2147483647 [2], 
and this method 
https://github.com/openstreetmap/osmosis/blob/master/core/src/main/java/org/openstreetmap/osmosis/core/util/LongAsInt.java#L30 
[3] will throw an exception "Cannot represent " + value + " as an 
integer." It is used in every IdTracker implementation, so id trackers 
will become unusable.


This will affect tag and area filters. Regional extracts that are 
made with osmosis will break. There is a comment at the start of each 
IdTracker class: "The current implementation only supports 31 bit 
numbers, but will be enhanced if and when required." I guess, now is 
the time. Can anybody fix that? There must be a reason why this hasn't 
done sooner.


Thanks for the heads up.  I could be wrong but I don't think this is 
an issue.


It is used by ListIdTracker and BitSetIdTracker so those
implementations will soon fail if you try to use them.  However, the
default implementation is now DynamicIdTracker which doesn't suffer
from this issue (I hope ;-).

The idTrackerType arguments could probably be removed from the
--bounding-box and --bounding-polygon tasks now because the default
implementation should be better than specifying one in the vast
majority of cases.


Thanks for clarification. I've checked and agree with you. But as the 
documentation states, ListIdTracker is still the default options for 
--used-node and --used-way: 
http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.41#--used-node_.28--un.29

Will those break?

I'd like idTrackerType removed, since after tomorrow using it will 
certainly cause osmosis to fail. I doubt this options was useful, and it 
only puzzled those who were learning osmosis, sometimes being the only 
option besides inPipe and outPipe.



IZ

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

Re: [osmosis-dev] 32-bit limit in IdTrackers

[Brett Henderson]

Hi IZ,

On 6 February 2013 20:05, Ilya Zverev  wrote:

> Hi! As some of you have read (http://lists.openstreetmap.**
> org/pipermail/dev/2013-**February/026495.html),
> in three days node ids are expected to surpass 2147483647, and this
> method https://github.com/**openstreetmap/osmosis/blob/**
> master/core/src/main/java/org/**openstreetmap/osmosis/core/**
> util/LongAsInt.java#L30will
>  throw an exception "Cannot represent " + value + " as an integer." It
> is used in every IdTracker implementation, so id trackers will become
> unusable.
>
> This will affect tag and area filters. Regional extracts that are made
> with osmosis will break. There is a comment at the start of each IdTracker
> class: "The current implementation only supports 31 bit numbers, but will
> be enhanced if and when required." I guess, now is the time. Can anybody
> fix that? There must be a reason why this hasn't done sooner.
>

Thanks for the heads up.  I could be wrong but I don't think this is an
issue.

It is used by ListIdTracker and BitSetIdTracker so those implementations
will soon fail if you try to use them.  However, the default implementation
is now DynamicIdTracker which doesn't suffer from this issue (I hope ;-).

DynamicIdTracker breaks the id range into chunks of 1024 and internally
uses either ListIdTracker or BitSetIdTracker for each of those chunks
depending on which is more efficient.  As a result, the largest number
either of those id trackers ever sees is 1023, the DynamicIdTracker adds a
base offset to those numbers to get the final number and it stores the base
number as a 64-bit long.

The idTrackerType arguments could probably be removed from the
--bounding-box and --bounding-polygon tasks now because the default
implementation should be better than specifying one in the vast majority of
cases.

I've checked the rest of the codebase for use of the LongToInt class.  It
is used internally by the Entity class to store a changeset id as a 32-bit
number instead of a 64-bit number.  That should be safe for a while yet.

Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev

[osmosis-dev] 32-bit limit in IdTrackers

[Ilya Zverev]

Hi! As some of you have read 
(http://lists.openstreetmap.org/pipermail/dev/2013-February/026495.html), 
in three days node ids are expected to surpass 2147483647, and this 
method 
https://github.com/openstreetmap/osmosis/blob/master/core/src/main/java/org/openstreetmap/osmosis/core/util/LongAsInt.java#L30 
will throw an exception "Cannot represent " + value + " as an integer." 
It is used in every IdTracker implementation, so id trackers will become 
unusable.


This will affect tag and area filters. Regional extracts that are made 
with osmosis will break. There is a comment at the start of each 
IdTracker class: "The current implementation only supports 31 bit 
numbers, but will be enhanced if and when required." I guess, now is the 
time. Can anybody fix that? There must be a reason why this hasn't done 
sooner.



IZ

___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev