On 12/13/2012 11:04 AM, Dominik Perpeet wrote:
It seems like osm2pgsql for windows used to be kept at
http://tile.openstreetmap.org/osm2pgsql.zip and is still from 2010.
I wonder if your version can be put up as a replacement for this at
the same location?
Maybe eventually. For now I have my own server I can use.
The idea behind the same location was so that any documentation or links
that still point to that location would automatically be up-to-date
again. That would probably be easier than finding any and all
documentation for windows and update them. Although perhaps it would be
good anyway to check if the rest still works.
Using MinGW or the Intel compiler seems to be one route? Coverting
things to C++ might be another?
Any suggestions what the most sensible route is in general for the
best portability?
I started removing the C99 code myself, but quit because it was too much
work (and a bit error-prone). Since I don't see a good reason to use C,
I'd choose the C++ route.
Sounds like the same reason I gave up on it too. But if it were of help,
I could finish it off fairly quickly.
How much work would it be to move things to C++? Any disadvantages for it?
The current code now has a configure check to test if fork is
available and if doesn't compile the multi-process code. I haven't
looked at the advantages and disadvantages of OpenMP, but perhaps
indeed that is an option. However, I suspect the big issue there are
the connections to postgresql, as each thread / process needs to setup
its own connections and transactions to postgresql.
Jochen's clang argument seems persuasive. I don't have much OSX
developing experience, so I wouldn't know. One reason I didn't add
Windows threaded code is that the postgresql connection stuff is a bit
scattered. I would be willing to refactor the code into C++ classes.
Once the connection variables are a bit more concentrated, it wouldn't
affect fork() and other threading systems could be implemented.
I was trying to remember why I chose the fork() route, as I started off
with threads. I think the reason was that part of the code (possibly
text-tree.c) used global or static variables and thus wasn't thread
safe. It was easier at the time to use fork() instead of fixing the rest
of the code to be thread safe.
However, going the fork() route has caused issues on the unix side as
well. Osm2pgsql uses a large amount of RAM for its node cache. Forking
the process then results in having e.g. many processes using 16GB each.
Theoretically that isn't a problem, as thanks to copy-on-write that
memory is shared and therefore only allocated once. However, as the OS
has no knowledge that that memory will never be written to and thus
remains shared, it may deny the fork request to prevent too much over
commit. Going back to threads would eliminate that issue.
So probably the best way forward in this case is to make the rest of the
code properly thread safe and use threads for everything instead of fork.
Once the code is refactored, there are alternatives to using openmp:
- Qt (personal preference, but big dependence)
- tbb (requires experience with tbb and can be a bit tricky to use
correctly)
- boost (don't really like using it for basic stuff like threads, also a
big dependence)
To my knowledge, all three have support for OSX. Qt is also nice because
it has different sql backends.
Without adding another library dependency, it should be possible to use
threads just for Windows. While it wouldn't break code for anyone
currently using osm2pgsql, it would lead to diverging paths in the
codebase which are a bit more difficult to maintain. For starters, I
think this would be the best option though - nobody likes too many
dependencies.
Yes, I think there is serious interest in this. And if reasonably
possible, I would like to get as many of those changes into the svn
master repository as possible to make it easier to compile on windows
in the future.
Ok, I'll look into it. I think once I switched to the Intel Compiler I
didn't have to change too much. The biggest hassle in Windows is
building and linking the dependencies anyway. =)
Dominik
_______________________________________________
dev mailing list
[email protected]
http://lists.openstreetmap.org/listinfo/dev
