Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
On Thu, 2011-06-02 at 03:12 +0200, Geoff McLane wrote: Hi Jason, When you say a 'long time', do you mean more than this, like DAYS? ;=(( Just trying to get an idea of how 'patient' one MUST be... before deciding something is really going 'wrong'... I have found that the standard set (from the old makeYSSY.sh script) takes about and hour to process. Once this has added data from 1:125000 topo's (landmass,river,stream,lake), OSM (freeway,truck,primary,secondary,service roads) then it extends to many hours just for a 2x2 degree chunk. I do not know, or remember, what OS you are using, but after you mentioned it, I decided to take a quick look at server and client in WIN32, and there is absolutely NO WAY they would EVER run in windows ;=(( I am on Gentoo 64 This also happens to be one of my favorite scenery build areas, although I always extend that out to YGIL ;=)) That is the e150s40 and e140s40 chunks... and the s30 group if I want to include YBBN... I plan on doing more once I build an interpreter for contours out of shape files. Anyone know the true meaning of the fit files? are these used instead of the DEM chopped files? Any chance we may see the harbor bridge in the scenery database ;=)) Sydney always looks a bit strange without it, and the Opera House, to name a few iconic indicators... maybe they are already there since I have not done a full check in a while... love too do it but I am not a modeler. :( (note to self, how about a dual camera rig with GPS that can build the wire frames Splinscan without a laser...) Jason Cox -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Geoff McLane wrote: Thanks. It is good to know that the - Default=x, where x = 0-2 is 'normal', but Chris reported that it was still running after 6 hours, or more... and still unable to exactly find where this is output, in the code... You won't be able to find Default in the sources as the output is Landclass = x, Default being Landmass. So there might be something wrong in this direction. And Chris, you can see some of the comments associated with the 2009 patch to the rlimits code you pointed to are NOT correct! Even at that time any attempt to limit memory use was abandoned, and the CPU timeout was doubled... Like it seems you have done, this 'limit' is really NOT applicable to a specific area generation, as apposed to a whole world generation, using server/client, so like you, I 'chop' this 'rlimit' code completely... I put the patch back in, too. Some, like Gijs, and myself, are working on a GUI to assist in this scenery generation, but the important issue IS making sure the TG tools work well, otherwise such a GUI is rather pointless... Exactly, and as long as we don't have a new terrain engine in FG, more efforts have to be put into TG, if we don't want to end up in a mess, scenery-wise. Chris -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Chris/Geoff, I normally see reams of these and other similar lines. I think that you are seeing some form of count for terrain type. My usual output has other textures mixed in there and only ever 3 as the number expressed. If you are using a hight level of detail (OSM residental) and processing a 10x10 then is will take a long time to complete and any tile that has a lot of detail will produce lots of the described output. I have also noticed that the generation takes a long time due to it being single threaded ( hogs a single core ). I have tried to use the server/client but found that it just doesn't appear to work anymore.. Jason Cox ( still building YSSY to YWLM ) On Wed, 2011-06-01 at 01:35 +0200, Geoff McLane wrote: Hi Chris, Glad the rough 'fixes' I provided worked a little for you... have yet to clone your 'papillon81' clone, and try it... but... While it is agreed on this list everyone seems to really _LOVE_ extreme brevity, (perhaps except me ;=)), your post just did not tell me enough ;=(( I searched the entire terragear-cs source for - Default=, and did not get a single hit, well one ... Default=%s\n, out_file), but that's obviously not it, so no idea where to look ;=(( Maybe the output is in simgear, or some of the other libraries that are included in fgfs-construct... Spaces and case are vitally important... But what is the console output immediately prior this 'continuous' output? Maybe that would provide more clues as to where it is in the processing... It is clear fgfs-construct is getting 'stuck' somewhere, but simply need more information... It is certainly _NOT_ 'normal' behavior, and historically (I assume Curt ;=)) implemented some draconian 'rlimit' - setrlimit(RLIMIT_CPU,timeout), to abort after a period of time, which is just NOT available in my WIN32 environment, to avoid such a 'forever' loop... At the time I understand, they were building the WHOLE WORLD world of scenery, and did not want the process 'stuck' on some bucket, or group of buckets... So while I also now play with a Ubuntu linux build, I try HARD to find 'other' solutions for WIN32... like finding, and fixing the 'reason' for such 'forever' loops... So with more information, maybe we can track down, and 'fix' it, forever ;=)) But, _BUT_, be aware, some others, who have tried my 'fixes', including myself ;=)), especially regarding the 'priorities' have succeeded in only producing really _MESSED_ up scenery - see - http://geoffair.org/tmp/mess-01.ppm or http://geoffair.org/tmp/mess-02.jpg and these were generated using my modified WIN32 fgfs construct exe - http://geoffair.org/fg/fgfs-054.htm#downloads so I feel I am very _FAR_ from the solution ;=(( Any input from others, with knowledge, would really HELP... Regards, Geoff. -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Geoff McLane wrote: It is certainly _NOT_ 'normal' behavior, and historically (I assume Curt ;=)) implemented some draconian 'rlimit' - setrlimit(RLIMIT_CPU,timeout), to abort after a period of time, which is just NOT available in my WIN32 environment, to avoid such a 'forever' loop... Hi, while i can't give you the exact output that I reported, right now (i'm not on my Computer where it happened), I have dug a bit deeper into the rlimit stuff. I used TG with the following patch applied (partly strange): http://git.overlays.gentoo.org/gitweb/?p=proj/gamerlay.git;a=blob;f=games- util/terragear-cs/files/terragear-cs- setrlimit.patch;h=bcb166cd1c6311d655416c4aacefa1b71bcd25c4;hb=e330979f290cc3b2259f124ddc9d9527d42280c5 To rule out any influence of it, I built TG without it, but the fgfs- construct process got interrupted very early, even with a small part of scenery to build. So it seems like this patch is needed one or the other way. Maybe with other settings. What do you use there? Chris -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Geoff, I applied both of your patches, see here: https://gitorious.org/papillon81/terragear-cs Until now I had no crashes or negative effects on the resulting scenery. However, there IS a problem, unrelated to your patches, for which I hope to get some help. I create large chunks of scenery with CORINE and OSM data. fgfs-construct runs quite some hours for it and i see the processing is fine. Then, after some time (say 6 hours), all I see in the console output are colums with Default=x where x is a number between 1 and 3. This goes on and on for hours and I stopped it yesterday. Is this supposed normal behaviour? Maybe the process is hanging somewhere, the CPU is buisy the whole time. Any ideas? Cheers Chris -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Hi Chris, Glad the rough 'fixes' I provided worked a little for you... have yet to clone your 'papillon81' clone, and try it... but... While it is agreed on this list everyone seems to really _LOVE_ extreme brevity, (perhaps except me ;=)), your post just did not tell me enough ;=(( I searched the entire terragear-cs source for - Default=, and did not get a single hit, well one ... Default=%s\n, out_file), but that's obviously not it, so no idea where to look ;=(( Maybe the output is in simgear, or some of the other libraries that are included in fgfs-construct... Spaces and case are vitally important... But what is the console output immediately prior this 'continuous' output? Maybe that would provide more clues as to where it is in the processing... It is clear fgfs-construct is getting 'stuck' somewhere, but simply need more information... It is certainly _NOT_ 'normal' behavior, and historically (I assume Curt ;=)) implemented some draconian 'rlimit' - setrlimit(RLIMIT_CPU,timeout), to abort after a period of time, which is just NOT available in my WIN32 environment, to avoid such a 'forever' loop... At the time I understand, they were building the WHOLE WORLD world of scenery, and did not want the process 'stuck' on some bucket, or group of buckets... So while I also now play with a Ubuntu linux build, I try HARD to find 'other' solutions for WIN32... like finding, and fixing the 'reason' for such 'forever' loops... So with more information, maybe we can track down, and 'fix' it, forever ;=)) But, _BUT_, be aware, some others, who have tried my 'fixes', including myself ;=)), especially regarding the 'priorities' have succeeded in only producing really _MESSED_ up scenery - see - http://geoffair.org/tmp/mess-01.ppm or http://geoffair.org/tmp/mess-02.jpg and these were generated using my modified WIN32 fgfs construct exe - http://geoffair.org/fg/fgfs-054.htm#downloads so I feel I am very _FAR_ from the solution ;=(( Any input from others, with knowledge, would really HELP... Regards, Geoff. -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
On Sat, 28 May 2011, Geoff McLane wrote: Hi all, I read (everywhere) that a vector erase invalidates all current iterators - see say - http://www.cplusplus.com/reference/stl/vector/erase/ so I really do not understand why genfans.cxx has lasted so long like it is ;=(( When it does a - tris.erase( t_current ); it should reset the iterators. The small patch attached does just that... Maybe somehow, on some platforms, the iterators are 'correctly' adjusted, but certainly NOT in WIN32, and I suspect the same in other platforms! I would respectfully request the attached patch be applied soonest, unless there is some other reasoning at play here. It certainly works better for me ;=)) And removes another reason why fgfs-construct can abort without apparent reason! Hi Geoff, If I recall correctly (it is still before my morning coffee here :) vector::erase() returns a new iterator that you can use to continue the iteration, so the below should work for continuing the iteration: t_current = tris.erase( t_current ); t_last = tris.end(); However, if you have more iterators around (than t_current and t_last) that will need to be adjusted perhaps a full restart is warranted. Cheers, Anders -- --- Anders Gidenstam WWW: http://www.gidenstam.org/FlightGear/ -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
On Sun, 2011-05-29 at 09:55 +0200, Anders Gidenstam wrote:
On Sat, 28 May 2011, Geoff McLane wrote:
Hi all,
I read (everywhere) that a vector erase invalidates
all current iterators - see say -
http://www.cplusplus.com/reference/stl/vector/erase/
so I really do not understand why genfans.cxx has lasted
so long like it is ;=((
When it does a -
tris.erase( t_current );
it should reset the iterators. The small patch attached
does just that...
Maybe somehow, on some platforms, the iterators are
'correctly' adjusted, but certainly NOT in WIN32, and I
suspect the same in other platforms!
I would respectfully request the attached patch be
applied soonest, unless there is some other reasoning
at play here.
It certainly works better for me ;=)) And removes
another reason why fgfs-construct can abort
without apparent reason!
Hi Geoff,
If I recall correctly (it is still before my morning coffee here :)
vector::erase() returns a new iterator that you can use to continue the
iteration, so the below should work for continuing the iteration:
t_current = tris.erase( t_current );
t_last = tris.end();
However, if you have more iterators around (than t_current and t_last)
that will need to be adjusted perhaps a full restart is warranted.
Cheers,
Anders
Hi Anders,
Thanks for the information... and before your
coffee as well ;=))
If erase does returns a 'next' valid iterator
after the erased item, then a slightly 'faster'
patch would indeed be, as you indicate :-
---
src/Lib/Optimize/genfans.cxx |3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/src/Lib/Optimize/genfans.cxx b/src/Lib/Optimize/genfans.cxx
index 166c354..9551088 100644
--- a/src/Lib/Optimize/genfans.cxx
+++ b/src/Lib/Optimize/genfans.cxx
@@ -225,7 +225,8 @@ opt_list TGGenFans::greedy_build( triele_list tris )
{
// t_current-get_n3()
//from master tri pool
// endl;
- tris.erase( t_current );
+ t_current = tris.erase( t_current );
+ t_last = tris.end();
} else {
++t_current;
}
--
1.7.0.4
since in that particular loop t_current and t_last
are the only iterators used... it compiles ok
like that, but yet to 'test' this change...
But also in this case, a full restart of the loop
would not 'cost' much more... just to be very sure...
Just hope one of the 'fixes' gets into mapserver
terragear cs soon ;=)) adding a little more
stability to the likes of fgfs-construct, as one
exe which uses this library...
Regards,
Geoff.
--
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery,
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now.
http://p.sf.net/sfu/quest-d2dcopy1
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Geoff McLane wrote: It certainly works better for me ;=)) And removes another reason why fgfs-construct can abort without apparent reason! Hi, you mean segfaults with no apparent reason? I experience them under Linux when building huge scenery chunks and if your patch improves the situation, I'll gladly give it a try. Chris -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
On Sun, 2011-05-29 at 11:45 +0200, Christian Schmitt wrote:
Geoff McLane wrote:
It certainly works better for me ;=)) And removes
another reason why fgfs-construct can abort
without apparent reason!
Hi,
you mean segfaults with no apparent reason? I experience them under Linux
when building huge scenery chunks and if your patch improves the situation,
I'll gladly give it a try.
Chris
Hi Chris,
Exactly ;=))
I am not sure if it is the WHOLE reason, but
under WIN32, where you get a 'small' indication
of where the problem occurred, even on the release
version - ie no debug symbols available - this is
certainly one, perhaps _NOT_ the only ;=((, where
a segfault can occur...
This part of the 'genfans' library is _NOT_ run on
every bucket being built... so MANY buckets can
be ok... ie no segfault... only in certain
situations...
If you do your own 'exe' build, please give the
patch a try... and advise if the situation
improves...
This would certainly help the 'maintainers' of
terragear cs to decide to incorporate the
patch... although, at the moment, it seems
very obvious to me...
Added to that... I have found another
'no-apparent-reason' segfault can be caused
by -
Re: src/BuildTiles/Clipper/priorities.cxx
ANOTHER patch I am working on for fgfs-contruct,
and currently testing, is the building of the
'area_types' array...
This is built from the reading of the
'default_priorities.txt' file, or such other
file as you designate via the '--priorities=file'
specific command...
The reading of this file creates the 'area_types'
array, which, no matter what, _MUST_ have at least
'TG_MAX_AREA_TYPES' members (= 128 in the current
build) ...
There are a number of functions, here and there,
having no 'knowledge' of the size of the 'area_types'
array, which use something like :-
for (i = 0; i TG_MAX_AREA_TYPES; i++) {
area_type_descriptor type = get_area_descriptor(i);
...
BUT the static function -
get_area_descriptor(AreaType area) -
(a) has a crazy, bad, stupid, range checking of -
if ( 0=area || area area_types.size() ) {
This _MUST_ be something more like :-
if ( area = 0 area area_types.size() ) {
(b) thus the array may not be less than
TG_MAX_AREA_TYPES in size()...
if ( area = 0 area area_types.size() ) {
return area_types[area];
} else {
SG_LOG(SG_GENERAL, SG_ALERT, unknown area code =
(int)area);
exit(-1);
}
With the current range checking of -
if ( 0=area || area area_types.size() ) {
this again means you can have a segfault when
'area' is above the area_types.size() ;=((
My quick patch for this is attached, but it presently
depends on the _LAST_ entry in 'default_priorities.txt',
or such other designated file, having a FINAL entry
of
Unknown other
to fill the array out to 128 with 'Unknown', of
priority value 7...
With these patches in place, I no longer get
'strange' aborts from fgfs-construct ;=)) but
welcome others testing, and reporting...
Regards,
Geoff.
attached: priorities-01.patch
--- C:\FGCVS\terragear-cs\src\BuildTiles\Clipper\priorities.cxx Sun Oct 10 10:12:53 2010
+++ C:\FG\TG2\terragear-cs\src\BuildTiles\Clipper\priorities.cxx Fri May 27 18:15:47 2011
@@ -31,6 +31,11 @@
#include priorities.hxx
+#ifndef TG_MAX_AREA_TYPES
+#define TG_MAX_AREA_TYPES 128 // FIXME also defined in
+// MergerClipper/clipper.hxx
+#endif // TG_MAX_AREA_TYPES
+
using std::ifstream;
using std::string;
using std::map;
@@ -75,8 +80,9 @@
in skipcomment;
in sliver_area_name;
in skipcomment;
+area_type_descriptor descriptor;
while ( !in.eof() ) {
-area_type_descriptor descriptor;
+//area_type_descriptor descriptor;
string type;
descriptor.kind = Other;
in descriptor.name;
@@ -109,6 +115,18 @@
sliver_target_area_type = get_area_type( sliver_area_name );
default_area_type = get_area_type( default_area_name );
+
+if (area_types.size() TG_MAX_AREA_TYPES) {
+SG_LOG(SG_GENERAL, SG_INFO, EEK: Area type count = area_types.size()
+ being less that MAX = TG_MAX_AREA_TYPES UGH! );
+while (area_types.size() TG_MAX_AREA_TYPES) {
+AreaType index = (AreaType)area_types.size();
+area_types.push_back(descriptor);
+area_names[descriptor.name]=index;
+SG_LOG(SG_GENERAL, SG_INFO, descriptor.name descriptor.kind);
+}
+return 2;
+}
return 1;
}
@@ -125,12 +143,15 @@
}
+//if ( 0=area || area area_types.size() ) {
+// the above should be AND nor OR ;=((
static area_type_descriptor get_area_descriptor( AreaType area ) {
-if ( 0=area || area area_types.size() ) {
+if (( area = 0 ) ( area area_types.size() )) {
return area_types[area];
} else {
SG_LOG(SG_GENERAL, SG_ALERT, unknown area code = (int)area);
exit(-1);
+// return
[Flightgear-devel] terragear - file src/Lib/Optimize/genfans.cxx
Hi all,
I read (everywhere) that a vector erase invalidates
all current iterators - see say -
http://www.cplusplus.com/reference/stl/vector/erase/
so I really do not understand why genfans.cxx has lasted
so long like it is ;=((
When it does a -
tris.erase( t_current );
it should reset the iterators. The small patch attached
does just that...
Maybe somehow, on some platforms, the iterators are
'correctly' adjusted, but certainly NOT in WIN32, and I
suspect the same in other platforms!
I would respectfully request the attached patch be
applied soonest, unless there is some other reasoning
at play here.
It certainly works better for me ;=)) And removes
another reason why fgfs-construct can abort
without apparent reason!
Regards,
Geoff.
patch:
diff -ur git\terragear-cs\src\Lib\Optimize\genfans.cxx local
\terragear-cs\src\Lib\Optimize\genfans.cxx
--- git\terragear-cs\src\Lib\Optimize\genfans.cxx Sun Oct 10 10:13:00
2010
+++ local\terragear-cs\src\Lib\Optimize\genfans.cxx Sat May 28 00:42:43
2011
@@ -213,6 +213,8 @@
// add this fan to the fan list
fans.push_back( node_list );
+// restart on delete !!!
+Re_Start:
// delete the triangles in best_fan out of tris and repeat
triele_list_iterator t_current = tris.begin();
triele_list_iterator t_last = tris.end();
@@ -226,6 +228,7 @@
//from master tri pool
// endl;
tris.erase( t_current );
+goto Re_Start;
} else {
++t_current;
}
--
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery,
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now.
http://p.sf.net/sfu/quest-d2dcopy1
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
