[Flightgear-devel] Automated builds tests
Hi everybody I'd like to hear thoughts from the FG community about setting up a system to perform builds execute a suite of tests on FlightGear, all automatically. Right now I've experimented a bit with buildbot, a neat continuous integration tool used by Mozilla and other projects, and I have a system that can: * check-out from various repositories * build all FlightGear components * perform rudimentary tests on the FG simulator just built, like verifing the output on the command line and starting the simulator. Now the next step would be to go airborne! And there are two issues to resolve before take-off: 1) how to drive the input of the simulator 2) how to read its state For the second one, I've seen examples of reading the property tree from an external process, so we should be set, but the solution to driving the sim's input is still not clear. Specifically, I'd want to drive it as similarly as possible as when it's controlled from a keyboard, not go through the property tree to force FGFS into certain conditions. By the way, the current setup works on Ubuntu x86-64, but buildbot is easily extensible and supports Windows and MacOS platforms, so this could become a cross-platform testing tool for the project. Thanks, Tom Tom-cat -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Automated builds tests
On Tue, 4 Aug 2009 00:00:39 -0700, Tom wrote in message c9e106b9090804l27fef105g5b047571a6c75...@mail.gmail.com: Hi everybody I'd like to hear thoughts from the FG community about setting up a system to perform builds execute a suite of tests on FlightGear, all automatically. Right now I've experimented a bit with buildbot, a neat continuous integration tool used by Mozilla and other projects, and I have a system that can: * check-out from various repositories * build all FlightGear components * perform rudimentary tests on the FG simulator just built, like verifing the output on the command line and starting the simulator. Now the next step would be to go airborne! And there are two issues to resolve before take-off: 1) how to drive the input of the simulator 2) how to read its state For the second one, I've seen examples of reading the property tree from an external process, so we should be set, but the solution to driving the sim's input is still not clear. ..why not feed it a flight plan to fly on autopilot. Specifically, I'd want to drive it as similarly as possible as when it's controlled from a keyboard, not go through the property tree to force FGFS into certain conditions. ..fgfs offers an --enable-auto-coordination option to ease keyboard control issues on e.g. laptops with exotic no-numpad keyboard layouts. By the way, the current setup works on Ubuntu x86-64, but buildbot is easily extensible and supports Windows and MacOS platforms, so this could become a cross-platform testing tool for the project. Thanks, Tom Tom-cat -- ..med vennlig hilsen = with Kind Regards from Arnt... ;o) ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Automated builds tests
Are we talking about validating the build process and checking that FG runs, or about checking the validity of the simulation? For the former the suggested buildbot , or similar, approach, perhaps with a very simple autopilot guided flight, would be adequate. Simulation validity checking is another issue, and given the (I hope I do not offend anyone) rather basic flight models used for most aircraft models in the FG library as well as the limited availability of accurate predicted response data is probably not attainable by a project such as Flightgear. As a now retired flight simulator professional most of my time was devoted to checking and validating my simulations before they would be accepted for training (by CAA/FAA) or for flight handling research (by my company's aerodynamics department). Each aircraft model required tests tailored to the use that the simulation was designed for. _ From: Curtis Olson [mailto:curtol...@gmail.com] Sent: 04 August 2009 12:37 To: FlightGear developers discussions Subject: Re: [Flightgear-devel] Automated builds tests On Tue, Aug 4, 2009 at 2:00 AM, Tom P zomm...@gmail.com wrote: Hi everybody I'd like to hear thoughts from the FG community about setting up a system to perform builds execute a suite of tests on FlightGear, all automatically. Right now I've experimented a bit with buildbot, a neat continuous integration tool used by Mozilla and other projects, and I have a system that can: * check-out from various repositories * build all FlightGear components * perform rudimentary tests on the FG simulator just built, like verifing the output on the command line and starting the simulator. Now the next step would be to go airborne! And there are two issues to resolve before take-off: 1) how to drive the input of the simulator 2) how to read its state For the second one, I've seen examples of reading the property tree from an external process, so we should be set, but the solution to driving the sim's input is still not clear. Specifically, I'd want to drive it as similarly as possible as when it's controlled from a keyboard, not go through the property tree to force FGFS into certain conditions. By the way, the current setup works on Ubuntu x86-64, but buildbot is easily extensible and supports Windows and MacOS platforms, so this could become a cross-platform testing tool for the project. Hi Tom, Because, of variations in flight dynamics models, possible variations in weather conditions, possible variations in frame rates, etc., simply replaying a series of keyboard commands is probably not going to lead to repeatable results. I suspect the replayed flight could diverge quite quickly and quite substantially from the original flight. If you want to test the simulator during flight, I really think you will have the most luck under some sort of scripted autopilot control. You should think about exactly what you are trying to measure and validate. As soon as you fire up the sim and start the aircraft moving, you've suddenly moved into the world of flight dynamics and you are looking at the physics/mathematics model of the aircraft. That's a good thing to look at though. One idea to consider is to setup a series of scripted flight tests that parallel the FAA simulator certification tests. I've gone through the Level 3 FTD set of tests and automated them for work (so I can't share the resulting scripts unfortunately) but it was an interesting process. For instance, configure some specific weather conditions, start the aircraft out at some particular altitude and speed. Setup the aircraft with a specific weight and CG. Configure the throttle for some particular RPM, keep the wings straight and level. Now measure the rate of climb (descent) you observe once the phugoid settles out. Another test involved setting up straight and level flight at certain known conditions, then commanding full rudder input while maintaining the same heading and altitude (steady state side slip.) Measure the resulting bank angle, amount of required aileron input, and side slip angle. This is all very interesting stuff, but it tends to focus you more on the validity of the aircraft model and less on the validity of the simulator code. The complexity of this in combined with the complexity of everything else you could be testing and evaluating is quite staggering. That said, having a few key spot or sanity checks can't hurt either. Regards, Curt. -- Curtis Olson: http://baron.flightgear.org/~curt/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ Flightgear-devel mailing list
Re: [Flightgear-devel] [Jsbsim-devel] Missing header file: simgear/math/SGMath.hxx
Jon S. Berndt wrote: Erik Hofman wrote: Apperently this was needed to compile with MSVC 9 (patch was added by Fred twice). I probably should have made a test build before committing to CVS. Which wouldn't have revealed the problem.. it might get picked up at another location for me. I'd like to back out that change and then see the error messages that result in an MSVC 9 compile of FlightGear. Maybe we can track it down. I don't recall that the JSBSim code should have changed so much to require the addition of those two header files. Good idea, I'll crosspost it to flightgear-devel in case fred doesn't read this list. Erik -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] shape-decode crash
Hi all,
shape-decode has been crashing on me for a certain shapefile and I can
not figure out why. I am using terragear-cs downloaded today from the
GIT repository. The rest of the toolchain runs fine.
shape-decode runs for a while and stops with the following error:
[...]
distance = 239.828
0.623843, 46.9914, 0
distance = 7.9
0.626076, 46.9898, 0
min = -2930.42, -592.671, 200 max = 0.62615, 46.997, -200
Bucket min = -180:0, -593:2
Bucket max = 0:2, 46:7
polygon spans tile boundaries
dx = 722 dy = 5117
terminate called after throwing an instance of 'sg_exception'
Aborted
A run under gdb yielded the following:
(gdb) run --max-segment 500 --line-width 6
/storage1/flightgear/scenery-dev/terrain/custom_road_rail_from_OSM/france/fg_railways
Railroad Railroad log 2 log2
Starting program:
/storage1/flightgear/scenery-dev/terragear/install/terragear-cs/bin/shape-decode
--max-segment 500 --line-width 6
/storage1/flightgear/scenery-dev/terrain/custom_road_rail_from_OSM/france/fg_railways
Railroad Railroad log 2 log2
[Thread debugging using libthread_db enabled]
[New Thread 0x7f866f54b6f0 (LWP 10926)]
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f866f54b6f0 (LWP 10926)]
0x7f866e447015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x7f866e447015 in raise () from /lib/libc.so.6
#1 0x7f866e448b83 in abort () from /lib/libc.so.6
#2 0x7f866ecebf94 in __gnu_cxx::__verbose_terminate_handler () from
/usr/lib/libstdc++.so.6
#3 0x7f866ecea396 in ?? () from /usr/lib/libstdc++.so.6
#4 0x7f866ecea3c3 in std::terminate () from /usr/lib/libstdc++.so.6
#5 0x7f866ecea4aa in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x0041175a in tgChopNormalPolygon (pa...@0x7fff7756f220,
poly_ty...@0x7fff7756f130, sha...@0x7fff7756eda0, preserve3d=false) at
chop-bin.cxx:222
#7 0x00405b50 in processLine (psShape=0x202d870,
work_d...@0x7fff7756f220, poly_ty...@0x7fff7756f130, linewidth=6) at
shape-decode.cxx:545
#8 0x0040a5f1 in main (argc=4, argv=0x7fff7756f3c8) at
shape-decode.cxx:920
(gdb) list *0x0041175a
0x41175a is in tgChopNormalPolygon(std::string const, std::string
const, TGPolygon const, bool) (chop-bin.cxx:208).
203 SG_LOG( SG_GENERAL, SG_DEBUG, Bucket min = b_min );
204 SG_LOG( SG_GENERAL, SG_DEBUG, Bucket max = b_max );
205
206 if ( b_min == b_max ) {
207 // shape entirely contained in a single bucket, write
and bail
208 clip_and_write_poly( path, index, poly_type, b_min,
shape, preserve3d );
209 return;
210 }
211
212 SGBucket b_cur;
This happens with the following shapefile:
http://www.mguillaud.net/fg/fg_railways.tgz
Note that I can run some other shapefiles through shape-decode without
problem. That particular file was generated by PostGIS using pgsql2shp.
It does not appear to be invalid.
Any advice ?
Regards,
Maxime
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] shape-decode crash
On Tue, Aug 4, 2009 at 8:11 AM, Maxime Guillaud m...@mguillaud.net wrote:
Hi all,
shape-decode has been crashing on me for a certain shapefile and I can
not figure out why. I am using terragear-cs downloaded today from the
GIT repository. The rest of the toolchain runs fine.
shape-decode runs for a while and stops with the following error:
[...]
distance = 239.828
0.623843, 46.9914, 0
distance = 7.9
0.626076, 46.9898, 0
min = -2930.42, -592.671, 200 max = 0.62615, 46.997, -200
Bucket min = -180:0, -593:2
A bucket coordinate of -593:2 suggests that this shapefile may contain some
bogus data. (or there could be a bug leading up to this) but I believe the
portion prior to the : represents a whole degree coordinate of the bucket,
so this should range from -180 to +179 for latitude and -90 to +89 for
longitude. -593 is completely outside of possible reality. So the question
is, is this a problem with the shapefile data, or some problem processing
the data in the shape-decode code.
Regards,
Curt.
Bucket max = 0:2, 46:7
polygon spans tile boundaries
dx = 722 dy = 5117
terminate called after throwing an instance of 'sg_exception'
Aborted
A run under gdb yielded the following:
(gdb) run --max-segment 500 --line-width 6
/storage1/flightgear/scenery-dev/terrain/custom_road_rail_from_OSM/france/fg_railways
Railroad Railroad log 2 log2
Starting program:
/storage1/flightgear/scenery-dev/terragear/install/terragear-cs/bin/shape-decode
--max-segment 500 --line-width 6
/storage1/flightgear/scenery-dev/terrain/custom_road_rail_from_OSM/france/fg_railways
Railroad Railroad log 2 log2
[Thread debugging using libthread_db enabled]
[New Thread 0x7f866f54b6f0 (LWP 10926)]
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f866f54b6f0 (LWP 10926)]
0x7f866e447015 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x7f866e447015 in raise () from /lib/libc.so.6
#1 0x7f866e448b83 in abort () from /lib/libc.so.6
#2 0x7f866ecebf94 in __gnu_cxx::__verbose_terminate_handler () from
/usr/lib/libstdc++.so.6
#3 0x7f866ecea396 in ?? () from /usr/lib/libstdc++.so.6
#4 0x7f866ecea3c3 in std::terminate () from /usr/lib/libstdc++.so.6
#5 0x7f866ecea4aa in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x0041175a in tgChopNormalPolygon (pa...@0x7fff7756f220,
poly_ty...@0x7fff7756f130, sha...@0x7fff7756eda0, preserve3d=false) at
chop-bin.cxx:222
#7 0x00405b50 in processLine (psShape=0x202d870,
work_d...@0x7fff7756f220, poly_ty...@0x7fff7756f130, linewidth=6) at
shape-decode.cxx:545
#8 0x0040a5f1 in main (argc=4, argv=0x7fff7756f3c8) at
shape-decode.cxx:920
(gdb) list *0x0041175a
0x41175a is in tgChopNormalPolygon(std::string const, std::string
const, TGPolygon const, bool) (chop-bin.cxx:208).
203 SG_LOG( SG_GENERAL, SG_DEBUG, Bucket min = b_min );
204 SG_LOG( SG_GENERAL, SG_DEBUG, Bucket max = b_max );
205
206 if ( b_min == b_max ) {
207 // shape entirely contained in a single bucket, write
and bail
208 clip_and_write_poly( path, index, poly_type, b_min,
shape, preserve3d );
209 return;
210 }
211
212 SGBucket b_cur;
This happens with the following shapefile:
http://www.mguillaud.net/fg/fg_railways.tgz
Note that I can run some other shapefiles through shape-decode without
problem. That particular file was generated by PostGIS using pgsql2shp.
It does not appear to be invalid.
Any advice ?
Regards,
Maxime
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus
on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
--
Curtis Olson: http://baron.flightgear.org/~curt/
--
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] shape-decode crash
Curtis Olson wrote: A bucket coordinate of -593:2 suggests that this shapefile may contain some bogus data. (or there could be a bug leading up to this) but I believe the portion prior to the : represents a whole degree coordinate of the bucket, so this should range from -180 to +179 for latitude and -90 to +89 for longitude. -593 is completely outside of possible reality. So the question is, is this a problem with the shapefile data, or some problem processing the data in the shape-decode code. Regards, Curt. Thanks for spotting this, Curt ! Looking at the output of shape-decode, it looks like the data read from the shapefile is reasonable: Point 5 (0.609977, 46.997) Point 6 (0.605342, 46.9964) Point 7 (0.601051, 46.9955) However, in the following processing steps the coordinates for point 7 appear wrong: point = 6 0.605333, 46.9964, 0 - -2828.93, -570.523, 0 The relevant excerpt of the log is here: http://www.mguillaud.net/fg/log2 Anyone familiar with the internals of shape-decode (I am not) can comment on this ? Regards, Maxime -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Multithreading support
On Monday 03 Aug 2009, Curtis Olson wrote: I'll toss in a couple thoughts. Running on 4 processors (quad-core AMD 64 bit machine) and 4 dual-head nvidia cards we split the render task up into a bunch of subthreads. The overall CPU load was pretty balanced and each CPU ran at about 40-60% utilization. That's interesting. Could you elaborate on that a little more i.e. did you split a single scene into 'render boxes' or were you, in effect, running four discrete but 'collaborative' instances, each just looking in a different direction? I don't know all the solid conclusions we can derive from this, but it seems to me that until we have a single CPU that is fully saturated, adding cores and adding threads will not buy us anything in terms of end-to-end performance. Drawing the scene still is (and always has been) our main bottleneck by far. What proportion of the drawing task is occupied with assembling the scene objects, compared with passing the assembled data to the video card for display? (I'm just wondering about the scope for box-rendering here) One of the big problems I had with FG was its pseudo asynchronous operation, which still meant that the rates at which you could run things like the FDM, autopilot and Nasal were effectively limited by the frame rate and which could lead to an aircraft being stable on one system but unstable on another. I would really like to see these subsystems running on their own cores, or even more preferably, on their own networked hardware, so that I could run them at much higher and guaranteed rates. I think the bottom line is that FG is simply going to have to face up to this issue at some point. A few people here have been bringing this topic up for a few years and now that multicore processors are the norm it's clear that the issue isn't going to go away. Like it or not, and I mean no offence or criticism by this, the current FG architecture is now obsolete. While single core and single processor systems were the norm it was fine - the software design was well fitted to the systems on which it ran - but parallel processing has always been the way things were going to go. It has been inevitable ever since super-computer designers realised that the only way that ever increasing performance could be achieved was by parallelism and now it's well and truly on the desktop. Of course, it's not going to be easy, but denying it won't make the issue go away. Tied in with this, I think, is that FG is still very much a developer-led project, so the developers, by and large, only work on things that interest them. However, the FG user base is constantly growing, and will continue to grow, and at some point FG will have to become user-led if it is to become more than just a stage for its developers to perform upon. FG will, if it has not already, reach the point where it is bigger than its developers and the project should dictate what the developers do, rather than the other way around. I think its a sad fact of reality but I think that FG has reached the point where FG development needs to be actively managed and directed, and not just left to the whims and desires of its developers. Like I said, I mean no criticism of the developers by this; they have achieved an immense amount by getting FG to this point, but in getting FG this far they've made FG into a bigger thing that needs handling differently if it isn't going to stagnate and be left behind. LeeE -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Multithreading support
On Tue, Aug 4, 2009 at 11:05 AM, leee l...@spatial.plus.com wrote: That's interesting. Could you elaborate on that a little more i.e. did you split a single scene into 'render boxes' or were you, in effect, running four discrete but 'collaborative' instances, each just looking in a different direction? This was a single instance of FlightGear rendering to 8 separate windows. What proportion of the drawing task is occupied with assembling the scene objects, compared with passing the assembled data to the video card for display? (I'm just wondering about the scope for box-rendering here) I'm not sure I fully understand your question, but a lot of this is dependent on how much of the opengl pipeline is implemented in hardware versus libraries/drivers on a particular video card and operating system. (Quality and efficiency of the actual implementation also factors in here.) There can be huge variability here across systems. One of the big problems I had with FG was its pseudo asynchronous operation, which still meant that the rates at which you could run things like the FDM, autopilot and Nasal were effectively limited by the frame rate and which could lead to an aircraft being stable on one system but unstable on another. I would really like to see these subsystems running on their own cores, or even more preferably, on their own networked hardware, so that I could run them at much higher and guaranteed rates. I think the bottom line is that FG is simply going to have to face up to this issue at some point. What you are referring to here is less of a threaded system and more of a real time system. You want guarantees that a function will execute at a specific fixed rate. Threads don't automatically give you that, in fact they don't give you that at all. You need a real time system with some sort of external timmer trigger and an interrupt service routine to get really tight timings. Personally I've seen several threaded applications that *thought* they were getting their routines to run at a particular rate until they dug in and actually timed what was going on and throught a bit more about how their code was structured and what the thread primatives they were using actually offered. A few people here have been bringing this topic up for a few years and now that multicore processors are the norm it's clear that the issue isn't going to go away. Like it or not, and I mean no offence or criticism by this, the current FG architecture is now obsolete. The only point I would make here is that threading and timing is an extremely snakey area to deal with. Most people I've talked to don't understand the nuances of what's really going on nearly as well as they imagine they do. I certainly admit to gaps and weaknesses in my own understanding. And I certainly don't mean to imply that anyone participating in this discussion falls under that same description. Here's my question to you: Once you've split the flight dynamics (for example) off into a separate thread/process running on it's own dedicated core, what mechanism do you propose to use to force it to run at a specific fixed and guaranteed rate? And this is an honest question, bucause the generic cross-platform timing mechanisms I'm aware have few guarantees. You might have the best luck with a busy/wait type architecture, but then you are needlessly burning CPU most of the time, since the flight dynamics will consume maybe 0.0001% of the dedicated CPU. Or do you somehow setup a counter and a compare register and use that to trigger an interrupt at a fixed interval and run an iteration of the flight dynamics out of the interrupt service routine? But can you do that in a cross-platform independent way? Can you even get access to these low level mechanisms from most modern operating systems? While single core and single processor systems were the norm it was fine - the software design was well fitted to the systems on which it ran - but parallel processing has always been the way things were going to go. It has been inevitable ever since super-computer designers realised that the only way that ever increasing performance could be achieved was by parallelism and now it's well and truly on the desktop. And the big challenge with modern super computer clusters is to find the parallelism in your task (as much that exists) or to redesign the task to improve the parallelism. (Well that and how to keep 2000 8-core machines running at the same time in a closet from reaching temperatures so hot they melt their way through to the earth's core.) But you are faced with really fast processors and in comparison, very slow interconnects between nodes ... even with fast/low latency networks like myrinet and infiniband. Of course, it's not going to be easy, but denying it won't make the issue go away. What I have always hoped to communicate is that we need solid justification before adding more complex
Re: [Flightgear-devel] Multithreading support
On Tue, 4 Aug 2009, leee wrote: One of the big problems I had with FG was its pseudo asynchronous operation, which still meant that the rates at which you could run things like the FDM, autopilot and Nasal were effectively limited by the frame rate and which could lead to an aircraft being stable on one system but unstable on another. I would really like to see these subsystems running on their own cores, or even more preferably, on their own networked hardware, so that I could run them at much higher and guaranteed rates. Moving these tasks into different threads would only bring you a cart load of indeterminism due to the whims of the OS scheduler, page faults, interrupts, bus contention or network latency etc etc.. The crucial observation you missed here is that these tasks need to run interleaved at high (120+ Hz) guaranteed rates in /simulated/ time not necessarily in real time. (Sampling pilot inputs needs to be done at what rate? 30Hz?). Interleaving them in one thread is really the very best you can do assuming your processor core is fast enough (and I'd be surprised if it isn't, e.g. JSBSim needs in the order of 1% of the capacity of a not so current core to run a typical model in real-time). IMHO the one important threading benefit is if we could get all of the rendering off the main simulation loop, meaning that the model runs independent of the presentation. (Ok, expensive environment eye candy like the traffic manager or wild fire CA would also be beneficial to move into threads - but they don't have tight synchronization needs wrt the main aircraft.) Apart from that the potential for improvements with threading I can see is in the rendering system, and that's perhaps a more general OSG and OpenGL vendor playing field than a FlightGear one. Cheers, Anders -- --- Anders Gidenstam WWW: http://www.gidenstam.org/FlightGear/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Multithreading support
On Tuesday 04 Aug 2009, Curtis Olson wrote: On Tue, Aug 4, 2009 at 11:05 AM, leee l...@spatial.plus.com wrote: That's interesting. Could you elaborate on that a little more i.e. did you split a single scene into 'render boxes' or were you, in effect, running four discrete but 'collaborative' instances, each just looking in a different direction? This was a single instance of FlightGear rendering to 8 separate windows. Ok - ta. What proportion of the drawing task is occupied with assembling the scene objects, compared with passing the assembled data to the video card for display? (I'm just wondering about the scope for box-rendering here) I'm not sure I fully understand your question, but a lot of this is dependent on how much of the opengl pipeline is implemented in hardware versus libraries/drivers on a particular video card and operating system. (Quality and efficiency of the actual implementation also factors in here.) There can be huge variability here across systems. Fair enough. To be honest, the question was at the limits of my understanding. What inspired it though is that when I'm rendering any of my 3D stuff the rendering process is distributed across several systems - the single scene is split into many separate boxes, and because the performance of the different systems in my render farm varies widely, I typically split the scene up into 20x10 boxes. Now this is all ray-traced software rendering, not hardware rendering, and there is an additional overhead because the scene needs to be split up and the subsequent results combined, but the bottom line is that this technique can allow much more processing power to be used and certainly enough to compensate for the overhead. I've honestly no idea though, how far, or even if this technique can be applied to h/w rendering. One of the big problems I had with FG was its pseudo asynchronous operation, which still meant that the rates at which you could run things like the FDM, autopilot and Nasal were effectively limited by the frame rate and which could lead to an aircraft being stable on one system but unstable on another. I would really like to see these subsystems running on their own cores, or even more preferably, on their own networked hardware, so that I could run them at much higher and guaranteed rates. I think the bottom line is that FG is simply going to have to face up to this issue at some point. What you are referring to here is less of a threaded system and more of a real time system. You want guarantees that a function will execute at a specific fixed rate. Threads don't automatically give you that, in fact they don't give you that at all. You need a real time system with some sort of external timmer trigger and an interrupt service routine to get really tight timings. Personally I've seen several threaded applications that *thought* they were getting their routines to run at a particular rate until they dug in and actually timed what was going on and throught a bit more about how their code was structured and what the thread primatives they were using actually offered. I'm not really thinking in terms of 'threading' at all, which I think is a very limited and half-house sort of technique. But neither though do I think it needs to be thought of as a pure real time system. Rather, I'm thinking in terms of the external FDM mechanism already present in FG. Running the FDM on it's own hardware system doesn't need to be any more real time than the FDM running within FG on the same system but because it's not going to be limited by the frame rate it could safely be run much faster and with proportionately more consistency than with FG. If you're running it at say 100Hz within FG I would expect to be able to run it several times faster, if not tens of times faster if the system it was running on wasn't spending most of its time rendering. You'll still get a variation in the rate that the FDM runs at but I suspect that the variation would be about the same in absolute terms. Let's say that if we get a variation of +/- 10 iteration difference per second running within FG it would probably be about the same running on its own system, but as we're running at a higher rate the difference is proprotionally smaller, perhaps down from 10% to 1%. A few people here have been bringing this topic up for a few years and now that multicore processors are the norm it's clear that the issue isn't going to go away. Like it or not, and I mean no offence or criticism by this, the current FG architecture is now obsolete. The only point I would make here is that threading and timing is an extremely snakey area to deal with. Most people I've talked to don't understand the nuances of what's really going on nearly as well as they imagine they do. I certainly admit to gaps and weaknesses in my own understanding. And I certainly don't
Re: [Flightgear-devel] Multithreading support
Anders Gidenstam wrote: IMHO the one important threading benefit is if we could get all of the rendering off the main simulation loop, meaning that the model runs independent of the presentation. (Ok, expensive environment eye candy like the traffic manager or wild fire CA would also be beneficial to move into threads - but they don't have tight synchronization needs wrt the main aircraft.) This might actually be better left to a multiplayer-like client process that feeds one or more FlightGear servers, instead of placing it in it's own thread within FlightGear. Erik -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Multithreading support
leee wrote: I'm not really thinking in terms of 'threading' at all, which I think is a very limited and half-house sort of technique. But neither though do I think it needs to be thought of as a pure real time system. Rather, I'm thinking in terms of the external FDM mechanism already present in FG. Running the FDM on it's own hardware system doesn't need to be any more real time than the FDM running within FG on the same system but because it's not going to be limited by the frame rate it could safely be run much faster and with proportionately more consistency than with FG. If you're running it at say 100Hz within FG I would expect to be able to run it several times faster, if not tens of times faster if the system it was running on wasn't spending most of its time rendering. You'll still get a variation in the rate that the FDM runs at but I suspect that the variation would be about the same in absolute terms. Let's say that if we get a variation of +/- 10 iteration difference per second running within FG it would probably be about the same running on its own system, but as we're running at a higher rate the difference is proprotionally smaller, perhaps down from 10% to 1%. I agree here, putting the FDM in it's own process would be a good idea. Erik -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] [DRAFT] generic input devices and hotplug support
Hi, I am currently playing with the input subsystem. This was triggered because I have two devices that currently do not work with FlightGear, neither Linux nor Windows. 1. A set of rudder pedals that does not get recognized as a joystick from joydev, because it reports it's axes as RX,RY and Z and the lack of an X-Axis stops joydev from using it as a joystick. 2. A 3dconnexion (http://www.3dconnexion.com/) SpaceNavigator 3D-Mouse which is basically a 6-axes/2buttons/1LED device. Both devices are HID devices and are handled by the Linux input/event driver. For example, the SpaceNavigator shows up in /proc/bus/input/devices as I: Bus=0003 Vendor=046d Product=c626 Version=0110 N: Name=3Dconnexion SpaceNavigator P: Phys=usb-:00:1d.1-1/input0 S: Sysfs=/devices/pci:00/:00:1d.1/usb2/2-1/2-1:1.0/input/input9 U: Uniq= H: Handlers=event9 B: EV=20017 B: KEY=3 0 0 0 0 B: REL=3f B: MSC=10 B: LED=100 My idea is to extend the FGInput class so it can use the devices presented thru the event interface. While at it, I'd also like to add hotplug support for devices, plugged in or out while FlightGear runs. What I have so far locally is a system that can use the input devices. Events and devices are configured with xml config files like this, similiar to joystick config files (just one event shown here for better readability): PropertyList name3Dconnexion SpaceNavigator/name event namerel-y-translate/name binding commandproperty-adjust/command propertysim/current-view/field-of-view/property factor type=double0.01/factor min type=double10.0/min max type=double80.0/max wrap type=boolfalse/wrap /binding /event /PropertyList See it in action here: http://www.youtube.com/watch?v=b8Me8d9UZLA The black knob can be twisted around and shifted along the x/y/z axes. These movements are translated into events which adjust the current-view direction and the field of view. The cool blue LED-light is controlled by FlightGear and indicates gear-down (YES! It's bidirectional communication). Adding/removing of devices is reported by HAL and DBUS on Linux and DirectInput on Win. I don't know how Macs deal with that, but I am certain there is a way for the macs, too. While I really like the new functionality and I see many advantages from moving into this direction, there are also some disadvantage and I'd like to open the discussion before I commit anything. Advantages: - Adding additional and generic support for a wide variety of devices - Named events, no more axis numbers and confusion between Linux/Win/Mac - Bidirectional communication (switch on/off LED's, probably force feedback etc.) - Might obsolete one plib dependency Disadvantages: - Introduces platform dependency, individual implementations needed for Linux/Win/Mac - Adds new library dependency (dbus/hal on linux DirectInput on Win, ??? on Mac) What do you think? Are the new features worth introducing new dependencies and platform dependent modules? Torsten P.S.: The GIMP also uses these techniques. Go to the settings dialog and select Input Devices. -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Automated builds tests
Hi Curt That's very advanced stuff about FAA simulator certification tests and testing all aspects of flight dynamics. As a starter, I was thinking about something simpler than validating the accuracy of the flight models. And yes, it cannot be just a script replaying a sequence of commands in a dumb way, we need a feedback loop between driving the input and reading the internal state of the simulator. Once it's clear how to drive the input, these are some of the tests that come to mind: - start the simulator - it starts in X seconds, and the test fails if the time is above a specific threshold (say 50 secs) - take a profile of the start sequence (I'm thinking about using oprofile on Linux) - shows that we spend X % of the startup time in some routine, loading scenery, and Y % starting subsystems - with the airplane on the ground - check that input is working, moving surfaces and reading back the specific property - start engine(s) - oh, yeah, it would be ideal to have a unified way to start engines on all planes, in addition to the realistic way, of course! - check parking brakes - it would be really nice to always start with brakes engaged, it's an item on my wishlist - rev engines up and down - take off - use an extremely simple feedback loop to - maintain direction on the runway (we know the angle) - and attitude during take off - exit loop once we are above a certain altitude - take screen snapshot (it could be the final step of each test) - once airborne, - gear in - insert autopilot - test wing leveler - test heading control - test altitude control - ... you get the picture The nice thing is that if we make this generic enough, we could use it to test the simulator with most of the planes released with FlightGear (or in CVS for that matter). So, to get back to my question, is there a way to drive the simulator input from an external process? Thanks Tom Tom-cat Curtis Olson wrote: On Tue, Aug 4, 2009 at 2:00 AM, Tom P zomm...@gmail.com mailto:zomm...@gmail.com wrote: Hi everybody I'd like to hear thoughts from the FG community about setting up a system to perform builds execute a suite of tests on FlightGear, all automatically. Right now I've experimented a bit with buildbot, a neat continuous integration tool used by Mozilla and other projects, and I have a system that can: * check-out from various repositories * build all FlightGear components * perform rudimentary tests on the FG simulator just built, like verifing the output on the command line and starting the simulator. Now the next step would be to go airborne! And there are two issues to resolve before take-off: 1) how to drive the input of the simulator 2) how to read its state For the second one, I've seen examples of reading the property tree from an external process, so we should be set, but the solution to driving the sim's input is still not clear. Specifically, I'd want to drive it as similarly as possible as when it's controlled from a keyboard, not go through the property tree to force FGFS into certain conditions. By the way, the current setup works on Ubuntu x86-64, but buildbot is easily extensible and supports Windows and MacOS platforms, so this could become a cross-platform testing tool for the project. Hi Tom, Because, of variations in flight dynamics models, possible variations in weather conditions, possible variations in frame rates, etc., simply replaying a series of keyboard commands is probably not going to lead to repeatable results. I suspect the replayed flight could diverge quite quickly and quite substantially from the original flight. If you want to test the simulator during flight, I really think you will have the most luck under some sort of scripted autopilot control. You should think about exactly what you are trying to measure and validate. As soon as you fire up the sim and start the aircraft moving, you've suddenly moved into the world of flight dynamics and you are looking at the physics/mathematics model of the aircraft. That's a good thing to look at though. One idea to consider is to setup a series of scripted flight tests that parallel the FAA simulator certification tests. I've gone through the Level 3 FTD set of tests and automated them for work (so I can't share the resulting scripts unfortunately) but it was an interesting process. For instance, configure some specific weather conditions, start the aircraft out at some particular altitude and speed. Setup the aircraft with a specific weight and CG. Configure the throttle for some particular RPM, keep the wings straight and level. Now measure the rate of climb (descent) you observe once the phugoid settles out. Another test involved setting up straight and level flight at certain known conditions, then commanding full rudder input
Re: [Flightgear-devel] [DRAFT] generic input devices and hotplug support
Hi Torsten, Hi, I am currently playing with the input subsystem. This was triggered because I have two devices that currently do not work with FlightGear, neither Linux nor Windows. 1. A set of rudder pedals that does not get recognized as a joystick from joydev, because it reports it's axes as RX,RY and Z and the lack of an X-Axis stops joydev from using it as a joystick. 2. A 3dconnexion (http://www.3dconnexion.com/) SpaceNavigator 3D-Mouse which is basically a 6-axes/2buttons/1LED device. Both devices are HID devices and are handled by the Linux input/event driver. For example, the SpaceNavigator shows up in /proc/bus/input/devices as I: Bus=0003 Vendor=046d Product=c626 Version=0110 N: Name=3Dconnexion SpaceNavigator P: Phys=usb-:00:1d.1-1/input0 S: Sysfs=/devices/pci:00/:00:1d.1/usb2/2-1/2-1:1.0/input/input9 U: Uniq= H: Handlers=event9 B: EV=20017 B: KEY=3 0 0 0 0 B: REL=3f B: MSC=10 B: LED=100 My idea is to extend the FGInput class so it can use the devices presented thru the event interface. While at it, I'd also like to add hotplug support for devices, plugged in or out while FlightGear runs. What I have so far locally is a system that can use the input devices. Events and devices are configured with xml config files like this, similiar to joystick config files (just one event shown here for better readability): PropertyList name3Dconnexion SpaceNavigator/name event namerel-y-translate/name binding commandproperty-adjust/command propertysim/current-view/field-of-view/property factor type=double0.01/factor min type=double10.0/min max type=double80.0/max wrap type=boolfalse/wrap /binding /event /PropertyList See it in action here: http://www.youtube.com/watch?v=b8Me8d9UZLA The black knob can be twisted around and shifted along the x/y/z axes. These movements are translated into events which adjust the current-view direction and the field of view. The cool blue LED-light is controlled by FlightGear and indicates gear-down (YES! It's bidirectional communication). Adding/removing of devices is reported by HAL and DBUS on Linux and DirectInput on Win. I don't know how Macs deal with that, but I am certain there is a way for the macs, too. While I really like the new functionality and I see many advantages from moving into this direction, there are also some disadvantage and I'd like to open the discussion before I commit anything. Advantages: - Adding additional and generic support for a wide variety of devices - Named events, no more axis numbers and confusion between Linux/Win/Mac - Bidirectional communication (switch on/off LED's, probably force feedback etc.) - Might obsolete one plib dependency Disadvantages: - Introduces platform dependency, individual implementations needed for Linux/Win/Mac - Adds new library dependency (dbus/hal on linux DirectInput on Win, ??? on Mac) if I am not totally wrong than hal will be skipped in favor of devicekit in the near future by probably all Linux distributions. Fedora 11 is already using some parts of it. What do you think? Are the new features worth introducing new dependencies and platform dependent modules? Torsten P.S.: The GIMP also uses these techniques. Go to the settings dialog and select Input Devices. Matthias -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] [DRAFT] generic input devices and hotplug support
On Tuesday 04 August 2009 19:54:23 Torsten Dreyer wrote: I am currently playing with the input subsystem. This was triggered because I have two devices that currently do not work with FlightGear, neither Linux nor Windows. My idea is to extend the FGInput class so it can use the devices presented thru the event interface. While at it, I'd also like to add hotplug support for devices, plugged in or out while FlightGear runs. What do you think? Are the new features worth introducing new dependencies and platform dependent modules? I'm really pleased to see someone looking at this - I think it's an area where the current way of doing things is beginning to creak and no longer ideal. As you probably recall I found this last year when I switched to a Saitek X52 Pro stick/throttle/MFD setup with a significant number of buttons, axes and controllable LEDs. Brief thread from the time... http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg18466.html While I did eventually manage (with help from various quarters) to hack together enough parts to make everything on the stick work pretty much as I wanted, and the amazing flexibility of FG (which usually seems to provide several workarounds for any given problem) was highlighted, the process was far from optimal. Personally, I'd be very much in favour of this development; the dependencies are hardly exotic on either the Windows or Linux side and presumably something similar is readily available on OSX. Cheers, AJ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Automated builds tests
Hi Alan The initial objective is to validate the build process and perform basic checks on a running FG. Checking the accuracy of the simulation, especially from the point of view of flight models, is beyond the scope at this point, but it's not impossible once the test infrastructure evolves. Tom Alan Teeder wrote: Are we talking about validating the build process and checking that FG runs, or about checking the validity of the simulation? For the former the suggested buildbot , or similar, approach, perhaps with a very simple autopilot guided flight, would be adequate. Simulation validity checking is another issue, and given the (I hope I do not offend anyone) rather basic flight models used for most aircraft models in the FG library as well as the limited availability of accurate predicted response data is probably not attainable by a project such as Flightgear. As a now retired flight simulator professional most of my time was devoted to checking and validating my simulations before they would be accepted for training (by CAA/FAA) or for flight handling research (by my company's aerodynamics department). Each aircraft model required tests tailored to the use that the simulation was designed for. *From:* Curtis Olson [mailto:curtol...@gmail.com] *Sent:* 04 August 2009 12:37 *To:* FlightGear developers discussions *Subject:* Re: [Flightgear-devel] Automated builds tests On Tue, Aug 4, 2009 at 2:00 AM, Tom P zomm...@gmail.com mailto:zomm...@gmail.com wrote: Hi everybody I'd like to hear thoughts from the FG community about setting up a system to perform builds execute a suite of tests on FlightGear, all automatically. Right now I've experimented a bit with buildbot, a neat continuous integration tool used by Mozilla and other projects, and I have a system that can: * check-out from various repositories * build all FlightGear components * perform rudimentary tests on the FG simulator just built, like verifing the output on the command line and starting the simulator. Now the next step would be to go airborne! And there are two issues to resolve before take-off: 1) how to drive the input of the simulator 2) how to read its state For the second one, I've seen examples of reading the property tree from an external process, so we should be set, but the solution to driving the sim's input is still not clear. Specifically, I'd want to drive it as similarly as possible as when it's controlled from a keyboard, not go through the property tree to force FGFS into certain conditions. By the way, the current setup works on Ubuntu x86-64, but buildbot is easily extensible and supports Windows and MacOS platforms, so this could become a cross-platform testing tool for the project. Hi Tom, Because, of variations in flight dynamics models, possible variations in weather conditions, possible variations in frame rates, etc., simply replaying a series of keyboard commands is probably not going to lead to repeatable results. I suspect the replayed flight could diverge quite quickly and quite substantially from the original flight. If you want to test the simulator during flight, I really think you will have the most luck under some sort of scripted autopilot control. You should think about exactly what you are trying to measure and validate. As soon as you fire up the sim and start the aircraft moving, you've suddenly moved into the world of flight dynamics and you are looking at the physics/mathematics model of the aircraft. That's a good thing to look at though. One idea to consider is to setup a series of scripted flight tests that parallel the FAA simulator certification tests. I've gone through the Level 3 FTD set of tests and automated them for work (so I can't share the resulting scripts unfortunately) but it was an interesting process. For instance, configure some specific weather conditions, start the aircraft out at some particular altitude and speed. Setup the aircraft with a specific weight and CG. Configure the throttle for some particular RPM, keep the wings straight and level. Now measure the rate of climb (descent) you observe once the phugoid settles out. Another test involved setting up straight and level flight at certain known conditions, then commanding full rudder input while maintaining the same heading and altitude (steady state side slip.) Measure the resulting bank angle, amount of required aileron input, and side slip angle. This is all very interesting stuff, but it tends to focus you more on the validity of the aircraft model and less on the validity of the simulator code. The complexity of this in combined with the complexity of everything else you could be testing and evaluating is quite staggering. That said, having a
Re: [Flightgear-devel] [DRAFT] generic input devices and hotplug support
I Just found out that my notebook's lid switch is an input device: I: Bus=0019 Vendor= Product=0005 Version= N: Name=Lid Switch P: Phys=PNP0C0D/button/input0 S: Sysfs=/devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input5 U: Uniq= H: Handlers=event5 B: EV=21 B: SW=1 So - FlightGear now pauses when I close my notebook's lid. This could be very useful when running FlightGear at work and the boss suddenly shows up on a short final... -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] [DRAFT] generic input devices and hotplug support
That's very clever! No complaints from my end if you want to pursue this. I suspect this would open up FlightGear to a lot of new and interesting input hardware, and I bet many cockpit builders would welcome generic HID support. Regards, Curt. On Tue, Aug 4, 2009 at 3:37 PM, Torsten Dreyer wrote: I Just found out that my notebook's lid switch is an input device: I: Bus=0019 Vendor= Product=0005 Version= N: Name=Lid Switch P: Phys=PNP0C0D/button/input0 S: Sysfs=/devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input5 U: Uniq= H: Handlers=event5 B: EV=21 B: SW=1 So - FlightGear now pauses when I close my notebook's lid. This could be very useful when running FlightGear at work and the boss suddenly shows up on a short final... -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel -- Curtis Olson: http://baron.flightgear.org/~curt/ -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
