Re: [Flightgear-devel] File sizes in fgdata, clean up needed?
Am 18.03.11 20:00, schrieb ThorstenB: Indeed, fgdata/master is becoming way too big though. But we can only solve this by splitting our current repository - and then push the different parts to fresh git repositories. Splitting fgdata was planned anyway. The new --fg-aircraft options was the first step to make this possible. I'm just not sure what the status of splitting fgdata is though... cheers, Thorsten Splitting is a good idea, sorry I didn’t realize that this is on the way right now. I splitted fgdata locally today because I want to distribute a snapshot of my new OSX launcher (FlightGear 2.2 for 10.5/6 intel) with some basic fgdata included, for testing purposes. Now fgx full comes with very basic fgdata around 800 MB. The .dmg to download takes ~500 MB. From the aircrafts I only included 737-100 b1900d bo105 c172p Citation-Bravo Concorde ec135 followme Generic Instruments Instruments3d MPCarrier Pushback seahawk Sikorsky-76C Storch ufo UH-1 wrightFlyer1903 ZLT-NT I am looking forward to include a better selection. I just wanted to have a good selection for a starting point with working aircrafts and a good mix of types. Maybe you see something important missing here, but when a new (and probably well discussed) repo with default fg aircrafts comes around, I will switch to this selection of course. Thanks, Yves -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] File sizes in fgdata, clean up needed?
On 18.03.2011 17:50, HB-GRAL wrote: Today I checked the current fgdata/Aircraft folder for sizes. It’s about 4,3 GB here. Nice. Some of this files are .gz already, when I .gz the rest I get another 100 MB, or in other words I get two MiG-15 or another p51d. Nice statistics! Maybe this motivates some of us to be a bit more considerate before committing something to the repo. However, remember that dropping any file from the repository wouldn't help at all now: a git repository never shrinks - it can only grow... It's an especially bad idea to drop files and resubmit a smaller/compressed version in a futile attempt to shrink the repository. git contains a complete history of every file. If you dropped files and resubmitted a smaller version, the local fgdata directories may at first appear smaller - but if you checked the total size (that is including the hidden .git folders) you'll see that the total size was actually increased... And another warning: the complete history issue also affects personal git branches on gitorious. So, if an a/c designer adds 19 versions of an image file to his private branch and then placed a merge request to fgdata/master - then the merge will actually copy the history of all 19 file versions to the master branch - even the history of files which were already dropped on the private branch. So, in such cases it's a good idea to not actually git merge the complete personal branch to our master - but to simply take a copy of the latest version of the a/c files and to submit them to master using a fresh commit (I think that's what our fgdata-merge-committers do anyway - at least I hope so...). Or maybe any git expert knew if there was a git-merge-without-history command? Indeed, fgdata/master is becoming way too big though. But we can only solve this by splitting our current repository - and then push the different parts to fresh git repositories. Splitting fgdata was planned anyway. The new --fg-aircraft options was the first step to make this possible. I'm just not sure what the status of splitting fgdata is though... cheers, Thorsten -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] File sizes in fgdata, clean up needed?
And another warning: the complete history issue also affects personal git branches on gitorious. So, if an a/c designer adds 19 versions of an image file to his private branch and then placed a merge request to fgdata/master - then the merge will actually copy the history of all 19 file versions to the master branch - even the history of files which were already dropped on the private branch. So, in such cases it's a good idea to not actually git merge the complete personal branch to our master - but to simply take a copy of the latest version of the a/c files and to submit them to master using a fresh commit (I think that's what our fgdata-merge-committers do anyway - at least I hope so...). Or maybe any git expert knew if there was a git-merge-without-history command? You can merge a branch using one commit with git merge --squash branch_to_merge git commit The merge --squash applies the changes from all the commits to the current branches working tree and stages them. It also creates a commit message that concatenates all the commit messages for the ones that are being merged. git commit then allows you to edit that message (and commit of course). Basically it does exactly what you're after :) Cheers, Luuk -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] File sizes in fgdata, clean up needed?
On Friday, March 18, 2011 09:50:50 AM HB-GRAL wrote: Hi all Today I checked the current fgdata/Aircraft folder for sizes. It’s about 4,3 GB here. Nice. Now some statistics (and this is no critcs on aircrafts of course, I like all the development and improvements a lot!): We have 372 aircrafts in the directory. 22 of this aircrafts have more than 30 MB and this 22 aircraft gives 1,1 GB of the aircraft folder. Number One is p51d with 104.5 MB (!). 50 MB in the models folder comes from GIMP .xcf-files and from Blender files (.blend). Do we distribute this files uncompressed? i. e. compressing .xcf to .gz will give a 1,2 file instead of a 4,3 MB. I think 12 MB vs. 43 MB. Number Two is 767-300 with 82,2 MB. I. e. this one comes with widely used .wav-sounds in the cabin ;-) This sounds, or better short loops, take 17,3 MB here. One livery (VRN.png) takes 6.3 MB. Number three is MiG-15, a really nice one, with a lot of instruments, and it seems like every byte is used here. I am looking deeper into the files and I see a radio-tune.wav which has 3.5 MB for 10 seconds of sound and 10 seconds of silence ;-) Some models like IAR80 have liveries with 13 MB .png-files. Totally we distribute 18 blender files with the directory. This is only 16,4 MB. Not much. But we distribute also 310 MB of original GIMP files. Some of this files are .gz already, when I .gz the rest I get another 100 MB, or in other words I get two MiG-15 or another p51d. Cheers, Yves I think Yves has several good points. First many of these advanced models have working files that are not actually used when the model is being flown in sim. The p51d GIMP files and blender files are two examples. Now there are valid reasons for these to be source controlled. For example the gimp files in the p51d/Models directory are complex multi layer files that are intended to make working with the resulting textures easier and they do indeed do that. Reading Yves comments I think one of the things he hionted was not so much that these working files made GIT bigger but that that they made the distribution size to users bigger and really served no purpose for users other than wasting disk space and bandwidth. This is a valid concern at least if the size of these working files is significant and in some of these case they are. A look at p51d/Models clearly shows that the three big space users are (in order of the highest space useage) the working files mostly in the form of high resolution multi layer textures, 3D models and the actual textures. In the case of the p51d all 3D models are AC3D files and many of the textures except some newer ones are *.rgb files. These are not the most compact formats and changing these could reduce the size of the model significantly but the 800 pound gorrila is still the working files. In the case of the p51d, and I suspect that this is true for most models, the working files could be located anywhere in the file system tree. And perhaps it makes sense to have a directory with a standard name that is used for these types of files that is always excluded (somehow?) when regular user gets a copy but is included for GIT clones. In the case of the p51d this would cut the size of the distributed copy almost in half. The 3D and texture parts of the p51d are now fairly far along and will not grow too much more even though there is still 3D work that needs to be done. It's size will only grow by perhaps 10% as it 3D model and FDM are finalized. There may be other models that get implemented for more complex aircraft that could result in significantly larger models. I suspect that the 100 meg p51d to 310+ meg IAR80 sort of represents the size range we will be seeing for really advanced highly detailed models with a few really careful modelers being able to bring these numbers down to lower levels while achiving the same effective level of detail like the Mig-15 does. I really think we are only seeing the begining of a period where are will see more efforts to take existing models to the next level and where we will start seeing new additions that enter GIT already in a very advanced state. As this process unfolds we will see many more models that approach the size of the ones listed by Yves. We don't want to discourage that work but if we can create a set of best practices we can perhaps help those working on this stuff create these highly detailed aircraft while using less space for the files needed to support this work. Hal -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net
Re: [Flightgear-devel] File sizes in fgdata, clean up needed?
I've been threatening for some time to break up the aircraft portion of fgdata into several repositories by using some git magic. It's true that the current repository size is out of hand. I would encourage people to check in their source files whenever possible. I'd also encourage people to add new aircraft to their own repositories on gitorious and think about moving past the central repository model. Tim On Fri, Mar 18, 2011 at 9:48 PM, Hal V. Engel hven...@gmail.com wrote: On Friday, March 18, 2011 09:50:50 AM HB-GRAL wrote: Hi all Today I checked the current fgdata/Aircraft folder for sizes. It’s about 4,3 GB here. Nice. Now some statistics (and this is no critcs on aircrafts of course, I like all the development and improvements a lot!): We have 372 aircrafts in the directory. 22 of this aircrafts have more than 30 MB and this 22 aircraft gives 1,1 GB of the aircraft folder. Number One is p51d with 104.5 MB (!). 50 MB in the models folder comes from GIMP .xcf-files and from Blender files (.blend). Do we distribute this files uncompressed? i. e. compressing .xcf to .gz will give a 1,2 file instead of a 4,3 MB. I think 12 MB vs. 43 MB. Number Two is 767-300 with 82,2 MB. I. e. this one comes with widely used .wav-sounds in the cabin ;-) This sounds, or better short loops, take 17,3 MB here. One livery (VRN.png) takes 6.3 MB. Number three is MiG-15, a really nice one, with a lot of instruments, and it seems like every byte is used here. I am looking deeper into the files and I see a radio-tune.wav which has 3.5 MB for 10 seconds of sound and 10 seconds of silence ;-) Some models like IAR80 have liveries with 13 MB .png-files. Totally we distribute 18 blender files with the directory. This is only 16,4 MB. Not much. But we distribute also 310 MB of original GIMP files. Some of this files are .gz already, when I .gz the rest I get another 100 MB, or in other words I get two MiG-15 or another p51d. Cheers, Yves I think Yves has several good points. First many of these advanced models have working files that are not actually used when the model is being flown in sim. The p51d GIMP files and blender files are two examples. Now there are valid reasons for these to be source controlled. For example the gimp files in the p51d/Models directory are complex multi layer files that are intended to make working with the resulting textures easier and they do indeed do that. Reading Yves comments I think one of the things he hionted was not so much that these working files made GIT bigger but that that they made the distribution size to users bigger and really served no purpose for users other than wasting disk space and bandwidth. This is a valid concern at least if the size of these working files is significant and in some of these case they are. A look at p51d/Models clearly shows that the three big space users are (in order of the highest space useage) the working files mostly in the form of high resolution multi layer textures, 3D models and the actual textures. In the case of the p51d all 3D models are AC3D files and many of the textures except some newer ones are *.rgb files. These are not the most compact formats and changing these could reduce the size of the model significantly but the 800 pound gorrila is still the working files. In the case of the p51d, and I suspect that this is true for most models, the working files could be located anywhere in the file system tree. And perhaps it makes sense to have a directory with a standard name that is used for these types of files that is always excluded (somehow?) when regular user gets a copy but is included for GIT clones. In the case of the p51d this would cut the size of the distributed copy almost in half. The 3D and texture parts of the p51d are now fairly far along and will not grow too much more even though there is still 3D work that needs to be done. It's size will only grow by perhaps 10% as it 3D model and FDM are finalized. There may be other models that get implemented for more complex aircraft that could result in significantly larger models. I suspect that the 100 meg p51d to 310+ meg IAR80 sort of represents the size range we will be seeing for really advanced highly detailed models with a few really careful modelers being able to bring these numbers down to lower levels while achiving the same effective level of detail like the Mig-15 does. I really think we are only seeing the begining of a period where are will see more efforts to take existing models to the next level and where we will start seeing new additions that enter GIT already in a very advanced state. As this process unfolds we will see many more models that approach the size of the ones listed by Yves. We don't want to discourage that work but if we can create a set of best practices we can perhaps help those working on this stuff create these highly detailed
