I wrote most of a patch to have Blender export .obj files using C code inside Blender. It saved a lot of time when exporting big files. But I got bogged down in getting all the texture outputs, etc., right. If someone is interested in trying to finish it, I could send the patch I had so far.
- Howard On Mon, Mar 10, 2014 at 3:41 AM, Martijn Berger <[email protected]>wrote: > Hi all, > > Just a side note: > You should be careful to compare blenders obj importer to something awesome > that you have written yourself. There is one thing that can be done to > speed it up greatly but that breaks entities that are split ( started and > finished elsewhere ). > Blenders obj importer needs to build a list now and make sure it got > everything before it can do the actual object import. If you convert this > to a generator you save a ton of memory and get a dramatic speedup. All > "normal" obj files will load but some exotics won't. It is little things > like this that make the importer slow also besides being written in python. > And it is little things like this that will take a couple of orders of > magnitude of your speedups if you implement an obj importer that has exact > feature parity. > > > > On Mon, Mar 10, 2014 at 8:05 AM, Fredrik hansson < > [email protected]> wrote: > > > you can always do as i did and grab some of the stanford models they > don't > > have uvs/normals but works anyway, > > the xyzrgb_dragon one is 574mb or so and takes 27seconds to load for me > or > > 7seconds if the file is in the file cache. > > > > can't say how long blender takes to do the same as it runs out of memory > > and crashes after a few minutes(32bit build) > > it did manage to do the parsing step however 82seconds. > > note that these numbere would be smaller if i was at home haven't got a > > ssd here at work. > > > > as to Dan's question what i do here is not to thread every line but > > instead load say 1mb of the file at a time split that up > > into sections of1-numcores of lines, that i then parse and stitch > > together in the original order since i know what order the chunks are. > > > > the main speedup i found however is to just load the entire file or parts > > of the file into memory and then parse. > > loading the entire thing is if im not mistaken the fastest but uses way > to > > much memory while if i load 1mb at a time, > > i actually use less memory than the file size in the end (338mb for the > > xyzrgb dragon) > > > > > > // Fredrik > > > > > > > > > > On Monday, March 10, 2014 5:50 AM, metalliandy < > > [email protected]> wrote: > > > > I'm sure I can get something to you, mate. Be aware that the files are > > often 900mb+ though > > > > I will upload something tomorrow. > > > > Cheers, > > > > -Andy > > > > On 10/03/2014 03:12, Sebastian A. Brachi wrote: > > > Andy, could you provide a link to an example to do some benchmarks? > > > I'd prefer to work with real user cases than a super-subdivided > suzzane. > > > > > > > > > On Sun, Mar 9, 2014 at 8:42 PM, metalliandy > > > <[email protected]>wrote: > > > > > >> I have been praying for a faster obj loader for years! > > >> I use massive objs on a regular basis (up to 900mb) so this would be > > >> amazing. IMHO all I/O plugins should be done in C/C++ as python just > > >> doesn't have the speed. > > >> FWIW, Blender takes about 30mins to export a 31.5m poly mesh while > > >> ZBrush takes around 2 mins for the same mesh. > > >> > > >> Cheers, > > >> > > >> -Andy > > >> On 09/03/2014 19:21, Fredrik hansson wrote: > > >>> hi, > > >>> i > > >>> have recently written a obj loader myself just for fun. now i did > > try > > >>> to > > >>> optimize it quite a bit by making it multithreaded and a few other > > >>> things. > > >>> i have put it up on https://github.com/FredrikHson/fast_obj_loader > > >>> now > > >>> granted this is a much simpler example than doing everything that > > >>> blender does > > >>> with pushing it into blenders internal structure and all that. > > >>> anyway tried it + blenders current importer. > > >>> some stats > > >>> blender total 17.8 sec. > > >>> 8.6sec. parse time > > >>> > > >>> mine: > > >>> sigle threaded 0.6seonds > > >>> multi threaded 0.26sec > > >>> > > >>> this is on a 36mb obj file on a ssd drive with 8threads. > > >>> > > >>> so yes it could probably benefit from being done in c/c++ the > question > > >> is still if its worth the trouble > > >>> i personally never import anything much heavier than that 36mb file > > >> myself due to > > >>> slow viewport/manipulation speeds after actually getting the file > into > > >> blender and 18seconds is > > >>> a bit annoying but i don't do it that often that its really much of > an > > >> issue. > > >>> what is much worse imo is export speeds ( i often forget export > > selected > > >> only) when dealing with > > >>> dense subsurfed meshes. > > >>> > > >>> // Fredrik > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On Saturday, March 8, 2014 3:30 AM, Sebastian A. Brachi < > > >> [email protected]> wrote: > > >>> Hi Paul, > > >>> I'm pretty satisfied with the performance of my importers right now, > > and > > >>> specially compared to max-script. > > >>> Turned out the major bottleneck was from UV data, since I wasn't > using > > >>> foreach_set which is a much efficient method than adding uv data in a > > >> loop. > > >>> check > > >>> > > >> > > > http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates > > >>> This is for video game data that usually isn't very heavy though > > >> (although > > >>> I'm also importing whole levels with hundreds of assets, all in ~40 > > >>> seconds), > > >>> and also not using an efficient method for unpacking half-floats > (used > > a > > >>> lot in video games for uvs). > > >>> > > >>> As I said before, I heard users complaining .obj importer performance > > vs > > >>> zbrush's for example in >500 mb data (sculpts mainly). > > >>> There might be room for improvement there, but consider the topics > > >>> discussed (would be good to rewrite it? is C an better option? can > > small > > >>> tweaks improve performance considerably?) > > >>> > > >>> Regards > > >>> > > >>> > > >>> > > >>> On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <[email protected]> > > >> wrote: > > >>>> Hi Sebastian, > > >>>> > > >>>> I read your interesting thread on blender import improvements. Did > you > > >>>> make any progress on this topic? Would this be something that a > > >>>> summer-of-code student can work on as well? > > >>>> > > >>>> Regards, > > >>>> Paul > > >>>> > > >>>> > > >>>> > > >>>> On 01/05/14 23:00, Sebastian A. Brachi wrote: > > >>>> > > >>>>> Hi all, > > >>>>> this is the first time writing to the list, I'd like to start > > learning > > >> a > > >>>>> lot and hopefully help to improve blender in some areas. > > >>>>> My main interest right now is Blender's import/export pipeline. > > >>>>> Currently I'm making an addon to import a lot of different formats > > from > > >>>>> different game engines and also rewriting/improving some existing > > ones > > >>>>> like > > >>>>> PSK/PSA and MD5. > > >>>>> > > >>>>> I'd like to research the best possible ways to import to blender; > my > > >> first > > >>>>> concern besides code style is speed in large or medium files, and I > > >> have a > > >>>>> couple questions. I've been reading > > >>>>> > http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO, > > >> and > > >>>>> the idea of using C/C++ modules is very interesting. Here are some > > >>>>> observations and questions for importing, > > >>>>> > > >>>>> (expect some misconceptions): > > >>>>> > > >>>>> > > >>>>> 1) Python file reading/parsing. > > >>>>> > > >>>>> *Seems fast enough to me in binary data, even with processing > > many > > >>>>> GB's. I > > >>>>> haven't tested XML/text data though (seems many users complain of > the > > >> obj > > >>>>> importer, but where are the main bottlenecks is unknown to me) > > >>>>> Also best practices for reading seems to improve the speed > (see > > >> 2)) > > >>>>> Q: C + Ctypes doesn't seem very good since we don't want to > > >> rewrite the > > >>>>> reading part in C if it's done in python, and compile dlls or OS > with > > >> the > > >>>>> addons, right? But if someone > > >>>>> like me would do it, does it seem like the best option > > because > > >> we can > > >>>>> still keep the mesh building or other data handling in more > readable > > >>>>> python? > > >>>>> > > >>>>> Q: Is it worth to investigate Cython/pypy for speed > improvements > > >> here > > >>>>> and > > >>>>> was it used in past addons pre 2.5? > > >>>>> I haven't done any tests so far and I'd like to know > opinions > > >> first, > > >>>>> haven't found more than a couple threads in the list that mention > it, > > >> and > > >>>>> a > > >>>>> few in BA: > > >>>>> e.g., > > >>>>> http://blenderartists.org/forum/showthread.php?278213- > > >>>>> TEST-Cython-performances-test > > >>>>> > > >>>>> > > >>>>> > > >>>>> 2) Python binary data unpack (struct module). > > >>>>> > > >>>>> * This seems to add a lot of overhead, specially in big > files. > > >> Also > > >>>>> best > > >>>>> practices allow for some speedups (like the use of struct.Struct or > > >>>>> list.extend). > > >>>>> Besides the main document in the API for best practices with > a > > >> few tips > > >>>>> in string handling when importing, I couldn't fine much info. > > >>>>> > > >>>>> Q: Is it worth to start/modify a document for best practices, > > and > > >> also > > >>>>> add benchmarks? Who could I ask to review it? > > >>>>> > > >>>>> * What if Blender could accept/interpret python bytes objects > > to > > >> build > > >>>>> geometry, avoiding the unpacking in python? E.g., reading a face > > index > > >>>>> buffer all at once, and just passing the count, > > >>>>> type (short in most cases) and bytes object to Blender. In > > case of > > >>>>> Vertex > > >>>>> buffer seems more complicated, since many parameters need to be > > >> defined, > > >>>>> such as stride, type of each element, > > >>>>> ignore vertex normals if included, etc. > > >>>>> > > >>>>> Q: Would this be a reasonable option to investigate, or even > > >> possible > > >>>>> to > > >>>>> do? > > >>>>> > > >>>>> * Another bottleneck is when binary data uses half-floats > (used > > >>>>> extensively in game formats for UV data). > > >>>>> The struct module doesn't have support for them (there is a > > >> patch > > >>>>> though: http://bugs.python.org/issue11734), so a python > > >>>>> function must be used instead. I'm using this one: > > >>>>> > > http://davidejones.com/blog/1413-python-precision-floating-point/which > > >>>>> is > > >>>>> pretty slow. > > >>>>> > > >>>>> Q: Is the python function optimal? I couldn't find better > ones. > > >>>>> Q: If not, is it feasible to do one of the following? : > > >>>>> a) Implement the patch mentioned above in our blender > python > > >>>>> b) Create the function in C and expose it to the python > API > > >>>>> Q: If b) is the best option, do you think is an OK task for > me, > > >> as a > > >>>>> first approach to Blender's C code? (no much experience in C > besides > > >>>>> college first year) > > >>>>> > > >>>>> > > >>>>> 3) Python data to blender data (converting lists to C arrays with > > >>>>> mesh.vertices.add, mesh.polygons.add, etc): > > >>>>> > > >>>>> *I've been doing a lot of tests but not much digging into C > > code. > > >> I > > >>>>> don't > > >>>>> seem to understand the process very well. > > >>>>> Using ctypes arrays, or array.array don't seem to have any > > >> performance > > >>>>> improvement when used in reading the file (it's actually a little > > >> slower) > > >>>>> nor building the mesh. > > >>>>> > > >>>>> Q: When using python data to import geometry, the python > > objects > > >> need > > >>>>> to > > >>>>> be converted to C arrays and that's the overhead right? > > >>>>> Q: When using c-like objects in python like ctypes array or > > >>>>> array.array, > > >>>>> they still need to be converted to C arrays, so that's why > > performance > > >> is > > >>>>> not improved? > > >>>>> Q: What could be done to avoid the conversion without using > C? > > >>>>> Something > > >>>>> like passing a pointer to a ctypes array instead? > > >>>>> > > >>>>> > > >>>>> Thanks! > > >>>>> Regards > > >>>>> _______________________________________________ > > >>>>> Bf-committers mailing list > > >>>>> [email protected] > > >>>>> http://lists.blender.org/mailman/listinfo/bf-committers > > >>>>> > > >>> _______________________________________________ > > >>> Bf-committers mailing list > > >>> [email protected] > > >>> http://lists.blender.org/mailman/listinfo/bf-committers > > >>> _______________________________________________ > > >>> Bf-committers mailing list > > >>> [email protected] > > >>> http://lists.blender.org/mailman/listinfo/bf-committers > > >>> > > >> _______________________________________________ > > >> Bf-committers mailing list > > >> [email protected] > > >> http://lists.blender.org/mailman/listinfo/bf-committers > > >> > > > _______________________________________________ > > > Bf-committers mailing list > > > [email protected] > > > http://lists.blender.org/mailman/listinfo/bf-committers > > > > > > > _______________________________________________ > > Bf-committers mailing list > > [email protected] > > http://lists.blender.org/mailman/listinfo/bf-committers > > _______________________________________________ > > Bf-committers mailing list > > [email protected] > > http://lists.blender.org/mailman/listinfo/bf-committers > > > _______________________________________________ > Bf-committers mailing list > [email protected] > http://lists.blender.org/mailman/listinfo/bf-committers > _______________________________________________ Bf-committers mailing list [email protected] http://lists.blender.org/mailman/listinfo/bf-committers
