Hi all, this is the first time writing to the list, I'd like to start learning a lot and hopefully help to improve blender in some areas. My main interest right now is Blender's import/export pipeline. Currently I'm making an addon to import a lot of different formats from different game engines and also rewriting/improving some existing ones like PSK/PSA and MD5.
I'd like to research the best possible ways to import to blender; my first concern besides code style is speed in large or medium files, and I have a couple questions. I've been reading http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO , and the idea of using C/C++ modules is very interesting. Here are some observations and questions for importing, (expect some misconceptions): 1) Python file reading/parsing. *Seems fast enough to me in binary data, even with processing many GB's. I haven't tested XML/text data though (seems many users complain of the obj importer, but where are the main bottlenecks is unknown to me) Also best practices for reading seems to improve the speed (see 2)) Q: C + Ctypes doesn't seem very good since we don't want to rewrite the reading part in C if it's done in python, and compile dlls or OS with the addons, right? But if someone like me would do it, does it seem like the best option because we can still keep the mesh building or other data handling in more readable python? Q: Is it worth to investigate Cython/pypy for speed improvements here and was it used in past addons pre 2.5? I haven't done any tests so far and I'd like to know opinions first, haven't found more than a couple threads in the list that mention it, and a few in BA: e.g., http://blenderartists.org/forum/showthread.php?278213-TEST-Cython-performances-test 2) Python binary data unpack (struct module). * This seems to add a lot of overhead, specially in big files. Also best practices allow for some speedups (like the use of struct.Struct or list.extend). Besides the main document in the API for best practices with a few tips in string handling when importing, I couldn't fine much info. Q: Is it worth to start/modify a document for best practices, and also add benchmarks? Who could I ask to review it? * What if Blender could accept/interpret python bytes objects to build geometry, avoiding the unpacking in python? E.g., reading a face index buffer all at once, and just passing the count, type (short in most cases) and bytes object to Blender. In case of Vertex buffer seems more complicated, since many parameters need to be defined, such as stride, type of each element, ignore vertex normals if included, etc. Q: Would this be a reasonable option to investigate, or even possible to do? * Another bottleneck is when binary data uses half-floats (used extensively in game formats for UV data). The struct module doesn't have support for them (there is a patch though: http://bugs.python.org/issue11734), so a python function must be used instead. I'm using this one: http://davidejones.com/blog/1413-python-precision-floating-point/ which is pretty slow. Q: Is the python function optimal? I couldn't find better ones. Q: If not, is it feasible to do one of the following? : a) Implement the patch mentioned above in our blender python b) Create the function in C and expose it to the python API Q: If b) is the best option, do you think is an OK task for me, as a first approach to Blender's C code? (no much experience in C besides college first year) 3) Python data to blender data (converting lists to C arrays with mesh.vertices.add, mesh.polygons.add, etc): *I've been doing a lot of tests but not much digging into C code. I don't seem to understand the process very well. Using ctypes arrays, or array.array don't seem to have any performance improvement when used in reading the file (it's actually a little slower) nor building the mesh. Q: When using python data to import geometry, the python objects need to be converted to C arrays and that's the overhead right? Q: When using c-like objects in python like ctypes array or array.array, they still need to be converted to C arrays, so that's why performance is not improved? Q: What could be done to avoid the conversion without using C? Something like passing a pointer to a ctypes array instead? Thanks! Regards _______________________________________________ Bf-committers mailing list [email protected] http://lists.blender.org/mailman/listinfo/bf-committers
