Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-19 Thread fruity
Hi ! I realized that i didn't put the result of the tests i did, so here they are, if it can help someone else ! Treating the maya part (getting / setting the weights) and the IO part (writing / reading the file) separately, i tried : *Maya : * cmds.skinPercent MFnSkinCluster.setWeights

Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-13 Thread fruity
Hi Marcus ! Thanks for your answer ! Well, i'm still working on the optimisation, i used json for the readable info (influences, etc), and cPickle for the weights array. But i think most of the optimisation should now come from how to export / import the values to the vertices. Exporting is

Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-13 Thread Marcus Ottosson
Hey @fruity, how did it go with this? Did you make any progress? :) ​I came to think of another constraint or method which to do what you’re after - in regards to random access. That is, being able to query weights for any given vertex, without (1) reading it all into memory and (2) physically

Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-06 Thread Marcus Ottosson
Cool :) About reading in a file, a little bit at a time, it’s not as difficult as it seems. f = open("100gb.json") At this point, you’ve opened a file of an imaginary 100 gigabytes. What is the effect on memory? 0. Nada. Now, if you were to this.. data = f.read() You’d be in trouble. Now

Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-06 Thread fruity
actually, the file would be split into 2 parts : the first one would be the infos that i want the user to be able to modify (a couple of lines), the second one would be a text of n lines (n being a number of vertices for a mesh, for instance), that the user will definetly not modify. So you're

Re: [Maya-Python] Re: [python] which library using to read / write huge amount of data

2016-10-06 Thread Marcus Ottosson
cPickle looks great If you want it to be human readable, you can’t encrypt, compress or otherwise obfuscate it, so both pickling and zlib is out. But think about that for a second. If you are talking about multi-megabyte/gigabytes worth of data, the mere quantity would make it uneditable