The answer is very helpful. I saw a way to solve the specific problem using other techniques but I tried to see anyway how to recover the computation. The key of the problem was a polynomial with more that 16 millions of monomials. I made the following approaches:
1. Save the object in a sobj file, load it in other session. The action of loading was stopped after more than one day of computation. 2. Compute the dictionnary and save it in a sobj file (less than 400Mb). In a new session, loading the dictionnary took less than 4 minutes. But as it was pointed in the previous message, this was not the point. Recovering the polynomial took too long, stopped after one day. 3. Compute the dictionnary and save it as text in a sage file (around 800Mb). In a new session, loading the dictionnary produced some errors about memory problems (I could reproduce it if necessary). 4. Save the polynomial as a chain in a sage file (around 700Mb). In a new session, reading this file produced other memory problems and stopped the session. 5. With the sobj file in 2, I broke the dictionnary in around 1600 dictionnaries with 10000 terms. This was very short time. Computing the 1600 polynomials of 10000 monomials took 12 minutes and summing them, 24 minutes. The last option takes around 40 minutes and It may exist better partitions to do this. I wonder if this approach could be made automatic. El viernes, 18 de enero de 2019, 4:50:09 (UTC+1), Nils Bruin escribió: > > On Thursday, January 17, 2019 at 3:37:36 PM UTC-8, Enrique Artal wrote: >> >> I made some computations, I skip the details for now, but the result was >> a rational function with rational coefficients and 13 indeterminates. The >> computation took around three hours and used a lot of memory so I made it >> using a script. The first time I saved the rational function in a pickle >> file, the second time in a sobj file. >> > > Both are actually pickles, but the standard python "pickle" defaults to an > older version of the protocol. Sage's "sobj" wrappers select a more modern > version of the protocol and a more compact (binary) file representation. > > >> The first one was 1.2Gb, the second one around 300Mb. The main issue is >> that opening a sage session (or using a script) takes much more time than >> computing (with pickle more than 12 hours, I stopped it, with sobj 10 hours >> and counting). >> > > The pickle format basically consists of a simple programming language with > instructions to build data structures. To get an idea what it does, you can > use, for instance, > > sage: R.<x,y>=QQ[] > sage: f=(x^2+y^2)/x > sage: import pickletools > sage: pickletools.dis(sage.misc.explainpickle.comp.decompress(dumps(f))) > > or, to get a more or less equivalent piece of sage code: > > sage: explain_pickle(dumps(f)) > > >> I did not try a text file, but I wonder which is the best strategy to >> save a heavy result and being able to load it using less time than the >> actual computation. >> > > The main thing you learn from the above exercise is that pickle in > principle does something fairly sensible for rational functions: It > constructs numerator and denominator via a dictionary. > > It may be that this happens with a lot of overhead, so perhaps you can > gain something if you do it via another route, but it may well be that the > bottleneck is in the sage-to-libsingular interface, and other construction > methods won't get around that. > > It's not always the case that reconstructing something from a file store > is faster than computing it. A silly example: > > a=2**(10**10) > > completes quite quickly, but writing the bit representation to a file and > reading it in will be considerable work. > > It's unlikely your example is as extreme as the one above, but your > experience indicates that the size of the polynomials involved in your case > (the size of the "sobj" file is a poor measure for that) is beyond what > sage has been tested with regularly. It looks like there's significant room > for improvement here. Pickle really should be able to do this. > >> Ideas would be appreciated. Thanks, Enrique Artal. >> > -- You received this message because you are subscribed to the Google Groups "sage-support" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-support+unsubscr...@googlegroups.com. To post to this group, send email to sage-support@googlegroups.com. Visit this group at https://groups.google.com/group/sage-support. For more options, visit https://groups.google.com/d/optout.
