Re: [sage-support] huge virtual memory size when launching 7.3

[Jeroen Demeyer]

After spending some time reading on the subject, I think I might have a 
solution to this "problem". It involves calling mmap() with PROT_NONE, 
which will require a patch to upstream PARI.


However, before implementing this, I would like a *strong commitment* 
from somebody to review my patch when I'm finished. Otherwise I'm just 
wasting my time.


--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 19:58, Jeroen Demeyer wrote:

* https://lwn.net/Articles/627557/


This last page indicates that one should use PROT_NONE to work around 
the issue you are having.


Note this nice quote:

> Sadly, the commit charge implications of MAP_NORESERVE are documented 
but silently broken, but the commit charge implications of PROT_NONE are 
undocumented and in theory mutable in future releases.


--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 14:04, Jonathan Bober wrote:

With overcommit_memory set to 2, I'm not sure that there is a right
thing to do. If I'm the only person in the world with this problem, then
I should keep my mouth shut and get with the times, but I don't think
this setup is so unreasonable.


Some pointers indicating that the Linux implementation of 
overcommit_memory=2 has issues:

* https://lkml.org/lkml/2005/8/8/46
* https://lwn.net/Articles/627557/

--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 19:01, William Stein wrote:

We should revert whatever trac ticket did this


The first version of this is from
https://trac.sagemath.org/ticket/19883


or at least make it an option to not allocate such a large address space.


That's easy to do. How would you see the user interface for this?


This huge vm
allocation is clearly an even worse hack dealing with PARI's very
hackish "memory management".


It is really not so bad as it looks. Let me recall that this is only 
about *virtual memory*. Normally, there are no consequences to 
allocating a huge amount of virtual memory. Apparently, there are 
unusual Linux settings in which it does cause problems. Unfortunately, I 
don't see an easy solution for this.



Making it the default is surprising.


I am open for suggestions, what would you make the default then?

--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 14:04, Jonathan Bober wrote:

That doesn't really solve the problem, which I phrased poorly. The
machine has an (antiquated) setup where, when a process requests X bytes
of memory, the kernel reserves X bytes of physical memory for it.


Well, in that case you would indeed "lose" 1/4 of your physical memory 
using the defaults that Sage uses for the PARI virtual stack. If you are 
unwilling to change your OS settings, then you can still manually change 
the PARi stack size with


from sage.libs.pari.all import pari
pari.allocatemem(4*10^6, 4*10^6)


Anyway, where exactly is this allocation coming from? Is it a default
PARI setting, or does it come from the way that Sage uses PARI?


It is because of the way how Sage uses PARI.


I mean, to whom should I address my "hate mail"? :)


That would be me...

Jeroen.


PS: The Sage <-> GAP interface has exactly the same problem, but with a 
smaller default memory allocation.


--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[William Stein]

>
>
> Anyway, where exactly is this allocation coming from? Is it a default PARI
> setting, or does it come from the way that Sage uses PARI? (I mean, to whom
> should I address my "hate mail"? :)
>

I think it is a problem in the way that Sage uses PARI.  I just tried "sage
-sh" with sage-7.3, then ran gp from the command line, and it uses very
little memory.Please continue to send "hate mail"...

We should revert whatever trac ticket did this, or at least make it an
option to not allocate such a large address space. This huge vm allocation
is clearly an even worse hack dealing with PARI's very hackish "memory
management".
Even back in 2005 when I was writing the original libpari hackish code, I
didn't seriously consider this "allocate huge vm" hack to deal with how
pari works, since that was even too hackish for me.  Making it the default
is surprising.

-- 
William (http://wstein.org)

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

[sage-support] Re: How can Sage and Python communicate via global variables?

[Nils Bruin]

On Monday, September 19, 2016 at 6:02:22 AM UTC-7, Paul Leopardi wrote:
>
> Hello,
> I am trying to use global variables to control the behaviour of Python 
> functions called from Sage. (Yes, I know there is probably a better way to 
> do it, but I am still interested in what's going on here.)
>
> If in foo.py I have
>
> def bar():
> print blah
>
> def baz():
> global zilch
> zilch = 0
>
> A "global" variable is a "module global" variable. Since you're always 
executing code that is in some module (that's what __main__ is for)

and, in Sage I do:
>
> sage: from foo import bar
> sage: global blah
> sage: blah = 1
> sage: bar()
>

If you spell that foo.blah = 1 it should work (perhaps after executing 
"import foo" in order to have the "foo module namespace" bound to the 
toplevel variable "foo")

 

> sage: from foo import baz
> sage: baz()
> sage: print zilch
>
>  
print foo.zilch

should work


> Is there a way in Sage that I can assign to a global variable that a 
> called Python function will recognize as a global variable?
> Thanks, Paul
>

This is just a python thing: you share global data across modules by making 
sure that this data is bound in a namespace to which both modules have 
access.

There is no really "global" namespace in sage. There's "toplevel", but that 
will generally not be the interactive namespace that a sage interface is 
using, given that ipython, jupyter, and sagenb run "above" your sage code.

You can try and hack your way to the "interactive" namespace. The toplevel 
"var" in sage does that for injection. That's generally fragile, though.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

[sage-support] How can Sage and Python communicate via global variables?

[Paul Leopardi]

Hello,
I am trying to use global variables to control the behaviour of Python 
functions called from Sage. (Yes, I know there is probably a better way to 
do it, but I am still interested in what's going on here.)

If in foo.py I have

def bar():
print blah

def baz():
global zilch
zilch = 0

and, in Sage I do:

sage: from foo import bar
sage: global blah
sage: blah = 1
sage: bar()

the result is:
---
NameError Traceback (most recent call last)
 in ()
> 1 bar()

/home/leopardi/sync/src/sage-sandbox/Boolean-Cayley-graphs/foo.py in bar()
  1 def bar():
> 2 print blah

NameError: global name 'blah' is not defined

If instead I do:

sage: from foo import baz
sage: baz()
sage: print zilch

the result is:
---
NameError Traceback (most recent call last)
 in ()
> 1 print zilch

NameError: name 'zilch' is not defined

Is there a way in Sage that I can assign to a global variable that a called 
Python function will recognize as a global variable?
Thanks, Paul

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jonathan Bober]

On Mon, Sep 19, 2016 at 12:13 PM, Jeroen Demeyer 
wrote:

> On 2016-09-19 13:02, Jonathan Bober wrote:
>
>> If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater
>> immediately grabs 2 GB of the virtual address space for itself, then
>> that seems like it leaves only 6 GB for malloc/sage_malloc/whatever
>> else, which would be effectively limiting the physical memory available
>> in some situations.
>>
>
> Then use a larger value for "ulimit -v" or add some swap space.
>
>
That doesn't really solve the problem, which I phrased poorly. The machine
has an (antiquated) setup where, when a process requests X bytes of memory,
the kernel reserves X bytes of physical memory for it. This is usually fine
because users usually use the memory they request, and when they don't,
well, there are 512 GB of it. But it does mean that every virtual memory
allocation has a physical memory cost. (It also has benefits, like when
someone screws up they usually notice immediately, instead of when the OOM
killer kills sshd or the login process.)


> This is not the default, and you probably wouldn't want to
>> run your laptop this way
>>
>
> But it is an advanced setting. Given that, I assume that you are clever
> enough to do the right thing.


With overcommit_memory set to 2, I'm not sure that there is a right thing
to do. If I'm the only person in the world with this problem, then I should
keep my mouth shut and get with the times, but I don't think this setup is
so unreasonable.

Anyway, where exactly is this allocation coming from? Is it a default PARI
setting, or does it come from the way that Sage uses PARI? (I mean, to whom
should I address my "hate mail"? :)


>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-support+unsubscr...@googlegroups.com.
> To post to this group, send email to sage-support@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 13:02, Jonathan Bober wrote:

If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater
immediately grabs 2 GB of the virtual address space for itself, then
that seems like it leaves only 6 GB for malloc/sage_malloc/whatever
else, which would be effectively limiting the physical memory available
in some situations.


Then use a larger value for "ulimit -v" or add some swap space.


This is not the default, and you probably wouldn't want to
run your laptop this way


But it is an advanced setting. Given that, I assume that you are clever 
enough to do the right thing.


--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jonathan Bober]

On Mon, Sep 19, 2016 at 11:04 AM, Jeroen Demeyer 
wrote:

> On 2016-09-19 11:54, Jonathan Bober wrote:
>
>> But in that case the memory does effectively get "used", if it is the
>> case that a quarter of the machine's memory is only available from the
>> PARI memory allocater. If I'm not using the PARI parts of Sage, is that
>> memory completely wasted?
>>
>
> There should not be any consequences apart from the artificially high
> value of virtual memory used. It certainly does not affect the physical
> memory or swap space which is effectively available.



> So the question becomes: what uses the "virtual memory used" value?
> Apparently, the Linux overcommit thing does use that value (which is
> surprising to me, I never really understood how this overcommit stuff
> works).
>
>
If I ulimit -v 8 GB, say, (which is 512/64), and the PARI allocater
immediately grabs 2 GB of the virtual address space for itself, then that
seems like it leaves only 6 GB for malloc/sage_malloc/whatever else, which
would be effectively limiting the physical memory available in some
situations.

This specific machine has overcommit_memory set to 2, which means that the
total amount of non-file-backed writable virtual memory that the kernel
will allocate is fixed (probably to somewhere around the physical size of
RAM). This is not the default, and you probably wouldn't want to run your
laptop this way, but is not so unreasonable for a shared computer with a
lot of RAM, where it usually doesn't cause problems. (It's a setting that
might get changed from the default after the first or second time someone
crashes the machine by using too much memory, especially if this happens on
a Friday afternoon and leaves the machine unavailable all weekend...)

When it does cause problems, it's usually because a process with a lot of
writable virtual memory allocated to it forks, as is happening here.

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

On 2016-09-19 11:54, Jonathan Bober wrote:

But in that case the memory does effectively get "used", if it is the
case that a quarter of the machine's memory is only available from the
PARI memory allocater. If I'm not using the PARI parts of Sage, is that
memory completely wasted?


There should not be any consequences apart from the artificially high 
value of virtual memory used. It certainly does not affect the physical 
memory or swap space which is effectively available.


So the question becomes: what uses the "virtual memory used" value? 
Apparently, the Linux overcommit thing does use that value (which is 
surprising to me, I never really understood how this overcommit stuff 
works).



Jeroen.

--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jonathan Bober]

On Mon, Sep 19, 2016 at 8:29 AM, Jeroen Demeyer 
wrote:

> Why don't you use ulimit -v to limit the per-process available memory?
> That would make sense when starting lots of processes even without the PARI
> non-issue.


Because I was being stupid. That seems like it should work ok.

But in that case the memory does effectively get "used", if it is the case
that a quarter of the machine's memory is only available from the PARI
memory allocater. If I'm not using the PARI parts of Sage, is that memory
completely wasted?

-- 
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

Re: [sage-support] Re: plotting ln(x) graph on sagemath

[Jori Mäntysalo]

On Sun, 18 Sep 2016, jack wrote:


Ubuntu16.04. 



P=plot(log((1+x)/(1-x)), (x, -1,1))
show(P)
gives a lengthy error message which ends with
ImportError: cannot import name scimath

I installed sage at /home/jack/Tools

One clue might be the initial message I get on initiating sage in a
terminal. It reads:


sys:1: RuntimeWarning: not adding directory '' to sys.path since everybody
can write to it.


I have no real clue, but I would start with

strace -e file ./sage 2> log-f

then write

plot(sin)

or something similar and then quit Sage. After that

fgrep tmp_ log-f

shows

open(".../.sage/temp/j-op7010/12072/tmp_n3Sn3X.png", ...) = 11

etc. and

fgrep EACCESS log-f

finds nothing.

--
Jori Mäntysalo

Re: [sage-support] Re: huge virtual memory size when launching 7.3

[Jeroen Demeyer]

Why don't you use ulimit -v to limit the per-process available memory? 
That would make sense when starting lots of processes even without the 
PARI non-issue.


--
You received this message because you are subscribed to the Google Groups 
"sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sage-support+unsubscr...@googlegroups.com.
To post to this group, send email to sage-support@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.