Re: ULIMIT: limiting user virtual address space (stack inluded)

[Valdis Klētnieks]

On Thu, 02 May 2019 09:06:09 -0400, Karaoui mohamed lamine said:

> According to the man page of "ulimit", it is possible to limit the virtual
> address space of a user process: https://ss64.com/bash/ulimit.html
> To limit the virtual address space to 2 GB:
> $ ulimit -SH -v 1048576

It's possible.  It also probably doesn't do what you think it does. In 
particular,
you may be looking for the -d (data) and/or -m (memory) flags.

Also, to limit it to 2G, you're going to need 'ulimit -v 2097152'

> However, it seems that "stack" allocation ignores this limit (I tried with
> both kernel 4.4 and 4.15).

I'm going to go out on a limb and say that what you think is happening
is not in fact what is happening.

> I found this article (patch) that seems to fix the problem:
> https://lwn.net/Articles/373701/

It's unclear what your actual problem is, but given that the patch
has been in the kernel for a decade, it's almost certainly not related.

> My questions are:
> - Why is the stack allocation not placed within the "ulimit" limit? (is it
> a bug?)

Rule of thumb:  Before asking why something happens, verify that it does
in fact happen.

Consider the following C program:

[~] cat > moby-stack.c
#include 

#define MOBYSTACK 16384
#define BIG 2048

int recurse(int level) {
double a[BIG]; /* chew up stack space */

if (level == MOBYSTACK) sleep(999);
recurse(level+1);
}

int main() {recurse(0); };
^D
[~] gcc moby-stack.c
[~] ulimit -v 16384
[~] ulimit -s 32768
[~] ./a.out
Segmentation fault (core dumped)

Examination of the core file shows it created 895 stack entries, for a total of
895 * 8 * 2048 or 14,663,680 or so bytes of stack. That's about what you'd
expect given a virtual limit of 16M, and a bit of space taken up by shared
libraries and so on. Re-running with a virtual limit of 32768 lets it get 1916
stack entries deep. Again, that's about where you'd expect the explosion to
happen with that virtual limit in place. So regardless of the stack limit, it
blows up when it gets to the -v limit.

Conclusion: The stack *is* included in the -v virtual limit.

> - Is there a way, in user space, to force the stack to be allocated at a
> certain address?

Sure.  mmap() yourself a chunk of anonymous pages at the address you want,
and call a little assembler stub that sets your registers.  You'll probably have
to ensure you allocate enough pages, or handle expanding onto new pages
yourself (including making sure there's free pages to grow into).  And of 
course,
you're basically stuck with that stack, returning to older functions on the old
stack is going to be *really* hard.  So you'll probably want to do that in 
main().

Google for 'linux green threads' for the gory details.  Java used them on Linux 
until 2017
or so.

> - Will the patch above (or similar) ever be integrated into the Linux
> kernel?

See above.  It's been in the kernel for a decade.


pgpruEvoYwKjm.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: ULIMIT: limiting user virtual address space (stack inluded)

[Greg KH]

On Thu, May 02, 2019 at 09:06:09AM -0400, Karaoui mohamed lamine wrote:
> Hi all,
> 
> According to the man page of "ulimit", it is possible to limit the virtual
> address space of a user process: https://ss64.com/bash/ulimit.html
> To limit the virtual address space to 2 GB:
> $ ulimit -SH -v 1048576
> 
> However, it seems that "stack" allocation ignores this limit (I tried with
> both kernel 4.4 and 4.15).
> 
> I found this article (patch) that seems to fix the problem:
> https://lwn.net/Articles/373701/
> 
> My questions are:
> - Why is the stack allocation not placed within the "ulimit" limit? (is it
> a bug?)
> - Is there a way, in user space, to force the stack to be allocated at a
> certain address?
> - Will the patch above (or similar) ever be integrated into the Linux
> kernel?

The patch above was integrated into the 2.6.33 kernel, and backported to
2.6.32.9 and fixed in 2.6.32.10.  All of those kernels were released in
2010, quite a long time ago.

Are you sure you are hitting the same "problem"?

thanks,

greg k-h

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

ULIMIT: limiting user virtual address space (stack inluded)

[Karaoui mohamed lamine]

Hi all,

According to the man page of "ulimit", it is possible to limit the virtual
address space of a user process: https://ss64.com/bash/ulimit.html
To limit the virtual address space to 2 GB:
$ ulimit -SH -v 1048576

However, it seems that "stack" allocation ignores this limit (I tried with
both kernel 4.4 and 4.15).

I found this article (patch) that seems to fix the problem:
https://lwn.net/Articles/373701/

My questions are:
- Why is the stack allocation not placed within the "ulimit" limit? (is it
a bug?)
- Is there a way, in user space, to force the stack to be allocated at a
certain address?
- Will the patch above (or similar) ever be integrated into the Linux
kernel?

Regards,
Mohamed
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: virtual address space allocation

[valdis . kletnieks]

On Tue, 31 Jul 2018 15:41:28 +0200, Babis Chalios said:

> I am working on the runtime system of a distributed memory programming
> language. For some reason we need to create a big hole on the address space of
> the application. We are trying to do so by using an mmap during the
> initialization of the runtime and then the runtime handles itself those
> addresses.

How big a big hole, exactly?  And when you say "For some reason", does that
mean "For reasons I don't understand", or "For reasons I don't want to explain"?
(Note that in American colloquial English, it usually means the first...)

> However, we do have problems if memory overcommitment is disabled.

Well, if your "big hole" plus everything already running is big enough to not
fit in (RAM+SWAP), and you disable overcommit, you're going to have a bad
day

> So here goes the question: Is there any other mechanism to allow us to
> do what we want to?

It's unclear what you're trying to *do*.  Trying to re-invent MPI?  If your
goal is RDMA, there's already drivers available for Infiniband and other
fabrics.



pgpH94vf4e22a.pgp
Description: PGP signature
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: virtual address space allocation

[Ruben Safir]

On 07/31/2018 09:41 AM, Babis Chalios wrote:
> I am working on the runtime system of a distributed memory programming
> language. For some reason we need to
> create a big hole on the address space of the application. We are trying
> to do so by using an mmap during the
> initialization of the runtime and then the runtime handles itself those
> addresses.


Oh ok

Allocate more memory

-- 
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002

http://www.nylxs.com - Leadership Development in Free Software
http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Madhu K]

Hi All,

Thank you so much for all your answers.

Regards,
Madhu

On Sun, Oct 2, 2016 at 8:22 AM, Arshad Hussain <arshad.su...@gmail.com>
wrote:

>
> On 30-Sep-2016, at 10:10 pm, Román Martínez <r...@riseup.net> wrote:
>
> > Hi,
> >
> > I compile main.s and it generates main.o. With objdump I can see:
> >
> >  <_start>:
> >0: b8 01 00 00 00  mov$0x1,%eax
> >5: bb 00 00 00 00  mov$0x0,%ebx
> >a: cd 80   int$0x80
> >
> > After link main.o it generates main. With objdump I now can see:
> >
> > 00400078 <_start>:
> >   400078: b8 01 00 00 00  mov$0x1,%eax
> >   40007d: bb 00 00 00 00  mov$0x0,%ebx
> >   400082: cd 80   int$0x80
> >
> > So, linker generates virtual address, doesn't it? But why it starts at
> 400078 and not in other any location? Is there any logic here? A virtual
> address can start at 0?
> >
> The linker script would normally leave out some area and _not_ start from
> 0. (although this is not
> an absolute necessity ). It will pick up and valid virtual address and
> assign start of .text,.bss and
> .stack. This is possible because we now have virtual memory. And every
> program thinks that it
> has _all_ the memory for itself.  So, 400078 is perfectly valid virtual
> address space and linker
> has chosen this for this executable.
>
> Also note that if the address space is getting shifted by little , this is
> because of the KASLR
> (layout randomisation) - If this is disabled then the linker would
> generate the same virtual
> address for all the executable every time.
>
>
>
> 
>
>
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Arshad Hussain]

On 30-Sep-2016, at 10:10 pm, Román Martínez <r...@riseup.net> wrote:

> Hi,
> 
> I compile main.s and it generates main.o. With objdump I can see:
> 
>  <_start>:
>0: b8 01 00 00 00  mov$0x1,%eax
>5: bb 00 00 00 00  mov$0x0,%ebx
>a: cd 80   int$0x80
> 
> After link main.o it generates main. With objdump I now can see:
> 
> 00400078 <_start>:
>   400078: b8 01 00 00 00  mov$0x1,%eax
>   40007d: bb 00 00 00 00  mov$0x0,%ebx
>   400082: cd 80   int$0x80
> 
> So, linker generates virtual address, doesn't it? But why it starts at 400078 
> and not in other any location? Is there any logic here? A virtual address can 
> start at 0?
> 
The linker script would normally leave out some area and _not_ start from 0. 
(although this is not
an absolute necessity ). It will pick up and valid virtual address and assign 
start of .text,.bss and 
.stack. This is possible because we now have virtual memory. And every program 
thinks that it 
has _all_ the memory for itself.  So, 400078 is perfectly valid virtual address 
space and linker
has chosen this for this executable.

Also note that if the address space is getting shifted by little , this is 
because of the KASLR
(layout randomisation) - If this is disabled then the linker would generate the 
same virtual 
address for all the executable every time.







___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Román Martínez]

Hi,

I compile main.s and it generates main.o. With objdump I can see:

 <_start>:
   0:   b8 01 00 00 00      mov$0x1,%eax
   5:   bb 00 00 00 00      mov$0x0,%ebx
   a:   cd 80           int$0x80

After link main.o it generates main. With objdump I now can see:

00400078 <_start>:
  400078:   b8 01 00 00 00      mov$0x1,%eax
  40007d:   bb 00 00 00 00      mov$0x0,%ebx
  400082:   cd 80           int$0x80

So, linker generates virtual address, doesn't it? But why it starts at
400078 and not in other any location? Is there any logic here? A
virtual address can start at 0?

Regards.

El mié, 28-09-2016 a las 19:02 +0530, Gadre Nayan escribió:
> > > Virtual addresses will be used only in case MMU is enabled, otherwise
for a processor an address is something it can put on the bus,
irrespective of physical or virtual.
> > So when your PC increments virtual address for a MMU enabled system,
they will get translated to physical.
> > > On 28 Sep 2016 4:29 p.m., "Prabhunath G" 
wrote:
> > > > > > > > > > > > > > > > > > The virtual addresses what you see in the 
> > > > > > > > > > > > > > > > > > output of objdump is
given/associated by Linker to every instruction and data symbol in
the data/bss section except for symbols in the stack section. It is
wrong to use generated in the context of Linker. When you initiate
$./a.out for execution, the kernel will take your start address
from the ELF header of a.out and place it on the PC (program
counter) or IP (instruction pointer) of the CPU, thereafter CPU
will start incrementing or generating virtual address for every
subsequent instructions. 
> > Regards,
> > Prabhu
> > 
> > 
> > > > > > On Wed, Sep 28, 2016 at 3:41 PM, Madhu K 
wrote:
> > > Hi Arun,
> > > Thanks for your response.
> > > 
> > > > > > > > > > > > > > > I will elaborate my question.Assume I have test.c 
> > > > > > > > > > > > > > > file, I
compiled test.c and generated the a.out ( Executable for linux ),
when I do objdump of a.out, we can see addresses( virtual address
) associated with each instruction, these instructions are
generated by whom?
> > > 
> > > Thanks
> > > 
> > > 
> > > > > > On Wed, Sep 28, 2016 at 11:35 AM, Arun Sudhilal  wrote:
> > > > Hello Madhu,
> > > > 
> > > > 
> > > > > > > > On Wed, Sep 28, 2016 at 10:36 AM, Madhu K  wrote:
> > > > 
> > > > > Hi All,
> > > > 
> > > > >
> > > > 
> > > > > > > > > This is to understand the Virtual address space.Basically who
generates the
> > > > 
> > > > > virtual addresses CPU or GNU compiler?
> > > > 
> > > > 
> > > > 
> > > > I didn't really get your question.
> > > > 
> > > > 
> > > > 
> > > > > > > > Linux kernel starts at a fixed location in virtual space. This
is
> > > > 
> > > > > > > > called PAGE_OFFSET. On a kernel split of 3GB/1GB, 32 system,
its is
> > > > 
> > > > > > > > 0xC000_. You can have a look at system.map file after
compiling
> > > > 
> > > > your kernel.
> > > > 
> > > > When cpu runs with MMU on, your cpu generates virtual address.
> > > > 
> > > > 
> > > > 
> > > > Regards,
> > > > 
> > > > Arun
> > > > 
> > > > 
> > > > 
> > > > >
> > > > 
> > > > > Thanks
> > > > 
> > > > > Madhu
> > > > 
> > > > >
> > > > 
> > > > > ___
> > > > 
> > > > > Kernelnewbies mailing list
> > > > 
> > > > > Kernelnewbies@kernelnewbies.org
> > > > 
> > > > > > > > > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbie
s
> > > > 
> > > > >
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ___
> > > 
> > > Kernelnewbies mailing list
> > > 
> > > Kernelnewbies@kernelnewbies.org
> > > 
> > > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > > 
> > > 
> > 
> > 
> > 
> > -- 
> > Regards,
> > Prabhunath G
> > Linux Trainer
> > Bangalore
> > 
> > 
> > 
> > 
> > 
> > 
> > ___
> > 
> > Kernelnewbies mailing list
> > 
> > Kernelnewbies@kernelnewbies.org
> > 
> > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> > 
> > 
> 
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[piyush moghe]

Please refer to following links, I hope this will help in
understanding of linker generated address and virtual process address
space:

http://www.linuxjournal.com/article/6463?page=0,0
http://www.tenouk.com/ModuleW.html

Regards.
Piyush

On Fri, Sep 30, 2016 at 10:39 PM,   wrote:
> Hi,
>
> I compile main.s and it generates main.o. With objdump I can see:
>
>  <_start>:
>0: b8 01 00 00 00mov$0x1,%eax
>5: bb 00 00 00 00mov$0x0,%ebx
>a: cd 80 int$0x80
>
> After link main.o it generates main. With objdump I now can see:
>
> 00400078 <_start>:
>   400078: b8 01 00 00 00mov$0x1,%eax
>   40007d: bb 00 00 00 00mov$0x0,%ebx
>   400082: cd 80 int$0x80
>
> So, linker generates virtual address, doesn't it? But why it starts at
> 400078 and not in other any location? Is there any logic here? A virtual
> address can start at 0?
>
> Regards.
>
> El 29.09.2016 09:25, arshad hussain escribió:
>
>
> On 28-Sep-2016, at 10:36 am, Madhu K  wrote:
>
> Hi All,
>
>
> The support comes right through from the architecture.
>
> Linear address: Also virtual address. This is Address CPU can address. This
> is 0 - 4GB under 32bit architecture.
> Physical address: This are actual memory installed in your system. Linear
> address have to be converted to
> physical address eventually. MMU translate linear address with the help of
> paging to physical address.
>
>
>
> This is to understand the Virtual address space.Basically who generates the
> virtual addresses CPU or GNU compiler?
>
>
> A program image generated by compiler & linker uses Virtual Address . During
> execution of the program CPU + MMU
> translates or maps each virtual address to physical address.
>
> Net-net this is architecture dependent and would highly recommend reading
> Intel's Architecture Manual for your reference.
>
>
> Thanks
> Madhu
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[roman]

Hi, 

I COMPILE MAIN.S and it generates MAIN.O. With objdump I can see: 

 <_start>: 
   0: b8 01 00 00 00mov$0x1,%eax 
   5: bb 00 00 00 00mov$0x0,%ebx 
   a: cd 80 int$0x80 

After LINK MAIN.O it generates MAIN. With objdump I now can see: 

00400078 <_start>: 
  400078: b8 01 00 00 00mov$0x1,%eax 
  40007d: bb 00 00 00 00mov$0x0,%ebx 
  400082: cd 80 int$0x80 

So, linker generates virtual address, doesn't it? But why it starts at
400078 and not in other any location? Is there any logic here? A virtual
address can start at 0? 

Regards. 

El 29.09.2016 09:25, arshad hussain escribió:

>> On 28-Sep-2016, at 10:36 am, Madhu K  wrote:
>> 
>> Hi All,
> 
> The support comes right through from the architecture. 
> 
> Linear address: Also virtual address. This is Address CPU can address. This 
> is 0 - 4GB under 32bit architecture.
> Physical address: This are actual memory installed in your system. Linear 
> address have to be converted to 
> physical address eventually. MMU translate linear address with the help of 
> paging to physical address.
> 
>> This is to understand the Virtual address space.Basically who generates the 
>> virtual addresses CPU or GNU compiler?
> 
> A program image generated by compiler & linker uses Virtual Address . During 
> execution of the program CPU + MMU 
> translates or maps each virtual address to physical address.
> 
> Net-net this is architecture dependent and would highly recommend reading 
> Intel's Architecture Manual for your reference.
> 
>> Thanks
>> Madhu
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> 
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[arshad hussain]

> On 28-Sep-2016, at 10:36 am, Madhu K  wrote:
> 
> Hi All,

The support comes right through from the architecture. 

Linear address: Also virtual address. This is Address CPU can address. This is 
0 - 4GB under 32bit architecture.
Physical address: This are actual memory installed in your system. Linear 
address have to be converted to 
physical address eventually. MMU translate linear address with the help of 
paging to physical address.


> 
> This is to understand the Virtual address space.Basically who generates the 
> virtual addresses CPU or GNU compiler?

A program image generated by compiler & linker uses Virtual Address . During 
execution of the program CPU + MMU 
translates or maps each virtual address to physical address.

Net-net this is architecture dependent and would highly recommend reading 
Intel’s Architecture Manual for your reference.

> 
> Thanks
> Madhu
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies


___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Gadre Nayan]

Virtual addresses will be used only in case MMU is enabled, otherwise for a
processor an address is something it can put on the bus, irrespective of
physical or virtual.

So when your PC increments virtual address for a MMU enabled system, they
will get translated to physical.
On 28 Sep 2016 4:29 p.m., "Prabhunath G"  wrote:

> The virtual addresses what you see in the output of objdump is
> given/associated by Linker to every instruction and data symbol in the
> data/bss section except for symbols in the stack section. It is wrong to
> use generated in the context of Linker.
> When you initiate *$./a.out* for execution, the kernel will take your
> start address from the ELF header of *a.out* and place it on the PC
> (program counter) or IP (instruction pointer) of the CPU, thereafter CPU
> will start incrementing or generating virtual address for every subsequent
> instructions.
>
> Regards,
> Prabhu
>
>
> On Wed, Sep 28, 2016 at 3:41 PM, Madhu K  wrote:
>
>> Hi Arun,
>>
>> Thanks for your response.
>>
>> I will elaborate my question.Assume I have test.c file, I compiled test.c
>> and generated the a.out ( Executable for linux ), when I do objdump of
>> a.out, we can see addresses( virtual address ) associated with each
>> instruction, these instructions are generated by whom?
>>
>> Thanks
>>
>> On Wed, Sep 28, 2016 at 11:35 AM, Arun Sudhilal 
>> wrote:
>>
>>> Hello Madhu,
>>>
>>> On Wed, Sep 28, 2016 at 10:36 AM, Madhu K  wrote:
>>> > Hi All,
>>> >
>>> > This is to understand the Virtual address space.Basically who
>>> generates the
>>> > virtual addresses CPU or GNU compiler?
>>>
>>> I didn't really get your question.
>>>
>>> Linux kernel starts at a fixed location in virtual space. This is
>>> called PAGE_OFFSET. On a kernel split of 3GB/1GB, 32 system, its is
>>> 0xC000_. You can have a look at system.map file after compiling
>>> your kernel.
>>> When cpu runs with MMU on, your cpu generates virtual address.
>>>
>>> Regards,
>>> Arun
>>>
>>> >
>>> > Thanks
>>> > Madhu
>>> >
>>> > ___
>>> > Kernelnewbies mailing list
>>> > Kernelnewbies@kernelnewbies.org
>>> > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>> >
>>>
>>
>>
>> ___
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>>
>
>
> --
> Regards,
> Prabhunath G
> Linux Trainer
> Bangalore
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Prabhunath G]

The virtual addresses what you see in the output of objdump is
given/associated by Linker to every instruction and data symbol in the
data/bss section except for symbols in the stack section. It is wrong to
use generated in the context of Linker.
When you initiate *$./a.out* for execution, the kernel will take your start
address from the ELF header of *a.out* and place it on the PC (program
counter) or IP (instruction pointer) of the CPU, thereafter CPU will start
incrementing or generating virtual address for every subsequent
instructions.

Regards,
Prabhu


On Wed, Sep 28, 2016 at 3:41 PM, Madhu K  wrote:

> Hi Arun,
>
> Thanks for your response.
>
> I will elaborate my question.Assume I have test.c file, I compiled test.c
> and generated the a.out ( Executable for linux ), when I do objdump of
> a.out, we can see addresses( virtual address ) associated with each
> instruction, these instructions are generated by whom?
>
> Thanks
>
> On Wed, Sep 28, 2016 at 11:35 AM, Arun Sudhilal 
> wrote:
>
>> Hello Madhu,
>>
>> On Wed, Sep 28, 2016 at 10:36 AM, Madhu K  wrote:
>> > Hi All,
>> >
>> > This is to understand the Virtual address space.Basically who generates
>> the
>> > virtual addresses CPU or GNU compiler?
>>
>> I didn't really get your question.
>>
>> Linux kernel starts at a fixed location in virtual space. This is
>> called PAGE_OFFSET. On a kernel split of 3GB/1GB, 32 system, its is
>> 0xC000_. You can have a look at system.map file after compiling
>> your kernel.
>> When cpu runs with MMU on, your cpu generates virtual address.
>>
>> Regards,
>> Arun
>>
>> >
>> > Thanks
>> > Madhu
>> >
>> > ___
>> > Kernelnewbies mailing list
>> > Kernelnewbies@kernelnewbies.org
>> > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>> >
>>
>
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>


-- 
Regards,
Prabhunath G
Linux Trainer
Bangalore
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Alexander Kapshuk]

On Wed, Sep 28, 2016 at 8:06 AM, Madhu K  wrote:
> Hi All,
>
> This is to understand the Virtual address space.Basically who generates the
> virtual addresses CPU or GNU compiler?
>
> Thanks
> Madhu
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

In no way does this response claim to be based on personal expert
knowledge, but if my understanding of Chapter 2 (Page tables) of
https://pdos.csail.mit.edu/6.828/2016/xv6/book-rev9.pdf is correct, it
is the hardware that performs the mapping of physical memory, RAM, to
virtual addresses on x86:
Quote:
As a reminder, x86 instructions (both user and kernel) manipulate
virtual addresses. The machine’s RAM, or physical memory, is indexed
with physical addresses. The x86 page table hardware connects these
two kinds of addresses, by mapping each virtual address to a physical
address.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Greg KH]

On Wed, Sep 28, 2016 at 03:44:28PM +0530, Madhu K wrote:
> Hi Greg,
> 
> As I understand CPU's work is Fetch, Decode and Execute. How can CPU generate
> virtual address.please correct me if my understanding is wrong.

What exactly do you mean by "generate virtual address"?

Is this a homework assignment?

greg k-h

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Madhu K]

Hi,

Even I read somewhere, during compilation, compiler will take the linker
script as input and it will generate the virtual addresses.

Thanks,
Madhu

On Wed, Sep 28, 2016 at 2:44 PM, Augusto Mecking Caringi <
augustocari...@gmail.com> wrote:

> On Wed, Sep 28, 2016 at 6:06 AM, Madhu K  wrote:
> > Hi All,
> >
> > This is to understand the Virtual address space.Basically who generates
> the
> > virtual addresses CPU or GNU compiler?
>
> Hi,
>
>  In my view, the linker.
>
>  The compiler normally generates absolute addresses (starting from
> 0), creating object code.
>
>  Then the linker generates an executable by combining different
> object files together and assigning virtual addresses.
>
>  Finally, is CPU (MMU) job (with the kernel helping) to translate
> these virtual addresses to physical ones.
>
>  Best regards,
>
> --
> Augusto Mecking Caringi
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Madhu K]

Hi Arun,

Thanks for your response.

I will elaborate my question.Assume I have test.c file, I compiled test.c
and generated the a.out ( Executable for linux ), when I do objdump of
a.out, we can see addresses( virtual address ) associated with each
instruction, these instructions are generated by whom?

Thanks

On Wed, Sep 28, 2016 at 11:35 AM, Arun Sudhilal  wrote:

> Hello Madhu,
>
> On Wed, Sep 28, 2016 at 10:36 AM, Madhu K  wrote:
> > Hi All,
> >
> > This is to understand the Virtual address space.Basically who generates
> the
> > virtual addresses CPU or GNU compiler?
>
> I didn't really get your question.
>
> Linux kernel starts at a fixed location in virtual space. This is
> called PAGE_OFFSET. On a kernel split of 3GB/1GB, 32 system, its is
> 0xC000_. You can have a look at system.map file after compiling
> your kernel.
> When cpu runs with MMU on, your cpu generates virtual address.
>
> Regards,
> Arun
>
> >
> > Thanks
> > Madhu
> >
> > ___
> > Kernelnewbies mailing list
> > Kernelnewbies@kernelnewbies.org
> > https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Madhu K]

Hi Greg,

As I understand CPU's work is Fetch, Decode and Execute. How can CPU
generate virtual address.please correct me if my understanding is wrong.

Thanks
Madhu

On Wed, Sep 28, 2016 at 11:39 AM, Greg KH  wrote:

> On Wed, Sep 28, 2016 at 10:36:59AM +0530, Madhu K wrote:
> > Hi All,
> >
> > This is to understand the Virtual address space.Basically who generates
> the
> > virtual addresses CPU or GNU compiler?
>
> CPU.
>
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Augusto Mecking Caringi]

On Wed, Sep 28, 2016 at 6:06 AM, Madhu K  wrote:
> Hi All,
>
> This is to understand the Virtual address space.Basically who generates the
> virtual addresses CPU or GNU compiler?

Hi,

 In my view, the linker.

 The compiler normally generates absolute addresses (starting from
0), creating object code.

 Then the linker generates an executable by combining different
object files together and assigning virtual addresses.

 Finally, is CPU (MMU) job (with the kernel helping) to translate
these virtual addresses to physical ones.

 Best regards,

-- 
Augusto Mecking Caringi

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Greg KH]

On Wed, Sep 28, 2016 at 10:36:59AM +0530, Madhu K wrote:
> Hi All,
> 
> This is to understand the Virtual address space.Basically who generates the
> virtual addresses CPU or GNU compiler?

CPU.

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Virtual Address Space

[Arun Sudhilal]

Hello Madhu,

On Wed, Sep 28, 2016 at 10:36 AM, Madhu K  wrote:
> Hi All,
>
> This is to understand the Virtual address space.Basically who generates the
> virtual addresses CPU or GNU compiler?

I didn't really get your question.

Linux kernel starts at a fixed location in virtual space. This is
called PAGE_OFFSET. On a kernel split of 3GB/1GB, 32 system, its is
0xC000_. You can have a look at system.map file after compiling
your kernel.
When cpu runs with MMU on, your cpu generates virtual address.

Regards,
Arun

>
> Thanks
> Madhu
>
> ___
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Virtual Address Space

[Madhu K]

Hi All,

This is to understand the Virtual address space.Basically who generates the
virtual addresses CPU or GNU compiler?

Thanks
Madhu
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Regarding virtual address space of the kernel...

[Naman shekhar Mishra]

I've been trying to understand this for a while with little success so any help 
would be hugely appreciated...

Suppose I have 512 MB of physical RAM available in the system. The top 1 GB 
part of the 4 GB virtual address space maps *directly* starting from the 0 
physical address (mapped by kernel). (Question 1) **Doesn't that mean that the 
physical RAM is not enough to support this  mapping (I mean, only 512 MB will 
be directly mapped be the kernel. Moreover, no physical space will be left for 
user space mapping)? And even if I have 1 GB of physical RAM installed, where 
would all the processes get mapping from since this this 1 GB will be mapped in 
the address space of the kernel? **

I read in a book:
The address space after PAGE OFFSET is reserved for the kernel and
this is where the complete physical memory is mapped (eg. if a system
has 64mb of RAM, it is mapped from PAGE OFFSET to PAGE OFFSET
+ 64mb).
(Question 2, somewhat related to Q1)**Does this mean that no physical memory is 
left to be mapped by user space virtual memory (i.e. if all physical memory is 
mapped by address above PAGE_OFFSET, what is mapped by address below 
PAGE_OFFSET)?**

(Question 3)**Also, why can't all memory, (PAGE_OFFSET to PAGE_OFFSET + 1 GB) 
be mapped to this 64 MB. After all pages can be swapped out and new pages can 
be brought in. 


(Question 4)**Also, when we say that kernel can map 896 MB (ZONE_NORMAL), does 
this 896 MB refer to virtual space or physical space?**
Little (ok, very) confused so kindly explain...  Thank you. 
___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Regarding virtual address space of the kernel...

[Dave Hylands]

Hi Naman,

On Fri, Jun 17, 2011 at 1:34 AM, Naman shekhar Mishra
namanshekharmish...@yahoo.in wrote:
 I've been trying to understand this for a while with little success so any
 help would be hugely appreciated...
 Suppose I have 512 MB of physical RAM available in the system. The top 1 GB
 part of the 4 GB virtual address space maps *directly* starting from the 0
 physical address (mapped by kernel). (Question 1) **Doesn't that mean that
 the physical RAM is not enough to support this  mapping (I mean, only 512 MB
 will be directly mapped be the kernel. Moreover, no physical space will be
 left for user space mapping)? And even if I have 1 GB of physical RAM
 installed, where would all the processes get mapping from since this this 1
 GB will be mapped in the address space of the kernel? **

Let's suppose that you're using an ARM (just because that's what I'm
familiar with), and further lets' suppose that yours 512 Mb of memory
occupies 512 MB starting at address 0x4000 (rather arbitrary - but
I wanted to pick something that wasn't zero).

So your physical memory will be at 0x4000 - 0x5fff
The kernel will map the 512Mb virtual space 0xc000-0xdfff to
the 512 Mb physical space 0x4000-0x5fff.

When a user mode process starts, the kernel will allocate pages from
it's memory, and setup additional virtual mappings into that memory.

 I read in a book:
 The address space after PAGE OFFSET is reserved for the kernel and
 this is where the complete physical memory is mapped (eg. if a system
 has 64mb of RAM, it is mapped from PAGE OFFSET to PAGE OFFSET
 + 64mb).

That's true, when the physical memory fits into that space. There are
also alternative mappings where you can change the split. By default
userspace gets 3Gb virtual space, and kernel gets 1Gb. You can also
2/2 and 1/3 splits as well.

 (Question 2, somewhat related to Q1)**Does this mean that no physical memory
 is left to be mapped by user space virtual memory (i.e. if all physical
 memory is mapped by address above PAGE_OFFSET, what is mapped by address
 below PAGE_OFFSET)?**

No the kernel allocates pages from its memory to give to each
user-mode process. This page has a reasonable picture:
http://www.xml.com/ldd/chapter/book/ch13.html

The memory starting at PAGE_OFFSET is what are being referred to as
kernel logical addresses.

user pages, and additional kernel memory (allocated using kmap/kunmap)
can use memory beyond the logical space.

 (Question 3)**Also, why can't all memory, (PAGE_OFFSET to PAGE_OFFSET + 1
 GB) be mapped to this 64 MB. After all pages can be swapped out and new
 pages can be brought in.

Swapping only works when you have external storage to swap to. Many
devices running linux (like your typical Android phone) have zero
swap. I'm also pretty sure that only user-mode pages are swapped.

When you're using swap, you can have more virtual memory in use than
physical memory. In your example, if we had 64 Mb of physical memory
and say 512Mb of virtual memory allocated. 64 Mb of the 512 Mb would
be mapped to physical pages, and the remainder would be mapped to
swap pages located on some external device. Accessing one of the
swap pages would cause a page-fault and the kernel would need to
swap-out one of the physical pages and then swap in the page which
caused the page fault.

 (Question 4)**Also, when we say that kernel can map 896 MB (ZONE_NORMAL),
 does this 896 MB refer to virtual space or physical space?**

The 896 Mb is the limit on the kernel logical memory space, when you
use a 3Gb user/1Gb kernel split.

-- 
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

___
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies