Hi Zachary, Hi All,
I would like to point out a few things we definitely will need even though
.NET doesn't have/support it.
- We clearly need to seperate the allocator (heap) and the GC
We could use private non-collected heaps in different scenarios to achieve
additional performance. I already have three places in my mind, being kernel
singleton objects, jit temporary objects and something like the exchange heap
provided by Singularity. These heaps should use a defined interface so we could
bind various garbage collectors on top of them (even though the GC only needs
two features from this interface: Determine if a reference is in the heap and
to release a reference in the heap itself.)
- We will need multiple (GC'ed) heaps to allocate objects from
I see this primarily to reduce lock contention and fragmentation. I want the
JIT to be able to allocate a private heap to store its intermediate
representation in and
throw it away immediately after I'm done - all without GCing the ton of
objects we're going to allocate there. I'm thinking of supporting this
scenario similarly as MS
has done with the constrained execution regions in .NET 2.0 and following.
- Memory image
I've been thinking a bit about Adams requests for everything is an object
style of development and have come to the conclusion that this may really be
feasable, if
we can support above scenario. We could emit already instantiated objects into
special sections of the executable file and load them into memory and have the
heap/GC take over later. This could dramatically speed up kernel boot times if
we don't have to setup everything from scratch - this advantage should
clearly be used.
- We need to be able to dump physical and potentially virtual memory to disk
for hibernation
I know this is really far away, but in contrast to other open source operating
system I clearly want perfect power management to be top priority. Dumping
physical and virtual memory to disk is a requirement for different scenarios
not only hibernation. I'm not sure how much it'll affect you, but we
should think about it. Your opinion or knowledge on this would be interesting.
Mike
Zachary Gorden schrieb:
Right, I see this is what happens when I'm out of touch for a few
days. Anyways, the whole single or multiple address space design issue
can be dealt with later. Page tables are going to be implemented
regardless (we need them for paging purposes anyways) and I intend to do
MAS for the time being (since it actually simplifies a few things for
us). We can experiment with SAS on x64, once we get there.
Anyways, as pointed out by Grover, there has to be something at the lowest
level of the OS to handle the basic memory allocations and whatnot.
Anything dealing with objects and the like need something already existing to
sit atop of and that is what I am trying to implement now. We can save
all the fancy object stuff for the GC, or we could go the NT route and have
everything as objects. Regardless, we still need to get the foundations
set before we can get too ambitious.
On Mon, Jul 21, 2008 at 5:07 PM, Sander van Rossen <[EMAIL PROTECTED]>
wrote:
We also need to decide how to implement virtual memory.
Before we where talking about using a single page-table (or maybe just
a handful) for the entire system, because this would allow us to not
slow down the system with page-table switches.. but on a 32 bit system
with 4gb memory this is pretty hard to do.
We could drop the "single page-table" idea, decide to go 64-bit only,
or something else.. but we need to define it.
On Sun, Jul 20, 2008 at 9:23 AM, grover <[EMAIL PROTECTED]> wrote:
> Hi Adam,
> something like that could work, but there is still the problem that you
have
> to locate the object in memory in a heap, which will later on be
maintained
> by the GC. There are some low-level services, which IMHO just can't be
> wrapped in object instances - as they are essentially singletons. Memory
> management should be split into five pieces IMHO:
> 1. A physical memory manager, which gives out 4KB pages of physical address
> space
> 2. A process memory manager, which manages virtual memory and acquires
> physical pages on demand
> 3. A pager, which moves virtual pages between disk and physical memory
> 4. A heap manager, which is responsible for the actual object allocation
> 5. A GC which returns regions to the heap
> At least #1,3 are singletons with respect to system lifetimes. #2,4,5 are
> per-process singletons.
> Maybe we could think about your design of using adapter or strategy objects
> to wrap the access to these services in your design? This way the adapters
> could implement the interfaces you need and abstract away the access to
the
> actual service implementations.
> Mike
> Am 20.07.2008 um 00:34 schrieb Adam Stevenson:
>
> Howdy Mike,
>
> I am thinking what we need to do is figure out what format the objects are
> going to need to look like in memory when a memory manager / GC are in
> place. This way, when the MM and GC come online during the boot up,
they
> can "take over" objects that were previously not managed, and
the initial
> memory manager can keep a list of memory allocated that is not yet been
> assigned to a memory manager - thus making it easy to identify objects.
> Think this approach could work? If so, we just need to figure out
how we
> want to structure the unmanaged memory / objects, so that their structures
> can be compatible.
>
> -Adam
>
> On Sat, Jul 19, 2008 at 5:21 AM, Matthijs ter Woord
> <[EMAIL PROTECTED]> wrote:
>>
>> Ah, you guys are trying to instantiate the memory manager. Hmm..
>>
>>
>>
>>
>> From: [EMAIL PROTECTED]
>> [mailto:[EMAIL PROTECTED] On Behalf Of
>> grover
>> Sent: zaterdag 19 juli 2008 11:31
>> To: sharpos-developers@lists.sourceforge.net
>> Subject: Re: [SharpOS Developers] Memory manager designand
>> overallcompartmentalization
>>
>>
>>
>> Matthijs, you're right. However it still doesn't solve the
problem that
>> you want to new a memory manager without having a memory manager to
allocate
>> one from ;) SharpOS doesn't have a GC with the current memory
management
>> facilities either. I see it more of a structural problem than a
technical
>> one and thus I see it more like a design issue.
>>
>>
>>
>> Mike
>>
>>
>> ________________________________
>>
>> Von: [EMAIL PROTECTED]
>> [mailto:[EMAIL PROTECTED] Im Auftrag von
>> Matthijs ter Woord
>> Gesendet: Samstag, 19. Juli 2008 11:04
>> An: sharpos-developers@lists.sourceforge.net
>> Betreff: Re: [SharpOS Developers] Memory manager designand
>> overallcompartmentalization
>>
>> An option is maybe this:
>>
>>
>>
>> Make a fake memory manager, which only is able to allocate (ie, it
never
>> takes back memory). For most basic operations you don't use too
much
>> objects, so you can live without a GC for a while too. This way you
can
>> start on Object support, then interface support, while not having
issues
>> with GC and memory manager.
>>
>>
>>
>> We do it that way in Cosmos. We don't have a memorymanager capable
of
>> freeing memory, neither do we have a GC (we have the code in place to
plug
>> one in, but don't have one. Ralf just started on this)?.
>>
>>
>>
>> From: [EMAIL PROTECTED]
>> [mailto:[EMAIL PROTECTED] On Behalf Of
>> grover
>> Sent: zaterdag 19 juli 2008 10:16
>> To: sharpos-developers@lists.sourceforge.net
>> Subject: Re: [SharpOS Developers] Memory manager design and
>> overallcompartmentalization
>>
>>
>>
>> Hi Adam,
>>
>>
>>
>> the AOT does support interfaces. There were some bugs, but they
shouldn't
>> stop us there. There's only one problem: For interfaces to work
you need
>> object instances and new/GC... Which causes us a kind of chicken &
egg
>> problem. At least at the level Zachary is working at, we need to stay
with
>> static classes. I understand your goals, but unless you can give me a
hint
>> how you want to manage it without the chicken&egg situation I
don't have any
>> idea.
>>
>>
>>
>> Mike
>>
>>
>> ________________________________
>>
>> Von: [EMAIL PROTECTED]
>> [mailto:[EMAIL PROTECTED] Im Auftrag von
>> Adam Stevenson
>> Gesendet: Freitag, 18. Juli 2008 06:44
>> An: sharpos-developers@lists.sourceforge.net
>> Betreff: Re: [SharpOS Developers] Memory manager design and
>> overallcompartmentalization
>>
>> Howdy Zachary and All,
>>
>> Yes, I believe that we do need to do a good restructruring. I
have a
>> bunch of ideas I have been hashing out in my sandbox, but I there are
parts
>> that I am not sure exactly what is going on (some parts get cryptic,
any
>> understand the process of preparing the image fully?). I first
think we
>> need a high level diagram of all the major components, so we can start
>> figuring out where we can make plugable modules. From my own
project work,
>> and my experience with biologcial systems, we need to slowly evolve
this and
>> not break it. (I know duh).
>>
>> First does thing support interfaces yet? Because I don't
see much of them
>> in the code base. If not, I say we work on making a list of
abstract
>> classes that we can start keying in as modules. I know Grover
has been
>> working on a lot of this, and we need to get his input on the AOT
plugable
>> architecture. Anyone going to be around Sunday afternoon (CST) -
or most
>> anytime Monday (minus 3:00 to 5:00 CST)? I can sit down online
with anyone
>> and start diagraming some of this stuff and work a bit in team mode -
>> identify document, and review. And then we can send out whatever
documents
>> we get done via the list serve, and continue to move on ot the next.
Anyone
>> up for it?
>>
>> As for what I have been working on is a bunch of class processing
stuff,
>> and memory structuring of components. I want to build an
architecture that
>> you could say is "fractal" - thus allowing for the same
archtiecture whether
>> a JIT is being ran on top of a JIT (pretty dang inefficient) or is
actaul
>> the JIT. To do this, everything needs to be turned into an
interface, to
>> allow for components to shift up and down. I have started
looking into how
>> to structure this namespace and working on a prototype base, but it
is not
>> compable yet (on my to do list for next week). But bottom line,
this is
>> still a long way off from where the AOT and SharpOS is now, so this is
by
>> now means a fully workable design that SharpOS can be converted too.
Plus
>> there are some differences in concepts too with the idea of runtime
and OS
>> that are going to have to be hashed out.
>>
>> So personally, we need to look at all of us "newbies' fully
understanding
>> what we have, and second we slowly turn this thing modular and make a
list
>> of all the modules and what their requirements are, so the code base
can be
>> evolved in small parts versus changing one thing causes a bunch of
ripple
>> effects throughout the code. My guess, a lot of this modularity
already
>> exists, but we just need to uncover dividing points. So back to
Sunday and
>> Monday for me. Anyone game? And thoughts on how we want to
start breaking
>> up this code base into smaller understandable pieces? Oh
Zachary, I got
>> both those books on Memory Management and Garbage Collection - need to
>> finish them, but if you have concepts to reference, just let me know
page
>> numbers.
>>
>> Adam Stevenson
>>
>> Graduate Student
>> Department of Biology
>> Texas A&M University
>> College Station, TX 77843, USA
>>
>> On Wed, Jul 16, 2008 at 10:03 AM, Zachary Gorden
>> <[EMAIL PROTECTED]> wrote:
>>
>> Just a clarification, when I say memory manager, I mean from the page
>> tables to the physical memory table to the allocations. It seems
that
>> people keep confusing what I am referring to whenever I say MM.
This is
>> mostly an informational thing to let people know what's going
through my
>> head and the fact that I'm not dead and still working/planning.
>>
>> The memory manager as it stands suffers from a somewhat awkward design
>> that makes it difficult to correct certain items. Not exactly
anyone's
>> fault, as SharpOS needed a MM and that's what it got. The
gist is, trying
>> to fix the issues in here and/or optimizing it won't do much in
the long
>> term, which is one reason you guys haven't seen much activity
from me.
>>
>> I've been mostly consulting a few ROS developers on how to
bootstrap a
>> page table based MM, the initial loading and stuff, so that's the
>> explanation for the lack of code. For the time being, I've
gotten most of
>> the conceptual things explained and will be writing out some of my own
code
>> as well as examining just what kind of support for paging exists right
now.
>> And actually, in the process of trying to look at the current code,
another
>> thought occurred to me. Currently, the ADC is, to put it simply,
horribly
>> organized. It's basically all of the AD code in one spot,
without any
>> actual segmentation. That is definitely going to screw with us
down the
>> line, and it is likely to not be limited to the ADC.
>>
>> Because of the above reasons, I am most definitely going to need to
create
>> a branch (once I figure out how), as obviously doing surgery on trunk
is
>> going to result in incredible instabilities and breakages. But
as far as
>> the restructuring goes, I think a lot of us would benefit from such
changes,
>> including people already working with their own branches.
I'd like to hear
>> from other people on the restructuring idea, as I know I'm not the
only one
>> who noticed the issues.
>>
>>
-------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> SharpOS-Developers mailing list
>> SharpOS-Developers@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>>
>>
>>
>>
-------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> SharpOS-Developers mailing list
>> SharpOS-Developers@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
>
http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
> SharpOS-Developers mailing list
> SharpOS-Developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> SharpOS-Developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>
>
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-Developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-Developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers
