Howdy Mike and Everyone,
It is late here, 4:30 am, so I am not going to be up for much longer, but I
wanted to get some ideas and some direction out on the table.
First, when I wake up, and after I go to church (and maybe take a short nap)
I am starting a document - which I want to lead to a group publication, on
the design strategy for building the SharpOS Memory Manager. I think we all
agree is the next big step in this project. If I am wrong in this, please
correct me and we can redirect our efforts elsewhere.
I would like to place the document and its contents on the website, which is
down again. This problem has to be solved and stay solved. XFury, what do
we need to do / contribute / find / help with / pray for / donate / find
rumpelstiltskin and trade our first born for (well maybe not the first
born...) to get the server in working order and for it to stay in working
order. I am thinking since redmine keeps crashing, we can install a
separate wiki, and have that as our main page, and then link to redmine
under a different hosting account to improve stability? And what is the
current configuration of the server? I am not asking to cause issues, just
so that we can help provide input to solve it and get the problem off of our
collective to-do list plate.
Second, speaking of to do list, I want to gather a list of to-do items that
everyone thinks as a group we need to do. I would then like to enumerate
them, and get everyones feedback on them all, and collect the feedback into
another document/wiki page "July To-Do List", and assign priorities which we
can collectively work on as a group. I don't feel this to do list has to be
fixed in any way or become law, but it will allow us all the ability to see
what needs to be done from their different points of views and perspectives.
Ok, with that said, I am going to be heading to bed soon (the underlying
them of this email... sleep), but I want to get the "virutal" meeting
started. When I wake up and get back from the morning activities I will try
to reply and start organizing responses. Till then, please everyone that
reads this, send in your perspective and issues - it is time for this online
meeting to take shape.
Oh one more thing, since I want to go to bed soon (sleep...) , I am going to
save a lot of my thoughts for the memory management till later, but I would
like for us to start making a list of sources from which we can all
reference to aid us in our design of the memory manager(s) (I am thinking we
can make simpler ones and build on the implementations as we go, to help
keep momentum going in the project - and to keep "market share" - really I
am not competitive at all ;-) ) Right now I am using the following
references, but I know there is a TON of more material out there. As we
cannot collect it all, I was wondering what you all see as the first set of
reference material we all need to be up to speed on? So far what I am using
for reference is the following:
1. The Minix Book - Operating Systems Design and Implementation (3rd Ed)
by Tanenbaum and Woodhull.
2. Memory Management - Algorithms and Implemnation in C/C++ by Bill Blunden
3. Garbage Collection - Algorithms for Automatic Dynamic Memory Management
(really for the next stage, but it helps)
4. The Intel microprocessors : 8086/8088, 80186/80188, 80286, 80386, 80486,
Pentium, Pentium Pro processor, Pentium II, Pentium III, and Pentium 4 :
architecture, programming, and interfacing
I also would like to find some good papers on the material - I need to scan
the references in Memory Management (#2), and I would like good
recommendations for any other really good x86 books and working with its
memory protection parts; any one have the links to the formal documents on
the stuff?
I know there a good couple of websites out there on OS design, anyone have
the links to those handy so we can push them there - they are burried
somewhere on my other computer, but someone has to have them more handy than
I, and links to Memory Management?
Any other internet link / source / paper / book / piece of bedtime reading
material that needs to be read by all of us to get memory management off of
our plates?
Ok, with that rant said, please all, hit me with your replies and I will see
you when my cerebral cortex is working again. Remember that the forum is
open, lets get what needs to be done out currently out into discussion so we
can work as a International Code Hacking Commando Team and get it done.
Adam to bed he said....
On Sun, Jul 20, 2008 at 2:23 AM, grover <[EMAIL PROTECTED]> wrote:
> Hi Adam,
> something like that could work, but there is still the problem that you
> have to locate the object in memory in a heap, which will later on be
> maintained by the GC. There are some low-level services, which IMHO just
> can't be wrapped in object instances - as they are essentially singletons.
> Memory management should be split into five pieces IMHO:
>
> 1. A physical memory manager, which gives out 4KB pages of physical address
> space
> 2. A process memory manager, which manages virtual memory and acquires
> physical pages on demand
> 3. A pager, which moves virtual pages between disk and physical memory
> 4. A heap manager, which is responsible for the actual object allocation
> 5. A GC which returns regions to the heap
>
> At least #1,3 are singletons with respect to system lifetimes. #2,4,5 are
> per-process singletons.
>
> Maybe we could think about your design of using adapter or strategy objects
> to wrap the access to these services in your design? This way the adapters
> could implement the interfaces you need and abstract away the access to the
> actual service implementations.
>
> Mike
>
> Am 20.07.2008 um 00:34 schrieb Adam Stevenson:
>
> Howdy Mike,
>
> I am thinking what we need to do is figure out what format the objects are
> going to need to look like in memory when a memory manager / GC are in
> place. This way, when the MM and GC come online during the boot up, they
> can "take over" objects that were previously not managed, and the initial
> memory manager can keep a list of memory allocated that is not yet been
> assigned to a memory manager - thus making it easy to identify objects.
> Think this approach could work? If so, we just need to figure out how we
> want to structure the unmanaged memory / objects, so that their structures
> can be compatible.
>
> -Adam
>
> On Sat, Jul 19, 2008 at 5:21 AM, Matthijs ter Woord <
> [EMAIL PROTECTED]> wrote:
>
>> Ah, you guys are trying to instantiate the memory manager. Hmm..
>>
>>
>>
>> *From:* [EMAIL PROTECTED] [mailto:
>> [EMAIL PROTECTED] *On Behalf Of *grover
>> *Sent:* zaterdag 19 juli 2008 11:31
>>
>> *To:* sharpos-developers@lists.sourceforge.net
>> *Subject:* Re: [SharpOS Developers] Memory manager designand
>> overallcompartmentalization
>>
>>
>> Matthijs, you're right. However it still doesn't solve the problem that
>> you want to new a memory manager without having a memory manager to allocate
>> one from ;) SharpOS doesn't have a GC with the current memory management
>> facilities either. I see it more of a structural problem than a technical
>> one and thus I see it more like a design issue.
>>
>>
>> Mike
>>
>>
>> ------------------------------
>>
>> *Von:* [EMAIL PROTECTED] [mailto:
>> [EMAIL PROTECTED] *Im Auftrag von *Matthijs
>> ter Woord
>> *Gesendet:* Samstag, 19. Juli 2008 11:04
>> *An:* sharpos-developers@lists.sourceforge.net
>> *Betreff:* Re: [SharpOS Developers] Memory manager designand
>> overallcompartmentalization
>>
>> An option is maybe this:
>>
>>
>> Make a fake memory manager, which only is able to allocate (ie, it never
>> takes back memory). For most basic operations you don't use too much
>> objects, so you can live without a GC for a while too. This way you can
>> start on Object support, then interface support, while not having issues
>> with GC and memory manager.
>>
>>
>> We do it that way in Cosmos. We don't have a memorymanager capable of
>> freeing memory, neither do we have a GC (we have the code in place to plug
>> one in, but don't have one. Ralf just started on this)….
>>
>>
>> *From:* [EMAIL PROTECTED] [mailto:
>> [EMAIL PROTECTED] *On Behalf Of *grover
>> *Sent:* zaterdag 19 juli 2008 10:16
>> *To:* sharpos-developers@lists.sourceforge.net
>> *Subject:* Re: [SharpOS Developers] Memory manager design and
>> overallcompartmentalization
>>
>>
>> Hi Adam,
>>
>>
>> the AOT does support interfaces. There were some bugs, but they shouldn't
>> stop us there. There's only one problem: For interfaces to work you need
>> object instances and new/GC... Which causes us a kind of chicken & egg
>> problem. At least at the level Zachary is working at, we need to stay with
>> static classes. I understand your goals, but unless you can give me a hint
>> how you want to manage it without the chicken&egg situation I don't have any
>> idea.
>>
>>
>> Mike
>>
>>
>> ------------------------------
>>
>> *Von:* [EMAIL PROTECTED] [mailto:
>> [EMAIL PROTECTED] *Im Auftrag von *Adam
>> Stevenson
>> *Gesendet:* Freitag, 18. Juli 2008 06:44
>> *An:* sharpos-developers@lists.sourceforge.net
>> *Betreff:* Re: [SharpOS Developers] Memory manager design and
>> overallcompartmentalization
>>
>> Howdy Zachary and All,
>>
>> Yes, I believe that we do need to do a good restructruring. I have a
>> bunch of ideas I have been hashing out in my sandbox, but I there are parts
>> that I am not sure exactly what is going on (some parts get cryptic, any
>> understand the process of preparing the image fully?). I first think we
>> need a high level diagram of all the major components, so we can start
>> figuring out where we can make plugable modules. From my own project work,
>> and my experience with biologcial systems, we need to slowly evolve this and
>> not break it. (I know duh).
>>
>> First does thing support interfaces yet? Because I don't see much of them
>> in the code base. If not, I say we work on making a list of abstract
>> classes that we can start keying in as modules. I know Grover has been
>> working on a lot of this, and we need to get his input on the AOT plugable
>> architecture. Anyone going to be around Sunday afternoon (CST) - or most
>> anytime Monday (minus 3:00 to 5:00 CST)? I can sit down online with anyone
>> and start diagraming some of this stuff and work a bit in team mode -
>> identify document, and review. And then we can send out whatever documents
>> we get done via the list serve, and continue to move on ot the next. Anyone
>> up for it?
>>
>> As for what I have been working on is a bunch of class processing stuff,
>> and memory structuring of components. I want to build an architecture that
>> you could say is "fractal" - thus allowing for the same archtiecture whether
>> a JIT is being ran on top of a JIT (pretty dang inefficient) or is actaul
>> the JIT. To do this, everything needs to be turned into an interface, to
>> allow for components to shift up and down. I have started looking into how
>> to structure this namespace and working on a prototype base, but it is not
>> compable yet (on my to do list for next week). But bottom line, this is
>> still a long way off from where the AOT and SharpOS is now, so this is by
>> now means a fully workable design that SharpOS can be converted too. Plus
>> there are some differences in concepts too with the idea of runtime and OS
>> that are going to have to be hashed out.
>>
>> So personally, we need to look at all of us "newbies' fully understanding
>> what we have, and second we slowly turn this thing modular and make a list
>> of all the modules and what their requirements are, so the code base can be
>> evolved in small parts versus changing one thing causes a bunch of ripple
>> effects throughout the code. My guess, a lot of this modularity already
>> exists, but we just need to uncover dividing points. So back to Sunday and
>> Monday for me. Anyone game? And thoughts on how we want to start breaking
>> up this code base into smaller understandable pieces? Oh Zachary, I got
>> both those books on Memory Management and Garbage Collection - need to
>> finish them, but if you have concepts to reference, just let me know page
>> numbers.
>>
>> Adam Stevenson
>>
>> Graduate Student
>> Department of Biology
>> Texas A&M University
>> College Station, TX 77843, USA
>>
>> On Wed, Jul 16, 2008 at 10:03 AM, Zachary Gorden <
>> [EMAIL PROTECTED]> wrote:
>>
>> Just a clarification, when I say memory manager, I mean from the page
>> tables to the physical memory table to the allocations. It seems that
>> people keep confusing what I am referring to whenever I say MM. This is
>> mostly an informational thing to let people know what's going through my
>> head and the fact that I'm not dead and still working/planning.
>>
>> The memory manager as it stands suffers from a somewhat awkward design
>> that makes it difficult to correct certain items. Not exactly anyone's
>> fault, as SharpOS needed a MM and that's what it got. The gist is, trying
>> to fix the issues in here and/or optimizing it won't do much in the long
>> term, which is one reason you guys haven't seen much activity from me.
>>
>> I've been mostly consulting a few ROS developers on how to bootstrap a
>> page table based MM, the initial loading and stuff, so that's the
>> explanation for the lack of code. For the time being, I've gotten most of
>> the conceptual things explained and will be writing out some of my own code
>> as well as examining just what kind of support for paging exists right now.
>> And actually, in the process of trying to look at the current code, another
>> thought occurred to me. Currently, the ADC is, to put it simply, horribly
>> organized. It's basically all of the AD code in one spot, without any
>> actual segmentation. That is definitely going to screw with us down the
>> line, and it is likely to not be limited to the ADC.
>>
>> Because of the above reasons, I am most definitely going to need to create
>> a branch (once I figure out how), as obviously doing surgery on trunk is
>> going to result in incredible instabilities and breakages. But as far as
>> the restructuring goes, I think a lot of us would benefit from such changes,
>> including people already working with their own branches. I'd like to hear
>> from other people on the restructuring idea, as I know I'm not the only one
>> who noticed the issues.
>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> SharpOS-Developers mailing list
>> SharpOS-Developers@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>>
>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win great
>> prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the
>> world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> SharpOS-Developers mailing list
>> SharpOS-Developers@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>>
>>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
>
> http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________
> SharpOS-Developers mailing list
> SharpOS-Developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> SharpOS-Developers mailing list
> SharpOS-Developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sharpos-developers
>
>
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
SharpOS-Developers mailing list
SharpOS-Developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sharpos-developers
