Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
On 2016-10-18 01.51, Colin Finck wrote: > Am 14.10.2016 um 05:55 schrieb Neo Love: >>> * We buy additional HDDs every year and continue to host them ourselves >>> just next to the newer ISOs. >> Sounds right to me.. HDDs are not too expensive. >> I'm willing to pitch in, despite my decrepit pension. >> Speaking of which, maybe we could all pitch in to build a new buildserver? > Having additional servers is not a problem of hardware costs, but of > hosting and logistics. Oh.. It sounded like there was a hardware issue, since you didn't have enough RAM and the build server was too slow. And yes, of course running costs will always dominate over hardware investment (which can be cheap using Asian sources). > Our critical services are all hosted on dedicated servers at a decent > hosting provider. These servers handle standard tasks very well. > ... > Therefore, we also bought some specialized servers. They have always > been hosted in countries where electricity and fast internet connections > are cheap and where we have a project member to take care of them. Yes, web/FTP/mail services can be had quite cheaply these days. It makes sense to move the bandwidth load for e.g. web off our own machines. I just thought that all build server(s) and test server(s) were our own machines due to the special programming requirements they have. South Korea, Ukraine, and Russia have well developed internet infrastructure at low/competitive cost (per my research at least). Hmm.. Guess that's why you pointed sideways at Aleksey :) > But as you can imagine, this can never match the availability of a real > hosting provider. Availability, yes.. But requirement-wise I doubt a provider like AWS or Akamai is a feasible way to go (economically). They would most likely charge an arm and a leg to get all our special programming on their (virtual) machines, even if they would be able to manage it (in an agnostic manner). > In the next few days, I'm expecting some more servers to be set up at > our project member's location. If that turns out well, we can think > about getting additional storage for them. Rented machines? Or is someone lending us the use of hardware (and/or bandwidth) out of the goodness of their hearts? :) > I'm still open for alternative/failover hosting solutions though as this > shouldn't depend on a single location and person. Thailand, where I live, have a consumer peak speed of 58 Mb/s (avg ~10), and electricity is fairly cheap; 6-13 US cent/kWh. (A home 10M/640k ADSL connect like mine is ~16 US $/month.) If you'd like, I can look into cost and availability for a fixed line or fiber. The idea with a self-built CD changer archive was something I just came up with since I like to tinker myself, and thought it would be fun to build.. Never mind ;) Best Regards L. > > > - Colin > > ___ > Ros-dev mailing list > Ros-dev@reactos.org > http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Am 14.10.2016 um 05:55 schrieb Neo Love: >> * We buy additional HDDs every year and continue to host them ourselves >> just next to the newer ISOs. > > Sounds right to me.. HDDs are not too expensive. > I'm willing to pitch in, despite my decrepit pension. > > Speaking of which, maybe we could all pitch in to build a new buildserver? > I built my latest workstation for just over 500 euro. > (Xeon E3 3.1GHz, 16 GB RAM, 2x1 TB HDD) Having additional servers is not a problem of hardware costs, but of hosting and logistics. Our critical services are all hosted on dedicated servers at a decent hosting provider. These servers handle standard tasks very well. But as soon as we have special requirements (e.g. lots of RAM, HDD space or multiple physical servers for VM testing), it is impossible or too expensive. Therefore, we also bought some specialized servers. They have always been hosted in countries where electricity and fast internet connections are cheap and where we have a project member to take care of them. But as you can imagine, this can never match the availability of a real hosting provider. In the next few days, I'm expecting some more servers to be set up at our project member's location. If that turns out well, we can think about getting additional storage for them. I'm still open for alternative/failover hosting solutions though as this shouldn't depend on a single location and person. - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
On 2016-10-05 03.10, Colin Finck wrote: > Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory > modules, our Buildserver has enough RAM for the Win7 buildslave VM now. (y) > Unfortunately, our HDDs were also lacking space, so I had to move the > public "bootcd_old" folder from iso.reactos.org to another (private) > place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. > Any idea how to deal with them in the long run? I can basically think of > 4 possibilities: > > * We remove ISOs older than 4 years, because nobody would really do > regression-testing with them. A reasonable option, imo. > * We buy additional HDDs every year and continue to host them ourselves > just next to the newer ISOs. Sounds right to me.. HDDs are not too expensive. I'm willing to pitch in, despite my decrepit pension. Speaking of which, maybe we could all pitch in to build a new buildserver? I built my latest workstation for just over 500 euro. (Xeon E3 3.1GHz, 16 GB RAM, 2x1 TB HDD) > * We find a free file hosting service for OSS projects that can cope > with these amounts of data. Not sure SF.net is the right choice here.. > > * We choose a paid data storage service like Amazon AWS. Oh god, no.. Amazon is a nightmare security-wise. It's practically impossible to define reasonable firewall rules for Amazon clients, since AWS shifts their IP range around all the time. > As always, comments are very welcome! And if fun is permissible.. Someone who likes to tinker could have fun building a CD changer archive :) Some Al rails, stepper motors, drive belts, servos, a normal BR burner drive, and an Arduino or two to control the mechanics and take commands from the PC. E.g. pater noster stores don't require too much space. While it would be slow to fetch old stuff, it would practically never run out of storage space. Best Regards L. > > > - Colin > > ___ > Ros-dev mailing list > Ros-dev@reactos.org > http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Am 05.10.2016 um 19:48 schrieb Christoph von Wittich: > What about deduplication? We would need to test if this yields any good results on our already compressed data. Even if we extract the currently 7zip-compressed ISOs, the "reactos.cab" inside the ISOs would still be a compressed file. What deduplication solution do you suggest anyway? ZFS? - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
What about deduplication? Am 04.10.2016 um 22:10 schrieb Colin Finck: Hi all, Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory modules, our Buildserver has enough RAM for the Win7 buildslave VM now. I've fired it up and while builds are much slower than before, they are at least being done again! Note that Doxygen, ISO hosting, and all three buildslaves are on a single HDD-backed server right now, so don't expect any performance miracles. I'm counting on Aleksey here to get us our remaining servers back this month :) Unfortunately, our HDDs were also lacking space, so I had to move the public "bootcd_old" folder from iso.reactos.org to another (private) place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. Any idea how to deal with them in the long run? I can basically think of 4 possibilities: * We remove ISOs older than 4 years, because nobody would really do regression-testing with them. * We buy additional HDDs every year and continue to host them ourselves just next to the newer ISOs. * We find a free file hosting service for OSS projects that can cope with these amounts of data. Not sure SF.net is the right choice here.. * We choose a paid data storage service like Amazon AWS. As always, comments are very welcome! - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC and GCC Windows Buildslaves are back - old ISOs are not
Hi Colin, On our side at triplecheck we are increasing the infrastructure, exactly with the intention of filling it up with software. Would happily make available 1Tb of long term storage or more as needed to save the old ISOs at no cost. The only nuisance is that we can't give direct access to the storage for security reasons (it is cut off from the Internet). Still, it is possible to download through a request mechanism that puts the needed files on a public server for download. We are negotiating the infrastructure growth, around the 1st of November should be possible to confirm that we can host these files. Nuno On 04/10/16 21:10, Colin Finck wrote: Hi all, Thanks to Dmitry Chapyshev's generous donation of rare FB-DIMM memory modules, our Buildserver has enough RAM for the Win7 buildslave VM now. I've fired it up and while builds are much slower than before, they are at least being done again! Note that Doxygen, ISO hosting, and all three buildslaves are on a single HDD-backed server right now, so don't expect any performance miracles. I'm counting on Aleksey here to get us our remaining servers back this month :) Unfortunately, our HDDs were also lacking space, so I had to move the public "bootcd_old" folder from iso.reactos.org to another (private) place. It contained BootCD ISOs from 2009 to 2012, totalling 421GB. Any idea how to deal with them in the long run? I can basically think of 4 possibilities: * We remove ISOs older than 4 years, because nobody would really do regression-testing with them. * We buy additional HDDs every year and continue to host them ourselves just next to the newer ISOs. * We find a free file hosting service for OSS projects that can cope with these amounts of data. Not sure SF.net is the right choice here.. * We choose a paid data storage service like Amazon AWS. As always, comments are very welcome! - Colin ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev -- Managing Director, TripleCheck GmbH twitter @nn81 | UK +44 7453 648423 | DE +49 162 327 2929 http://triplecheck.net ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] msvc build update
Hello, Would you care to comment on unefficient regarding the trap rewrite, please? The code is more efficient than previous assembly code, and more finely tuned than Windows' own code, which wasn't updated since 1989 when it was first designed (and is full of non-pairing, hazardous operations that were efficient only on a 486). I also resent the hackish statement. Perhaps GCC Extended Assembly is hard to understand? I would be happy to walk you through it. Finally, in any team project, it is efficient to subscribe to SVN notifications and keep track of the project status. This way it would've been clear that a trap rewrite was in progress, and we could have worked together. -r Hi, I'm being asked about the msvc build and decided to answer in the list to keep you all informed on the status. Today I submitted a non working update to my branch: svn://svn.reactos.org/reactos/branches/jcatena-branch useful for reference for those interested in msc builds. Before I got a svn branch where to commit, I got ntoskrnl built by msvc9 and booting to desktop, Based on build 44678. This is not synchable with svn, but I have put it in my server if someone wants to take a look: ftp://diwaves.com/reactos/ntos_44678_local.zip After I got a svn branch, I started to make the same happen without changing so much. Unfortunately this was in the middle of the trap handling rewrite. This is being done in a really weird, unportable, hackish and unefficient way, and I had a lot of problems to have the port booting. Finally, after much more work than expected, I got the port working but with some weird temporary hacking I didn't like, and then wanted to sync with more current version, only to find that the trap handlers were substantially changed again, in an even more unportable way. So instead of keep debugging and hacking a code that i would pretty much drop at the end, I started rewriting it yesterday and I expect to have it working in less than a week. This will save efforts at the end. Initially I wanted to commit only a working branch, but since it's taking time and other devs may benefit from what's done even if it is not complete, i decided to have the working not synchable version based on 44678 available at my server, and the non booting svn branch r45369 commited. I recommend taking a look at platf.h and cpu-c.h, these files contain platform and compiler dependent definitions and macros, that make most inline assembly and compiler conditionals elsewhere unnecessary. Hope you find this useful. Jose Catena DIGIWAVES S.L. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] msvc build update
Hi, Don't feel offended, the port to c is a very good thing and indeed better in the most part than the previous asm implementation. I value your work very much. But there are some aspects badly implemented: 1) It is not possible to write a good stub in an inline function nor in a regular macro due to c preprocessor limitations. Your implementation trust the optimizer to remove constant conditionals, but it does not work as well as you may think. Have you seen the generated code? It has a lot of unnecessary branching. I have implemented it as an x-macro, where I can use the preprocessor to select the options to be generated and it does not generate any run time conditional branching. 2) You have added many inline assembly parts in several files that should be kept as portable as possible. These things are better put together in architecture dependent files/directories. Furthermore, most are unnecessary, my single trap stub in a single x-macro replaces most of your inlines. 3) Since the pointer to the trap frame is passed to handlers, why do you pass as extra parameters information that is already there? Access to members of the KTRAP_FRAME is just as fast as access to local variables or parameters, but saves on additional copies and allow more efficient register usage. Furthermore using regparam(3) for functions called from inline assembly, when actually only one param was needed, was really a very bad idea. And you pass the same parameters to child handlers in cascade, again and again. This is very unefficient. Today I simplified the syscall and fastsyscall handlers removing a lot unnecessary parameter copying and cascading, saving a lot of cycles and memory, while making it much easier to read and understand. 4) Why did you resort to code patching to encode the PKINTERRUPT parameter, instead of generating a static variable named together with the stub, for example? Wasn't it hackish? I hope you understand that you won't save even a cycle that way, so why to use such an unportable and complex method? The worst thing is the indiscriminate and spread use of inline assembly and compiler specific features. Please think in placing such things in platform specific includes and try to make them as general and reusable as possible, instead of making the code base very difficult to port and maintain. Please take this as constructive criticism and not as any personal offense. We all can always learn better if we can accept criticism. Jose Catena DIGIWAVES S.L. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
-Original Message-
From: ros-dev-boun...@reactos.org [mailto:ros-dev-boun...@reactos.org] On
Behalf Of KJK::Hyperion
Post them on bugzilla, assign them to me and Cc sginsb...@reactos.org
Well, I submitted my first bug patch to bugzilla.
Before I submit more, I'd like to know if I did it correctly. I have many
and I'll wait to know if should submit them this way.
Couldn't cc to sginsb...@reactos.org, bugzilla doesn't know him.
Based on the large amount and severity of bugs in the msvc build perhaps
nobody is using it but me, plz let me know if you don't want any fixes to
msvc build submitted.
The submitted bug patch is as follows:
Bug# 5071 intrin_i.h: sgdt lgdt fixed for msvc
In intrin_i.h there are two inline functions that are defined differently
based on __GNUC__ or _MSV_VER. The _MSC_VER ones make ntoskrnl crash early
on boot.
As these functions were written, sgd lgdt stored and loaded the gdt
to/from the pointer variable passed instead of the location the pointer
points to.
Fixed and tested.
Patch follows:
intrin_i.h
Ke386GetGlobalDescriptorTable(OUT PVOID Descriptor)
{
- __asm sgdt [Descriptor]
+ __asm {
+ mov ebx, Descriptor
+ sgdt [ebx];
+ }
}
Ke386SetGlobalDescriptorTable(IN PVOID Descriptor)
{
- __asm lgdt [Descriptor]
+ __asm {
+ mov ebx, Descriptor;
+ lgdt [ebx];
+ }
}
Jose Catena
DIGIWAVES S.L.
___
Ros-dev mailing list
Ros-dev@reactos.org
http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
I recommend changing the convention such that Descriptor is a pointer to the
pointer -- this way the functions can remain one-liners and not introduce
register side-effects (especially since you're choosing ebx -- trashing a
nonvolatile).
On 2009-12-30, at 11:46 AM, Jose Catena wrote:
-Original Message-
From: ros-dev-boun...@reactos.org [mailto:ros-dev-boun...@reactos.org] On
Behalf Of KJK::Hyperion
Post them on bugzilla, assign them to me and Cc sginsb...@reactos.org
Well, I submitted my first bug patch to bugzilla.
Before I submit more, I'd like to know if I did it correctly. I have many
and I'll wait to know if should submit them this way.
Couldn't cc to sginsb...@reactos.org, bugzilla doesn't know him.
Based on the large amount and severity of bugs in the msvc build perhaps
nobody is using it but me, plz let me know if you don't want any fixes to
msvc build submitted.
The submitted bug patch is as follows:
Bug# 5071 intrin_i.h: sgdt lgdt fixed for msvc
In intrin_i.h there are two inline functions that are defined differently
based on __GNUC__ or _MSV_VER. The _MSC_VER ones make ntoskrnl crash early
on boot.
As these functions were written, sgd lgdt stored and loaded the gdt
to/from the pointer variable passed instead of the location the pointer
points to.
Fixed and tested.
Patch follows:
intrin_i.h
Ke386GetGlobalDescriptorTable(OUT PVOID Descriptor)
{
- __asm sgdt [Descriptor]
+ __asm {
+ mov ebx, Descriptor
+ sgdt [ebx];
+ }
}
Ke386SetGlobalDescriptorTable(IN PVOID Descriptor)
{
- __asm lgdt [Descriptor]
+ __asm {
+ mov ebx, Descriptor;
+ lgdt [ebx];
+ }
}
Jose Catena
DIGIWAVES S.L.
___
Ros-dev mailing list
Ros-dev@reactos.org
http://www.reactos.org/mailman/listinfo/ros-dev
Best regards,
Alex Ionescu
___
Ros-dev mailing list
Ros-dev@reactos.org
http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
As intended in my last post, better work and proof than discussion. Do you agree? You made too many assumptions, forget for a while that you already know everything if you want to learn something. What an audio workstation needs is that the audio events don't delay beyond a predictable maximum. The processing of such events will be always a fraction of the delay. So if the maximum latency is for example 1ms, lower priority rt tasks won't be delayed more than a fraction of a millisecond for this cause. This does not necessarily affect throughput of anything, and anyway never without a good reason. An audio workstation will configure the audio driver to have higher priority DPCs than disk, and disk higher than network, USB, video... A network server will give priority to NIC then to disk then whatever. Is the current scheme better, when all NIC, audio driver, disk will suffer because the mouse DPCs? Proper RT scheduling without anything violating it (like DPCs) can only help in every scenario, how couldn't it? How could the current scheme be more efficient in latency, throughput or anything than this one? Are you joking or what? The rt priorities don't have much to do with throughput, the tasks demanding lowest latency are the ones that will also do its job very quickly. If a rt task does not finish in a very limited time, it does not comply and its priority will be changed to time sharing range, this is one of the keys in my scheme. The useful statistics are very easy to gather efficiently. The threaded DPCs are not a solution when it depends on all drivers employing it (which is not a reality and will not be), and does not address the need of configuration of the priorities scheme by the user. Nor Win7 solves the risks of user configurable real time priorities. MMCSS is not that thing, it is just a try to provide a reserved high priority class to user mode, but it is anyway below *any* and *all* DPCs. What happened with WaveRT in Vista is a demonstration of the absolute lack of knowledge of what realtime scheduling is and the effects of latency, and being separate of the scheduler itself, is a proof of the lack of understanding in the very same regard. And very related to the scheduler too. If it was so hard to learn for them after endless discussions, I wouldn't expect you would a more complex timings relationship, much less knowing your I know it all attitude. You make me laugh, so if my scheduler performs better than the one in Win7 (not only better than the ones in ReactOS) you will support its inclusion? LOL. My aim is well beyond Win7 performance anyway. I'll work on it, and if I success as I expect, you will support its inclusion or not as you wish. But please don't ask me to discuss in that tone and attitude, we all know you're the smartest guy with the biggest dick and I won't waste time challenging you, ok? I don't know Arun Kishan, who is he? MS is so huge... Most ppl I treated is in the wdm area. Also as a team of audio developers leaded by Ron Kuper (Cakewalk) we have been working with them at all levels. After a decade or so the improvements are valuable, we got the KS interfaces published, the nasty priority inversion algorithm removed, an so much more. And still, so far from the right thing. If I did alone a scheduler that solved elegantly and efficiently the same problems (and mine is not as large as yours by far), how a company with the resources of MS doesn't? It is not that they don't have great people, it is not that they don't get well founded and documented requests, it is not a black hand... is the very nature of marketing, every user will find some icons nicer than others, but hardly understands what low latency means. If not, look how you saw at it: if MS didn't how you...? If a developer skilled in windows internals doesn't know about real time concepts... Shields down. It is fun but I don't want to waste time in useless discussions. I'll keep working as I said, and you all will know how it goes. The only big obstacle in the way is the very limited time I have for it, not any of arguments. Not that changing the DPC handling and sync will be easy, but the model I'm trying to implement works great, I have already used it in very different projects. This is not a fool idea of the latest frickie. Back to msvc build fixing... Cheers, Jose Catena DIGIWAVES S.L. -Original Message- From: ros-dev-boun...@reactos.org [mailto:ros-dev-boun...@reactos.org] On Behalf Of Alex Ionescu Sent: Tuesday, 22 December, 2009 06:36 To: ReactOS Development List Subject: Re: [ros-dev] MSVC Sounds like you want make Threaded DPCs (which exist to fill the niche you talked about) the default model. Are you aware of Threaded DPCs? Why not just go down that route? Your idea of targeting DPC-heavy drivers to lower their throughput is what MMCSS attempted (and still does) to do back in Vista -- it partly lead to a catastrophe as suddenly network card traffic on heavy audio I/O machines dropped
Re: [ros-dev] MSVC
Jose Catena wrote: If it was so hard to learn for them after endless discussions, I wouldn't expect you would a more complex timings relationship, much less knowing your I know it all attitude. You make me laugh, so if my scheduler performs better than the one in Win7 (not only better than the ones in ReactOS) you will support its inclusion? LOL. My aim is well beyond Win7 performance anyway. I'll work on it, and if I success as I expect, you will support its inclusion or not as you wish. But please don't ask me to discuss in that tone and attitude, we all know you're the smartest guy with the biggest dick and I won't waste time challenging you, ok? Can I ask which part of Alex's email you found offensive? I'm confused as to why you felt the need be rude and offensive. Please don't turn our mailing list into a place to pick arguments, this is what IRC is for. Ged. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
Jose Catena wrote: I could post a more concise description of the plan, but what I intend ultimately to achieve is a fully working and tested ntoskrnl that could run on regular xp or s2003 too (to verify full compatibility). As a general rule, we prefer fixes to features, and even then we prefer features found in Windows to entirely new features. As an example of why we try to avoid new features: running DPCs in pre-emptible real-time threads raises serious issues, like the risk of deadlocks between high priority real-time applications and DPCs that used to run at a higher priority but don't anymore. You'll need a framework for compatibility settings to make it work with existing drivers: at that point you'll have solved one problem and created two new ones If you want to make a lasting contribution to the project, work on something that's aligned to the goals of the project So maybe instead of writing the details and discussing it, we may wait till I have it working and speak again afterwards. This way I won't be wasting the time of anyone else until the objectives are achieved and can be verified. One doesn't usually start a discussion about something and at the same time dismiss the need to discuss it. If you came here to brag, you came to the wrong place ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
One doesn't usually start a discussion about something and at the same time dismiss the need to discuss it. If you came here to brag, you came to the wrong place May I kindly suggest that he came here to ask if you are interested in what he wants to do ? Just to know if he has the support of the team. That's what it looks like from a non-programmer's point of view. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
Ø May I kindly suggest that he came here to ask if you are interested in what he wants to do ? Just to know if he has the support of the team. That's what it looks like from a non-programmer's point of view. 1) I was simply asking if I should submit fixes and how. 2) I was telling what Im doing for your information. I didnt ask for integration of my project in ReactOS, Ill only ask for it if I finish it successfully. 3) No, Im not asking for support in that sense, I intend to write the new scheduler alone. I dont ignore that it will take a lot of work, but I couldnt expect any involvement by the ReactOS team. 4) I didnt ask for discussion or help. Im well aware of the difficulties and solutions, well beyond the well intentioned objections raised. I hope this clears any confusion and ends any discussion on the subject. Jose Catena DIGIWAVES S.L. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
Jose Catena wrote: well beyond the well intentioned objections raised. Sorry, the position of official project asshole is already taken. Try maybe applying for respectful newbie instead ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
Jose Catena wrote: Also, I don't know if I should submit patches for msvc build, I mantain the makefile-based builder, which is the core build backend AND supports Microsoft compilers (sort of). Ged Murphy maintains the new Visual Studio project generator (which is based on the makefile-based builder), while the old project generator is a total orphan I suppose I should even if it is not the official way, because I find msvc/gnuc conditionals there, but the msvc part is broken in many places. yes, it's supposed to work. Yes, we accept fixes. No, you can't link with the Microsoft linker when using the makefile-based builder (yet), because I didn't write the build rules for it (yet). Yes, you will be able to, eventually, and then use the makefile-based Visual Studio projects I fixed ntoskrnl and its dependencies issues (many in crt) for msvc. Please let me know if I should submit such patches or not, and what would be the proper way to submit these and any other patches. I suppose that initially I should send them to someone for review, right? Post them on bugzilla, assign them to me and Cc sginsb...@reactos.org I'm very pleased with the improvements in kd and windbg support. I was working on it and found that I wasted some efforts. No problem, I'll check svn more frequently for changes, but I'd like to know if there is a list somewhere with a description of who is currently doing or planning to do what, to avoid doing the same as someone else again. sginsb...@reactos.org and tkreu...@reactos.org are your men. I'd still recommend using IRC though, as most of the developers hang out there and then incorporating to it a new high performance scheduler Alex Ionescu will wear your spleen like a hat for this. Discuss it with him first if you want the remotest possibility of your scheduler being accepted in the tree ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
If your new implementation: 1) Is better than what Windows does today (hint: it's nearly lockless in Win 7, and O(1) since 2003) in every single way (ie: not sacrificing 50% of desktop users for 10% of server users). AND 2) Maintains full compatibility with Windows applications (and I expect you to TEST this), drivers, etc in every way. I promise you I will wholeheartedly support its inclusion in ReactOS. In fact, I will do even more than just that. On 2009-12-21, at 5:53 PM, Jose Catena wrote: Post them on bugzilla, assign them to me and Cc sginsb...@reactos.org Thanks you. I'll post all together after tests completion, including verification of that I didn't break rosbe/gnuc build in any way. sginsb...@reactos.org and tkreu...@reactos.org are your men. I'd still recommend using IRC though, as most of the developers hang out there Well, perhaps I'll try IRC sometime, but based in previous experiences, I don't think it's an efficient communication channel for things like this. I hope we could manage well enough with these e-mail lists or direct e-mail with these that you kindly provided. Alex Ionescu will wear your spleen like a hat for this. Discuss it with him first if you want the remotest possibility of your scheduler being accepted in the tree Hehehe, I won't discuss much with him. I'll send to this list an explanation of what I intend to do, why, and how. The possibility of overcoming the real-time scheduling limitations of windows (mostly due to DPC handling, whose mere existence is one of the effects of an incapable scheduler), is in my eyes one of the most appealing aspects of reactos. I have been developing mostly for automation systems and pro audio, and I know well the problem and how to solve it. If a windows compatible os fixes such limitations, which is what I intend to do, I can assure you those industries will be very interested. In any case, if it is not accepted initially, perhaps at a later time, after you can see a working implementation, much simpler than current one, yet much more powerful and efficient. But if it's still not accepted, I'm still willing to do it privately and I hope in such a case you won't have any problem with me using reactos sources for that. Best regards and thanks you very much for answering my questions. Jose Catena DIGIWAVES S.L. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev Best regards, Alex Ionescu ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
I'm very glad you hear you saying that, Alex. Those are indeed requirements my implementation will have to comply with, and I think it is totally feasible. It should improve realtime class latency very meaningfully without making any other parameter worse, although I also expect to achieve improvements in other parameters too. What I intend to do is to fully comply with the real time paradigm, by using some very efficient solutions I have been using to eliminate the risks of rt priorities misuses without breaking the paradigm. For me that's the easy thing (and proven already), although some related ppl at MS had a very hard time to understand it. The real challenge is to handle DPCs as preemptable real time threads, fully respecting assigned rt priorities. This is what would fix the problem definitely, but since currently DPCs are not preemtable, there is a possibility of breaking some drivers because access serialization or sync issues. But I expect this possibility will be in real world very low, as there should not be interactions between DPCs created by different drivers out of system control, and ultimately, the new scheduler will be very flexible, allowing to configure each driver's DPCs parameters separately, so DPC preemption could be disabled for each particular driver or for all of them. Probably the default config will be fully compatible with current behavior, but I expect that enabling DPC preemption by lowering the priorities of selected driver's DPCs should work fine with the most or almost all drivers. I could post a more concise description of the plan, but what I intend ultimately to achieve is a fully working and tested ntoskrnl that could run on regular xp or s2003 too (to verify full compatibility). So maybe instead of writing the details and discussing it, we may wait till I have it working and speak again afterwards. This way I won't be wasting the time of anyone else until the objectives are achieved and can be verified. P.D: Win7 has again improved the scheduler in the right direction (and fixed a related big mistake in the Vista's WaveRT model), but the DPC thing still kills the latency and they don't want to change that. I already discussed it with some key ppl at MS: a few understood it, but even those didn't think there were good chances of convincing decision makers anytime soon. It is considered a very low priority, if not null at all. But for all pro audio manufacturers, and some other niches like automation and control, it is a top priority. Best regards, Jose Catena DIGIWAVES S.L. -Original Message- From: ros-dev-boun...@reactos.org [mailto:ros-dev-boun...@reactos.org] On Behalf Of Alex Ionescu Sent: Tuesday, 22 December, 2009 04:10 To: ReactOS Development List Subject: Re: [ros-dev] MSVC If your new implementation: 1) Is better than what Windows does today (hint: it's nearly lockless in Win 7, and O(1) since 2003) in every single way (ie: not sacrificing 50% of desktop users for 10% of server users). AND 2) Maintains full compatibility with Windows applications (and I expect you to TEST this), drivers, etc in every way. I promise you I will wholeheartedly support its inclusion in ReactOS. In fact, I will do even more than just that. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev
Re: [ros-dev] MSVC
Sounds like you want make Threaded DPCs (which exist to fill the niche you talked about) the default model. Are you aware of Threaded DPCs? Why not just go down that route? Your idea of targeting DPC-heavy drivers to lower their throughput is what MMCSS attempted (and still does) to do back in Vista -- it partly lead to a catastrophe as suddenly network card traffic on heavy audio I/O machines dropped to a mere fraction of full network bandwidth on a GigE network. What techniques have you developed to address these issues? Other DPC throttling on I/O leads to priority inversion and inheritance problems, due to the heavy inter-dependency of Windows services and drivers. How do you plan on solving these challenges? Most scalability bottlenecks in the 2003 scheduler are related to the dispatcher lock -- multiple highly threaded real time applications would encounter increased latency on many-core systems. How do you intend to solve this issue without removing the lock as well (which would cause multiple layer/version violations in ReactOS and take nearly as much time as removing it in Win7 took)? How do you intend to balance real-time thread usage with multiple user/TS scenarios? Also, I'm not sure what WaveRT/scheduler mode you are talking about. WaveRT was mostly changed to properly support Event-Mode in Win7, which is an issue entirely separate to scheduler changes. Which key people have you spoken with? Have you spoken with Arun Kishan? On 2009-12-21, at 11:40 PM, Jose Catena wrote: I'm very glad you hear you saying that, Alex. Those are indeed requirements my implementation will have to comply with, and I think it is totally feasible. It should improve realtime class latency very meaningfully without making any other parameter worse, although I also expect to achieve improvements in other parameters too. What I intend to do is to fully comply with the real time paradigm, by using some very efficient solutions I have been using to eliminate the risks of rt priorities misuses without breaking the paradigm. For me that's the easy thing (and proven already), although some related ppl at MS had a very hard time to understand it. The real challenge is to handle DPCs as preemptable real time threads, fully respecting assigned rt priorities. This is what would fix the problem definitely, but since currently DPCs are not preemtable, there is a possibility of breaking some drivers because access serialization or sync issues. But I expect this possibility will be in real world very low, as there should not be interactions between DPCs created by different drivers out of system control, and ultimately, the new scheduler will be very flexible, allowing to configure each driver's DPCs parameters separately, so DPC preemption could be disabled for each particular driver or for all of them. Probably the default config will be fully compatible with current behavior, but I expect that enabling DPC preemption by lowering the priorities of selected driver's DPCs should work fine with the most or almost all drivers. I could post a more concise description of the plan, but what I intend ultimately to achieve is a fully working and tested ntoskrnl that could run on regular xp or s2003 too (to verify full compatibility). So maybe instead of writing the details and discussing it, we may wait till I have it working and speak again afterwards. This way I won't be wasting the time of anyone else until the objectives are achieved and can be verified. P.D: Win7 has again improved the scheduler in the right direction (and fixed a related big mistake in the Vista's WaveRT model), but the DPC thing still kills the latency and they don't want to change that. I already discussed it with some key ppl at MS: a few understood it, but even those didn't think there were good chances of convincing decision makers anytime soon. It is considered a very low priority, if not null at all. But for all pro audio manufacturers, and some other niches like automation and control, it is a top priority. Best regards, Jose Catena DIGIWAVES S.L. -Original Message- From: ros-dev-boun...@reactos.org [mailto:ros-dev-boun...@reactos.org] On Behalf Of Alex Ionescu Sent: Tuesday, 22 December, 2009 04:10 To: ReactOS Development List Subject: Re: [ros-dev] MSVC If your new implementation: 1) Is better than what Windows does today (hint: it's nearly lockless in Win 7, and O(1) since 2003) in every single way (ie: not sacrificing 50% of desktop users for 10% of server users). AND 2) Maintains full compatibility with Windows applications (and I expect you to TEST this), drivers, etc in every way. I promise you I will wholeheartedly support its inclusion in ReactOS. In fact, I will do even more than just that. ___ Ros-dev mailing list Ros-dev@reactos.org http://www.reactos.org/mailman/listinfo/ros-dev Best
