Holy war (was: libpager multi-client support)

[Olaf Buddenhagen]

Hi,

On Mon, Oct 17, 2016 at 09:51:41PM -1000, Brent W. Baccala wrote:

> I've been thinking more about the data structures involved with the new
> libpager code, and they're complex enough that I'd like to write them in
> C++11.

Yay, flamebait! ;-)

Two years ago, I would have said something along these lines: C++ does
certainly have some strong point; but at the same time it's a very ugly
language, which is not universally considered an improvement. Even *if*
we agreed it was slightly better all in all, it's probably not worth the
costs.

But that was two years ago. Today my reply is: why even consider C++,
when there is Rust?

Rust offers the same level of efficiency; about the same level of
expressiveness (I'd say higher, but I might be biased); much, *much*
better safety; and while the initial learning curve is a bit steep, it
doesn't have any of the hidden complexity and countless pitfalls.

Rust's slogan is, "fast, reliable, productive: pick three" -- and it
delivers. I honestly believe it is a game-changer; and in my opinion the
Hurd would do good to start using Rust sooner rather than later. It
would gain a major edge: not only from the inherent productivity boost;
but also from increased attractiveness to developers.

While the Hurd wouldn't be the first operating system employing Rust, it
would be (to the best of my knowledge) the first general-purpose system
to employ Rust as a tool to meet its own unique goals, rather than just
"it's written in Rust" as the *only* selling point.

There is just one little problem right now: Rust hasn't been ported to
the Hurd yet. Whoopsie... ;-)

-antrik-

Re: libpager multi-client support

[Richard Braun]

On Tue, Oct 18, 2016 at 01:56:48PM -1000, Brent W. Baccala wrote:
> Can it be done in C?  Sure.  But following my principle of asking how to
> write the code in the simplest manner, this one is obvious: STL containers
> win.

I'm not saying it can't be done. I'm saying it's not that hard.

-- 
Richard Braun

Re: libpager multi-client support

[Brent W. Baccala]

On Tue, Oct 18, 2016 at 2:37 AM, Richard Braun  wrote:

>
> From my fifteen years or so experience working in various projects with
> C++ and other languages, my opinion is that C++ is never the best
> choice. It makes you trade lines of code for increased complexity in
> understanding and following the code. You should either go with C for
> excellent control, or another higher level langguage for e.g. user
> interfaces and tasks that don't require the highest performance.
>

My opinion is that C++ features have to be used very, very judiciously to
avoid the problems you describe.  It's very easy to fall into a "C++
mindset" and try to write everything in "C++ style".  My solution has been
to look at each problem and ask myself how to write the code in the
simplest manner.


> I really don't think the problem you describe would be so hard to solve
> in C.
>

Well, let's see.  To handle arbitrary numbers of clients, we have to
dynamically allocate all those queues and lists.  We have to copy them when
adding new clients or removing old ones (more dynamic allocation).  To
constrain memory utilization, we have detect when they're no longer used
and deallocate them.

Can it be done in C?  Sure.  But following my principle of asking how to
write the code in the simplest manner, this one is obvious: STL containers
win.

agape
brent

Re: libpager multi-client support

[Richard Braun]

On Mon, Oct 17, 2016 at 09:51:41PM -1000, Brent W. Baccala wrote:
> What do you guys think?

>From my fifteen years or so experience working in various projects with
C++ and other languages, my opinion is that C++ is never the best
choice. It makes you trade lines of code for increased complexity in
understanding and following the code. You should either go with C for
excellent control, or another higher level langguage for e.g. user
interfaces and tasks that don't require the highest performance.

I really don't think the problem you describe would be so hard to solve
in C.

-- 
Richard Braun

Re: libpager multi-client support

[Brent W. Baccala]

Aloha -

I've been thinking more about the data structures involved with the new
libpager code, and they're complex enough that I'd like to write them in
C++11.  The pagemap has to contain a list of all clients with copies of a
page, and a queue of clients waiting to access the page.  To keep the
memory footprint under control, the pagemap itself should be an array of
pointers to structures that are reused if multiple pages have the same
"clientèle", so they also need a usage counter, plus we need to make copies
of them (say, when a new client is added), which requires making copies of
their constituent lists and queues.  STL containers and shared pointers
seem like a good choice.

C linkage allows backward compatible use of the library.

The biggest drawback that I see is that memory usage might get out of
hand.  A std::shared_ptr, for example, is four times bigger than a regular
pointer.  After reading http://gamedev.stackexchange.com/questions/268 I
think it might not be so bad, but I'm not thrilled about it, either.  We're
basically paying for code simplicity with memory.

Also, programs like ext2fs would be linked against the standard C++
library, but I don't know if that's really such a problem.

Obviously, introducing C++ into libpager would probably open the door to
large scale use of C++ in Hurd.  This might not be such a bad thing.  I
suspect that serialization of RPCs could be done with templates,
eliminating a lot of the need for MIG.

What do you guys think?

agape
brent

Re: workflow with Debian patches and Git repositories (was: libpager multi-client support)

[Brent W. Baccala]

On Sat, Oct 15, 2016 at 11:17 PM, Kalle Olavi Niemitalo  wrote:

>
> I also have a "hurd-debian" working tree.  I extracted that from
> the Debian source package, so that I got a .pc directory with the
> correct state information.  I then added a .git directory cloned
> from "git://anonscm.debian.org/pkg-hurd/hurd.git", so that I can
> view the history and make local commits.
>
>
This is the step that most raises my hackles.  You tack a .git directory
onto the unpacked Debian source package?

It looks to me like that Debian git tree contains an unpacked snapshot of
the savannah git tree.  Various commits there are labeled "new upstream
snapshot"; I suppose that's how changes to savannah get imported?  And
those other .tar.gz files that make up the Debian source package are
unpacked into it as well?  Are they snapshots taken from the incubator git
tree?

Is the Debian git tree composed exclusively of pieces pulled from various
git trees on savannah?

How does the Debian source package actually get built?  Is there a script?

Could we reorganize the Debian git tree to import the savannah git trees as
submodules?  Or has this approach been deliberately rejected?

And then, of course, the Debian patches are checked into git as files, not
(git) patches.  I was just reading about "git-buildpackage", which manages
Debian patches by converting them back and forth to git patches on a
dedicated branch.  You keep this branch local and rebase it to apply the
patches to a new location.  Sounds a little crazy, but interesting.

Just trying to get my mind around all this.

agape
brent

workflow with Debian patches and Git repositories (was: libpager multi-client support)

[Kalle Olavi Niemitalo]

"Brent W. Baccala"  writes:

> My first question is for advice on managing my workflow.  I'm using the
> Debian hurd package, which adds patches on top of a snapshot taken from
> savannah's git tree.  I think I want to work on the Debian-ized code, since
> that's what's installed on my system, but that leaves me without git.  It's
> just a snapshot, not an actual git repository.  Any advice?

Here is what I did.  It's cumbersome but at least it works.

I have a "hurd-upstream" working tree with a repository that I
cloned from the "git://git.sv.gnu.org/hurd/hurd.git".  I applied
the API-relevant Debian patches as separate commits, in the same
order as in debian/patches/series:

debian/patches/exec_filename_exec.patch
debian/patches/exec_filename_fs.patch
debian/patches/exec_filename_use.patch
debian/patches/exec_filename_fix.patch

I set an "in-debian" branch at that point, and then made my own
commits on top of them in the "master" branch.  I can then build
Debian-compatible executables from these branches.

I also have a "hurd-debian" working tree.  I extracted that from
the Debian source package, so that I got a .pc directory with the
correct state information.  I then added a .git directory cloned
from "git://anonscm.debian.org/pkg-hurd/hurd.git", so that I can
view the history and make local commits.

When I want to make a private Debian package with my Hurd
changes, I run "git format-patch -o
DIRECTORY/hurd-debian/debian/patches/kon in-debian..master" in my
"hurd-upstream" repository, add the names of the patch files to
debian/patches/series, describe the changes in debian/changelog,
and build it with dpkg-buildpackage.  This gives me proper Debian
source and binary packages.  In debian/changelog, I use "local"
as the distribution and append ".kon.1" or another number to the
version.  I keep my own patches in the debian/patches/kon
directory so that I can easily delete them before generating new
ones from my "hurd-upstream" repository.

The commit messages in my "hurd-upstream" repository have DEP-3
(http://dep.debian.net/deps/dep3/) "Bug", "Bug-Debian", or
"Forwarded" fields at the bottom if applicable.  The "Subject"
and "From" fields are generated by git format-patch.

Those working trees and repositories are inside the virtual
machine.  I also have clones of those repositories in the host in
case the file system in the VM gets corrupted, and off-site
backups of both the qcow2 disk images and the host-side
repositories.  (I did not have backups of my Hurd activity
between May and July, and lost my unpublished patches when the
SSD failed.)  I omit the build trees from backups though.

Re: libpager multi-client support

[Kalle Olavi Niemitalo]

"Brent W. Baccala"  writes:

> Second, I can't find anywhere in the hurd source tree where this function
> is actually used.

Commit 84cf9c0f312637b670cc87224ff7e7c4da659e36 on 2013-09-17
removed the ufs directory, in which the offer_data function used
to call pager_offer_page.  The last argument of pager_offer_page
always pointed to a page that was part of the global zeroblock,
which was the size of a filesystem block and filled with zeroes.

> Third, why would we want this feature?  Why would a memory server ever want
> to send an unsolicited page to a client?

One of the offer_data calls was preceded by a comment:

  /* Make sure that any pages of this block which just became allocated
 don't get paged in from disk. */

I don't know how ext2fs makes sure of that, or whether
pager_offer_page might again be needed for the same purpose in
the future.

libpager multi-client support

[Brent W. Baccala]

aloha -

I've started looking at adding multi-client support to libpager.

My first question is for advice on managing my workflow.  I'm using the
Debian hurd package, which adds patches on top of a snapshot taken from
savannah's git tree.  I think I want to work on the Debian-ized code, since
that's what's installed on my system, but that leaves me without git.  It's
just a snapshot, not an actual git repository.  Any advice?

Then, should I wait until I've got it working and send the changes to this
list as a comprehensive set of patches?  That seems like what others have
done in the past.

I've reviewed the existing code, and I have a problem with the function
pager_offer_page().

First, the API is problematic, since pager_offer_page() is a call from the
memory server (i.e, ext2fs) into the libpager library, instructing the
library to offer a page to the client (i.e, the kernel) that hasn't been
solicited by the client.  The problem is that the function parameters don't
indicate which client to offer the page to.

Second, I can't find anywhere in the hurd source tree where this function
is actually used.

Third, why would we want this feature?  Why would a memory server ever want
to send an unsolicited page to a client?

So, I propose deprecating pager_offer_pager() by simply removing it from
the library.

Any objections?

agape
brent