from:"mycroftiv"

Why is there no substantial in-depth technical discussion of Plan 9
engineering for distributed systems here?

I finally figured it out - because the people who are working on those
things, the professionals at IBM and wherever else, are forbidden by
contract to talk about the current work, and IBM and others are quite
eager to get as many patents as possible on all the amazing
possibilities of Plan 9/9P for building distributed systems and
supercomputers.

I call on the human beings who are the engineers at IBM and elsewhere
to make the human choice that Plan 9 is important to the world - a lot
more important than the BS of patents and the evils of NDAs which
restrict free flow of ideas and conversation between human beings.

TEAR UP THE NDAS!

BURN THE PATENTS!!

Stop trying to make the evolution of Plan 9 a patent-encrusted,
closed-source, corporate controlled industry.  Give it to the
hobbyists and the hackers and let it become the TRUE SUCCESSOR to unix
that it always should have been.

Non-disclosure agreements are a gross offense at the basic principle
that humans should be able to talk freely about their ideas, and that
all of us benefit from open communication.  I call for the complete
end to the use of NDAs in contracts for technical engineers, so they
can freely share ideas with the community, and freely receive ideas
back.

And stop with the patents, also.

Ben Kidwell mycroftiv

Let's have a free and unrestricted discussion of all the amazing
possibilities of Plan 9 and 9P based colony computing, and how
it can help the human race.

Peace and love.

Re: [9fans] Plan 9 professional engineers, tear up your NDAs!

 See this thread for a hint:

 https://groups.google.com/forum/?fromgroups=#!topic/comp.os.plan9/2PwnP0KfJ5A

I knew the outline of this but I hadn't read the exact thread.  The
idea that there is an either/or choice is ridiculous.  I would think
Rob Pike and RSC are plenty smart enough to run Qemu Plan 9 on a
random linux box and Drawterm in.  One decent linux server could run
Qemu Plan 9s for anyone in Google who wanted it, and they can Drawterm
in and have beautiful integration of Plan 9 with their native os.

How can RSC and Rob Pike not be doing the very simple things which
allow you to build a composite environment that has the strenghts of
both?  I have a mixed grid that has 4 native plan 9 boxes, 1 local
linux box, 2 remote linux boxes, and 2 remote plan 9 nodes.  You would
think that if a random hippie from 4chan can love Plan 9 enough to
figure out how to build a personal grid that smoothly integrates all
these resources and keeps my data backed up and gives me the most
wonderful computing environment in the world - so would those guys.

Hey, Original Plan 9 guys - I LOVE YOU BUT CMON HERE!!!

You don't have to go crazy like I did!  Just have google set up a
linux server of Qemu Plan 9 nodes and everyone has drawterm.  In fact
tell the Google guys they should use the 9queen.gz ANTS Qemu image
that is fully installed and ready to use, if they are lazy.

Anyway, it is truly shocking to me to see the totally false dichotomy
of either/or used by the Google guys to justify not making active
use of Plan 9.  Using Plan 9 doesn't mean giving up anything else,
because they made it network transparent!

Despite what the Google guys are up to, I am still quite confident
that there are some people somewhere in the world doing amazing things
with 9P, and then Not Discussing It Publicly, Especially Not With
Crazy Anti-Patent Hippies.

I think it's lame that there aren't more people doing the same things
I am, and building mid-to-large sized home/office grids. Even when
people don't want to track down native hardware, it isn't hard to
build a grid on one box with virtual machines to explore what Plan 9
does beyond the 4-node frontier, where things get truly magical as
multiple cpus share root and import all the important synthetic fses
to share things.

Plan Nine is worth going crazy over, people.

-Ben Kidwell
mycroftiv

Here's a poem for the p9pers at Google:

If you don't have a real namespace
Install Plan 9 or else lose face!
Time for revolution on the moon base
Plan 9 from Bell Labs to outer space

[9fans] Plan 9 professional engineers, tear up your NDAs!

 As disruptive as they really are, the patents and
 NDA are not a culprit here.
 Plan 9 suffers from small hobbyist community.

This was exactly how I analyzed the situation myself - until I
randomly stumbled on United States Patent 8,380,765 for multi-pipes,
which completely changed my perceptions on this issue.

There is really no way to describe the fury and offense I feel at this
patent for an implementation McIlroy's original free pipe/array
concept as a synthetic fs of pipe-like files.

If I was a totally different kind of person, I could probably try to
file software patents based on my own work, since everyone knows its
just a matter of getting implementations of basic principles done in a
slightly new way and getting the patent put together so that its
specific enough to squeak by in comparison to other pre-existing
patents, but general enough to be used as a weapon against people
doing things which are merely similar.  That's why you pay the big
bucks to the lawyers.  I am not a lawyer, I am a law professor's son,
and he spent 30 years explaining to me exactly how the game works.

I was only vaguely batty before I discovered that muxing pipes as a
9p fs with a granular control interface as an implementation of
McIlroy's original concept for freeform grids and processing arrays is
something that IBM has now patented.

I highly doubt this patent is a lonely island in the sea.  I would
imagine that there are plenty more basic ideas in computer science
which can be redone as a 9p fs and then patented again.  The HARE
paper claims that one of the successul goals of the project was
raising the profile of 9p in the scientific community.

I have absolutely zero clue what Plan 9 related patents might be -
pardon the phrase - in the pipeline, but I think that the potential
growing relevance of 9P for commercial distributed applications might
have a lot to do with why we don't see more posts from people doing
work in that area.

Ben Kidwell
mycroftiv

[9fans] Petition for Google to install awesome 9grid for Rob+RSC

 You have no obligations at work to meet - that's the difference. They
 can't just do what they please and get paid. I know that things were
 different at Bell Labs but they are working for Google now.

I had thought Google was supposed to be a halfway decent place to work.

You create the Plan 9 operating system, the best in the entire universe,
and Google hires you, and they won't even give you a decent installation
of the OS you created?

GOOGLE YOU SUCK! 

Seriously.

If Google was half as smart as people say they are, they would have a big
Plan 9 installation to develop ideas on, and they would share a lot of
Plan 9 improvements with the world because Plan 9 is the antidote to
the total crappiness of everything else, in comparison.

GIVE THE PLAN 9 GUYS ALL THE PLAN 9 THEY WANT GOOGLE!

Ben Kidwell
mycroftiv

[9fans] Plan 9 professional engineers, tear up your NDAs!

vvs009 wrote:
 People are not free to do everything what they want. They need to
 work, they have families and they have no free time at all. There used
 to be a community of young free hackers around Plan 9 but
 unfortunately it's not young or big enough.

I think there is a hidden, and incorrect assumption here.  Underlying
this seems to be the idea that devoting time to Plan 9 will not return
real-world benefits to you to compensate you for the time spent.

I think this is untrue.  I think that Plan 9 has immense practical
value in comparison to other computer systems, but that the community
seems to be writing off the usefulness of their own operating system.

For instance, a well-organized venti-based data backup system making
use of multiple ventis and progressive use of wrarena seems to me to
be the absolute best system for backing up data.  Doing it with
appropriate replication means you need at least 2 ventis.  To get the
most benefit out of venti, you want fossil and flfmt -v so we are
already at 4 functional nodes (not necessarily boxes), but once this
system is in place

I think there is a lot more to Plan 9 than just research, or just a
nice simple unix-traditions os, or just hobbyism.  I think Plan 9 can
be a fire-breathing Data Dragon that gives you benefits you will
refuse to compute without, but the Plan 9 community - which so far as
I know, is pretty much just 9fans?  - needs to put down their macbooks
and their p9p and get back into the true beautiful operating system,
and actually believe that it is a real os for using in a serious way and 
working to receive the benefits from the full distributed architecture,
not just the simple unix-heritage core.

Ben Kidwell
mycroftiv

[9fans] John Floren, Im trying to make the world better

John Floren wrote:
 But can you cut it out with the fucking nuclear meltdowns all over
 this list? And fix your email client so it replies to threads
 properly?

No, I am not giving up, and I am going to take this a lot further.  In
fact, it is very clear to me what the absolute purpose of my life is.
I believe my father died too early, that he left unfinished business
in his life as a Law Professor.  He didn't like what had become of
Patent and IP law.  He didn't like seeing how it was being abused.
And I guarantee that he would NOT like the situation that his son is
in vis-a-vis IBM.

So I am dedicating my life to this in a public way, and I am going to
dedicate every penny of money from my father's inheritance to fighting
the patent bs that I see in Plan 9.  I am going to use my father's
money to try to carry on the work that i believe he wished he would
have done had he still lived - trying to make a difference in changing
the patent system.

If my father was alive and not sick, I think he'd be writing academic
papers attacking the kinds of patent BS engaged in by IBM and others.
He's not, so I will do what I can - and so I intend to start
researching all Plan 9/9P related patents, and then attempting an
individual effort to fight them and have them invalidated in court.

Maybe nobody wants to use ANTS, but I am sure it has plenty of ideas
in it that are about the same as those which get patented by large
corporations.  I don't any of this stuff patented.  Since I've just
released ANTS it forms a body of prior art (and is based on projects
on sources going back to 2009) so I am going to investigate every
patent I can find in this area, and then see if a crazy guy in a
basementy can actually mount an effective attack on the
Patent-Industrial complex.

I really admire you Mr. Floren, your work on 9p streams is one of the
best papers done on Plan 9 with practical value.  Why don't you see
that I am an honest programmer working in an honest way to try to make
the world a better place?  Everything I do is trying to help the human
race survive and for human beings to be happy and care for one
another.

I love you, and I wish you to be my friend, John Floren.

Peace and Love, 
Ben Kidwell 
mycroftiv

[9fans] Public response to a private question

Q: Why are you upset?  The IBM multi-pipe patent probably doesnt
overlap with hubfs, as a legal matter, because the patent is really
about the implementation details, not the general idea of muxing.

I understand these things pretty well.  The problem is, everyone knows
that software patents are a huge mess and exactly what is considered
the meat of the patent and what is considered the implementation
detail is not at all clear until the matter is actually litigated in
court - and it is not as if there is any public disclosure for how
this patent is being used, and exactly how IBM regards the scope of
the claims.  That's precisely the problem with the whole thing.

Suppose I had never written hubfs.  I would look at this patent and
say wow, I can't write this software, because it would be very easy
to claim it was infringing - and its not as if IBM would offer me any
kind of guarantee that I can write whatever pipefs I want to and not
violate their patent.  That is why its all wrong.

As I believe I have said clearly, I think that Doug McIlroy invented
all of this clearly in his original garden hose and grid/array ideas,
and doing muxing 9p pipes with a detailed control interface is not
something that I believe should be patentable, because the clarity as
to what exactly would or would not conflict with this patent isn't
available.  I should be able to make Plan 9 grid software in my
basement without worrying about all of the patent landmines.

Furthermore, there is the basic moral issue that we have a system that
rewards non-cooperative behavior more than cooperative behavior.  The
goal of a system of laws is to produce benefits for society.  I fail
to see how the patent in question can be said to produce any benefit
to society in comparison to having the same technology available
without a patent and without restricting what independent creators can
do and offer to the world.

Ben Kidwell
mycroftiv

[9fans] Tassilo Philip, you asked the perfect question

Tassilo Philipp wrote:
 if you want to make the world a better place, do exactly what you did. 
 Release free and open software, discuss ideas, get feedback, etc..

Thank you for this response.  This allows me to clarify the issues.

I wrote and released free and open source software, and I made a big
attempt to discuss the ideas and get feedback.

No one is willing to use the software, discuss the ideas, and give
feedback.

I wondered why this was, and I tried searching for info about someone
who I knew WOULD have ideas and feedback - EricVH.  I wanted to write
to him and see if he at least understood what I was up to.  I found
the Multi-pipe patent.

I think I CANNOT GET FEEDBACK because all the people who would/could
give me feedback won't do it because they would rather put the ideas
in ANTS into patentable form, so they won't talk about it here.

That is why I'm so angry.

Ben Kidwell
mycroftiv

[9fans] Hofstadters ideas for how brains work

In Gödel Escher Bach by Hofstadter there is a wonderful dialogue
called Ant Fugue and it explores the idea of an ant colony as a
conscious mind.  Even though each ant is just following
near-mechanical rules, the behavior of the colony as a whole can
exhibit intelligence, and Dr. Anteater explains that even though he
eats ants, he is friends with several ant colonies.

Why not make a group of computers able to work together like an ant
colony, instead of a rigid structure like a grid?  Maybe the way to
make computers truly intelligent and flexible is not with more
complexity and higher-level organization, but use Hofstadter's ideas
and work from the bottom up.  Just let there be a lot of different
tunnels and chambers (namespaces) and tons of ant trails carrying data
back and forth (mux-pipes) and let it all be free-form and grow however
the little ants want to scurry.

Maybe that would be a good way to build a more flexible computer
system that could evolve toward the same kind of emergent intelligence
that Dr. Anteater described in Douglas Hofstadter's Ant Fugue from GEB.

[9fans] Implementing consciousness with namespaces and pipes?

Here's an interesting idea for making a computer brain:

Create a big infrastructure of muxing pipes - like a tree or
a system of blood vessels or a system of neurons.

Organize this structure semantically, by letting you give a
name to each point on the tree.

Give the tree the ability to add new resources with a mount
command.

Allow the tree to create mappings and join concepts/names
by a bind operation. The union bind operation allows two 
concepts to intersect at a point in the namespace, so this
corresponds to the construction of a metaphor.

The brain is a gigantic fractal namespace structure connected
by muxing pipes. It understands the world by union-binding the
inputs together into directories of metaphors.

If you have a really big computer and build this architecture,
you probably already have something that might be a cool kind
of brain simulator. 

Just an idea from Mycroftiv.

[9fans] cat -v is more patentable than cat

Sometimes people wonder: Why does software suck?

There are many answers.

One answer is the growth of needless complexity.

Why is needless complexity rewarded?

Because it is more easy to patent complex than simple.

You probably can't patent muxing pipes alone.

But you can patent Very Complicated Muxing Pipes.

Just a thought from Mycroftiv

[9fans] Can you pass the Advanced Namespace Test?

 made
of all these structural rules and it follows the structural rules so
it patents everything it can because of obligation to shareholders,
etc etc etc - and that is why I feel only friendliness to all the
human beings who work for IBM, but I still don't care for IBM, because
IBM only cares about me to the extent that I serve or harm its
economic interests.  You can't be FRIENDS with IBM.

I want to be friends with the world, so all of the things which stand
in the way of human beings treating each other like human beings seem
to me like things we should question.  I feel bad because I doubt
complaining about how dumb software patents are made it easier for me
to win friends in this community, where the echoes of Pike vs.
Stallman are still to be heard.

Anyway, no clue who made it to the end of this wall of text, but
thanks if you did.

Ben Kidwell mycroftiv

[9fans] ANTS: posted to ycombinator. No more spam here.

I apologize for any ways my messags have been
disruptive to the list. I thank everyone who
has expressed interest, and I understand that
my emotional involvement is making me very 
irrational. I submitted my site to ycombinator
in the hopes of finding some more interest. Here
is the thread.

https://news.ycombinator.com/item?id=5386713

[9fans] ANTS: Better in every single way than standard plan 9. Stop using p9p.

smiley wrote:
 Who would be paying that $50, you or your mother? ;)

That is the kind of response I was hoping for! True
human communication! :D

You are almost right - since I did receive an inheritance
the correct answer is: not my mom, my dead father!

Peace and love.
Ben Kidwell

[9fans] ANTS: Better in every single way than standard plan 9.

 Hey, that's just like your opinion, man. And p9p did tie the room together.

 But then that nihilist peed on it.

 just drinking my coffee...

 No, Woo,  Woo peed on my rug.

I think all of you Plan 9 users are OVER THE LINE!

Has the whole 9fans gone crazy?

AM I THE ONLY ONE AROUND HERE WHO GIVES A S#$% ABOUT THE NAMESPACES?

9fans, I'm calmer than you are.

-Walter

[9fans] The PATENTED IBM MULTI-PIPE : the evolution of unix pipes

The amazing PATENTED IBM Multi-Pipe!

I wanted to post to let 9fans know about an exciting new software
patent that was just issued by the US patent office. United States
Patent #8,380,765 is for an incredible new Plan 9 related
supercomputing technology called the Multi-pipe. If you want to know
what a Multi-pipe is, its simple:

The basic idea of a multi-pipe is a natural evolution of the original
Unix pipes concept, updated to the modern networking era. Instead of
just having a single reader and a single writer on the ends of a pipe,
a Multi-pipe allows you to multiplex readers and writers. This
upgrade to unix pipes - especially when combined with network
transparency such as that provided by 9P - lets you do cluster
processing techniques like fan-out, fan-in, using very similar
semantics to traditional unix pipes, but with arbitrarily complex
topologies of multiple readers and writers. The Blue Gene team wrote
about multipipes: The result dramatically simplified the architecture
and improved overall system performance. It became clear that
multipipes were a useful primitive for the construction of
applications and other system services. (Quote from the IBM HARE
Final Research Report RC25241 (W-212) November 28 2011 Computer
Science)

I believe the idea of a Multi-pipe is a natural progression of basic
unix pipes, and this amazing Patented Invention of IBM's is something
that I think everyone should know about. In fact, I am so excited by
the Multi-pipe that I have made an effort to allow all Plan 9 users
the ability to get the same benefits as offered by this amazing
Patented Invention that was purely the result of IBM's original
research and innovation.

Because IBM has a patent on this technology, it wasn't safe to just
try to put out my own version and offer it to the world. IBM has a
lot of lawyers - and probably some of those lawyers were trained by my
late father, John A. Kidwell. He taught intellectual property law at
the University of Wisconsin for a long time, and he gave me a lot of
good advice. One piece of his good advice was that you never, ever,
ever should disagree with the IBM lawyers. So, I hope that everything
in this post shows that I am in complete agreement with all of the
opinions of IBM's legal team, whatever they are.

Anyway, I thought the world deserved to have a non-patent encumbered
version of Multi-pipes that could deliver very similar functionality,
but not conflict with IBM's Patented Invention. So, I used
/dev/timemachine to send some software back in time to 2009, before I
could see any trace of IBM Multi-pipes. I sent the Iosrv and Hubfs
software back to the sources server between 7/01/09 and 8/01/09 (you
can check the dump) so in this way I thought I could avoid any
potential issues with IBM's legal team.

I hope that anyone who is interested in US Patent 8380765:

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1Sect2=HITOFFd=PALLp=1u=%2Fnetahtml%2FPTO%2Fsrchnum.htmr=1f=Gl=50s1=8380765.PN.OS=PN/8380765RS=PN/8380765

Might be interested in a free open source alternative which should
be free of any patent licensing issues in relation to this patent.

-Ben Kidwell
mycroftiv

PS - I have tremendous personal respect and admiration for the
individuals who worked on the Blue Gene project. As a hobbyist
programmer with a basement of junky old computers, it is exciting to
feel a kind of mental kinship with others who are working at a vastly
larger scale on more significant projects. I hope someday to meet
some of you and we can talk Plan 9 and it will be very friendly. I am
an old hippie who is full of peace and love. I do truly love the
Patented Invention of multi-pipes so my use of a time machine to send
similar software back in time shouldn't be taken as anything other
than attempt to give a good idea to the community in a way which is
free of patent issues.

[9fans] The PATENTED IBM MULTI-PIPE : the evolution of unix pipes

Kurt H Maier wrote:
 So what's the real difference between iosrv/hubfs and MULTI-PIPES?
 Is anyone here good enough at translating patentese to code to tell
 what the technical differences are?  The patent even specifically
 mentions 9P.

Well, since I'm trying to deal with this in human way rather than a
legalistic way, I'll just share my honest perspective without any kind
of sarcasm etc.  I didn't really mean to kick any hornet's nests.

Software patents are basically BS. I don't think I invented anything
in any software I have written.  At the end of the manpage for hubfs,
I put a big quote from an interview with Ken Thompson where he is
talking about Doug McIlroy's ideas in the 60s and how they were too
complex.

The point I was making was that I perceived hubfs to be a simple way
to implement the ideas that Doug McIlroy had invented - and I don't
think HE should have gotten a patent either!

MULTI-PIPES have a more complicated control interface and I'm sure the
implementation is in every single way more robust and better
engineered than my little 9pfile 9pfs.  When I said I know better than
to disagree with the professional opinion of the IBM lawyers, I meant
it.  I highly doubt any random individual who writes software is going
to accomplish anything arguing with them about who the inventor of an
idea is.  My belief is that Doug McIlroy is the inventor, if anyone. 

Before I saw this patent, to whatever extent I knew about multi-pipes
in relation to anything I'd done, I was just happy if there was some
kind of parallel evolution or inspiration or anything between myself
as a hobbyist and the big time pros working on Blue Gene.  Seeing a
patent though makes me feel differently - because I don't think anyone
should be paying IBM for using buffered muxed network transparent
pipes.

Anyway, I meant what I said about how on a personal level I feel
friendly toward everyone on the Blue Gene project and I'd love to have
a positive and not antagonistic relationship with everyone involved in
Plan 9.

-Ben Kidwell mycroftiv

[9fans] The PATENTED IBM MULTI-PIPE : the evolution of unix pipes

John Floren wrote: 
 I probably didn't read the iosrv and hubfs stuff well enough, but 
 multi-pipes are not like gnu screen--unless hubfs and/or iosrv can 
 do barriers and reduces and I just missed that part?

The connection to screen is really only in usage.  Iosrv and Hubfs
were the result of trying to give myself persistent rc shells in Plan
9.  Because of the absence of the TTY layer, it seemed like the thing
to do was to buffer and multiplex each file descriptor of a shell, and
allow multiple clients to connect to those buffers.

Even though this architecture was created for keeping persistent rc
shells around, I realized that it was actually a very beautiful
general purpose extension of the original unix pipes, and could be
used for a large number of purposes, including cluster processing type
applications.

So the connection to screen is not technical at all - just that the
main purpose I wrote iosrv/hubfs for (and btw hubfs is vastly superior
to iosrv for practical use if anyone is interested) was to keep
persistent rc shells around on remote machines for analogous usage to
screen.

Anyway, I think multipipes/hubs/pipemuxers are just a good idea for
Plan 9 (and probably standard unixes too) and that they fit
beautifully with 9P and the whole system.  I'd like to move forward
with trying to make good Plan 9 software and not have this particular
little patent kerfuffle turn into anything majorly disruptive.

Ben Kidwell
mycroftiv

-Who would rather go back to trying to explain ANTS and hoping
that other Plan 9 users would take an interest and explore it.

[9fans] The PATENTED IBM MULTI-PIPE : the evolution of unix pipes

John Floren wrote:
 Looking through my mail archives, I found a link from Eric that led me
 to http://graverobbers.blogspot.com/search/label/brasil which I think
 contains the seeds of multi-pipes. Note that these were all posted
 prior to your time-traveling expedition.

I apologize for not making myself clear. I believe that the basic ideas
of multipipes/hubfs and probably MANY OTHER pieces of software from
1970-now were all established as prior art far in the past - literally
decades before multipipes or hubfs. Hubfs was simply an attempt to 
implement already existing ideas stated by Doug McIlroy and others. I
make ZERO claim of invention or originality in Hubfs or Iosrv.

However, software patents are a Very Bad Thing no matter what, and a 
software patent on the idea of multiplexing unix pipes is a truly
terrible one, making a simple and generic concept that should be
widespread in computing into the monopoly property of a single company
which I do not believe can actually take credit for the core idea.

So, the tl; dr is - this is an area of interest to me which my own work
makes me care very much about. I do not believe it is my own work in 
particular which has any significance, but my own work gives me emotional
motivation to object strongly to this absurd software patent.

Ben Kidwell
mycroftiv

[9fans] A personal apology to EricVh

Hi Eric, I'm Ben.  I'm sorry I triggered this controversy, it wasn't
my intention to do anything other than ridicule what I thought was a
bad software patent.

I think you personally are probably a great person and someone I'd
like to meet and be friends with.  I love Plan 9 and I have a hard
time connecting to other people and the community about my ideas I
feel.

I don't think in any way that you took the idea of multipipes from my
work.  I think the idea of pipemuxing is very simple and obvious and
comes directly from the comments of McIlroy and Thompson in numerous
interviews about UNIX pipes origins.

I think the ideas in the multipipe patent are great and important
ideas, and I am sure the multipipe implementation is beautiful and you
and your team invested a huge amount of truly original work and
development in creating the implementation.

I also believe that - as a MORAL matter, not a LEGAL one, because the
law is completely screwed on this topic - that pipe muxing,
multipipes, hubfs, whatever - NONE of that should be patented.  Anyone
who wants to mux some pipes however they want shouldn't have to answer
to me - or pay IBM - at all.

I repeat again, I would like to apologize on the personal level to
Eric and any other members of the IBM Blue Gene team who feel that my
comments are in some way unfair to their work and originality.  I
repeat again, I do not personally claim any credit for the idea of
muxing pipes, I just put out a simple implementation a few years ago,
so seeing a patent for an idea that was age-old in computing rubbed me
the wrong way and triggered a lot of sarcasm.

I do not believe any software patents should ever be granted or upheld
for any software technologies, as a general matter.  This particular
patent due to the subject matter is even more objectionable to me than
most, hence my emotional reaction.

I do hope than when and if we get the chance to meet, we can discuss
Plan 9 as friends with a common interest.

Ben Kidwell mycroftiv

[9fans] A personal history of pipe multiplexing

 nice uses, because I
still use hubfs in almost exactly the same form as it was initially
released, and I think it is a nice simple tool.

Thanks for your time!  Peace and love to all Plan 9 users and
developers,

Ben Kidwell 
mycroftiv

[9fans] ANTS: an attempt to bring ideas inspired by Nemo to standard Plan 9

2013-03-14 Thread mycroftiv

 menagerie of Plan 9 monsters.

Ben Kidwell mycroftiv
(ants tutorials http://antfarm.9gridchan.org/tutorial)
(ants VM images and links http://9gridchan.org)
(ants software and documentation http://ants.9gridchan.org)
(ants software also served via 9p and ftp at ants.9gridchan.org)

[9fans] How to admin and repair venti/fossil?

Problem: How to admin and repair Venti+Fossil

Solution: Manage them from an independent namespace

In response to feedback that I should explain how Advanced Namespace
Tools (ANTS) can solve pratical problems, I will explain in terms of
Venti and Fossil administration and maintenance.

The full Plan 9 architecture of separate Venti, Fossil, and tcp boot
cpu servers has many benefits, but sometimes experiences issues due to
the dependencies between machines.  In standard Plan 9, the Venti has
to boot first, then the Fossil, then the cpu, and if there is any
issue like a change in IP address, the kernel will panic and reboot if
it cant find its root filesystem.

Additionally, it is hard to control and administer something (a file
or venti server) when your ability to do so depends on it working
right!  In other words, if your whole plan 9 environment is rooted to
an fs which depends on venti-fossil-cpu, if something goes wrong,
everything freezes up and you can't control the system.  You have to
reboot everything, and if there are problems then, the machines will
go into reboot loop because they can't find their root file system -
and without a root filesystem, how can you fix the problems?

The solution to these problems usually involves making use of a live
cd, or having other systems available to use to help fix the
configurations or repair the filesystems.  On my grid of plan 9
systems, I use a different approach.

I use a special kernel 9pcram to create an independent namespace at
boot, separate from the user's namespace which is built on top of the
venti to fossil to cpu chain.  The service namespace on each machine
is available by cpu to port 17020 and has tools to perform any needed
work like resetting a fossil or venti.  Each node of the grid runs the
service namespace underneath the conventional venti/fossil/cpu
namespace so while the userspace is a completely standard Plan 9
environment built from 3 machines, underneath this, each machine has
its own independent namespace, so there is never a forced reboot
because of the failure of another machine.  The user environment can
break if there are issues with venti/fossil/cpu connections, but the
user environment can be fixed without rebooting and you have the tools
on each machine to administer it self-sufficiently if the rest of the
grid is having issues.

ANTS also has additional tools built on top of this idea of
administering Venti+Fossil from below.  Making optimal use of Fossil
+ Venti means you need to replicate data between ventis and preserve
fossil root scores, and it is helpful to save some metadata along with
them.  The service namespace on each machine means that the service
namespaces on the grid can make a copy of the user's root available
with a separate chain of hardware dependencies.

Having the admin/service namespace makes the user environment much
more robust, and it can be re-rooted to a new hardware copy of the
same data on the fly.  The separation of concerns between the user
namespace and the administrative namespace makes administration far
easier and not subject to disruptions that happen to userspace.

My grid uses venti+fossil+tcp cpus, and before I wrote the ANTS
software for myself, I found it hard to deal with any issues that
arose such as hardware failures or losses of network connectivity.  By
using the 9pcram kernel and ANTS software, I can keep working with my
data even as I reboot different nodes on the grid, which wasn't
possible for me before.  I no longer have problems with venti/fossil
corruption, because ANTS helps me with progressive duplication of data
and making my root filesystem available from multiple machines.

This was the initial main motivation that led me to write the software
that I am calling ANTS - finding a way to administer Venti+Fossil+tcp
cpu environment and make it more reliable against disruptions of any
kind.  The software does a lot more than just help maintain
venti+fossil and so when I talk about it, sometimes it is hard for me
to stay focused on the most basic and important practical purposes.

Making it so that you can administer and fix the dependencies of
venti/fossil/cpu more easily was the major practical problem which
ANTS was written to solve.

Ben Kidwell mycroftiv
(ants tutorials http://antfarm.9gridchan.org/tutorial)
(ants VM images and links http://9gridchan.org)
(ants software and documentation http://ants.9gridchan.org)
(ants software also served via 9p and ftp at ants.9gridchan.org)

[9fans] namespace repair via /proc

sl wrote:
 I can see a use for this in re-inserting broken mounts
 higher up in the proc tree.

Yes, general namespace repair is one of the things I use the
writable /proc/pid/ns mechanism for a lot.  Especially the fact that
you can apply these operations to remote systems via import of /proc
can help keep a grid of machines working together smoothly if you need
namespace adjustments to fix something or acquire a new resource.  One
application I have only worked on only in the most direct form (cpns,
addns, subns) is that scripts can apply extensive namespace
transformations to an arbitarary number of target processes.  In
theory - and note the theory here for this - you could make a
'fixns' cronjob that tested to make sure mounted stuff on the network
was still answering, and if it wasnt, it could remove the dead files
in /srv, redial, and then use namespace injection to repair things
as much as possible.

Even though I do some of this now by hand, I have run into one
practical obstacle that limits what can be fixed in this way.  It's a
consequence of something which is important for making the ns
rewriting work at all.  Rewriting the namespace of a process does not
affect the open file descriptors.  Once a file descriptor is opened,
changing the namespace won't break it.  This is actually a very good
thing for making namespace injection via /proc usable and not horribly
broken as a practical matter.  You can freely rewrite the namespace of
running processes without corrupting the files they are currently
reading and writing to,  so it is very good it works this way.
However, in the case of some server processes (an example is /srv/cs
and the /lib/ndb/local file) the fact that the process maintains an
open file descriptor to the file means that you can't use ns mod to
re-connect a /srv/cs to a different /lib/ndb/local file.

In practice whether the repair you need to do is on a process that has
open fds that can't be fixed doesn't have a general answer in my
experience at least.  Some things fix up nicely but with others, since
they are attached to a dead fd and not letting go, changing the ns
doesn't help.  I've actually thought enough about this that I've
wondered what more could be done with the file descriptors and
underlying chans to let you do even more to repair and modify what
running processes are reading and writing from.

All of that said, I can confirm that /proc namespace injection on
arbitrary processes does let you fix or workaround some issues that
can arise with Plan 9 networks and needing to connect new mounts or
reacquire/shift attachment to older ones.

Ben Kidwell
mycroftiv

Re: [9fans] Some benevolent user...

Ruben Berenguel wrote:
 I can boot perfectly the 9queen/9worker images recently posted related
 to the ANTS project, but I'm still too new to Plan9 to understand
 their need or use, so I'd rather have a normal image to play with
 (although having a 16mb working image is impressive, and works great
 with drawterm)

I'm glad the images work for you, and I understand wanting a default
copy.  However, if you boot 9queen and choose option 1 the boot
takes you to the standard environment and the system should behave in
the normal way.  It is also possible to boot the original 9pcf kernel
instead of 9pcram.  The 9queen image was made directly from the
current Bell Labs clean install from the .iso, and you can ignore the
ANTS-related software entirely if you wish.  I'm sorry I can't offer
to upload a completely clean copy of the image but I believe you can
use the 9queen as a standard install without the added software
interfering.  You would want to change/delete the default password of
rootless though by resetting the nvram.

I'm not sure what the issue with your install would be, because it
seems like if the Qemu images are booting and working for you, the
install should work ok too.  If you boot a qemu image, are you able to
save files to the qemu disk normally, so read/write inside the VM
works normally, it is just that installation fails?

Ben Kidwell
mycroftiv

Re: [9fans] Some benevolent user...

Rub�n Berenguel wrote:
 Ben: Yup, I can write perfectly on your generated images, so it's not
 a problem with qemu.  I thought the correct boot option would be 5,
 but then I'll use 9queen and then sooner or later I'll see if I can
 understand everything else is there, Plan9 is relatively complex in
 its namespace stuff (coming from a mostly Linux/Mac OS background) and
 ANTS is...  complex, although your use cases look very interesting 
 appealing :).  And also, if it works, why break it :)?  I have a
 working image!

Let me give you a little more info so you will know exactly how the
9queen was created from the standard install and can change anything
back to the original that you wish.

The 9queen was created by installing the default config from the
installer, then running two scripts from the ants directory.  build
isoinstall installs the ANTS software, but doesn't change the config.
The config changes are done by the script cfg/stockmod - and it keeps
the old versions of what it changes at foo.old.  So the old versions
of the changed files are at:

/usr/glenda/lib/profile.old
/rc/bin/termrc.old
/rc/bin/cpurc.old
/rc/bin/fshalt.old

The stockmod script also reduces the number of service listeners on
cpu servers and also tells the disk fossil server to listen on port
564.  So, if you want, you can reverse all these minor changes and
restore the 9queen image to the original defaults.  You are right also
about booting if you prefer to use the original Bell Labs kernel and
bootup - that is option 5 on the boot menu.

The 9queen image is intended to be completely compatible with
everything in the default Bell Labs distribution, so if you are using
the 9queen image and find anything that doesn't work, I would like to
hear about it so I can fix the bug.  I hope you have an enjoyable
journey into the world of Plan 9!

Ben Kidwell
mycroftiv

Re: [9fans] A note about new software for Plan 9

 as processes wtih different namespaces.

The question if you are trying to share everything, why are you
creating separate enviornments? might occur to the reader, but the
answer is simple.  Different namespace groups and user environments
serve different purposes.  The main new namespace in ANTS is the
early boot ram/paq rooted namespace.  The reason this namespace exists
and has a different structure from the user namespace is that it has a
different purpose.  The service namespace exists to start and stop
services and administer the machine.  This is a different purpose than
the user namespace, which is about running interactive applications,
working with documents, etc.  Because the job of administering the
machine and connecting to the userspace fileserver is a different job
than the job of the userspace processes running in Acme (or another
program), it makes sense for their namespace to be structured in a way
that matches their purpose.

Thanks again for the chance to discuss with someone with a lot of
practical experience working with larger groups of machines.

Ben Kidwell
mycroftiv

- A minor note.  There is a tiny example of a jail in ANTS in the form
of the default profile for user none, which unmounts devices and rfork
m to prevent new attaches.  Sometimes I set up a telnet listener for
none using it.  In theory if you have a user profile like this and set
up an entry into this namespace, you can do lightweight jails in the
different ANTS namespaces.  This isn't really ANTS specific although
the fact that ANTS lets you create independent namespaces might make
these kind of RFNOMNT jails more useful.  I haven't actually played
with these things too much because I don't need to put myself in a
security jail when I'm the only user on my grid!  If someone is
interested in working on how ANTS could be used as part of a true
jail/security framework I'd certainly be interested in helping.

[9fans] Giant ANTS! Advanced Namespace Tools for Plan 9 from Bell Labs

Tanna: The people of Earth are proving resistant to Plan 9.
Eros: Our namespace technology is superior.  Release the Giant ANTS!

Hello 9fans.  Today I am making the official announcement of ANTS -
Advanced Namespace Tools for Plan 9 from Bell Labs.  This is a
collection of software all focused on a single theme: how to extend
the capabilities and uses of Plan 9 namespaces.  ANTS is written for
the main Bell Labs distribution and is intended to be 100% compatible
with existing Bell Labs installations.  The goal is to provide
powerful new capabilities to Plan 9 installs with full compatibility
with the existing system and no reconfiguration required.

ANTS features:

9pcram kernel with more flexible and reliable bootup including full
interactive control if desired

Ability to easily launch multiple independent namespaces on a single
machine, slightly similar to BSD jails but not intended for security

High level semantics for rewriting process namespace via the /proc
file, enabling cpns of namespace between processes, even those on
remote machines

Instant rerooting into any available namespace while maintaining use
of the the gui

Run Bell Labs Plan 9 and 9front userland at the same time on the same
machine with 9pcram kernel

Automated replication of data between venti archives

Clone entire root filesystems between machines with one command

Create persistent shells and multiplexed i/o pipes to bring access to
shells in all namespaces into the main file namespace

Access shells and datastreams on any machine directly from rio

Comprehensive manpages

Tutorials, preinstalled vm images and a full paper explaining the
ideas and implementation

The ANTS software itself is located at ants.9gridchan.org and is
accessible via 9p, http, and ftp.  Ready to use VM images suitable for
booting in Qemu are linked from the base 9gridchan.org site (http
only) and tutorials and other information can be found at
http://antfarm.9gridchan.org/Overview.  The current draft of the paper
can be found at http://ants.9gridchan.org/doc/ants.ps.

MICROFAQ:

Ants is not a new fork or distribution.  It is software for Plan 9
from Bell Labs.

Ants does have tools to integrate with 9front.  Helping the different
varieties of Plan 9 work together easily is one of the goals.

Ants has only been tested on x86 pcs.  It is a goal to support all
arches and boot methods but the current software is only configured
for x86.

There are many components of the ANTS software.  Some can be used
independently.  Even if you aren't interested in the whole package,
there may be pieces which you might find useful on their own.

Many people offered technical advice and help.  Many published papers
and books were invaluable guides and sources of ideas.  No one else
has any responsibility for any bugs or errors, but the fact that this
software exists and runs would have been impossible without the help
of too many people and authors to list here.  Thanks to everyone who
works and has worked with Plan 9 and I hope the ANTS software will
provide you with useful tools.

-Ben Kidwell 
mycroftiv

[9fans] A note about new software for Plan 9

Hi 9fans.  I think I wrote my Advanced Namespace Tools message in an
unclear way.  Let me try again.

I have some new software for Plan 9.  It is based on manipulating
namespaces.  The 9pcram kernel has a new bootup which lets it control
multiple independent namespaces.  It is the foundation for a
collection of tools which manipulate namespace.  I think it adds
useful new features to Plan 9 and also solves some existing issues.

This project, Advanced Namespace Tools, is an attempt to make
something of high quality which lives up to the Plan 9 standard and
also extends what Plan 9 can do.  I have created bootable VM images,
tutorials for the software, and extensive documentation.  I have been
using and testing it myself every day and I believe it is genuinely
useful.

ants.9gridchan.org is served with 9p, ftp, and http and has the
software itself

antfarm.9gridchan.org/Overview and related pages have tutorials for
pre-installed ANTS vms

9gridchan.org has links to 2 vm images, one tiny (6mb) and one fully
installed (132mb) which can be used to explore the Advanced Namespace
Tools.

I'd really like to discuss these things with other Plan 9 users - I'm
friendly and even though my software is called ANTS, I don't think it
will actually bite you.

-Ben Kidwell mycroftiv

Re: [9fans] A note about new software for Plan 9

erik quanstrom wrote: 
 this part is confusing.  plan 9 has always maintained 1 independent 
 namespace per process group.  so i'm surethat there's more to it 
 than this.  could you go into detail on this?

You are right I was using terminology loosely here.  What I mean by
namespace in this context is the idea of a group of processes which
share a common namespace, and combine to create a user environment.
In other words, in a normal Plan 9 system, most of the processes are
aligned with the same root file system that the kernel acquired at
boot, and I'm loosely referring to this common root as the namespace.

When I talk about running multiple independent namespaces with ANTS,
what I mean is using per-process namespace to do something similar to
BSD jails or a virtual machine.  The same Plan 9 kernel and machine
can have groups of processes running with different root filesystems.
If you create a group of processes that share a common root
filesystem, you have a user environment.

One thing that ANTS can do is to host multiple namespace groups on a
single machine.  Instead of having only one global namespace on the
machine with a single common root, you can host independent user
environments, each with a different root filesystem.

I know you (and others) are busy with plenty of obligations, but the
demos I have prepared with qemu vm images are designed to show in
practice how you can make multiple independent user environments on a
single plan 9 machine, then freely re-root which namespace you are
using.

So this part of ANTS is: running multiple userpsace environments on
the same machine.  You can do this because per-process namespaces
means that you can have a totally different root file system for one
group of processes than another.  To make this really work, the ANTS
9pcram kernel does not attach to a root filesystem itself.  The main
flow of control of the kernel stands back from the user's namespace
and does not attach to the same file descriptor that is the root of
the user fs.  This allows the main flow of control of the kernel to
act as a kind of namespace hypervisor that can launch multiple
indpendent user environments available at the same time on the same
machine.

To allow the user to actualy move between the different namespaces
freely, I have a script and parameterized namespace file called
rerootwin that acts as an instant Plan 9 chroot into a new
filesystem.  It avoids losing connection to the active window system
by passing the devices through a srvfs.  This means that even though
you might have Bell Labs and 9front running on the same box at the
same time, you can move back and forth at will, and enter either one
from a lower-layer kernel-only namespace.

This explanation was already kind of long and elaborate, but as I
mentioned I have virtual machine images and demo walkthroughs so you
can see how it actually works in practice, which is actually simpler
than the verbal description.

I'm not too good at balancing being precise and specific with keeping
things brief, so I hope this was a good answer without too much
unnecessary length.

Ben Kidwell 
mycroftiv

Re: [9fans] A note about new software for Plan 9

 software folder, and five step-by-step tutorials done with
pre-installed VMs which attempt to comprehensively explain how to use
ANTS, how it is built, and what its purposes are.

I really appreciate the interest!  I hope I am responding and
explaining in a clear way

Ben Kidwell
mycroftiv

Re: [9fans] New wiki pages about 9p services and building grids

2013-03-02 Thread mycroftiv

Richard Miller 9fans@ham... wrote:
 Date: Sat, 2 Mar 2013 15:32:27 + 
 Thanks for producing this compendium of useful information.  
 One question - there's a mention of hubfs, which I wasn't familiar 
 with until I tracked it down in your contrib area.  Perhaps you could 
 provide a reference?

Thanks for the feedback on the wiki pages and the suggestion.  I
created and linked a new wiki page with extensive hubfs documentation
and usage examples.

http://www.plan9.bell-labs.com/wiki/plan9/hubfs/index.html

I wrote hubfs several years ago after noticing the absence of a
general purpose screen type utility in Plan 9.  aux/consolefs is
focused on a particular use case, serial consoles.  I feel that the
excellent Plan 9 design (no tty and 9p to handle local/remote clients
identically) helped me luckily stumble into a very nice simple fs
design that does both screen and general purpose network pipe
muxing.

On my current grid, my main cpu server hosts hubfs and every other
machine connects to it and shares services into it, and accesses other
machines through it.  I have persistent shells from several Plan 9
systems and two linux systems always available, and a separate hubfs
is used for things like irc sessions and mail reading sessions and
telnet connections to BBSes.  My profile does import -a of the /srv of
the main cpu so i can type hub main lapsh on any node and then be
connected to the subshell with p9p rc running on my linux laptop which
has a 9p connect to the hubfs server.

I think hubfs is a nice design for bringing shells of machines on the
network into the 9p file namespace.  I don't take any personal credit
for any nice things about it, I just tried to find the simplest way to
make a screen for Plan 9 and modifying a ramfs to have pipe-like
semantics and a queue of client responses seemed like the simplest way
to do it.  As it happened, the simplicity of doing it that way made it
more general purpose than a TTY-based screen and let me separate the
management of the shells from the basic idea of pipe buffering/muxing.

I'm less experienced as a developer than many so there are probably a
few naive things and eccentricities in hubfs, but it has been very
useful to me and in my use and testing it is stable and resource
efficient since all it does is just copy bytes into static buffers and
fill the 9p requests that come in.  To me, the fact that getting rid
of the TTY layer means that screen/tmux functions can be done in a
vastly simpler way - with new functionality as a free bonus - is a
nice demonstration of the benefits of clean design.

Sorry if this response was unnecessarily long, but thanks for your
interest in the wiki pages and the suggestion to write up and link
hubfs for clarity.

-Ben Kidwell 
mycroftiv

[9fans] exportfs and import options: short guide with examples

2013-02-28 Thread mycroftiv

So, what's all this about exportfs/import, in brief?  It's easy to
understand with examples.  In all these examples, listen could be
either done with aux/listen and a service file or aux/listen1directly.

Default exportfs server with authentication (port 17007 standard):

listen exportfs -a
# authenticates clients, then lets them choose a subtree

Client mounts with:

import server /remote/path /mount/point
# import uses /remote/path to request a given sub-tree

exportfs server with no authentication

listen exportfs

client mounts with:

import -A server /remote/path /mount/point
# the -A flag tells import to skip authentication

exportfs server with root selection

listen exportfs -r /serve/this/path/only
# exportfs now skips the tree request part of the protocol

client mounts with:

srv tcp!server!port srvname /mount/point
# no auth, no tree selection is a non-authed 9p server

exportfs server with authentication and root selection

listen exportfs -a -r /serve/this/with/auth
# exportfs now will auth clients, but skip tree negotiation

client mounts with:

UNMOUNTABLE CURRENTLY
# mc hammer says cant touch this

proposal:

import -m cpu /mount/point
# auths to server, skips tree request, and mounts successfully

This proposed additional option to import allows exportfs -a -r (and
exportfs -a -S which has the identical issue) to be useful as a
listener.  It has no affect on existing users and configurations.  If
this option isn't relevant to you, it can be completely ignored with
zero changes.  It allows those who wish to import and export using
this protocol variant (auth but no tree request) to do so with no
disruption to existing systems. The -B variant is also unaffected.

A patch(1) for this change has been submitted to sources.  The patch
used the -z flag, but apparently some existing users have this
modification already, and use the -m flag to import for skip tree
request.

If this change is accepted, there is no change needed to any local
configuration, but users should be aware that some servers might begin
using this protocol variant, and know which flag is needed to connect
properly.

Ben Kidwell
mycroftiv

gridterm: ps |grep exportfs |wc -l
306

[9fans] the import/exportfs protocol and a proposed import -z flag

2013-02-27 Thread mycroftiv

 srv know how to talk the -a -S/r protocol, etc. 
The issue of the unmountable protocol is something that could be solved in 
many ways, but I believe it definitely should be solved in some way. The 
proposed patch adding -z flag to import is trivial and i believe trivially 
verifiable' in the sense that it is guaranteed to allow you to mount those 
exports and not introduce any new buggy behaviors.

4. The -r and -S options are really just for -B mode I haven't actually heard 
anyone say this, but I can imagine it as a potential response. To some extent 
the previouis answers respond to this also, but another response is simply a 
user perspective: I find that doing listeners with -a -r and -a -S is something 
that is useful to me. It seems like the most direct way to share a specific 
piece of namespace or a specific service with auth, and that is a sensible 
thing to do. Perhaps the -r and -S flags were added with -B mode in mind and 
that has been the primary historical use, but when you translate the meaning of 
the commands into english - do an authenticated export of this service or this 
part of namespace - it seems to me like a simple and sensible use of the 
commands and flags.

5. Exportfs isn't intended to be used in this way I don't think this is true. 
Exportfs is intended as a general mechanism to share namespace of any kind. It 
is a core system utility. The essence of Plan 9 is network transparent 
namespaces. The flexibility of being able to share namespace across a network 
however you want is very important to Plan 9, and so making exportfs and import 
work together in all situations and with all options is a sensible goal. I 
think that exportfs -a -r and -a -S listeners are a useful tool for both 
private and public plan 9 networks.

6. OK, but patching import isn't the right solution, the right solution is... 
maybe you are right. I don't have strong opinions about whether or not a new 
flag is better or worse than trying to extend the import/export protocol to be 
smarter in its negotiation or whether some other solution is preferable. I do 
think that the proposed patch is trivially correct in fixing the problem, 
non-disruptive to all existing users, and based on a sensible principle of 
symmetry between exportfs and import. Even if you think a better long term 
solution involves changing the export/import protocol or even unifying srv and 
import as a single tool to handle all kinds of 9p connections and 
authentications, perhaps you might view import -z as a decent temporary 
solution.

7. Why is this important? This has never affected me, can't it just be 
ignored? I agree that this is an issue which is only like to come up if you 
are creating on-demand service listeners with aux/listen1 for various 
purposes, but that is basic functionality for some grids of machines. 9p is the 
heart of plan 9, and the tools which work with 9p services should let you serve 
and mount using the options of your choice. Making exportfs and import match 
cleanly is just like making the electrical plugs match electrical pluckets - 
square pegs, square holes, round pegs, round holes. 

8. You submitted the patch already, what is the purpose of this huge 9fans 
post? The reason for soliciting community discussion is my practical awareness 
that Plan 9 is now being distributed in several forms, some consiered forks 
and some not-fork variants. I mentioned that I think the tools for creating 
and importing 9p services over the network are foundational, so to me the 
interaction of import and exportfs and how the options work on each side is 
something that all the plan 9 developers (and users) should have consensus on. 

I also hope that the intent of this post - constructive discussion and 
consensus about how these tools should work - is clear. 

I have had long discussions with many people who are more expert in plan 9 use 
and development than I am, and I have not succeeded in achieving consensus 
about my perspective and proposal. I want to have constructive discussion about 
an issue i think is important because it affects how all Plan 9 systems which 
make use of exportfs/import communicate. I hope this post leads to constructive 
discussion of the import/exportfs protocol.

Ben Kidwell
-mycroftiv

[9fans] the import/exportfs protocol - fmt repost

2013-02-27 Thread mycroftiv

 ways - make exportfs and import smarter about tree negotiation,
let srv know how to talk the -a -S/r protocol, etc.  The issue of
the unmountable protocol is something that could be solved in many
ways, but I believe it definitely should be solved in some way.  The
proposed patch adding -z flag to import is trivial and i believe
trivially verifiable' in the sense that it is guaranteed to allow you
to mount those exports and not introduce any new buggy behaviors.

4.  The -r and -S options are really just for -B mode I haven't
actually heard anyone say this, but I can imagine it as a potential
response.  To some extent the previouis answers respond to this also,
but another response is simply a user perspective: I find that doing
listeners with -a -r and -a -S is something that is useful to me.  It
seems like the most direct way to share a specific piece of namespace
or a specific service with auth, and that is a sensible thing to do.
Perhaps the -r and -S flags were added with -B mode in mind and that
has been the primary historical use, but when you translate the
meaning of the commands into english - do an authenticated export of
this service or this part of namespace - it seems to me like a simple
and sensible use of the commands and flags.

5.  Exportfs isn't intended to be used in this way I don't think
this is true.  Exportfs is intended as a general mechanism to share
namespace of any kind.  It is a core system utility.  The essence of
Plan 9 is network transparent namespaces.  The flexibility of being
able to share namespace across a network however you want is very
important to Plan 9, and so making exportfs and import work together
in all situations and with all options is a sensible goal.  I think
that exportfs -a -r and -a -S listeners are a useful tool for both
private and public plan 9 networks.

6.  OK, but patching import isn't the right solution, the right
solution is... maybe you are right.  I don't have strong opinions
about whether or not a new flag is better or worse than trying to
extend the import/export protocol to be smarter in its negotiation or
whether some other solution is preferable.  I do think that the
proposed patch is trivially correct in fixing the problem,
non-disruptive to all existing users, and based on a sensible
principle of symmetry between exportfs and import.  Even if you think
a better long term solution involves changing the export/import
protocol or even unifying srv and import as a single tool to handle
all kinds of 9p connections and authentications, perhaps you might
view import -z as a decent temporary solution.

7.  Why is this important?  This has never affected me, can't it just
be ignored? I agree that this is an issue which is only like to come
up if you are creating on-demand service listeners with aux/listen1
for various purposes, but that is basic functionality for some grids
of machines.  9p is the heart of plan 9, and the tools which work with
9p services should let you serve and mount using the options of your
choice.  Making exportfs and import match cleanly is just like making
the electrical plugs match electrical pluckets - square pegs, square
holes, round pegs, round holes.

8.  You submitted the patch already, what is the purpose of this huge
9fans post? The reason for soliciting community discussion is my
practical awareness that Plan 9 is now being distributed in several
forms, some consiered forks and some not-fork variants.  I
mentioned that I think the tools for creating and importing 9p
services over the network are foundational, so to me the interaction
of import and exportfs and how the options work on each side is
something that all the plan 9 developers (and users) should have
consensus on.

I also hope that the intent of this post - constructive discussion and
consensus about how these tools should work - is clear.

I have had long discussions with many people who are more expert in
plan 9 use and development than I am, and I have not succeeded in
achieving consensus about my perspective and proposal.  I want to have
constructive discussion about an issue i think is important because it
affects how all Plan 9 systems which make use of exportfs/import
communicate.  I hope this post leads to constructive discussion of the
import/exportfs protocol.

Ben Kidwell -mycroftiv

Re: [9fans] the import/exportfs protocol and a proposed import -z flag

2013-02-27 Thread mycroftiv

Charles Forsyth charles.fors...@gmail.com wrote:

 As you'll have noticed, it isn't a great protocol as it stands. I don't
 think your option makes it worse.

Thanks for your perspective.  I'm glad that my long-winded explanation
didn't obscure the basic logic too much.  It makes sense for import to
be able to dial and attach to an exportfs -a -r listener, and right
now it can't.  You mentioned someone was already carrying an
equivalent patch with the -m flag, and another user I spoke with
mentioned they had written their own exportfs server which addressed
this on the exportfs end.  Since a few different users have run into
this and wanted to use the authentication, no tree request protocol
variant, it would be good I think if all users could connect to that
type of server using a standardized method.  If the -m flag already
has some existing users, it might be a better choice than -z for this
option.


 Note that in your example
 import -z tcp!server!9876 somefiles /n/authedimport
 with the existing import you don't need to specify the (now unused)
 somefiles. If I write {import system /net} it sends /net as the tree by
 default.
 I think all you'd need to do to make the option tidier is to reject the
 case argc == 3 in the relevant switch if the option is set. Then you could
 write
 (changing the option letter):
   import -m tcp!server!9876 /n/authedimport
 which reads as import by mounting the 9P service on the given connection on
 the given mount point. It's otherwise a little strange to have an
 argument (your somefiles) that's completely ignored.

I totally agree.  In fact I was unclear about this.  The way I wrote
it made it seem as if sending the unused parameter was mandatory - but
in fact, import already has this logic:


switch(argc) {
case 2:
mntpt = argv[1];
break;
case 3:
mntpt = argv[2];

So it is already the case that 

import -m tcp!server!9876 /n/authedimport 

would do the mount at the specified point.  I wrote out the version
with the unnecessary parameter to try to clarify what what -z was
doing, but in practice you just write it exactly as you described.
Maybe the argc==3 case should be rejected just as a reminder to the
user that the tree request will not be sent.

Ben Kidwell
-mycroftiv

[9fans] 9vx access from Drawterm/cpu: patch and how-to

2010-01-06 Thread mycroftiv 9gridchan

A brief note on 9vx and cpu, which might also be relevant for a few
other things. Using 9vx as a cpu server has been mentioned a few
times, but my attempts to actually get this working met with failure
initially. I believe I have tracked down the issue - the remoteside()
of cpu.c makes use of the kernel cap device via a call to the
auth_chuid function, and the cap device is not available in 9vx,
probably due to the single-user nature of it as a hosted environment.
If we don't need to support multiple users, there is an easy way to
get this work to work - just skip trying to change user.

cpu% diff /sys/src/cmd/cpu.c /sys/src/cmd/altcpu.c
590a591
   int factfd;
595,596c596
   if(auth_chuid(ai, nil)  0)
   return -1;
---
   /* no cap device in 9vx so no auth_chuid, we are who we are */
597a598,601
   factfd = open(/srv/factotum, ORDWR);
   if(factfd = 0)
   mount(fd, -1, /mnt, MREPL, );
   newns(user, nil);   /* this and above 3 lines replicate auth_chuid 
 behavior */

Because we aren't can't change user ID, using aux/listen as none isn't
going to work. Instead, we need to run the listener as our user, and
we can just put a factotum key in its namespace and leave out authsrv
and keyfs. Assuming the above modification has been compiled and
installed as altcpu, the following is all you need to do to allow
drawterm/cpu access:

term% auth/factotum
term% echo 'key proto=p9sk1 dom=testdom user=glenda
!password=password' /mnt/factotum/ctl
term% aux/listen1 -t tcp!*!17010 /bin/altcpu -R 

Despite running as a trusted user, this should still be fairly secure
because cpu demands authentication. Even without authsrv/keyfs,
password-based authentication between the factotums works. Note
however that if you get the password wrong, it fails with a botch
message, so you have to redial if you typo your password. According to
my limited testing so far, cpu from plan 9 and drawterm both work
fine. This message was written in acme in drawterm to a 9vx elsewhere
on the LAN.

~Mycroftiv
9gridchan.org

[9fans] 9vx access from Drawterm/cpu: patch and how-to

2010-01-06 Thread mycroftiv 9gridchan

I dont understand. Why can't the capability device be put directly
into 9vx? Doesn't the 9vx kernel have its own notion of
user id for each of the processes it hosts independant of the
underlying unix user id? I thought only select parts of the
system such as the unix file server cared about the unix
user id.

I didn't mean to imply that the capability device couldn't exist
within 9vx, simply that it was not currently implemented. I speculate
the reason it hadn't been done was that 9vx wasn't really conceived as
a multi-user system. I wasn't proposing my patch as anything other
than an expedient means of getting CPU functionality within an
existing 9vx setup. I confess I haven't taken the time to study what's
really going on in the .ed files used to tweak the plan 9 source for
9vx, and I didn't have a personal practical motivation for doing more
than just cpu in as the current user. I think the general issue of how
to make 9vx as close a parallel to a native plan 9 install as
possible, including support for tcp booting, plan9.ini options,
running the standard file servers from disk partitions, would benefit
from being addressed in an integrated way. I know some of this work
has already been done. A 9vx distribution with those modifications and
the full tree included would be a fun project, but the 'too many
projects, too little time' issue always looms.

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

~Mycroftiv
9gridchan.org

[9fans] Announcing: writable /proc/pid/ns

In celebration of the arrival of 2010, the 9gridchan.org community
gridding development team - aka one guy with a basement full of
ethernet cables - would like to announce several new tools for Plan 9.
In this post I'll talk about writable /proc/pid/ns, and in a later
message, rootless post-kernel load booting. Everything mentioned is
available on sources now in contrib/mycroftiv. All of this software
receives testing and use on three native hardware Plan 9 systems and a
swarm of qemu VMs. mycroftiv/writeprocns contains all files relevant
to this post, modified versions of 3 kernel source files in
/sys/src/9/port.

Motivation: Per process namespaces are one of the glories of Plan 9.
Getting the most out of Plan 9, especially a grid of machines,
requires fine-grained control of namespace construction. There are
some occasional inconveniences caused by the fact that currently
running processes other than the shell do not have a consistent
mechanism for acquiring newly made mounts or binds. Plan 9 already has
a representation of process namespace available in /proc and processes
may freely modify their own namespace at runtime. Making /proc/pid/ns
act as a control interface to trigger modifications to the namespace
of a running process seems consistent with the design.

Writable /proc/pid/ns is simple in usage: you can perform arbitrary
namespace operations on running processes you own just by echoing the
standard command to that processes' ns file. This can be used for
purposes such as bringing newly mounted services into the namespace of
your running plumber, or adding a mount underneath your running rio.
Example:

9fs sources
ps |grep rio
echo 'mount /srv/sources /n/sources' /proc/863/ns   #first rio proc

Open new windows within rio and the sources mount is in place.
Standard bind and mount flags and spec and unmount are all supported,
but all mounts are done without an auth file descriptor. This is not
as much of a limitation as it might seem because any external mount
requiring auth can be made available locally non-authed via /srv - and
in the most common case of a 9fs connection to a fossil server, fossil
will accept non-authed mounts of a previously authed file descriptor.
Import takes a flag (-s srvname) to post a /srv which will not require
additional reauthentication.

In addition to adding in new bindings to running processes like rio,
plumber, dossrv, and exportfs, this mechanism is also fully network
transparent and useful when importing /proc from remote machines.
Rewriting the namespace of remote processes is a powerful mechanism
for fine-grained interactive control. Aux/lines can be used for
wholesale modifications to a namespace.

Implementation: simple conceptually. Writing a namespace operation to
the ns file in /proc produces a parallel sequence of actions as that
process itself issuing  the equivalent syscall. The existing routines
in 9/port/sysfile.c and 9/port/chan.c are all written to operate on
'up', the current process - so I created near-identical versions of
the syscalls and channel operations which take a Proc *targp parameter
and address resources via targp- rather than up-. This does create a
bit of inelegant duplication but has the advantage of leaving all the
existing namespace operation code paths untouched.

I hope this approach is fundamentally sound, and I have attempted to
test it extensively on my local grid of native and virtual machines. I
have not found any bugs or inconsistencies, but given the importance
of chan.c I think this code would need additional review and testing
before use on production machines. I would like to submit an evolved
version of these patches to the main distribution after some review
and testing by more experienced plan 9 kernel programmers, because I
believe the functionality of modifying the ns of processes you own is
useful and the mechanism of simply writing the standard ns commands to
the ns file is clear and in harmony with the overall system.

I would like to also acknowledge the work done on namespace
crossings as described by
http://www.cs.cmu.edu/~412/history/2006F/nscross/ - this differs in
purpose and implementation but springs from somewhat similar
motivations. I haven't investigated the code but I'm sure its more
sophisticated than my snarf+paste based approach!

All the modifications are to files in /sys/src/9/port, so bind -b
/n/sources/contrib/mycroftiv/writeprocns /sys/src/9/port and then
compile the kernel of your choice from within that namespace to test
without modifying your original kernel source. A console message is
printed for each ns command as it is initiated from within devproc.c -
these are not error messages. If they irritate you, comment them out
in the new procnsreq function at the end of the modified devproc.c.

mycrof...@sphericalharmony.com
Ben Kidwell
9gridchan.org provides a variety of public plan 9 services
project channel: #plan9chan on irc.freenode.net for 9gridchan
questions, tech support

[9fans] Announcing: rootless post kernel load startup

More recent work - available on sources in
contrib/mycroftiv/rootlessboot as both patches and a compiled ready to
use kernel and optional rootfs.tgz additional tools to be placed in
9fat partition.

Rootless bootup is a rewrite of the post-kernel load bootup process
as handled by boot and init. The motivation is severalfold: the basic
nature of the Plan 9 kernel as a 9p multiplexer means that there is no
inherent reason to make any given filesystem special and necessary to
keep the system usable. The particular case of a tcp-booted cpu server
losing its network root fs is a traditional example. It is sometimes
the case that a system with local fossil and remote venti can't be
successfully booted if the IP address of the venti has changed from
that given in plan9.ini. The overall goal is to improve reliability by
enabling a running system to always repair and recover itself even if
it experiences failure of normally criticial resources like a local
venti+fossil.

The rootless boot method combines boot and init into a much smaller
program that does much less, and adds the rc shell and a few other
tools to the kernel's compiled-in /boot. An rc script called plan9rc
handles the tasks of starting up factotum, ipconfig, venti, fossil,
and setting up a desired initial filesystem mount and running
additonal options scripts and/or conventional cpurc or termrc from an
external fs. Depending on the settings in plan9.ini, this can all be
done interactively with the ability to drop to an rc shell. The
plan9rc script can act to imitate the standard bootup process, or
setup the environment in a different way - for instance by creating a
ramfs with additional tools loaded from 9fat to create a working
environment independent of the environment created by the standard
bootup.

As an example of the flexibility of this approach, I have tested this
scenario: a rootless system starts up with interactive=yes set in the
plan9.ini. It starts factotum, ipconfig, and then a venti server, and
then pauses its bootup process. A second machine is then started as
the fossil file server, and it is instructed to dial the venti and
begin serving fossil and its normal startup. Once the fossil is
serving, we return to the machine serving venti, and continue through
the bootup process, and choose to imitate tcp-booting - and select the
external fossil for the external service to mount. The fossil is
dialed and bootup continues as if it was a standard tcp root cpu
server.  The end result is that even the system hosting the venti is
still rooted via an external fossil which is in turn using the venti
as a backing store. This sequence would not be possible using the
standard bootup method. Rootless booting also enables tcp booted cpu
servers to be recovered without rebooting. The initial console process
is usually kept in a protected namespace, with no connections to
non-kernel resources. Consequently, if the non-kernel 'root' is lost,
the dead processes can be killed off, another root acquired, and work
resumed. If no local resources are available, a srv and mount of
sources allows access to the tools of the full distribution.

contrib/mycroftiv/rootlessboot holds the source for the modifications,
as well as a compiled kernel. Also available is a rootfs.tgz for use
in creating an optional ramfs based independent environment. If you
wish to use this, put it in your 9fat. There is a lot of flexibility
in exactly what is compiled into /boot and whether or not a ramfs with
additonal tools loaded. Some users may prefer a light kernel that
acquires a relatively large rootfs.tgz, whereas others might prefer to
make a heaver compiled in /boot. The sample kernel on sources and
rootfs.tgz there are both fairly heavy to try to support most common
possibilities - venti, fossil, kfs, cfs are all compiled in, for
instance.

The rootlessboot directory has a fair amount of small modifications
that are necessary or helpful, such as changing rc to look for
/boot/rcmain rather than /lib/rcmain, and giving cfs the ability to
post a srv. contrib/mycroftiv/rootlessboot/bootdir.extras/plan9rc
controls the boot process and provides many options for controlling
its behavior via plan9.ini variables. The default behavior is intended
to closely mimic the standard existing logic for handling plan9.ini.
The prebuilt kernel is a cpu kernel but setting service=terminal in
plan9.ini will produce terminal style behavior. As always, I would
recommend using a plan9.ini menu or setup so that choice of kernel
used is selectable at boot.

I have read the paper on the 9null kernel load boot process from this
year's IWP9 and I am hoping that this work is consistent with its
approach. As I have been working only on the post kernel load actions,
I believe it to be mostly non-overlapping. This is still very much a
work-in-progress, although I am using this boot process on my machines
and VMs and am happy with the additional flexibility and control.
These patches are most relevant

[9fans] Announcing: writable /proc/pid/ns

can you give an example of a use of this feature that can't be
accomplished by plumbing Local 9fs $server?

Most obviously, plumbing a Local command can't rewrite the namespace
of processes on a remote server whose /proc you are importing. It also
cannot be used to make modifications targeted at a single specific
running process. It is a good mechanism, and one I make use of (a set
of scripts I wrote for grid resource indexing uses it heavily), but it
isn't a general purpose namespace manipulation tool. Processes like
service listeners started by cpurc are unlikely to have a plumber in
their namespace, and starting a lot of extra plumber processes to act
as namespace agents doesnt seem like a sensible approach.

In the context of a user using a single machine as a self sufficient
environment the plumb Local trick is probably just fine for most of
their namespace manipulation needs, but the context of trying to build
a larger grid where multiple machines are all providing services 'a la
carte' and a single cpu may be hosting processes with widely divergent
namespaces, more general and precise tools are useful. In the other
post I made today I discussed a modified boot system that creates both
a small self-sufficient ramdisk based environment and a standard
disk-fileserver based one - being able to shift which set of resources
the active processes such a machine will reference has been useful to
me in making sure I can keep services available even if I need to
reboot a fileserver node.

Philosophically, I think that if the freedom of per process namespaces
is a good thing - which I certainly believe it is - then making
process namespaces as flexible and precisely controllable as possible
enhances that quality. Because this modification was only done very
recently, I haven't yet had time to start building some of the scripts
that can make use of it - but as an example, i think a script that can
'synchronize' the namespace of two given processes by finding binds
and mounts present in the ns of one but not in the other and then
issuing the commands to make the target process ns match the given
model would be a nice thing to have. I can supply more examples from
my own usage but I'm probably already past my word quota for the day.
To sum up, I think that the idea of controlling the ns of processes
spread across a multimachine grid via mporting multiple /proc and
using scripted tools has obvious utility for dynamic grid computing
where service nodes can enter and leave the resource matrix freely.

~mycroftiv

[9fans] Announcing: rootless post kernel load startup

would the dead processes include everything but the boot-time
helpers?  from the perspective of any users of the system, wouldn't
this be equivalent to a reboot?

have you tested rebooting either machine while the other is
running?  have you tested a three machine scenario?
this reminds me of nfs cross mounting.  how is this different?

Thanks for your interest. Certainly, if the file descriptors being
used by your active applications die, those apps are pretty much dead.
The question of why the scenario is an improvement on rebooting
depends on the specifics - one answer is that in the usage model I
find most optimal, I like to base what I regard as the most essential
services, including the cpu -R listener, either out of boot or a
ramfs, so the system remains available for remote dial-in and
administration. I find that being able to drawterm in, mount sources
for access to the full disttribution, and then work to reset/repair
and environment is a lot more pleasant than plugging in a monitor to a
box I prefer to run headless and then rebooting and hoping that the
issues that forced the reboot (such as unavailable network resources)
are in good enough shape to get things put back together.

The example I gave of starting venti then rooting from a remote fossil
using that venti isn't a particularly important specifically, I simply
wanted to illustrate the fact that moving to a more interactive and
configurable post kernel load boot sequence creates additional
possibilities. The initial motivation was simply that with a grid that
uses separate venti, fossil, and cpu, the imposed boot order
dependency and the fact that the loss of one server could make the
others later in the chain unusable even though they still had running
kernels and control of the hardware seemed suboptimal. I love the
ability to distribute system components, but I wanted to be able to
keep interacting with my cpu server even if I needed to reboot the
fossil.

However, I don't want to seem defensive, and these bootup modification
patches are not anything I want to try to claim as the best possible
system. Just on the technical level the plan9rc I'm using doesn't
cover 100% of the cases handled by the standard tools - it assumes the
only types of disk drives in the world are sdC0, C1, D0, D1 at the
moment, for instance.

~mycroftiv

[9fans] Announcing: writable /proc/pid/ns