Re: [ANNOUNCE] Darkstar Development Project

[Thomas Graichen]

Larry McVoy <[EMAIL PROTECTED]> wrote:
> On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
>> > Err, "faster"?  The following is the moral equiv of 4 kernel updates
>> > which had nothing to do using BitKeeper instead of CVS.  The local copy
>> > was in San Francisco and the remote copy is Cort's machine in New Mexico
>> > over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
>> > CVS tree they can try to get comparable numbers?
>> 
>> Try: http://innominate.org/~tgr/projects/lksr/

> Thanks, that was helpful.  Comparison numbers for a null update of the 2.3
> kernel, which means you update and then update again, timing the second update
> to get some idea of the system's best case throughput, are:

> CVS: 139.5 seconds
> BK:1.6 seconds

> The BK tree is the 2.3 kernel tree maintained by FSMlabs.

larry - this one is a bit unfair i think: the innominate.org tree
runs right now on a 200mhz pentium and is quite a bit worse
connected to you than the bk tree - also it's a "synthetic"
tree which contains for instance 100+ tags in the 2.3
tree which might make it a bit slow too ...

but all that does not mean that bk is bad - haven't had a look at
it so far - i just wanted to say: better avoid such comparisions
- i think the mozilla idea (from some mails later) side by side
will give a much better comparision

t

-- 
[EMAIL PROTECTED]
technical director   innominate AG
clustering & securitynetworking people
tel: +49.30.308806-13  fax: -77   http://innominate.de
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Thomas Graichen]

Larry McVoy [EMAIL PROTECTED] wrote:
 On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
  Err, "faster"?  The following is the moral equiv of 4 kernel updates
  which had nothing to do using BitKeeper instead of CVS.  The local copy
  was in San Francisco and the remote copy is Cort's machine in New Mexico
  over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
  CVS tree they can try to get comparable numbers?
 
 Try: http://innominate.org/~tgr/projects/lksr/

 Thanks, that was helpful.  Comparison numbers for a null update of the 2.3
 kernel, which means you update and then update again, timing the second update
 to get some idea of the system's best case throughput, are:

 CVS: 139.5 seconds
 BK:1.6 seconds

 The BK tree is the 2.3 kernel tree maintained by FSMlabs.

larry - this one is a bit unfair i think: the innominate.org tree
runs right now on a 200mhz pentium and is quite a bit worse
connected to you than the bk tree - also it's a "synthetic"
tree which contains for instance 100+ tags in the 2.3
tree which might make it a bit slow too ...

but all that does not mean that bk is bad - haven't had a look at
it so far - i just wanted to say: better avoid such comparisions
- i think the mozilla idea (from some mails later) side by side
will give a much better comparision

t

-- 
[EMAIL PROTECTED]
technical director   innominate AG
clustering  securitynetworking people
tel: +49.30.308806-13  fax: -77   http://innominate.de
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[Rik van Riel]

On Thu, 14 Sep 2000, David A. Gatwood wrote:
> On Wed, 13 Sep 2000, mberglund wrote:
> 
> > Because Linux already runs on a couple of platforms even NetBSD does not
> > run on,
> 
> And vice-versa, last I checked.  Let's be fair here.  :-)

Indeed. Until Linux has support for the Sega Dreamcast
NetBSD is still the system of choice for ultra-cheap
CPU power ;)

> Indeed, the linux port I work most with is very nearly a linux
> kernel running on top of a BSD one (Mach, to be precise, but it
> shows its BSD heritage well).  I've borrowed code from the *BSDs
> frequently, and I've always let both them and the monolithic
> Linux guys borrow my code whenever it was helpful, because I
> believe in the free flow of ideas among open-sourcers.
> 
> Our biggest strength would be in embracing all that is different
> and unique yet open, and sharing resources and ideas.  Don't
> fear the daemon.  Use it.  Learn from it.

Indeed. It would be so good for everybody if the Linux
developers and the developers from the 4 BSDs would talk
to each other more and develop good ideas together, in a
friendly competition with each other ...

[With the design of the new VM, we had a lot of help from
Matt Dillon - who made the FreeBSD VM - and we're of course
giving back any ideas we come across so FreeBSD VM can be
improved too. It is my personal hope that both Linux and
FreeBSD VM will raise the bar for virtual memory management
a little bit over the next year or so.]

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
   -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/   http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[mberglund]

On Thu, 14 Sep 2000, David A. Gatwood wrote:

> On Wed, 13 Sep 2000, mberglund wrote:
> 
> > Because Linux already runs on a couple of platforms even NetBSD does not
> > run on,
> 
> And vice-versa, last I checked.  Let's be fair here.  :-)
> 
> 
> > PS: If FreeBSD DID run on PPC and S390 and offer the company I work for
> 
> NetBSD runs on lots of PPC machines, although I don't think it runs on the
> S390.  Their ofppc port doesn't have a supported models list on the main
> NetBSD site
> 
> 
> > all the growth potential Linux does, there is a good chance that I would
> > already be there. I don't want this to happen, so I am hoping that
> > instead, we can raise the level of integration and upgradability here. I
> > would hate to have to trade my penguin in for a devil!
> 
> I agree with the goal, although perhaps not the reason for it.  I think
> that's the wrong way to look at it. I think all the free 'NIXers should
> see this as an opportunity to take free code and concepts and leverage it
> to our advantage, while at the same time, giving back those advantages to
> the groups that gave them to us.  The Penguin and the Daemon need not be
> mortal enemies.
> 
> Indeed, the linux port I work most with is very nearly a linux kernel
> running on top of a BSD one (Mach, to be precise, but it shows its BSD
> heritage well).  I've borrowed code from the *BSDs frequently, and I've
> always let both them and the monolithic Linux guys borrow my code whenever
> it was helpful, because I believe in the free flow of ideas among
> open-sourcers.
> 
> I dream of a day when little black and white children and little red
> children can... oops... wrong speech  ;-)  But you get the idea.  Our
> biggest strength would be in embracing all that is different and unique
> yet open, and sharing resources and ideas.  Don't fear the daemon.  Use
> it.  Learn from it.
> 
> 
> David
> 

You are a wise and learned man. I will study in your light.

Thanks,
Matt

Unix is best described as an old, sturdy tree.
It is well structured, always growing, and has passed the test of time.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Wed, 13 Sep 2000, mberglund wrote:

> Because Linux already runs on a couple of platforms even NetBSD does not
> run on,

And vice-versa, last I checked.  Let's be fair here.  :-)


> PS: If FreeBSD DID run on PPC and S390 and offer the company I work for

NetBSD runs on lots of PPC machines, although I don't think it runs on the
S390.  Their ofppc port doesn't have a supported models list on the main
NetBSD site


> all the growth potential Linux does, there is a good chance that I would
> already be there. I don't want this to happen, so I am hoping that
> instead, we can raise the level of integration and upgradability here. I
> would hate to have to trade my penguin in for a devil!

I agree with the goal, although perhaps not the reason for it.  I think
that's the wrong way to look at it. I think all the free 'NIXers should
see this as an opportunity to take free code and concepts and leverage it
to our advantage, while at the same time, giving back those advantages to
the groups that gave them to us.  The Penguin and the Daemon need not be
mortal enemies.

Indeed, the linux port I work most with is very nearly a linux kernel
running on top of a BSD one (Mach, to be precise, but it shows its BSD
heritage well).  I've borrowed code from the *BSDs frequently, and I've
always let both them and the monolithic Linux guys borrow my code whenever
it was helpful, because I believe in the free flow of ideas among
open-sourcers.

I dream of a day when little black and white children and little red
children can... oops... wrong speech  ;-)  But you get the idea.  Our
biggest strength would be in embracing all that is different and unique
yet open, and sharing resources and ideas.  Don't fear the daemon.  Use
it.  Learn from it.


David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Wed, 13 Sep 2000, mberglund wrote:

 Because Linux already runs on a couple of platforms even NetBSD does not
 run on,

And vice-versa, last I checked.  Let's be fair here.  :-)


 PS: If FreeBSD DID run on PPC and S390 and offer the company I work for

NetBSD runs on lots of PPC machines, although I don't think it runs on the
S390.  Their ofppc port doesn't have a supported models list on the main
NetBSD site


 all the growth potential Linux does, there is a good chance that I would
 already be there. I don't want this to happen, so I am hoping that
 instead, we can raise the level of integration and upgradability here. I
 would hate to have to trade my penguin in for a devil!

I agree with the goal, although perhaps not the reason for it.  I think
that's the wrong way to look at it. I think all the free 'NIXers should
see this as an opportunity to take free code and concepts and leverage it
to our advantage, while at the same time, giving back those advantages to
the groups that gave them to us.  The Penguin and the Daemon need not be
mortal enemies.

Indeed, the linux port I work most with is very nearly a linux kernel
running on top of a BSD one (Mach, to be precise, but it shows its BSD
heritage well).  I've borrowed code from the *BSDs frequently, and I've
always let both them and the monolithic Linux guys borrow my code whenever
it was helpful, because I believe in the free flow of ideas among
open-sourcers.

I dream of a day when little black and white children and little red
children can... oops... wrong speech  ;-)  But you get the idea.  Our
biggest strength would be in embracing all that is different and unique
yet open, and sharing resources and ideas.  Don't fear the daemon.  Use
it.  Learn from it.


David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[mberglund]

On Thu, 14 Sep 2000, David A. Gatwood wrote:

 On Wed, 13 Sep 2000, mberglund wrote:
 
  Because Linux already runs on a couple of platforms even NetBSD does not
  run on,
 
 And vice-versa, last I checked.  Let's be fair here.  :-)
 
 
  PS: If FreeBSD DID run on PPC and S390 and offer the company I work for
 
 NetBSD runs on lots of PPC machines, although I don't think it runs on the
 S390.  Their ofppc port doesn't have a supported models list on the main
 NetBSD site
 
 
  all the growth potential Linux does, there is a good chance that I would
  already be there. I don't want this to happen, so I am hoping that
  instead, we can raise the level of integration and upgradability here. I
  would hate to have to trade my penguin in for a devil!
 
 I agree with the goal, although perhaps not the reason for it.  I think
 that's the wrong way to look at it. I think all the free 'NIXers should
 see this as an opportunity to take free code and concepts and leverage it
 to our advantage, while at the same time, giving back those advantages to
 the groups that gave them to us.  The Penguin and the Daemon need not be
 mortal enemies.
 
 Indeed, the linux port I work most with is very nearly a linux kernel
 running on top of a BSD one (Mach, to be precise, but it shows its BSD
 heritage well).  I've borrowed code from the *BSDs frequently, and I've
 always let both them and the monolithic Linux guys borrow my code whenever
 it was helpful, because I believe in the free flow of ideas among
 open-sourcers.
 
 I dream of a day when little black and white children and little red
 children can... oops... wrong speech  ;-)  But you get the idea.  Our
 biggest strength would be in embracing all that is different and unique
 yet open, and sharing resources and ideas.  Don't fear the daemon.  Use
 it.  Learn from it.
 
 
 David
 

You are a wise and learned man. I will study in your light.

Thanks,
Matt

Unix is best described as an old, sturdy tree.
It is well structured, always growing, and has passed the test of time.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[mberglund]

On Wed, 13 Sep 2000, Stephen Frazier wrote:

> Does *BSD run on S390?

Ummm, hello, NO they do not. This is at least PART of the point. I
believe now is my chance to get out my clue-by-four.

FreeBSD has a GREAT idea(and really NetBSD as well). We might want to
consider adopting that idea. In this case, if we work toward a standard
base system across platforms it will be that much more simple to scale
up. Unix has notoriously been a pain to move from platform to platform
because of 'little differences here and there'. 

Lets remove this obstacle, however small. Make it a little easier to
migrate up to your 'big iron'. 

Because Linux already runs on a couple of platforms even NetBSD does not
run on, it seems logical to take their good ideas and steal them while we
have the edge Otherwise, one day we will wake up and they WILL run on
S390 and the like.

Currently it would be easier for us to organise just a little more than
it would be for them to get the those platforms up and running. I doubt it
will be that way forever.
 
This is an issue of using those good ideas that we might lack, and in this
case, I am certain we lack this kind of and quality of userland and system
and kernel integration.

We need some help getting the website built and getting some of this kind
of material documented. If we can do this, the word may travel a little
better. And (hopefully) clearer.

I hope this more clearly explains some of ideas and why thier ideas may be
good ones. And if not, give the BSD update system a shot. If you are an
admin I am sure you will like at least some of the functionality.


Later,
Matt


PS: If FreeBSD DID run on PPC and S390 and offer the company I work for
all the growth potential Linux does, there is a good chance that I would
already be there. I don't want this to happen, so I am hoping that
instead, we can raise the level of integration and upgradability here. I
would hate to have to trade my penguin in for a devil!

Unix is best described as an old, sturdy tree.
It is well structured, always growing, and has passed the test of time.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Ralf Baechle]

On Tue, Sep 12, 2000 at 12:21:16AM +0200, Jamie Lokier wrote:

> > Other than the initial repository create (aka cvs checkout), BK
> > *never* moves an entire file across the wire.  Never means never and
> > includes the process of deciding what to do.  CVS moves whole files
> > just to discover there is nothing to do.
> 
> Yes, CVS over rsync is much better for that.  IMHO the ideas underlying
> the rsync protocol are most cool.

Except that this still doesn't make CVS a distributed system, so it's only
good for checkouts ...

  Ralf
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[Stephen Frazier]

Does *BSD run on S390?

Stephen Frazier
Oklahoma Department of Corrections

-Original Message-
On Wed, 13 Sep 2000, Geert Uytterhoeven wrote:

I just thought I would remind all the nay-sayers that the *BSD world has
been doing something along this line since at least 1990. While some of
the programs have changed, the functionality has remained largely the
same.



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Geert Uytterhoeven]

On Tue, 12 Sep 2000, Ralf Baechle wrote:
> On Mon, Sep 11, 2000 at 02:39:42PM -0700, Larry McVoy wrote:
> > On the other hand, if you do a
> >
> > find . -type f | xargs touch
> > time cvs update .
> >
> > it will melt down your DSL line for what seems forever.  I killed it after
> > 20 minutes, I have better things to do with my bandwidth.   It's pretty
> > clear that CVS is comparing timestamps so if your files get modified at
> > all, it's going to transfer them to see what needs to be updated.  The
> > same sort of "touch all, then update" operation in BK has no effect on
> > performance, BK doesn't do its work that way.
> 
> >From some random Linux source tree's .hdepend:
> 
> /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
>/usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
>/usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
> @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
> /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
>$(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
> @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
> [...]
> 
> Linux, born to be CVS worst case ...

Which also hinders compiling kernels in `cp -rl'd read-only trees...

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Jamie Lokier wrote:
> > @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
> > [...]
> .cvsignore

Oops, ignore me 'cos I obviously didn't read what I replied to. 

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Michael Elizabeth Chastain]

Jamie Lokier writes:

> .cvsignore

Not so fast.  Read your .hdepend file, notice which files that it's
touching, and then notice that those are real source files
(include/linux/*.h and include/asm/*.h).

Michael
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Ralf Baechle wrote:
> >From some random Linux source tree's .hdepend:
> /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
>/usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
>/usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
> @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
> /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
>$(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
> @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
> [...]
> 
> Linux, born to be CVS worst case ...

.cvsignore

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Ralf Baechle wrote:
 From some random Linux source tree's .hdepend:
 /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
/usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
/usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
 @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
 /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
$(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
 @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
 [...]
 
 Linux, born to be CVS worst case ...

.cvsignore

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Michael Elizabeth Chastain]

Jamie Lokier writes:

 .cvsignore

Not so fast.  Read your .hdepend file, notice which files that it's
touching, and then notice that those are real source files
(include/linux/*.h and include/asm/*.h).

Michael
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Jamie Lokier wrote:
  @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
  [...]
 .cvsignore

Oops, ignore me 'cos I obviously didn't read what I replied to. 

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Geert Uytterhoeven]

On Tue, 12 Sep 2000, Ralf Baechle wrote:
 On Mon, Sep 11, 2000 at 02:39:42PM -0700, Larry McVoy wrote:
  On the other hand, if you do a
 
  find . -type f | xargs touch
  time cvs update .
 
  it will melt down your DSL line for what seems forever.  I killed it after
  20 minutes, I have better things to do with my bandwidth.   It's pretty
  clear that CVS is comparing timestamps so if your files get modified at
  all, it's going to transfer them to see what needs to be updated.  The
  same sort of "touch all, then update" operation in BK has no effect on
  performance, BK doesn't do its work that way.
 
 From some random Linux source tree's .hdepend:
 
 /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
/usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
/usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
 @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
 /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
$(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
 @touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
 [...]
 
 Linux, born to be CVS worst case ...

Which also hinders compiling kernels in `cp -rl'd read-only trees...

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

RE: [ANNOUNCE] Darkstar Development Project

[Stephen Frazier]

Does *BSD run on S390?

Stephen Frazier
Oklahoma Department of Corrections

-Original Message-
On Wed, 13 Sep 2000, Geert Uytterhoeven wrote:

I just thought I would remind all the nay-sayers that the *BSD world has
been doing something along this line since at least 1990. While some of
the programs have changed, the functionality has remained largely the
same.



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Keith Owens]

CC: trimmed to just l-k.

On Tue, 12 Sep 2000 10:02:59 +0200, 
Ralf Baechle <[EMAIL PROTECTED]> wrote:
>From some random Linux source tree's .hdepend:
>
>/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
>   /usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
>   /usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
>@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
>[...]
>
>Linux, born to be CVS worst case ...

FWIW, this will be fixed by the Makefile redesign for 2.5.  The kernel
source tree will become read only (for various reasons) so nothing will
touch .h files anymore.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Ralf Baechle]

On Mon, Sep 11, 2000 at 02:39:42PM -0700, Larry McVoy wrote:

> On the other hand, if you do a
> 
> find . -type f | xargs touch
> time cvs update .
> 
> it will melt down your DSL line for what seems forever.  I killed it after
> 20 minutes, I have better things to do with my bandwidth.   It's pretty
> clear that CVS is comparing timestamps so if your files get modified at
> all, it's going to transfer them to see what needs to be updated.  The
> same sort of "touch all, then update" operation in BK has no effect on
> performance, BK doesn't do its work that way.

>From some random Linux source tree's .hdepend:

/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
   /usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
   /usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
   $(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
[...]

Linux, born to be CVS worst case ...

  Ralf
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> No matter what you do with rsync, there is no bloody way you can even
> come close to a single 32K disk read and then a 6K over the wire transfer.
> At least, I can't think of one, can you?

rsync will transfer a bunch of file modification times and, where they
don't match, a bunch of checksums.  This is still a few bytes of state
for every file.

The obvious extension to rsync would be to make the rolling checksum
hierarchical -- successively approximating smaller and smaller regions,
and treating the whole set of files as a single data stream for the
algorithm.  "Null update" would then always reduce to transferring a
single cryptographic-level checksum -- better than 6K and still works if
you lose the Changeset file. :-)

> The fundemental observation is as the tree size/age grows, the amount of
> change you make to it stays relatively constant but the updates grow with
> tree size.  One human can only make so much change, but many can make a 
> lot.  BK takes advantage of that and does the hard work when you do hard
> work, not every time you update.

Yes that's a good thing.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Ralf Baechle]

On Mon, Sep 11, 2000 at 02:39:42PM -0700, Larry McVoy wrote:

 On the other hand, if you do a
 
 find . -type f | xargs touch
 time cvs update .
 
 it will melt down your DSL line for what seems forever.  I killed it after
 20 minutes, I have better things to do with my bandwidth.   It's pretty
 clear that CVS is comparing timestamps so if your files get modified at
 all, it's going to transfer them to see what needs to be updated.  The
 same sort of "touch all, then update" operation in BK has no effect on
 performance, BK doesn't do its work that way.

From some random Linux source tree's .hdepend:

/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
   /usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
   /usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h: \
   $(wildcard /usr/people/ralf/src/linux/linux/include/config/sgi/sn0/n/mode.h)
@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/arch.h
[...]

Linux, born to be CVS worst case ...

  Ralf
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Keith Owens]

CC: trimmed to just l-k.

On Tue, 12 Sep 2000 10:02:59 +0200, 
Ralf Baechle [EMAIL PROTECTED] wrote:
From some random Linux source tree's .hdepend:

/usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h: \
   /usr/people/ralf/src/linux/linux/include/asm/mipsregs.h \
   /usr/people/ralf/src/linux/linux/include/asm/sn/addrs.h
@touch /usr/people/ralf/src/linux/linux/include/asm/sn/sn0/ip27.h
[...]

Linux, born to be CVS worst case ...

FWIW, this will be fixed by the Makefile redesign for 2.5.  The kernel
source tree will become read only (for various reasons) so nothing will
touch .h files anymore.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Horst von Brand wrote:

> 
> mberglund <[EMAIL PROTECTED]> said:
> 
> [...]
> 
> > License:
> > This project will not be restricted to any one license. If a piece
> > of software is desirable, but under an Artistic-, BSD-, or even
> > GPL-style license, it will be used, so long as it allows
> > free(no cost) distribution.
> 
> Can't do that. The pieces that go into this (gcc, the Linux kernel, and
> minor pieces) are under licences granted by _other_ people, which you can't
> just change at your whim. And I doubt it _very_ much indeed Linus or the
> FSF will allow you to distribute their respective properties under other
> licences than the GPL.

I'm fairly sure that's not what the original poster meant.  I think the
statement was intended to mean that free software would be included on the
CD/ftp site without a Debian-like aversion to anything "free but
commercial or closed source".


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Horst von Brand]

mberglund <[EMAIL PROTECTED]> said:

[...]

> License:
> This project will not be restricted to any one license. If a piece
> of software is desirable, but under an Artistic-, BSD-, or even
> GPL-style license, it will be used, so long as it allows
> free(no cost) distribution.

Can't do that. The pieces that go into this (gcc, the Linux kernel, and
minor pieces) are under licences granted by _other_ people, which you can't
just change at your whim. And I doubt it _very_ much indeed Linus or the
FSF will allow you to distribute their respective properties under other
licences than the GPL.

Good luck in any case!
-- 
Dr. Horst H. von Brand   mailto:[EMAIL PROTECTED]
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria  +56 32 654239
Casilla 110-V, Valparaiso, ChileFax:  +56 32 797513
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Bryan O'Sullivan]

l> If you can get me a tarball of the CVS repository, I'll import the
l> history into a BK tree and then we can do side by side tests.

I know BK is the best thing since sliced yams, but can you please take
this thread to some BK mailing list and do your performance
comparisons there?

http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jonathan Lemon]

On Mon, Sep 11, 2000 at 05:05:08PM -0700, Larry McVoy wrote:
> On Mon, Sep 11, 2000 at 06:49:43PM -0500, Jonathan Lemon wrote:
> > I don't know why I'm bothering to reply to this, but yes, if you're
> > trying to synchronize CVS source trees with only CVS, it will be slow.
> > Now, if you were to compare CVSup vs Bitkeeper, then things might get
> > more interesting.
> > --
> > Jonathan
> > 
> > (for those unaware of it, CVSup is a high-speed mechanism to
> > distribute CVS repositories, and uses several algorithms including
> > "rsync" to accomplish this.)
> 
> I'd be happy to do this.  I've already gone over the details in private
> mail with Dave Miller, he was suggesting something similar and I explained
> how much disk & net I/O you have to with each case and the BK case is
> dramatically less.  
> 
> The reason is that BK does all the work when you do a local commit, it
> captures the state of the entire tree once, at the time you commit your
> changes.  CVSup, rsync, CVS, etc., all have to look at the whole tree
> because there is no fanin/fanout like BK has with the ChangeSet file.
> You can play all the games you want, but the bottom line is that you
> have to look at some version of the data, be it all the inodes, or the
> actual data.  With BK, we've distilled the state of about 9,000 files
> in the Linux tree down to about 6,000 bytes.  We have to do a single
> roughly 32KB disk I/O to get that state and then we compress to 6K and
> transfer it across the wire.
> 
> No matter what you do with rsync, there is no bloody way you can even
> come close to a single 32K disk read and then a 6K over the wire transfer.
> At least, I can't think of one, can you?
> 
> We do just as much I/O on the commit, then we walk the tree, and diff
> against the checked in version, so if you have the entire tree editted,
> we'll diff the entire tree.  But that happens when you commit your
> changes, not every time you update.
> 
> The fundemental observation is as the tree size/age grows, the amount of
> change you make to it stays relatively constant but the updates grow with
> tree size.  One human can only make so much change, but many can make a 
> lot.  BK takes advantage of that and does the hard work when you do hard
> work, not every time you update.
> 
> It's just a different design, no offense is intended against CVS, we have
> all used and learned from CVS.  But just because CVS is useful doesn't mean
> it is the best answer.

Oh, no disagreement there, CVS has quite a few shortcomings, so I'm
in no way holding it up as anything near an ideal solution, it's just
what we have now.  And yes, the cvsup daemon will take quite a bit of
I/O and memory as well.  I merely pointing out that a comparision of
cvsup vs BK would be far more interesting than native CVS alone.
--
Jonathan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 06:49:43PM -0500, Jonathan Lemon wrote:
> I don't know why I'm bothering to reply to this, but yes, if you're
> trying to synchronize CVS source trees with only CVS, it will be slow.
> Now, if you were to compare CVSup vs Bitkeeper, then things might get
> more interesting.
> --
> Jonathan
> 
> (for those unaware of it, CVSup is a high-speed mechanism to
> distribute CVS repositories, and uses several algorithms including
> "rsync" to accomplish this.)

I'd be happy to do this.  I've already gone over the details in private
mail with Dave Miller, he was suggesting something similar and I explained
how much disk & net I/O you have to with each case and the BK case is
dramatically less.  

The reason is that BK does all the work when you do a local commit, it
captures the state of the entire tree once, at the time you commit your
changes.  CVSup, rsync, CVS, etc., all have to look at the whole tree
because there is no fanin/fanout like BK has with the ChangeSet file.
You can play all the games you want, but the bottom line is that you
have to look at some version of the data, be it all the inodes, or the
actual data.  With BK, we've distilled the state of about 9,000 files
in the Linux tree down to about 6,000 bytes.  We have to do a single
roughly 32KB disk I/O to get that state and then we compress to 6K and
transfer it across the wire.

No matter what you do with rsync, there is no bloody way you can even
come close to a single 32K disk read and then a 6K over the wire transfer.
At least, I can't think of one, can you?

We do just as much I/O on the commit, then we walk the tree, and diff
against the checked in version, so if you have the entire tree editted,
we'll diff the entire tree.  But that happens when you commit your
changes, not every time you update.

The fundemental observation is as the tree size/age grows, the amount of
change you make to it stays relatively constant but the updates grow with
tree size.  One human can only make so much change, but many can make a 
lot.  BK takes advantage of that and does the hard work when you do hard
work, not every time you update.

It's just a different design, no offense is intended against CVS, we have
all used and learned from CVS.  But just because CVS is useful doesn't mean
it is the best answer.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jonathan Lemon]

In article [EMAIL PROTECTED]> you write:
>On Mon, Sep 11, 2000 at 05:48:44PM -0500, Tony Mantler wrote:
>> 
>> "It's the latency, stupid". I wouldn't care to argue whether CVS is slower
>> than BK or not, but consider that if you had a router between you and the
>> CVS server that was dropping even 5% of your packets, or even just bumping
>> the latency by a quarter second (and I've seen routers that do that. evil
>> things), the timing numbers will jump *significantly*.
>
>Well, err, I guess I'm pretty stupid, but how about this?  If the lxr guy
>is listening to all this, he could download a copy of BK, clone the FSMlabs
>tree, then I'll do a null pull from him, and we'll have an apples to apples
>comparison, now won't we?
>
>I think it's a pointless thing to do, I already know that CVS transfers way
>more data to do the same operation so it's a foregone conclusion it will be
>slower.  But I'm very happy to do it if the lxr guy (sorry, I've forgotten
>your name, I apologize) is willing.  Send me private mail, it shouldn't take
>very long at all to get you set up.

I don't know why I'm bothering to reply to this, but yes, if you're
trying to synchronize CVS source trees with only CVS, it will be slow.
Now, if you were to compare CVSup vs Bitkeeper, then things might get
more interesting.
--
Jonathan

(for those unaware of it, CVSup is a high-speed mechanism to
distribute CVS repositories, and uses several algorithms including
"rsync" to accomplish this.)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 07:44:52PM -0400, James Lewis Nance wrote:
> On Mon, Sep 11, 2000 at 03:45:18PM -0700, Larry McVoy wrote:
> 
> > the 120MB for the checked out files and some mem for inodes.  But the
> > difference in price is reasonable and if we have to buy memory for the
> > kernel developers, we'll do it once we can afford to do it.  It's _really_
> > nice to measure your operations in seconds rather than minutes.
> 
> Larry,
> It would be interesting to see the speed difference between bk and cvs
> for the mozilla sources.  Doing a cvs update takes at least 10 minutes even
> if no files have changed.  It took significantly longer when I was using
> a modem instead of a DSL line.  You haven't benchmarked this case have you?

If you can get me a tarball of the CVS repository, I'll import the
history into a BK tree and then we can do side by side tests.  I know
Mitchell Baker somewhat, so if she is still there, you might ping her.
I'd be interested to see this as well, so please let me know if the
CVS tarball is available.  Just to be clear, I am not talking about the
results of a CVS checkout, I am talking about the actual CVS tree with
the RCS files - if I just imported the most recent versions, BK would
be unfairly faster because it would be storing a lot less history.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[James Lewis Nance]

On Mon, Sep 11, 2000 at 03:45:18PM -0700, Larry McVoy wrote:

> the 120MB for the checked out files and some mem for inodes.  But the
> difference in price is reasonable and if we have to buy memory for the
> kernel developers, we'll do it once we can afford to do it.  It's _really_
> nice to measure your operations in seconds rather than minutes.

Larry,
It would be interesting to see the speed difference between bk and cvs
for the mozilla sources.  Doing a cvs update takes at least 10 minutes even
if no files have changed.  It took significantly longer when I was using
a modem instead of a DSL line.  You haven't benchmarked this case have you?

Thanks,

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 05:48:44PM -0500, Tony Mantler wrote:
> 
> "It's the latency, stupid". I wouldn't care to argue whether CVS is slower
> than BK or not, but consider that if you had a router between you and the
> CVS server that was dropping even 5% of your packets, or even just bumping
> the latency by a quarter second (and I've seen routers that do that. evil
> things), the timing numbers will jump *significantly*.

Well, err, I guess I'm pretty stupid, but how about this?  If the lxr guy
is listening to all this, he could download a copy of BK, clone the FSMlabs
tree, then I'll do a null pull from him, and we'll have an apples to apples
comparison, now won't we?

I think it's a pointless thing to do, I already know that CVS transfers way
more data to do the same operation so it's a foregone conclusion it will be
slower.  But I'm very happy to do it if the lxr guy (sorry, I've forgotten
your name, I apologize) is willing.  Send me private mail, it shouldn't take
very long at all to get you set up.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

Note: trimmed the 390 list, they don't care according to Alan..

On Tue, Sep 12, 2000 at 12:21:16AM +0200, Jamie Lokier wrote:
> Larry McVoy wrote:
> > That's a benefit [for BK] of having changesets, I only need to compare
> > the ChangeSet file to know that 4 files were updated 2 were moved, and
> > 5 were created, then I move those *portions* of those files across the
> > wire.
> 
> What happens when I lose the ChangeSet file, or misplace it?

Life really sucks.  It's like losing a superblock, sort of.  We can 
reconstruct them but almost never do because people typically have 
more than once copy of a repository sitting around.  So you can copy
it back, do a "bk -r check -a" (the moral equiv of an fsck, no we don't
really have an fsck -y yet), and fix up the errors.  The revision control
files and the ChangeSet files point at each other and need to be in sync.

That's how we get all the performance, by the way, all we need to compare
is a tiny subset of the ChangeSet files (32KB out of 5MB in one tree, that's
pretty typical).  And we could compare a lot less, it just hasn't been
worth it to do so, we gzip that 32K down to less than 6K so...)
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Tony Mantler]

At 5:13 PM -0500 9/11/2000, Larry McVoy wrote:
[...]
>> > > > over a 384Kbits/sec link.
>
>That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,
>the bandwidth to FSMlabs.com and innominate.org seems to be identical,
>I suspect that my link is the bottleneck, not either of theirs.
>In order for the link to be the reason for the performance difference,
>innominate.org would need to be a .9Kbyte/second link.  I kinda doubt
>that seeing as a 128MB cvs checkout took 23 minutes (works out to around
>90Kbyte/sec uncompressed, so cvs must compress it, so I'd guesstimate
>they have around a 30Kbyte/sec link or so).
[...]

"It's the latency, stupid". I wouldn't care to argue whether CVS is slower
than BK or not, but consider that if you had a router between you and the
CVS server that was dropping even 5% of your packets, or even just bumping
the latency by a quarter second (and I've seen routers that do that. evil
things), the timing numbers will jump *significantly*.

The best way to test network performance between the two protocols would be
to get yourself a good ol'fasioned serial cable and connect your test
client and test server to eachother via PPP at about ~9600bps or so, *then*
do your tests. Equal (though low) latency, equal (definatley low)
bandwidth, equal server and client performance.

Of course, all the CVS work I do is either over local 10/100 ethernet or
through my 6d/1uMbit cablemodem (which actually gets 6d/1u, up here in the
great white north), so what do I care? 8)


Cheers - Tony 'Nicoya' Mantler :)


--
Tony "Nicoya" Mantler - Renaissance Nerd Extraordinaire - [EMAIL PROTECTED]
Winnipeg, Manitoba, Canada   --   http://nicoya.feline.pp.se/


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

David A. Gatwood wrote:
> > I'd love to see a filesystem feature where I could efficiently identify
> > "changed files", where "changed" is defined by last time this application
> > checked or something similar.
> 
> An in-filesystem revision number would do the trick.  Could be REALLY
> efficient.  All you have to do is stat and you know if the thing really
> changed, instead of just knowing a date that can be horked.  I like!  I
> like a lot!

Yes indeed.  There aren't many bits to play with in an ext2 inode so you
have to be careful, but it can be done.  Especially when you combine it
with the timestamp (there's no need to update a revision number if the
time changes).  I think this can even run over NFS (as a dubious
extension in the cookie or something ;-)

> Something else that'd be nice would be to make Linux deal better with NFS
> dates.  The date presented by an NFS server for files is its timestamp. 
> If the linux NFS client code were to subsequently check the stamp after
> performing an operation, it could use that to approximate the time error
> and this problem would about 99% disappear, with a fudge factor of under a
> second.  Comments?

AFS does something like that.  ntpd is better :-)

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

> > Fourth, non-null pulls are similarly faster, again, by design.  BK only
> > moves the data it needs to move.  That means if you have a 100GB file
> > in which you have changed one byte, BK will move on the order of 1 byte
> > to update that file.  And that's it - it doesn't compare the two files,
> > or read the two files, or in any way look at the two files to figure out
> > that they need to be updated.
> 
> This is an interesting point.  As far as changes are concerned, it should
> have lower network utilization.  However, in effect, it does compare the
> files.  It compared the file when you checked it into your local
> repository.  It's just shifting the burden of the comparison to a local
> repository instead of doing it over the network.  Faster, yes, but it
> still does the same thing, just in a different way that uses less
> bandwidth.

You just summed up one of the main reasons why BK is better.  We shifted
work to the client.  There is one server and many, many clients in a 
CVS tree.  The same problem solved using BK inherently distributes the
load over all the machines, moving 99% of to the clients and off loading
the server (and hence the network) almost completely.

BitKeeper isn't any faster than CVS when you do the same operation, or at
least not much faster.  If we both have 10,000 files we have to update,
we're both going to be doing a lot of work.  In fact, I suspect that CVS
is faster at that than BK is because it is just updating a clear text file
and we are updating a revision history - we have more work to do so it 
stands to reason we'd be slower.

BitKeeper is faster in practice because it isn't trying to talk to the
CVS server for all of the operations.  By far the vast majority of the
operations never reach out across the network.  So on a slow client,
BK is slow, on a fast client, BK is fast.  By contrast, with a slow CVS
server, both a slow and a fast client are slow.

So perhaps a better way to look at it is that if you can afford decent
hardware, your life should be quite pleasant using BK.  Our definition
of decent, for Linux kernel repositories, is a 466Mhz celeron with at
least 256MB, and you really want 384.  The kernel is about around 120MB
checked out, the revision history is about 160MB, and you need some mem
for the inodes.  CVS is lighter weight in that respect, it just needs
the 120MB for the checked out files and some mem for inodes.  But the
difference in price is reasonable and if we have to buy memory for the
kernel developers, we'll do it once we can afford to do it.  It's _really_
nice to measure your operations in seconds rather than minutes.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> We thought about this too (filesystems are where I got into kernel hacking),
> but dismissed it as a Linux only solution.  As much as I'd like it to be
> otherwise, BK is not a Linux only product.  Whatever we do needs to work on
> NT (shudder) as well as all the dinosaur Unixen.

NT has a facility to monitor changing files.  You could run a daemon :-)

> We do have more to work with because we have both the revision history and
> the checked out file in each work area.  So we can play games with those
> to try and simulate the "what's changed since" operator.  It's NFS which
> screws that up.  

Not just NFS.  It's me when I back-touch files to control Make (Emacs
backtouch-mode ;-), or rename files, or my CMOS clock breaks, or I have
to do an fsck.  The last applies especially to kernel development on one
machine...

Just checking 100GB of files _locally_ is pretty tedious.

Heck, just calling _stat_ on a typical source tree can be immensely time
consuming over NFS.  Like, a minute.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Tue, 12 Sep 2000, Jamie Lokier wrote:

> I'd love to see a filesystem feature where I could efficiently identify
> "changed files", where "changed" is defined by last time this application
> checked or something similar.

An in-filesystem revision number would do the trick.  Could be REALLY
efficient.  All you have to do is stat and you know if the thing really
changed, instead of just knowing a date that can be horked.  I like!  I
like a lot!

Something else that'd be nice would be to make Linux deal better with NFS
dates.  The date presented by an NFS server for files is its timestamp. 
If the linux NFS client code were to subsequently check the stamp after
performing an operation, it could use that to approximate the time error
and this problem would about 99% disappear, with a fudge factor of under a
second.  Comments?


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Larry McVoy wrote:

> That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,

I meant FSMlabs, but yeah.  ;-)


> Second of all, if you ask around, you'll find that I'm a performance guy
> more than anything and that I'm not given to skewing the numbers and *if*
> that was your point, I resent the implication.

No, wasn't my point.  Sorry if it came out that way.  My point was for
anyone reading not to jump to conclusions based on just those numbers, and
that I'd like to see some real-world tests on comparable systems.


> Fourth, non-null pulls are similarly faster, again, by design.  BK only
> moves the data it needs to move.  That means if you have a 100GB file
> in which you have changed one byte, BK will move on the order of 1 byte
> to update that file.  And that's it - it doesn't compare the two files,
> or read the two files, or in any way look at the two files to figure out
> that they need to be updated.

This is an interesting point.  As far as changes are concerned, it should
have lower network utilization.  However, in effect, it does compare the
files.  It compared the file when you checked it into your local
repository.  It's just shifting the burden of the comparison to a local
repository instead of doing it over the network.  Faster, yes, but it
still does the same thing, just in a different way that uses less
bandwidth.


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Tue, Sep 12, 2000 at 12:24:26AM +0200, Jamie Lokier wrote:
> Larry McVoy wrote:
> > We have a hack in BK for this, at least I think we do, where we can look at
> > the time stamps to notice that you haven't modified the files.  We don't 
> > use it because of NFS screwing up timestamps.  I suppose we could enable
> > it on a per repository basis so that if you knew you were running NTP or
> > some other thing to keep your time stamps right, then we could diff as
> > fast as we can stat.  
> 
> I'd love to see a filesystem feature where I could efficiently identify
> "changed files", where "changed" is defined by last time this application
> checked or something similar.  A bonus would be to propagate this up
> directory trees.  Yes I know about hard links, and it's not necessary to
> propagate up trees in those cases.  (The application can just remember
> paths of directories containing hard links, and not depend on the
> per-directory bits for those paths).

We thought about this too (filesystems are where I got into kernel hacking),
but dismissed it as a Linux only solution.  As much as I'd like it to be
otherwise, BK is not a Linux only product.  Whatever we do needs to work on
NT (shudder) as well as all the dinosaur Unixen.

We do have more to work with because we have both the revision history and
the checked out file in each work area.  So we can play games with those
to try and simulate the "what's changed since" operator.  It's NFS which
screws that up.  
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> We have a hack in BK for this, at least I think we do, where we can look at
> the time stamps to notice that you haven't modified the files.  We don't 
> use it because of NFS screwing up timestamps.  I suppose we could enable
> it on a per repository basis so that if you knew you were running NTP or
> some other thing to keep your time stamps right, then we could diff as
> fast as we can stat.  

I'd love to see a filesystem feature where I could efficiently identify
"changed files", where "changed" is defined by last time this application
checked or something similar.  A bonus would be to propagate this up
directory trees.  Yes I know about hard links, and it's not necessary to
propagate up trees in those cases.  (The application can just remember
paths of directories containing hard links, and not depend on the
per-directory bits for those paths).

There are ways to do it but not much enthusiasm last time I brought it
up.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> That's a benefit [for BK] of having changesets, I only need to compare
> the ChangeSet file to know that 4 files were updated 2 were moved, and
> 5 were created, then I move those *portions* of those files across the
> wire.

What happens when I lose the ChangeSet file, or misplace it?

> Other than the initial repository create (aka cvs checkout), BK
> *never* moves an entire file across the wire.  Never means never and
> includes the process of deciding what to do.  CVS moves whole files
> just to discover there is nothing to do.

Yes, CVS over rsync is much better for that.  IMHO the ideas underlying
the rsync protocol are most cool.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Tue, Sep 12, 2000 at 12:09:29AM +0200, Juan J. Quintela wrote:
> > "david" == David A Gatwood <[EMAIL PROTECTED]> writes:
> But I think that the null update is a "typical" usage, and more
> typical indeed a cvs diff (and how that it is spelled in bk).  I want
> to be able to use cvs diff for a whole tree, when I have changed only
> 2-5 files to be very fast, (i.e. speed of diff -r between hardlinkend
> trees).

We have a hack in BK for this, at least I think we do, where we can look at
the time stamps to notice that you haven't modified the files.  We don't 
use it because of NFS screwing up timestamps.  I suppose we could enable
it on a per repository basis so that if you knew you were running NTP or
some other thing to keep your time stamps right, then we could diff as
fast as we can stat.  
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 02:58:25PM -0700, David A. Gatwood wrote:
> > On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
> > > > Err, "faster"?  The following is the moral equiv of 4 kernel updates
> > > > which had nothing to do using BitKeeper instead of CVS.  The local copy
> > > > was in San Francisco and the remote copy is Cort's machine in New Mexico
> > > > over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> > > > CVS tree they can try to get comparable numbers?
> > >
> > > Try: http://innominate.org/~tgr/projects/lksr/
> >
> > Thanks, that was helpful.  Comparison numbers for a null update of the
> > 2.3 kernel, which means you update and then update again, timing the
> > second update to get some idea of the system's best case throughput,
> > are:
> >
> > CVS: 139.5 seconds
> > BK:1.6 seconds
> >
> > The BK tree is the 2.3 kernel tree maintained by FSMlabs.
> 
> All right, this is ridiculous.  Those statistics are worthless.  The BK
> tree was on a horribly fast network and the CVS repository was on a
> horribly slow one.  There's just no way that BK is that much faster than
> CVS in a comparable environment.
> 
> I'm sorry, but I can't believe those numbers.  It takes longer than 1.6
> seconds to stat all the kernel directories unless the BK machine has a
> huge disk cache.  

Calm down, David.  BK really is pretty fast.  Here's a point by point 
rebuttal to your mail.

First of all, I don't think you read the mail.  Let me quote:

> > > > over a 384Kbits/sec link.  

That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,
the bandwidth to FSMlabs.com and innominate.org seems to be identical,
I suspect that my link is the bottleneck, not either of theirs.
In order for the link to be the reason for the performance difference,
innominate.org would need to be a .9Kbyte/second link.  I kinda doubt
that seeing as a 128MB cvs checkout took 23 minutes (works out to around
90Kbyte/sec uncompressed, so cvs must compress it, so I'd guesstimate
they have around a 30Kbyte/sec link or so).

Second of all, if you ask around, you'll find that I'm a performance guy
more than anything and that I'm not given to skewing the numbers and *if*
that was your point, I resent the implication.

Third, BK is designed to be that much faster.  It's inherent in the
nature of a distributed repositories with changesets.  I can, and did,
reboot the machine, did the test again, and it took an average of 1.56
seconds per tree to do a null pull.

Fourth, non-null pulls are similarly faster, again, by design.  BK only
moves the data it needs to move.  That means if you have a 100GB file
in which you have changed one byte, BK will move on the order of 1 byte
to update that file.  And that's it - it doesn't compare the two files,
or read the two files, or in any way look at the two files to figure out
that they need to be updated.  It knows.  That's a benefit of having 
changesets, I only need to compare the ChangeSet file to know that 4 files
were updated 2 were moved, and 5 were created, then I move those *portions*
of those files across the wire.  Other than the initial repository create
(aka cvs checkout), BK *never* moves an entire file across the wire.
Never means never and includes the process of deciding what to do.  CVS
moves whole files just to discover there is nothing to do.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Juan J. Quintela]

> "david" == David A Gatwood <[EMAIL PROTECTED]> writes:

Hi
[stuff about unfair test]

I don't arguee if the test was fair or not.

david> and does not include a "null update", as that is an atypical usage pattern
david> for most trees that unfairly skews the test towards software or kernels
david> that does caching of file stat results, which has little bearing on
david> typical use.

But I think that the null update is a "typical" usage, and more
typical indeed a cvs diff (and how that it is spelled in bk).  I want
to be able to use cvs diff for a whole tree, when I have changed only
2-5 files to be very fast, (i.e. speed of diff -r between hardlinkend
trees).

Later, Juan.

-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> On the other hand, if you do a
> 
> find . -type f | xargs touch
> time cvs update .
> 
> it will melt down your DSL line for what seems forever.  I killed it after
> 20 minutes, I have better things to do with my bandwidth.   It's pretty
> clear that CVS is comparing timestamps so if your files get modified at
> all, it's going to transfer them to see what needs to be updated.  The
> same sort of "touch all, then update" operation in BK has no effect on
> performance, BK doesn't do its work that way.

The recommended way to do CVS in such situations is rsync your local
tree with somewhere closer to the CVS server.  Of course it's obvious
CVS should have something rsync-like built in.
I bet BK does exactly that :-)

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Larry McVoy wrote:

> 
> On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
> > > Err, "faster"?  The following is the moral equiv of 4 kernel updates
> > > which had nothing to do using BitKeeper instead of CVS.  The local copy
> > > was in San Francisco and the remote copy is Cort's machine in New Mexico
> > > over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> > > CVS tree they can try to get comparable numbers?
> >
> > Try: http://innominate.org/~tgr/projects/lksr/
> 
> Thanks, that was helpful.  Comparison numbers for a null update of the
> 2.3 kernel, which means you update and then update again, timing the
> second update to get some idea of the system's best case throughput,
> are: 
> 
> CVS: 139.5 seconds
> BK:1.6 seconds
> 
> The BK tree is the 2.3 kernel tree maintained by FSMlabs.

All right, this is ridiculous.  Those statistics are worthless.  The BK
tree was on a horribly fast network and the CVS repository was on a
horribly slow one.  There's just no way that BK is that much faster than
CVS in a comparable environment.

I'm sorry, but I can't believe those numbers.  It takes longer than 1.6
seconds to stat all the kernel directories unless the BK machine has a
huge disk cache.  It sounds like the BK server was a much more powerful
machine.

Also, the innominate server is WAY out.  It took _25_ hops to get to that
server from here.  It only took 11 to fsmlabs.  Since bitmover is in San
Jose as I am (or at least bitmover.com's server is), I suspect you will
see a similarly dramatic difference in the distance.  That's just not a
valid test by any measure.

I'd like someone to do a valid test of these two systems, as I'm curious
myself as to which is faster.  A valid test includes the following things:

1.  Raw checkout of a full tree
2.  Raw checkin of a full tree

and does not include a "null update", as that is an atypical usage pattern
for most trees that unfairly skews the test towards software or kernels
that does caching of file stat results, which has little bearing on
typical use.

Critically, the test should be with two otherwise identically configured
servers and two identically configured clients, across identical networks.
Only then do the tests mean anything.  I'd be very curious to see the
results.  :-)


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 02:08:42PM -0700, Larry McVoy wrote:
> 
> On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
> > > Err, "faster"?  The following is the moral equiv of 4 kernel updates
> > > which had nothing to do using BitKeeper instead of CVS.  The local copy
> > > was in San Francisco and the remote copy is Cort's machine in New Mexico
> > > over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> > > CVS tree they can try to get comparable numbers?
> >
> > Try: http://innominate.org/~tgr/projects/lksr/
> 
> Thanks, that was helpful.  Comparison numbers for a null update of the 2.3
> kernel, which means you update and then update again, timing the second update
> to get some idea of the system's best case throughput, are:
> 
> CVS: 139.5 seconds
> BK:1.6 seconds

In fairness to CVS, I ran this a third time and got 78 seconds, so the net
must have been busy.  It's still about a 50:1 performance difference.

On the other hand, if you do a

find . -type f | xargs touch
time cvs update .

it will melt down your DSL line for what seems forever.  I killed it after
20 minutes, I have better things to do with my bandwidth.   It's pretty
clear that CVS is comparing timestamps so if your files get modified at
all, it's going to transfer them to see what needs to be updated.  The
same sort of "touch all, then update" operation in BK has no effect on
performance, BK doesn't do its work that way.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
> > Err, "faster"?  The following is the moral equiv of 4 kernel updates
> > which had nothing to do using BitKeeper instead of CVS.  The local copy
> > was in San Francisco and the remote copy is Cort's machine in New Mexico
> > over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> > CVS tree they can try to get comparable numbers?
> 
> Try: http://innominate.org/~tgr/projects/lksr/

Thanks, that was helpful.  Comparison numbers for a null update of the 2.3
kernel, which means you update and then update again, timing the second update
to get some idea of the system's best case throughput, are:

CVS: 139.5 seconds
BK:1.6 seconds

The BK tree is the 2.3 kernel tree maintained by FSMlabs.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
> > Development:
> > In addition, by maintaining the system in CVS we can offer much
> > faster and convenient source updates than are currently available
> > from other Linux-based systems currently available.
> 
> Err, "faster"?  The following is the moral equiv of 4 kernel updates
> which had nothing to do using BitKeeper instead of CVS.  The local copy
> was in San Francisco and the remote copy is Cort's machine in New Mexico
> over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> CVS tree they can try to get comparable numbers?

Try: http://innominate.org/~tgr/projects/lksr/

Actually rather handy.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Troy Benjegerdes]

On Mon, 11 Sep 2000, Larry McVoy wrote:

> 
> On Mon, Sep 11, 2000 at 01:13:56PM -0400, mberglund wrote:
> > Name:
> > Darkstar - an integrated operating system based on the Linux kernel
> > and a stable set of tools.
> [...]
> > Development:
> > In addition, by maintaining the system in CVS we can offer much
> > faster and convenient source updates than are currently available
> > from other Linux-based systems currently available.
> 
> Err, "faster"?  The following is the moral equiv of 4 kernel updates
> which had nothing to do using BitKeeper instead of CVS.  The local copy
> was in San Francisco and the remote copy is Cort's machine in New Mexico
> over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
> CVS tree they can try to get comparable numbers?
> 
> At the risk of being slashdot-ed, you can browse these trees with BK's
> replacement for CVSweb at http://www.bitkeeper.com/bkweb

Cool... So can the latest version of the sources for BitKeeper be checked
out too, or do we just have to write a script to extract it from the BkWeb
changesets? :P

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 01:13:56PM -0400, mberglund wrote:
> Name:
> Darkstar - an integrated operating system based on the Linux kernel
> and a stable set of tools.
[...]
> Development:
> In addition, by maintaining the system in CVS we can offer much
> faster and convenient source updates than are currently available
> from other Linux-based systems currently available.

Err, "faster"?  The following is the moral equiv of 4 kernel updates
which had nothing to do using BitKeeper instead of CVS.  The local copy
was in San Francisco and the remote copy is Cort's machine in New Mexico
over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
CVS tree they can try to get comparable numbers?

At the risk of being slashdot-ed, you can browse these trees with BK's
replacement for CVSweb at http://www.bitkeeper.com/bkweb

$ for i in *
> do (cd $i; echo === $i ===; time bk pull )
> done
=== linux_2_2 ===
Nothing to pull.
0.02user 0.02system 0:01.04elapsed 3%CPU (0avgtext+0avgdata 0maxresident)k
=== linux_2_3 ===
Nothing to pull.
0.07user 0.03system 0:01.15elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
=== linuxppc_2_2 ===
Nothing to pull.
0.02user 0.05system 0:01.20elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
=== linuxppc_2_3 ===
Nothing to pull.
0.09user 0.01system 0:01.60elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k

-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

[ANNOUNCE] Darkstar Development Project

[mberglund]

Name:
Darkstar - an integrated operating system based on the Linux kernel
and a stable set of tools.

Purpose:
To build a source based operating system that ties together the
Linux kernel and a stable set of standard tools.  Essentially this
should bring together all of the tools required to build and use a
system that is consistent on multiple hardware platforms from
PDA's to Mainframes.

License:
This project will not be restricted to any one license. If a piece
of software is desirable, but under an Artistic-, BSD-, or even
GPL-style license, it will be used, so long as it allows
free(no cost) distribution.

Development:
Similar to the BSD development model, the main body of developers
(committers) will use CVS to make/track changes to the system. This
allows for centralized development by the team of developers
emphasizing peer review, code review, general correctness, and an
audit trail for future reference.
Commit privileges are given on a case-by-case basis to competent
people who continually provide code, fixes, and have a clue.  
In addition, by maintaining the system in CVS we can offer much
faster and convenient source updates than are currently available
from other Linux-based systems currently available.

Wanted:
We are seeking system administrators, kernel developers, and
userland contributors to help with the development and maintenance
of the system. Knowledge of CVS and make(8) is recommended, but not
absolutely required.

In addition, web-design and documentation people would be useful.

Status:
We currently have the build-system operational and building select
pieces of software and have been imported into the tree, though
development has stalled due to a lack of developers.  The current
CVS tree is available for viewing at:
.

Contact:
If you are interested in contributing, please email and/or subscribe
to <[EMAIL PROTECTED]>, the primary development mailing list.
In addition, the primary developers can be reached at:
Chris D. Faulhaber  <[EMAIL PROTECTED]>
Matt Berglund   <[EMAIL PROTECTED]>

We look forward to your interest,
Matt Berglund
<[EMAIL PROTECTED]>


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

[ANNOUNCE] Darkstar Development Project

[mberglund]

Name:
Darkstar - an integrated operating system based on the Linux kernel
and a stable set of tools.

Purpose:
To build a source based operating system that ties together the
Linux kernel and a stable set of standard tools.  Essentially this
should bring together all of the tools required to build and use a
system that is consistent on multiple hardware platforms from
PDA's to Mainframes.

License:
This project will not be restricted to any one license. If a piece
of software is desirable, but under an Artistic-, BSD-, or even
GPL-style license, it will be used, so long as it allows
free(no cost) distribution.

Development:
Similar to the BSD development model, the main body of developers
(committers) will use CVS to make/track changes to the system. This
allows for centralized development by the team of developers
emphasizing peer review, code review, general correctness, and an
audit trail for future reference.
Commit privileges are given on a case-by-case basis to competent
people who continually provide code, fixes, and have a clue.  
In addition, by maintaining the system in CVS we can offer much
faster and convenient source updates than are currently available
from other Linux-based systems currently available.

Wanted:
We are seeking system administrators, kernel developers, and
userland contributors to help with the development and maintenance
of the system. Knowledge of CVS and make(8) is recommended, but not
absolutely required.

In addition, web-design and documentation people would be useful.

Status:
We currently have the build-system operational and building select
pieces of software and have been imported into the tree, though
development has stalled due to a lack of developers.  The current
CVS tree is available for viewing at:
http://darkstar.fxp.org/cgi-bin/cvsweb.cgi/.

Contact:
If you are interested in contributing, please email and/or subscribe
to [EMAIL PROTECTED], the primary development mailing list.
In addition, the primary developers can be reached at:
Chris D. Faulhaber  [EMAIL PROTECTED]
Matt Berglund   [EMAIL PROTECTED]

We look forward to your interest,
Matt Berglund
[EMAIL PROTECTED]


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 01:13:56PM -0400, mberglund wrote:
 Name:
 Darkstar - an integrated operating system based on the Linux kernel
 and a stable set of tools.
[...]
 Development:
 In addition, by maintaining the system in CVS we can offer much
 faster and convenient source updates than are currently available
 from other Linux-based systems currently available.

Err, "faster"?  The following is the moral equiv of 4 kernel updates
which had nothing to do using BitKeeper instead of CVS.  The local copy
was in San Francisco and the remote copy is Cort's machine in New Mexico
over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
CVS tree they can try to get comparable numbers?

At the risk of being slashdot-ed, you can browse these trees with BK's
replacement for CVSweb at http://www.bitkeeper.com/bkweb

$ for i in *
 do (cd $i; echo === $i ===; time bk pull )
 done
=== linux_2_2 ===
Nothing to pull.
0.02user 0.02system 0:01.04elapsed 3%CPU (0avgtext+0avgdata 0maxresident)k
=== linux_2_3 ===
Nothing to pull.
0.07user 0.03system 0:01.15elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
=== linuxppc_2_2 ===
Nothing to pull.
0.02user 0.05system 0:01.20elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
=== linuxppc_2_3 ===
Nothing to pull.
0.09user 0.01system 0:01.60elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k

-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
  Development:
  In addition, by maintaining the system in CVS we can offer much
  faster and convenient source updates than are currently available
  from other Linux-based systems currently available.
 
 Err, "faster"?  The following is the moral equiv of 4 kernel updates
 which had nothing to do using BitKeeper instead of CVS.  The local copy
 was in San Francisco and the remote copy is Cort's machine in New Mexico
 over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
 CVS tree they can try to get comparable numbers?

Try: http://innominate.org/~tgr/projects/lksr/

Actually rather handy.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 02:08:42PM -0700, Larry McVoy wrote:
 
 On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
   Err, "faster"?  The following is the moral equiv of 4 kernel updates
   which had nothing to do using BitKeeper instead of CVS.  The local copy
   was in San Francisco and the remote copy is Cort's machine in New Mexico
   over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
   CVS tree they can try to get comparable numbers?
 
  Try: http://innominate.org/~tgr/projects/lksr/
 
 Thanks, that was helpful.  Comparison numbers for a null update of the 2.3
 kernel, which means you update and then update again, timing the second update
 to get some idea of the system's best case throughput, are:
 
 CVS: 139.5 seconds
 BK:1.6 seconds

In fairness to CVS, I ran this a third time and got 78 seconds, so the net
must have been busy.  It's still about a 50:1 performance difference.

On the other hand, if you do a

find . -type f | xargs touch
time cvs update .

it will melt down your DSL line for what seems forever.  I killed it after
20 minutes, I have better things to do with my bandwidth.   It's pretty
clear that CVS is comparing timestamps so if your files get modified at
all, it's going to transfer them to see what needs to be updated.  The
same sort of "touch all, then update" operation in BK has no effect on
performance, BK doesn't do its work that way.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Larry McVoy wrote:

 
 On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
   Err, "faster"?  The following is the moral equiv of 4 kernel updates
   which had nothing to do using BitKeeper instead of CVS.  The local copy
   was in San Francisco and the remote copy is Cort's machine in New Mexico
   over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
   CVS tree they can try to get comparable numbers?
 
  Try: http://innominate.org/~tgr/projects/lksr/
 
 Thanks, that was helpful.  Comparison numbers for a null update of the
 2.3 kernel, which means you update and then update again, timing the
 second update to get some idea of the system's best case throughput,
 are: 
 
 CVS: 139.5 seconds
 BK:1.6 seconds
 
 The BK tree is the 2.3 kernel tree maintained by FSMlabs.

All right, this is ridiculous.  Those statistics are worthless.  The BK
tree was on a horribly fast network and the CVS repository was on a
horribly slow one.  There's just no way that BK is that much faster than
CVS in a comparable environment.

I'm sorry, but I can't believe those numbers.  It takes longer than 1.6
seconds to stat all the kernel directories unless the BK machine has a
huge disk cache.  It sounds like the BK server was a much more powerful
machine.

Also, the innominate server is WAY out.  It took _25_ hops to get to that
server from here.  It only took 11 to fsmlabs.  Since bitmover is in San
Jose as I am (or at least bitmover.com's server is), I suspect you will
see a similarly dramatic difference in the distance.  That's just not a
valid test by any measure.

I'd like someone to do a valid test of these two systems, as I'm curious
myself as to which is faster.  A valid test includes the following things:

1.  Raw checkout of a full tree
2.  Raw checkin of a full tree

and does not include a "null update", as that is an atypical usage pattern
for most trees that unfairly skews the test towards software or kernels
that does caching of file stat results, which has little bearing on
typical use.

Critically, the test should be with two otherwise identically configured
servers and two identically configured clients, across identical networks.
Only then do the tests mean anything.  I'd be very curious to see the
results.  :-)


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
 On the other hand, if you do a
 
 find . -type f | xargs touch
 time cvs update .
 
 it will melt down your DSL line for what seems forever.  I killed it after
 20 minutes, I have better things to do with my bandwidth.   It's pretty
 clear that CVS is comparing timestamps so if your files get modified at
 all, it's going to transfer them to see what needs to be updated.  The
 same sort of "touch all, then update" operation in BK has no effect on
 performance, BK doesn't do its work that way.

The recommended way to do CVS in such situations is rsync your local
tree with somewhere closer to the CVS server.  Of course it's obvious
CVS should have something rsync-like built in.
I bet BK does exactly that :-)

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Juan J. Quintela]

 "david" == David A Gatwood [EMAIL PROTECTED] writes:

Hi
[stuff about unfair test]

I don't arguee if the test was fair or not.

david and does not include a "null update", as that is an atypical usage pattern
david for most trees that unfairly skews the test towards software or kernels
david that does caching of file stat results, which has little bearing on
david typical use.

But I think that the null update is a "typical" usage, and more
typical indeed a cvs diff (and how that it is spelled in bk).  I want
to be able to use cvs diff for a whole tree, when I have changed only
2-5 files to be very fast, (i.e. speed of diff -r between hardlinkend
trees).

Later, Juan.

-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 02:58:25PM -0700, David A. Gatwood wrote:
  On Mon, Sep 11, 2000 at 09:55:01PM +0200, Jamie Lokier wrote:
Err, "faster"?  The following is the moral equiv of 4 kernel updates
which had nothing to do using BitKeeper instead of CVS.  The local copy
was in San Francisco and the remote copy is Cort's machine in New Mexico
over a 384Kbits/sec link.  All 4 updates in 5 seconds.  Anyone have a
CVS tree they can try to get comparable numbers?
  
   Try: http://innominate.org/~tgr/projects/lksr/
 
  Thanks, that was helpful.  Comparison numbers for a null update of the
  2.3 kernel, which means you update and then update again, timing the
  second update to get some idea of the system's best case throughput,
  are:
 
  CVS: 139.5 seconds
  BK:1.6 seconds
 
  The BK tree is the 2.3 kernel tree maintained by FSMlabs.
 
 All right, this is ridiculous.  Those statistics are worthless.  The BK
 tree was on a horribly fast network and the CVS repository was on a
 horribly slow one.  There's just no way that BK is that much faster than
 CVS in a comparable environment.
 
 I'm sorry, but I can't believe those numbers.  It takes longer than 1.6
 seconds to stat all the kernel directories unless the BK machine has a
 huge disk cache.  

Calm down, David.  BK really is pretty fast.  Here's a point by point 
rebuttal to your mail.

First of all, I don't think you read the mail.  Let me quote:

over a 384Kbits/sec link.  

That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,
the bandwidth to FSMlabs.com and innominate.org seems to be identical,
I suspect that my link is the bottleneck, not either of theirs.
In order for the link to be the reason for the performance difference,
innominate.org would need to be a .9Kbyte/second link.  I kinda doubt
that seeing as a 128MB cvs checkout took 23 minutes (works out to around
90Kbyte/sec uncompressed, so cvs must compress it, so I'd guesstimate
they have around a 30Kbyte/sec link or so).

Second of all, if you ask around, you'll find that I'm a performance guy
more than anything and that I'm not given to skewing the numbers and *if*
that was your point, I resent the implication.

Third, BK is designed to be that much faster.  It's inherent in the
nature of a distributed repositories with changesets.  I can, and did,
reboot the machine, did the test again, and it took an average of 1.56
seconds per tree to do a null pull.

Fourth, non-null pulls are similarly faster, again, by design.  BK only
moves the data it needs to move.  That means if you have a 100GB file
in which you have changed one byte, BK will move on the order of 1 byte
to update that file.  And that's it - it doesn't compare the two files,
or read the two files, or in any way look at the two files to figure out
that they need to be updated.  It knows.  That's a benefit of having 
changesets, I only need to compare the ChangeSet file to know that 4 files
were updated 2 were moved, and 5 were created, then I move those *portions*
of those files across the wire.  Other than the initial repository create
(aka cvs checkout), BK *never* moves an entire file across the wire.
Never means never and includes the process of deciding what to do.  CVS
moves whole files just to discover there is nothing to do.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Tue, Sep 12, 2000 at 12:09:29AM +0200, Juan J. Quintela wrote:
  "david" == David A Gatwood [EMAIL PROTECTED] writes:
 But I think that the null update is a "typical" usage, and more
 typical indeed a cvs diff (and how that it is spelled in bk).  I want
 to be able to use cvs diff for a whole tree, when I have changed only
 2-5 files to be very fast, (i.e. speed of diff -r between hardlinkend
 trees).

We have a hack in BK for this, at least I think we do, where we can look at
the time stamps to notice that you haven't modified the files.  We don't 
use it because of NFS screwing up timestamps.  I suppose we could enable
it on a per repository basis so that if you knew you were running NTP or
some other thing to keep your time stamps right, then we could diff as
fast as we can stat.  
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
 That's a benefit [for BK] of having changesets, I only need to compare
 the ChangeSet file to know that 4 files were updated 2 were moved, and
 5 were created, then I move those *portions* of those files across the
 wire.

What happens when I lose the ChangeSet file, or misplace it?

 Other than the initial repository create (aka cvs checkout), BK
 *never* moves an entire file across the wire.  Never means never and
 includes the process of deciding what to do.  CVS moves whole files
 just to discover there is nothing to do.

Yes, CVS over rsync is much better for that.  IMHO the ideas underlying
the rsync protocol are most cool.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

Larry McVoy wrote:
 We have a hack in BK for this, at least I think we do, where we can look at
 the time stamps to notice that you haven't modified the files.  We don't 
 use it because of NFS screwing up timestamps.  I suppose we could enable
 it on a per repository basis so that if you knew you were running NTP or
 some other thing to keep your time stamps right, then we could diff as
 fast as we can stat.  

I'd love to see a filesystem feature where I could efficiently identify
"changed files", where "changed" is defined by last time this application
checked or something similar.  A bonus would be to propagate this up
directory trees.  Yes I know about hard links, and it's not necessary to
propagate up trees in those cases.  (The application can just remember
paths of directories containing hard links, and not depend on the
per-directory bits for those paths).

There are ways to do it but not much enthusiasm last time I brought it
up.

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Tue, Sep 12, 2000 at 12:24:26AM +0200, Jamie Lokier wrote:
 Larry McVoy wrote:
  We have a hack in BK for this, at least I think we do, where we can look at
  the time stamps to notice that you haven't modified the files.  We don't 
  use it because of NFS screwing up timestamps.  I suppose we could enable
  it on a per repository basis so that if you knew you were running NTP or
  some other thing to keep your time stamps right, then we could diff as
  fast as we can stat.  
 
 I'd love to see a filesystem feature where I could efficiently identify
 "changed files", where "changed" is defined by last time this application
 checked or something similar.  A bonus would be to propagate this up
 directory trees.  Yes I know about hard links, and it's not necessary to
 propagate up trees in those cases.  (The application can just remember
 paths of directories containing hard links, and not depend on the
 per-directory bits for those paths).

We thought about this too (filesystems are where I got into kernel hacking),
but dismissed it as a Linux only solution.  As much as I'd like it to be
otherwise, BK is not a Linux only product.  Whatever we do needs to work on
NT (shudder) as well as all the dinosaur Unixen.

We do have more to work with because we have both the revision history and
the checked out file in each work area.  So we can play games with those
to try and simulate the "what's changed since" operator.  It's NFS which
screws that up.  
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Larry McVoy wrote:

 That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,

I meant FSMlabs, but yeah.  ;-)


 Second of all, if you ask around, you'll find that I'm a performance guy
 more than anything and that I'm not given to skewing the numbers and *if*
 that was your point, I resent the implication.

No, wasn't my point.  Sorry if it came out that way.  My point was for
anyone reading not to jump to conclusions based on just those numbers, and
that I'd like to see some real-world tests on comparable systems.


 Fourth, non-null pulls are similarly faster, again, by design.  BK only
 moves the data it needs to move.  That means if you have a 100GB file
 in which you have changed one byte, BK will move on the order of 1 byte
 to update that file.  And that's it - it doesn't compare the two files,
 or read the two files, or in any way look at the two files to figure out
 that they need to be updated.

This is an interesting point.  As far as changes are concerned, it should
have lower network utilization.  However, in effect, it does compare the
files.  It compared the file when you checked it into your local
repository.  It's just shifting the burden of the comparison to a local
repository instead of doing it over the network.  Faster, yes, but it
still does the same thing, just in a different way that uses less
bandwidth.


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Tue, 12 Sep 2000, Jamie Lokier wrote:

 I'd love to see a filesystem feature where I could efficiently identify
 "changed files", where "changed" is defined by last time this application
 checked or something similar.

An in-filesystem revision number would do the trick.  Could be REALLY
efficient.  All you have to do is stat and you know if the thing really
changed, instead of just knowing a date that can be horked.  I like!  I
like a lot!

Something else that'd be nice would be to make Linux deal better with NFS
dates.  The date presented by an NFS server for files is its timestamp. 
If the linux NFS client code were to subsequently check the stamp after
performing an operation, it could use that to approximate the time error
and this problem would about 99% disappear, with a fudge factor of under a
second.  Comments?


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

  Fourth, non-null pulls are similarly faster, again, by design.  BK only
  moves the data it needs to move.  That means if you have a 100GB file
  in which you have changed one byte, BK will move on the order of 1 byte
  to update that file.  And that's it - it doesn't compare the two files,
  or read the two files, or in any way look at the two files to figure out
  that they need to be updated.
 
 This is an interesting point.  As far as changes are concerned, it should
 have lower network utilization.  However, in effect, it does compare the
 files.  It compared the file when you checked it into your local
 repository.  It's just shifting the burden of the comparison to a local
 repository instead of doing it over the network.  Faster, yes, but it
 still does the same thing, just in a different way that uses less
 bandwidth.

You just summed up one of the main reasons why BK is better.  We shifted
work to the client.  There is one server and many, many clients in a 
CVS tree.  The same problem solved using BK inherently distributes the
load over all the machines, moving 99% of to the clients and off loading
the server (and hence the network) almost completely.

BitKeeper isn't any faster than CVS when you do the same operation, or at
least not much faster.  If we both have 10,000 files we have to update,
we're both going to be doing a lot of work.  In fact, I suspect that CVS
is faster at that than BK is because it is just updating a clear text file
and we are updating a revision history - we have more work to do so it 
stands to reason we'd be slower.

BitKeeper is faster in practice because it isn't trying to talk to the
CVS server for all of the operations.  By far the vast majority of the
operations never reach out across the network.  So on a slow client,
BK is slow, on a fast client, BK is fast.  By contrast, with a slow CVS
server, both a slow and a fast client are slow.

So perhaps a better way to look at it is that if you can afford decent
hardware, your life should be quite pleasant using BK.  Our definition
of decent, for Linux kernel repositories, is a 466Mhz celeron with at
least 256MB, and you really want 384.  The kernel is about around 120MB
checked out, the revision history is about 160MB, and you need some mem
for the inodes.  CVS is lighter weight in that respect, it just needs
the 120MB for the checked out files and some mem for inodes.  But the
difference in price is reasonable and if we have to buy memory for the
kernel developers, we'll do it once we can afford to do it.  It's _really_
nice to measure your operations in seconds rather than minutes.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jamie Lokier]

David A. Gatwood wrote:
  I'd love to see a filesystem feature where I could efficiently identify
  "changed files", where "changed" is defined by last time this application
  checked or something similar.
 
 An in-filesystem revision number would do the trick.  Could be REALLY
 efficient.  All you have to do is stat and you know if the thing really
 changed, instead of just knowing a date that can be horked.  I like!  I
 like a lot!

Yes indeed.  There aren't many bits to play with in an ext2 inode so you
have to be careful, but it can be done.  Especially when you combine it
with the timestamp (there's no need to update a revision number if the
time changes).  I think this can even run over NFS (as a dubious
extension in the cookie or something ;-)

 Something else that'd be nice would be to make Linux deal better with NFS
 dates.  The date presented by an NFS server for files is its timestamp. 
 If the linux NFS client code were to subsequently check the stamp after
 performing an operation, it could use that to approximate the time error
 and this problem would about 99% disappear, with a fudge factor of under a
 second.  Comments?

AFS does something like that.  ntpd is better :-)

-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Tony Mantler]

At 5:13 PM -0500 9/11/2000, Larry McVoy wrote:
[...]
over a 384Kbits/sec link.

That's a 48Kbyte/sec link.  Hardly a "horribly fast network".  In fact,
the bandwidth to FSMlabs.com and innominate.org seems to be identical,
I suspect that my link is the bottleneck, not either of theirs.
In order for the link to be the reason for the performance difference,
innominate.org would need to be a .9Kbyte/second link.  I kinda doubt
that seeing as a 128MB cvs checkout took 23 minutes (works out to around
90Kbyte/sec uncompressed, so cvs must compress it, so I'd guesstimate
they have around a 30Kbyte/sec link or so).
[...]

"It's the latency, stupid". I wouldn't care to argue whether CVS is slower
than BK or not, but consider that if you had a router between you and the
CVS server that was dropping even 5% of your packets, or even just bumping
the latency by a quarter second (and I've seen routers that do that. evil
things), the timing numbers will jump *significantly*.

The best way to test network performance between the two protocols would be
to get yourself a good ol'fasioned serial cable and connect your test
client and test server to eachother via PPP at about ~9600bps or so, *then*
do your tests. Equal (though low) latency, equal (definatley low)
bandwidth, equal server and client performance.

Of course, all the CVS work I do is either over local 10/100 ethernet or
through my 6d/1uMbit cablemodem (which actually gets 6d/1u, up here in the
great white north), so what do I care? 8)


Cheers - Tony 'Nicoya' Mantler :)


--
Tony "Nicoya" Mantler - Renaissance Nerd Extraordinaire - [EMAIL PROTECTED]
Winnipeg, Manitoba, Canada   --   http://nicoya.feline.pp.se/


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

Note: trimmed the 390 list, they don't care according to Alan..

On Tue, Sep 12, 2000 at 12:21:16AM +0200, Jamie Lokier wrote:
 Larry McVoy wrote:
  That's a benefit [for BK] of having changesets, I only need to compare
  the ChangeSet file to know that 4 files were updated 2 were moved, and
  5 were created, then I move those *portions* of those files across the
  wire.
 
 What happens when I lose the ChangeSet file, or misplace it?

Life really sucks.  It's like losing a superblock, sort of.  We can 
reconstruct them but almost never do because people typically have 
more than once copy of a repository sitting around.  So you can copy
it back, do a "bk -r check -a" (the moral equiv of an fsck, no we don't
really have an fsck -y yet), and fix up the errors.  The revision control
files and the ChangeSet files point at each other and need to be in sync.

That's how we get all the performance, by the way, all we need to compare
is a tiny subset of the ChangeSet files (32KB out of 5MB in one tree, that's
pretty typical).  And we could compare a lot less, it just hasn't been
worth it to do so, we gzip that 32K down to less than 6K so...)
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 05:48:44PM -0500, Tony Mantler wrote:
 
 "It's the latency, stupid". I wouldn't care to argue whether CVS is slower
 than BK or not, but consider that if you had a router between you and the
 CVS server that was dropping even 5% of your packets, or even just bumping
 the latency by a quarter second (and I've seen routers that do that. evil
 things), the timing numbers will jump *significantly*.

Well, err, I guess I'm pretty stupid, but how about this?  If the lxr guy
is listening to all this, he could download a copy of BK, clone the FSMlabs
tree, then I'll do a null pull from him, and we'll have an apples to apples
comparison, now won't we?

I think it's a pointless thing to do, I already know that CVS transfers way
more data to do the same operation so it's a foregone conclusion it will be
slower.  But I'm very happy to do it if the lxr guy (sorry, I've forgotten
your name, I apologize) is willing.  Send me private mail, it shouldn't take
very long at all to get you set up.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[James Lewis Nance]

On Mon, Sep 11, 2000 at 03:45:18PM -0700, Larry McVoy wrote:

 the 120MB for the checked out files and some mem for inodes.  But the
 difference in price is reasonable and if we have to buy memory for the
 kernel developers, we'll do it once we can afford to do it.  It's _really_
 nice to measure your operations in seconds rather than minutes.

Larry,
It would be interesting to see the speed difference between bk and cvs
for the mozilla sources.  Doing a cvs update takes at least 10 minutes even
if no files have changed.  It took significantly longer when I was using
a modem instead of a DSL line.  You haven't benchmarked this case have you?

Thanks,

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 07:44:52PM -0400, James Lewis Nance wrote:
 On Mon, Sep 11, 2000 at 03:45:18PM -0700, Larry McVoy wrote:
 
  the 120MB for the checked out files and some mem for inodes.  But the
  difference in price is reasonable and if we have to buy memory for the
  kernel developers, we'll do it once we can afford to do it.  It's _really_
  nice to measure your operations in seconds rather than minutes.
 
 Larry,
 It would be interesting to see the speed difference between bk and cvs
 for the mozilla sources.  Doing a cvs update takes at least 10 minutes even
 if no files have changed.  It took significantly longer when I was using
 a modem instead of a DSL line.  You haven't benchmarked this case have you?

If you can get me a tarball of the CVS repository, I'll import the
history into a BK tree and then we can do side by side tests.  I know
Mitchell Baker somewhat, so if she is still there, you might ping her.
I'd be interested to see this as well, so please let me know if the
CVS tarball is available.  Just to be clear, I am not talking about the
results of a CVS checkout, I am talking about the actual CVS tree with
the RCS files - if I just imported the most recent versions, BK would
be unfairly faster because it would be storing a lot less history.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jonathan Lemon]

In article local.mail.linux-kernel/[EMAIL PROTECTED] you write:
On Mon, Sep 11, 2000 at 05:48:44PM -0500, Tony Mantler wrote:
 
 "It's the latency, stupid". I wouldn't care to argue whether CVS is slower
 than BK or not, but consider that if you had a router between you and the
 CVS server that was dropping even 5% of your packets, or even just bumping
 the latency by a quarter second (and I've seen routers that do that. evil
 things), the timing numbers will jump *significantly*.

Well, err, I guess I'm pretty stupid, but how about this?  If the lxr guy
is listening to all this, he could download a copy of BK, clone the FSMlabs
tree, then I'll do a null pull from him, and we'll have an apples to apples
comparison, now won't we?

I think it's a pointless thing to do, I already know that CVS transfers way
more data to do the same operation so it's a foregone conclusion it will be
slower.  But I'm very happy to do it if the lxr guy (sorry, I've forgotten
your name, I apologize) is willing.  Send me private mail, it shouldn't take
very long at all to get you set up.

I don't know why I'm bothering to reply to this, but yes, if you're
trying to synchronize CVS source trees with only CVS, it will be slow.
Now, if you were to compare CVSup vs Bitkeeper, then things might get
more interesting.
--
Jonathan

(for those unaware of it, CVSup is a high-speed mechanism to
distribute CVS repositories, and uses several algorithms including
"rsync" to accomplish this.)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Larry McVoy]

On Mon, Sep 11, 2000 at 06:49:43PM -0500, Jonathan Lemon wrote:
 I don't know why I'm bothering to reply to this, but yes, if you're
 trying to synchronize CVS source trees with only CVS, it will be slow.
 Now, if you were to compare CVSup vs Bitkeeper, then things might get
 more interesting.
 --
 Jonathan
 
 (for those unaware of it, CVSup is a high-speed mechanism to
 distribute CVS repositories, and uses several algorithms including
 "rsync" to accomplish this.)

I'd be happy to do this.  I've already gone over the details in private
mail with Dave Miller, he was suggesting something similar and I explained
how much disk  net I/O you have to with each case and the BK case is
dramatically less.  

The reason is that BK does all the work when you do a local commit, it
captures the state of the entire tree once, at the time you commit your
changes.  CVSup, rsync, CVS, etc., all have to look at the whole tree
because there is no fanin/fanout like BK has with the ChangeSet file.
You can play all the games you want, but the bottom line is that you
have to look at some version of the data, be it all the inodes, or the
actual data.  With BK, we've distilled the state of about 9,000 files
in the Linux tree down to about 6,000 bytes.  We have to do a single
roughly 32KB disk I/O to get that state and then we compress to 6K and
transfer it across the wire.

No matter what you do with rsync, there is no bloody way you can even
come close to a single 32K disk read and then a 6K over the wire transfer.
At least, I can't think of one, can you?

We do just as much I/O on the commit, then we walk the tree, and diff
against the checked in version, so if you have the entire tree editted,
we'll diff the entire tree.  But that happens when you commit your
changes, not every time you update.

The fundemental observation is as the tree size/age grows, the amount of
change you make to it stays relatively constant but the updates grow with
tree size.  One human can only make so much change, but many can make a 
lot.  BK takes advantage of that and does the hard work when you do hard
work, not every time you update.

It's just a different design, no offense is intended against CVS, we have
all used and learned from CVS.  But just because CVS is useful doesn't mean
it is the best answer.
-- 
---
Larry McVoy[EMAIL PROTECTED]   http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Jonathan Lemon]

On Mon, Sep 11, 2000 at 05:05:08PM -0700, Larry McVoy wrote:
 On Mon, Sep 11, 2000 at 06:49:43PM -0500, Jonathan Lemon wrote:
  I don't know why I'm bothering to reply to this, but yes, if you're
  trying to synchronize CVS source trees with only CVS, it will be slow.
  Now, if you were to compare CVSup vs Bitkeeper, then things might get
  more interesting.
  --
  Jonathan
  
  (for those unaware of it, CVSup is a high-speed mechanism to
  distribute CVS repositories, and uses several algorithms including
  "rsync" to accomplish this.)
 
 I'd be happy to do this.  I've already gone over the details in private
 mail with Dave Miller, he was suggesting something similar and I explained
 how much disk  net I/O you have to with each case and the BK case is
 dramatically less.  
 
 The reason is that BK does all the work when you do a local commit, it
 captures the state of the entire tree once, at the time you commit your
 changes.  CVSup, rsync, CVS, etc., all have to look at the whole tree
 because there is no fanin/fanout like BK has with the ChangeSet file.
 You can play all the games you want, but the bottom line is that you
 have to look at some version of the data, be it all the inodes, or the
 actual data.  With BK, we've distilled the state of about 9,000 files
 in the Linux tree down to about 6,000 bytes.  We have to do a single
 roughly 32KB disk I/O to get that state and then we compress to 6K and
 transfer it across the wire.
 
 No matter what you do with rsync, there is no bloody way you can even
 come close to a single 32K disk read and then a 6K over the wire transfer.
 At least, I can't think of one, can you?
 
 We do just as much I/O on the commit, then we walk the tree, and diff
 against the checked in version, so if you have the entire tree editted,
 we'll diff the entire tree.  But that happens when you commit your
 changes, not every time you update.
 
 The fundemental observation is as the tree size/age grows, the amount of
 change you make to it stays relatively constant but the updates grow with
 tree size.  One human can only make so much change, but many can make a 
 lot.  BK takes advantage of that and does the hard work when you do hard
 work, not every time you update.
 
 It's just a different design, no offense is intended against CVS, we have
 all used and learned from CVS.  But just because CVS is useful doesn't mean
 it is the best answer.

Oh, no disagreement there, CVS has quite a few shortcomings, so I'm
in no way holding it up as anything near an ideal solution, it's just
what we have now.  And yes, the cvsup daemon will take quite a bit of
I/O and memory as well.  I merely pointing out that a comparision of
cvsup vs BK would be far more interesting than native CVS alone.
--
Jonathan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Bryan O'Sullivan]

l If you can get me a tarball of the CVS repository, I'll import the
l history into a BK tree and then we can do side by side tests.

I know BK is the best thing since sliced yams, but can you please take
this thread to some BK mailing list and do your performance
comparisons there?

b
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[Horst von Brand]

mberglund [EMAIL PROTECTED] said:

[...]

 License:
 This project will not be restricted to any one license. If a piece
 of software is desirable, but under an Artistic-, BSD-, or even
 GPL-style license, it will be used, so long as it allows
 free(no cost) distribution.

Can't do that. The pieces that go into this (gcc, the Linux kernel, and
minor pieces) are under licences granted by _other_ people, which you can't
just change at your whim. And I doubt it _very_ much indeed Linus or the
FSF will allow you to distribute their respective properties under other
licences than the GPL.

Good luck in any case!
-- 
Dr. Horst H. von Brand   mailto:[EMAIL PROTECTED]
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria  +56 32 654239
Casilla 110-V, Valparaiso, ChileFax:  +56 32 797513
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/

Re: [ANNOUNCE] Darkstar Development Project

[David A. Gatwood]

On Mon, 11 Sep 2000, Horst von Brand wrote:

 
 mberglund [EMAIL PROTECTED] said:
 
 [...]
 
  License:
  This project will not be restricted to any one license. If a piece
  of software is desirable, but under an Artistic-, BSD-, or even
  GPL-style license, it will be used, so long as it allows
  free(no cost) distribution.
 
 Can't do that. The pieces that go into this (gcc, the Linux kernel, and
 minor pieces) are under licences granted by _other_ people, which you can't
 just change at your whim. And I doubt it _very_ much indeed Linus or the
 FSF will allow you to distribute their respective properties under other
 licences than the GPL.

I'm fairly sure that's not what the original poster meant.  I think the
statement was intended to mean that free software would be included on the
CD/ftp site without a Debian-like aversion to anything "free but
commercial or closed source".


Later,
David

-
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/