Re: Auto-install (was autoloaded...)

[Ask Bjoern Hansen]

On Fri, 16 Feb 2001, Stephen P. Potter wrote:

> Lightning flashed, thunder crashed and "Branden" <[EMAIL PROTECTED]> whisp
> ered:
> | For the list managers: Could we have a list apart from -language, so that we
> | don't bother all with this `par'-issue ??? Please?  Perhaps a list that
> | includes the issue on directory structure, and other issues related to
> | installation and deployment of Perl itself and Perl programs & modules.
> 
> How about a perl6-install list?  This discussion really doesn't fit into
> any of the current top level lists, so we can make a new top level and
> cover other installation issues as well.  Ask, can you make this, if the
> name is agreeable.

[EMAIL PROTECTED] already exists. 
http:[EMAIL PROTECTED]/


 - ask

-- 
ask bjoern hansen - http://ask.netcetera.dk/

Re: Auto-install (was autoloaded...)

[Andy Dougherty]

On Fri, 16 Feb 2001, Stephen P. Potter wrote:

> How about a perl6-install list?  This discussion really doesn't fit into
> any of the current top level lists, so we can make a new top level and
> cover other installation issues as well.  Ask, can you make this, if the
> name is agreeable.

There already is a perl6-build list.  Perhaps you can just use that?

-- 
Andy Dougherty  [EMAIL PROTECTED]
Dept. of Physics
Lafayette College, Easton PA 18042

Re: Auto-install (was autoloaded...)

[Stephen P. Potter]

Lightning flashed, thunder crashed and "Branden" <[EMAIL PROTECTED]> whisp
ered:
| For the list managers: Could we have a list apart from -language, so that we
| don't bother all with this `par'-issue ??? Please?  Perhaps a list that
| includes the issue on directory structure, and other issues related to
| installation and deployment of Perl itself and Perl programs & modules.

How about a perl6-install list?  This discussion really doesn't fit into
any of the current top level lists, so we can make a new top level and
cover other installation issues as well.  Ask, can you make this, if the
name is agreeable.

-spp

Re: Auto-install (was autoloaded...)

[Andy Dougherty]

On Tue, 13 Feb 2001, Branden wrote:

> Hello.
> 
> I'm working on the PDD for par. I would like to propose a standard directory
> structure for the files inside the archive, but I realise this depends
> greatly upon the directory structure of Perl itself.
> 
> How does Perl 5 manage its directory structure?

See the INSTALL file supplied with Perl5.6.0 or later.  If you have any
questions after that, let me know and I'll try to answer them.

> in. And Perl 5's directory structure is rather clumsy, I think we should set
> platform-independent conventions for that too, where it's possible. 

You might want to build on top of some of the ideas hinted at in
perl-5.6.0's "installstyle" Configure variable. See Porting/Glossary for
the full definition.

The directory structure may also be affected by the goal of allowing the
simultaneous installation of multiple versions of modules.  (I don't
recall the relevant RFC offhand, but I think there's was also a similar
idea in Larry's Atlanta speech.)

-- 
Andy Dougherty  [EMAIL PROTECTED]
Dept. of Physics
Lafayette College, Easton PA 18042

Re: Auto-install (was autoloaded...)

[Branden]

Hello.

I'm working on the PDD for par. I would like to propose a standard directory
structure for the files inside the archive, but I realise this depends
greatly upon the directory structure of Perl itself.

How does Perl 5 manage its directory structure?

Suppose $PERL is the base directory where Perl is installed (that would be
C:\PERL on Win and /usr/lib/perl5 or something like that on UNIX). Are
always modules stored on $PERL/lib ? And shared libraries for modules always
on $PERL/lib/auto ? What's $PERL/site for? What's the structure inside it?
Is there a standard directory for storing scripts?

I think this issue should be discussed along with `par', since we cannot
install something if we don't know the structure that something is stored
in. And Perl 5's directory structure is rather clumsy, I think we should set
platform-independent conventions for that too, where it's possible. Perhaps
we can write a separate PDD on directory structure, or put it all inside
par's PDD.

It would help if someone writes a snippet of a PDD on signing archives, etc.
so that I can include it on the PDD. I have really no experience on that, so
I probably would do it wrong. Anyone to volunteer?

For the list managers: Could we have a list apart from -language, so that we
don't bother all with this `par'-issue ??? Please?  Perhaps a list that
includes the issue on directory structure, and other issues related to
installation and deployment of Perl itself and Perl programs & modules.

- Branden

Re: Auto-install (was autoloaded...)

[Clayton Scott]

Branden wrote:
> And I'll probably ask you to use another naming/extension, like pp5 (par for
> perl 5), so that modules for both versions don't get mixed up (since they'll
> be incompatible).

That doesn't make sense. Either your script or your archive
 tool (par, pun, or CPAN or whatever) should know what perl interpreter
 to use without have to have a .pp5, .pp5.6, .pp5.6.1. The script
 should 'require 5.6' (and you'll get an error like 
 Perl 5.6 required--this is only version 5.00503, stopped at -e line 1.
 and your perl interpreter will deal with it. That and/or it would be
 nice if a metadata document came as a part of the archive that was
 easy to query for details e.g.

par --perl_version my_cool_script.par

or from the CPAN shell:
CPAN> install my_cool_script.par for 5.6

> Other than that, I believe developping it for Perl 5 is a very good idea,

Perl 5 is not dead and will not be for a long, long time.
 Building it first with perl 5 for perl 5 should be a no brainer.

Clayton

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 05:28:04PM -0300, Branden wrote:
> Could you point me to some URLs? Like .deb file format? What's the good info
> the have? What's dselect? How it works?

Start from sections 6, 7 and 8 of the Debian FAQ
http://www.debian.org/doc/FAQ/

dselect, aptitude and several other utilities all work like the CPAN
shell, except its for Linux programs, utilities, libraries and kernels
(and perl libraries).  It connects to a central repository (or
mirror), gets a listing of everything available, checks it against
what's already installed and gives you a nice listing.  It can
automatically upgrade, install, find dependencies, find conflicts,
etc... for many thousands of packages (I have 1000+ packages installed
on my laptop, and that's just scratching the surface) and about a
dozen different architectures.

apt-get provides something similar to "perl -MCPAN -e 'install Module::Foo'"


> > par can do something similar.  It can slap a copy of pun (and thus
> > perl) onto the archive.  Its not simple, and its platform dependent,
> > but its useful.  I'm more and more seeing par as a way of
> > embrace/extend/destroying perl2exe.
> >
> 
> Yes, useful, but I don't think this would be default. 

Oh, of course it wouldn't be the default.

> For this you would
> need a statically linked Perl, and probably with all extensions linked in
> too. And wouldn't be platform-independent. And it would be huge!!!

Actually, its smaller than you'd think.  From the little I've played
with perl2exe, it typically produces a 600-700K binary.  This was for
a short program which only contained a few modules, but still.  And
this was the demo version of perl2exe which doesn't try to save space.
Of course, I suspect its cheating and not using a static binary, but
assuming that certain Windows DLLs will be there.  I don't know about
the Unix version of perl2exe.


> And extracting documentation from it would be a problem. And it
> wouldn't merge with already existent Perl installation. And wouldn't
> be upgraded when modules or perl itself is upgraded. And...

I agree, the situation sucks.  Welcome to Windows.  Trust me, I've
been bashing perl2exe for years (or rather, the comedy of errors which
leads to its necessity).  Its only recently that I've had to deal with
a Windows office that its come in handy.


> SelfLoader could potentially be pre-processed the same way AutoLoader is...

That defeats the point of SelfLoader.  AutoLoader sticks its routines
into individual files and then loads them from there as needed, but
reading files is potentially expensive.  SelfLoader parses __DATA__
into a hash and evals them from there as needed.

AutoLoader is low startup cost for potentially high first run cost.
SelfLoader is for potentially high startup cost for low first run cost.
But they both do exectly the same thing externally.

Two options.  One, just provide a bogus SelfLoader which is a subclass
of AutoLoader and run AutoSplit over the files anyway.  IMHO the
performance difference between AutoLoader and SelfLoader will be
negligible compared to the cost of all this compression/decompression
and file reading going on.

Or, we could simply dup *DATA and stick the contents of __DATA__ into
another file (I think you suggested something similar).  The only
problem there is if they attempt to use DATA to seek() back and read
in the original program.  We can scan for that and warn about it.

Oh dear, I just had a horrible thought.  Filters.  Will Filter::exec
and friends still work with all this madness going on?  Can we shove
Lingua::Romana::Perligata into a par??!  A good pathological test
case, that.


> Are there already modules that handle this (RSA/IDEA/...) ? Perhaps we'll
> also want them to share the same interface so that we can do it flexible as
> well... For this one I don't know if we could support different key formats
> or if we should standardize on one...

The Crypt::* modules on CPAN are all there.  There are Crypt::RSA and
Crypt::IDEA floating around somewhere, haven't been posted on CPAN
because of legal issues.  They have a fairly common interface, so its
not a problem.

Get away from this whole "standardize" notion.  We're going to have
to abstract its usage internally anyway (there's already several ways
to do that with the Crypto modules) for design sanity, so we might as
well make it available to the user.  For the start, we'll pick one
that's easy.  In fact, we'll probably just call gpg or pgp to start
with for simplicity, then figure out a pure Perl version later.

Its unimportant.

Re: Auto-install (was autoloaded...)

[abigail]

On Mon, Feb 12, 2001 at 01:28:56PM -0600, Jarkko Hietaniemi wrote:
> On Mon, Feb 12, 2001 at 02:19:54PM -0500, [EMAIL PROTECTED] wrote:
> 
> > You have to do that anyway to solve the "what version of glibc are you
> > using" problem (and others).
> 
> *minirant*
> The world is not not glibc.  The world is not Linux.
> *end of minirant*
> 
> Either I am being even more grumpier than usual or you people
> are making even less sense than usual :-)


*microrant*
The world is not schwern
*end of microrant*


Abigail

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 06:56:47PM -0300, Branden wrote:
> I'd rather not have any kind of `script' that would be run on an
> installation, to avoid the `Memoize' kind of bug (couldn't find the
> reference), in which the install script had something like
> 
> # `rm -rf /`
> # This line above could be uncommented

Moot, since the par itself could contain a nasty program.

Re: Auto-install (was autoloaded...)

[James Mastros]

On Mon, Feb 12, 2001 at 06:56:47PM -0300, Branden wrote:
> James Mastros wrote:
> > magical "install" script in them that knows how to do special things with
> > files in that directory (like set up symlinks from the normal man dirs).
> 
> That probably should be in Perl's Config.pm, since Perl itself will have to
> install its own manpages. Perl should probably also include system-wide
> shared libraries location, and other things of this kind.
Yah.  In fact, I'm surprised it doesn't already.

> I'd rather not have any kind of `script' that would be run on an
> installation, to avoid the `Memoize' kind of bug (couldn't find the
> reference), in which the install script had something like
Oh, I think you're understanding me wrong -- the par being installed doesn't
get to have an install program, the directory being installed into does.
Thus, for ex, doc/lib could have a script that does pod2man and installs the
manpages.

(It's arguable that the par being installed should get to have an install
script, but it should be highly optional.)

-=- James Mastros
-- 
"All I really want is somebody to curl up with and pretend the world is a
safe place."
AIM: theorbtwo   homepage: http://www.rtweb.net/theorb/

Re: Auto-install (was autoloaded...)

[Branden]

James Mastros wrote:
> On Mon, Feb 12, 2001 at 01:03:31PM -0600, Jarkko Hietaniemi wrote:
> > The problem of unpacking, or in other words, installing, or in other
> > words, embedded hardwired paths is hard.  Think library paths: both
> > pure Perl libraries *and* shared libraries.
> True enough.  The way Linux package managers work I think we can go with:
> they rely on stuff being in well-known locations.
>
> In the Linux case this is because the packager (in the broad sense of the
> term) can rely on the target system's distro being "right" for the
package.
> In the case of Perl, we can rely on the parts inside the /usr/lib/perl (or
> whatever) tree being set up the "right way", and die, or even do the wrong
> thing, if the admin messed it up.
>
> In other words, if there's a place for everything, we just have to put
> everything in its place.
>
> The one rub with that is manpages: they need to be installed in a
> system-dependent location (with system-dependent names, even) in order for
> the "man" command to pick them up.  My best idea for this is to have some
> directories (IE man dirs, but it could be useful for other things) have a
> magical "install" script in them that knows how to do special things with
> files in that directory (like set up symlinks from the normal man dirs).
>

That probably should be in Perl's Config.pm, since Perl itself will have to
install its own manpages. Perl should probably also include system-wide
shared libraries location, and other things of this kind.

I'd rather not have any kind of `script' that would be run on an
installation, to avoid the `Memoize' kind of bug (couldn't find the
reference), in which the install script had something like

# `rm -rf /`
# This line above could be uncommented



> BTW, this plan would make it painful to do with perl5 setups, since they
> commonly have odd dir structures.
>

This is something that should be redesigned for Perl 6. And I think this
discussion goes along with the one about `par', since these directory
structures exist mainly for installing extensions on them...

- Branden

Re: Auto-install (was autoloaded...)

[Branden]

<[EMAIL PROTECTED]> wrote:
> Oh, I fully realize that *none* of this "self-extracting" nonsense is
> going to be cross-platform by any means.  For each variation of Unix
> you'll need a seperate par binary, but its no worse than C.  But Unix
> really isn't a problem.  Any Unix dist worth its weight in snot comes
> with Perl.
>
> The real use is for distributing to MacOS and Windows.  That's much
> easier.  One binary for 95/98/Me, one for NT/2000, one for MacOS, one
> for MacOS X and maybe one for DOS/3.1.  It might seem like alot of
> work, but its dirt cheap for cross-platform development.  (Of course,
> you'll still have to be careful about file operations and the like).
>

OK. Convinced me. We could even have one distribution of pre-built
statically-linked customized Perl builds for the platforms listed below
(Win/Mac) that already does automatically the same that `pun' does, but also
automatically knows that the executable itself is the `par'-file. This file
would also have sockets/zlib/... statically linked in for scripts that need
them.

`par' could get a command-line option to ``use an executable as container''.
This would create a .exe/.bin/whatever file, that would:
1. Copy the `container' file into that other file.
2. Append the resulting .par to that file.
3. Adjust things so that zip gets happy and knows it's being included in an
executable.
Running the file would run the script. Running it with some exoteric options
(like --par-install, --par-uninstall, etc.) would do the same as `pun' with
some shorter options



> Perl gets *creamed* on this issue.  Personally, I've had lots of ideas
> for little utilities to help with office-work killed because all the
> non-programmers run Windows and don't want to be bothered installing
> Perl.  No, its not a fight you can easily win.
>

Yeah, I've had many of these too. And never got to do them...



>
> Well... there is one part that will be cross-platform.  The problem
> of having Perl but not pun.

Won't be a problem with Perl 6.


>  That's simple enough, par can build an
> archive with a minimalist version of pun with the par attached as
> __DATA__ or something.
>

See my solution above.

- Branden

Re: Auto-install (was autoloaded...)

[James Mastros]

On Mon, Feb 12, 2001 at 01:03:31PM -0600, Jarkko Hietaniemi wrote:
> The problem of unpacking, or in other words, installing, or in other
> words, embedded hardwired paths is hard.  Think library paths: both
> pure Perl libraries *and* shared libraries.  
True enough.  The way Linux package managers work I think we can go with:
they rely on stuff being in well-known locations.

In the Linux case this is because the packager (in the broad sense of the
term) can rely on the target system's distro being "right" for the package.
In the case of Perl, we can rely on the parts inside the /usr/lib/perl (or
whatever) tree being set up the "right way", and die, or even do the wrong
thing, if the admin messed it up.

In other words, if there's a place for everything, we just have to put
everything in its place.

The one rub with that is manpages: they need to be installed in a
system-dependent location (with system-dependent names, even) in order for
the "man" command to pick them up.  My best idea for this is to have some
directories (IE man dirs, but it could be useful for other things) have a
magical "install" script in them that knows how to do special things with
files in that directory (like set up symlinks from the normal man dirs).

BTW, this plan would make it painful to do with perl5 setups, since they
commonly have odd dir structures.

  -=- James Mastros
-- 
"All I really want is somebody to curl up with and pretend the world is a
safe place."
AIM: theorbtwo   homepage: http://www.rtweb.net/theorb/

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Mon, Feb 12, 2001 at 02:41:01PM -0500, [EMAIL PROTECTED] wrote:
> Oh, I fully realize that *none* of this "self-extracting" nonsense is
> going to be cross-platform by any means.  For each variation of Unix

Whew!  I was starting to think I'm surrounded by tunnel visioned penguins.

> you'll need a seperate par binary, but its no worse than C.  But Unix
> really isn't a problem.  Any Unix dist worth its weight in snot comes
> with Perl.

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[schwern]

Oh, I fully realize that *none* of this "self-extracting" nonsense is
going to be cross-platform by any means.  For each variation of Unix
you'll need a seperate par binary, but its no worse than C.  But Unix
really isn't a problem.  Any Unix dist worth its weight in snot comes
with Perl.

The real use is for distributing to MacOS and Windows.  That's much
easier.  One binary for 95/98/Me, one for NT/2000, one for MacOS, one
for MacOS X and maybe one for DOS/3.1.  It might seem like alot of
work, but its dirt cheap for cross-platform development.  (Of course,
you'll still have to be careful about file operations and the like).

Perl gets *creamed* on this issue.  Personally, I've had lots of ideas
for little utilities to help with office-work killed because all the
non-programmers run Windows and don't want to be bothered installing
Perl.  No, its not a fight you can easily win.


Well... there is one part that will be cross-platform.  The problem
of having Perl but not pun.  That's simple enough, par can build an
archive with a minimalist version of pun with the par attached as
__DATA__ or something.

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 01:28 PM 2/12/2001 -0600, Jarkko Hietaniemi wrote:
>On Mon, Feb 12, 2001 at 02:19:54PM -0500, [EMAIL PROTECTED] wrote:
> > Perl binary with a built-in @INC prefix something like
> > "/tmp/XpErLXX" and then do some s/// madness over the
> > binary.
> >
> > Anyhow, this is easily solved by having pun run the scripts inside the
> > par as "perl -Ilib" or something similar.
>
>It's "easily done" but strangely the implementations seems to have
>left the main Perl distribution unscathed after all these years.

I'm pretty sure there are a couple of platforms in the core distribution 
that manage position independence, or moveability at least--I thought that 
the OS/2 port did this. (VMS and Windows don't count here, though both are 
position independent last I knew, requiring an environment variable or 
something similar)

> > Shared libraries are easy enough to solve, build perl staticly linked.
>
>Oh, great.

If things are being packaged up, it is the only option. Otherwise you have 
to go and unpack things into some 'known' location at runtime, or other odd 
things.

I'm not saying it's a *good* option, mind. It just may be the only one for 
a perl with extensions.


Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Mon, Feb 12, 2001 at 02:19:54PM -0500, [EMAIL PROTECTED] wrote:
> On Mon, Feb 12, 2001 at 01:03:31PM -0600, Jarkko Hietaniemi wrote:
> > The problem of unpacking, or in other words, installing, or in other
> > words, embedded hardwired paths is hard.  Think library paths: both
> > pure Perl libraries *and* shared libraries.  In theory this is easy:
> > the portable (and grody) solutions is to have a "long enough" filler
> > path(s) that is(are) then at install time binary patched.  In
> > practice, though, for some reason no one seems to have done this so
> > that this would be a widely used practice and well-known art.  
> 
> Looked at the ActiveState Unix installer lately?  They distribute a

*rant warning*
Sigh.  (Note to self: always remember to spell out everything in full
because someone will feel obliged to informe you.)  Yes, I know this.
That's what I referred to by "widely used".  ActiveState may be
big, especially in the Windows arena  but they are not "widely used".
zip is widely used.  .tgz is widely used.  Windows is widely used.
UNIX is widely used.  ActiveState, not.
*end of rant*

> Perl binary with a built-in @INC prefix something like
> "/tmp/XpErLXX" and then do some s/// madness over the
> binary.
> 
> Anyhow, this is easily solved by having pun run the scripts inside the
> par as "perl -Ilib" or something similar.

It's "easily done" but strangely the implementations seems to have
left the main Perl distribution unscathed after all these years.

> Shared libraries are easy enough to solve, build perl staticly linked.

Oh, great.

> You have to do that anyway to solve the "what version of glibc are you
> using" problem (and others).

*minirant*
The world is not not glibc.  The world is not Linux.
*end of minirant*

Either I am being even more grumpier than usual or you people
are making even less sense than usual :-)

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Branden]

<[EMAIL PROTECTED]> wrote:
> debs store alot of information that rpm doesn't, and it would be good
> to look at to steal good ideas.  Also, and most importantly, they have
> dselect, which is similar, but much more powerful, than CPAN and the
> CPAN shell.  That's something to look at.
>

Could you point me to some URLs? Like .deb file format? What's the good info
the have? What's dselect? How it works?



> Ever made a self-extracting archive?  Its just a zip file attached a
> little unzip binary.  Send that to someone, they run it and it
> decompresses itself!  chicken<->egg.
>

Having a self-extractable wouln't allow perl using its own modules and
configuration files to decide how a `par'-file should be handled.



> par can do something similar.  It can slap a copy of pun (and thus
> perl) onto the archive.  Its not simple, and its platform dependent,
> but its useful.  I'm more and more seeing par as a way of
> embrace/extend/destroying perl2exe.
>

Yes, useful, but I don't think this would be default. For this you would
need a statically linked Perl, and probably with all extensions linked in
too. And wouldn't be platform-independent. And it would be huge!!! And
extracting documentation from it would be a problem. And it wouldn't merge
with already existent Perl installation. And wouldn't be upgraded when
modules or perl itself is upgraded. And...

As you see, there are many reasons not to do this. Of course, there are many
more to do it, but I think we could leave them for `perl2exe'. If that's
very needed, we could add some parameters for `par' to do it, by including
the `par' archive to a statically-linked customized with-all-extensions
pre-built perl binary, to get the same result of `perl2exe', but I guess
that would yes be harmful for IndigoSTAR, and as _you_ said it, we don't
want to hurt them.



> > > I've been thinking about this... would dumping things into /tmp be
> > > considered cheating?
> >
> > Probably not, but avoiding it would be appreciated (for the objectives
> > above -- consider a diskless machine).
>
> Ok, we can have some flags and configuration bits figure that out.
>

With zip, wouldn't be necessary.



> > >  Loading a Perl module from a filehandle might
> > > screw with .
> >
> > As resource files can be attached to the archive, I think not allowing
> > __DATA__ wouldn't be such a big restriction. I don't know if we can find
> > other workarounds.
>
> This kills SelfLoader and unsplit AutoLoader modules and similar
> schemes.  AutoLoader isn't so much a problem, since we can just run
> AutoSplit over it.  SelfLoader will be tricky.
>

SelfLoader could potentially be pre-processed the same way AutoLoader is...
If Perl 6 is sufficiently smart, and it'll probably be, it will let us
`hook' another tied filehandle into __DATA__, and so we'll be able to do
some crazy stuff to make this work. Maybe we `dup' the filehandle that's
reading the file when __DATA__ is found, and we read from it (decompressing
on-the-fly) until we reach __END__, in which case we know we are at the
beginning of the data. Admitedly not efficient at all, but as said before,
efficiency is not the real objective of it, and we have many ways to work
around this.




> > > > | Merge many archives  | Yes | No  | Yes |
> > > > | in one   | | | |
> >
> > Take some scripts and lots of .par (containing modules needed by the
> > scripts) and make all this directly into one .par, without needing for
> > having the .pars uncompressed somewhere.
>
> Heh, simplest way to do that is to cheat and just decompress all the
> pars and put them back together.

Yes, but this is something `par' is expected to do. For example, suppose I
wrote a script foo.pl that requires Bar.pm and Baz.pm. I could go to CPAN,
and download Bar-1.0.noarch.par and Baz-3.54.noarch.par. Then I could

par -o ../foo-1.3.noarch.par ./bin/foo.pl ../Bar-1.0.noarch.par
../Baz-3.54.noarch.par

And `par' would automatically handle them for me. The main use of it is when
Bar/Baz was platform-dependent, so that I could build various foo pars, one
for which platform, only by having Bar/Baz pars for all those platforms.




>  There will be problems, for
> instance, with two pars that use the same modules but different
> versions.  Default behavior might be to use the newest version and
> throw a warning.  Otherwise our par could have several different lib/
> directories.  For instance:
>
> lib/ contains the unconflicting modules
> lib/CGI-2.49/  contains version 2.49 of CGI.pm
> lib/CGI-2.62/  contains version 2.62 of CGI.pm
>
> And then scripts which need 2.49 can be run with "perl -Ilib/CGI-2.49"
> and the other with "perl -Ilib/CGI-2.62".
>
> This is ok for running from within the par, but installing will get
> tricky under perl5.  perl6 will presumably have something to solve
> this (as noted in the recent lengthy discussion).
>

For par in Perl5 (pp5), I suggest w

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 01:03:31PM -0600, Jarkko Hietaniemi wrote:
> The problem of unpacking, or in other words, installing, or in other
> words, embedded hardwired paths is hard.  Think library paths: both
> pure Perl libraries *and* shared libraries.  In theory this is easy:
> the portable (and grody) solutions is to have a "long enough" filler
> path(s) that is(are) then at install time binary patched.  In
> practice, though, for some reason no one seems to have done this so
> that this would be a widely used practice and well-known art.  

Looked at the ActiveState Unix installer lately?  They distribute a
Perl binary with a built-in @INC prefix something like
"/tmp/XpErLXX" and then do some s/// madness over the
binary.

Anyhow, this is easily solved by having pun run the scripts inside the
par as "perl -Ilib" or something similar.

Shared libraries are easy enough to solve, build perl staticly linked.
You have to do that anyway to solve the "what version of glibc are you
using" problem (and others).

Re: Auto-install (was autoloaded...)

[James Mastros]

On Mon, Feb 12, 2001 at 04:01:31PM -0300, Branden wrote:
> > We'll just have to use something other than RSA most likely.
> Why? Problems with exporting cryptosystems? If that's it, how does
> Java/Netscape do it?
Nah, it's a pattent issue.  Netscape (and other .jar consumers, assumedly)
licenced code from RSA.  However, it shouldn't be a problem, since RSA's
pattent (in the US, anyway, and I don't think they pattented anywhere else)
has timed out.

-=- James Mastros
-- 
"All I really want is somebody to curl up with and pretend the world is a
safe place."
AIM: theorbtwo   homepage: http://www.rtweb.net/theorb/

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 04:50:53PM -0300, Branden wrote:
> > 2. "My end-users might not have Perl installed"  Bundling a Perl
> >interpretor with your program (until perlcc is viable)
> >
> 
> No. I don't expect Perl installation or any other otherwise executable or
> installation program to be bundled inside of archives. Correct if I'm wrong,
> but archives are meant for Perl scripts and Perl modules.

"Ment for"?  POD was also "ment for" documentation, but it doesn't
stop me from using it to embed test code.  Besides, the idea has only
been kicking around for what, a week now?  Nothing is nailed down.

Seems like a logical extension of the idea and the technology.  Once
you have something working that can bundle all the components of a
Perl program together, its just one more step to wrap an interpretor
on top of that.

Its a simple fact that lots of people are afraid to distribute Perl
programs because their customers/users don't have Perl installed and
aren't about to download 3.5 megs to run one program.  perl2exe has a
market.  I don't want to get into a big discussion about this because
its all been said before.  Once par's basic functionality works we'll
give a shot at a self-extracting par.


> No. One objectives of `par' is ``g) The archive file should be stored in a
> format that can be easily created and inspected with widely available
> external tools.'' (from alpha PDD).

pun will be able to do that.  par's version of "source hiding" will be
roughly akin to uuencoding your source.  Its not going to stop anyone
with half a brain, but it will please the suits.


> because even with perl2cc, bytecode and par all together, you won't be able
> to compile this all into an executable with scripts and all static libraries
> directly...

perlcc already tries to do just that.


> If someone cares to create this list? It should probably be
> [EMAIL PROTECTED], because I believe the main focus is on shipping it with
> Perl 6, althought I think we'll probably have a preliminary version on Perl
> 5.

There's nothing about par particularly related to perl6, except that
it would be nice if it was distributed with it.  Some of the potential
features in Perl 6 would be useful to par, but then they'll be useful
to just about every Perl program.

In the end, its just a name.

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

> par can do something similar.  It can slap a copy of pun (and thus
> perl) onto the archive.  Its not simple, and its platform dependent,
> but its useful.  I'm more and more seeing par as a way of
> embrace/extend/destroying perl2exe.
> 
> And I think we could squeeze something into 5.8.

Careful with that keyboard.

The problem of packaging and compressing is easy.

The problem of unpacking, or in other words, installing, or in other
words, embedded hardwired paths is hard.  Think library paths: both
pure Perl libraries *and* shared libraries.  In theory this is easy:
the portable (and grody) solutions is to have a "long enough" filler
path(s) that is(are) then at install time binary patched.  In
practice, though, for some reason no one seems to have done this so
that this would be a widely used practice and well-known art.  In your
answer remember other UNIX platforms than Linux and other platforms
than UNIX.
 
-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 06:35:17PM +, Nicholas Clark wrote:
> "par" stood for what?

Perl ARchive, just like jar (Java ARchive).  "par" will be the utility
to create pars.  To run a par, you'd use a seperate utility (so an
end-user doesn't have to carry around all the extra junk associated
with creating a par)  Unfortunately, "punar" is difficult to pronounce
so we will have to go with the shorter, easier to speak "pun". :P


> I'm assuming it's seen as a general (not just perl6) thing?

There seems to be two minds about this.  I think its perfectly
feasible to do now (perl2exe already does what par proposes) and
distribute seperately (and possibly with 5.8).  Others think its
better held as a Perl 6 utility and maybe make a prototype now.

Both can do their thing at the same time, doesn't matter.

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Mon, Feb 12, 2001 at 01:50:39PM -0500, [EMAIL PROTECTED] wrote:
> On Mon, Feb 12, 2001 at 04:01:31PM -0300, Branden wrote:

> > >  Loading a Perl module from a filehandle might
> > > screw with .
> > 
> > As resource files can be attached to the archive, I think not allowing
> > __DATA__ wouldn't be such a big restriction. I don't know if we can find
> > other workarounds.
> 
> This kills SelfLoader and unsplit AutoLoader modules and similar
> schemes.  AutoLoader isn't so much a problem, since we can just run
> AutoSplit over it.  SelfLoader will be tricky.

I found (no surprise with hindsight) that doing zips with source filters
screwed  (for want of a better term)
Doing them with sfio made  work just fine. You got 
uncompressed as you read it. PerlIO will work the same way
It's only going to screw modules that expect to be able to seek 
I've not yet met any of them

Autoloader works no problem once all the .al files are in a zip
(it uses require)

s/zip/something else/g to generalise

Nicholas Clark

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 04:01:31PM -0300, Branden wrote:
> I don't really see much _conceptual_ difference between rpm, deb, and the
> other package formats used by Linux.

debs store alot of information that rpm doesn't, and it would be good
to look at to steal good ideas.  Also, and most importantly, they have
dselect, which is similar, but much more powerful, than CPAN and the
CPAN shell.  That's something to look at.


> I don't know, are these the problems with MacPerl you're
> talking about ?

Yeah, just alot of little things.


> > par?  Binaries?  Bytecode?  Not until perlcc works, which won't be for
> > years if we assume it won't work until Perl 6.
> 
> I see no problem, since this is perl6-language...
> 
> I really think par/pun are Perl 6 things. I realise module distribution with
> Perl 5 is nasty, but I really don't think we can do something about it now.
> IMO, if it's not shipped with Perl, most layman users won't have it, and

Ever made a self-extracting archive?  Its just a zip file attached a
little unzip binary.  Send that to someone, they run it and it
decompresses itself!  chicken<->egg.

par can do something similar.  It can slap a copy of pun (and thus
perl) onto the archive.  Its not simple, and its platform dependent,
but its useful.  I'm more and more seeing par as a way of
embrace/extend/destroying perl2exe.

And I think we could squeeze something into 5.8.


> > I've been thinking about this... would dumping things into /tmp be
> > considered cheating?
> 
> Probably not, but avoiding it would be appreciated (for the objectives
> above -- consider a diskless machine).

Ok, we can have some flags and configuration bits figure that out.


> >  Loading a Perl module from a filehandle might
> > screw with .
> 
> As resource files can be attached to the archive, I think not allowing
> __DATA__ wouldn't be such a big restriction. I don't know if we can find
> other workarounds.

This kills SelfLoader and unsplit AutoLoader modules and similar
schemes.  AutoLoader isn't so much a problem, since we can just run
AutoSplit over it.  SelfLoader will be tricky.


> > > | Merge many archives  | Yes | No  | Yes |
> > > | in one   | | | |
>
> Take some scripts and lots of .par (containing modules needed by the
> scripts) and make all this directly into one .par, without needing for
> having the .pars uncompressed somewhere.

Heh, simplest way to do that is to cheat and just decompress all the
pars and put them back together.  There will be problems, for
instance, with two pars that use the same modules but different
versions.  Default behavior might be to use the newest version and
throw a warning.  Otherwise our par could have several different lib/
directories.  For instance:

lib/ contains the unconflicting modules
lib/CGI-2.49/  contains version 2.49 of CGI.pm
lib/CGI-2.62/  contains version 2.62 of CGI.pm

And then scripts which need 2.49 can be run with "perl -Ilib/CGI-2.49"
and the other with "perl -Ilib/CGI-2.62".

This is ok for running from within the par, but installing will get
tricky under perl5.  perl6 will presumably have something to solve
this (as noted in the recent lengthy discussion).


> > > | Signed archives  | No  | No  | Yes |
> 
> Well, you'll probably help me with this, if that's OK with you. Could you
> send me some text/URLs saying how it works so that I can figure out one way
> to do it and include it in the PDD.

Oh, I'm the last person to ask about crypto.  I don't even know how to
use pgp.  I just peeked into a jar and saw how they did it.
Basically, they have a manifest file like:

Manifest-Version: 1.0
Main-Class: bsh.Console

Name: bsh/lib/defaultImports
Digest-Algorithms: SHA MD5 
SHA-Digest: 20px9AdMKT+bm//JCDSIaMVES/c=
MD5-Digest: j+s36wWf8zHghz5I8jxURQ==

Name: bsh/lib/javaPackages
Digest-Algorithms: SHA MD5 
SHA-Digest: y52XNqru6tU8BXGybyzJ7Gvy49g=
MD5-Digest: q7Z7lIxNwxmKVXMBeEiN4A==

and then there's another file for storing RSA information, that's a
bit more complicated.  However, I figure tacking a pgp or gpg
signature onto each package in the manifest wouldn't be too hard.


> > We'll just have to use something other than RSA most likely.
> 
> Why? Problems with exporting cryptosystems? If that's it, how does
> Java/Netscape do it?

Actually, the RSA patent just expired, didn't it?  Even gpg has it.
Ok, then maybe we can use RSA.  OTOH, IDEA is right out.

As for export, the US recently relaxed its laws, especially relating
to Open Source projects (maybe because they assume no bunch of
unfunded geeks could out-think the NSA ;)
http://www.crypto.com/exports/, http://www.cdt.org/crypto/ and
http://www.cdt.org/crypto/admin/

Again, this is just like the compression format.  Its all flexible,
its not really important.

Re: Auto-install (was autoloaded...)

[Branden]

[EMAIL PROTECTED] wrote:
> What are we doing with it?  We are killing perl2exe.

Not exactly.

> The niches of:
>
> 1. "I don't want to use modules because the end-user might not have
> them installed"
>

Yes.

> 2. "My end-users might not have Perl installed"  Bundling a Perl
>interpretor with your program (until perlcc is viable)
>

No. I don't expect Perl installation or any other otherwise executable or
installation program to be bundled inside of archives. Correct if I'm wrong,
but archives are meant for Perl scripts and Perl modules. (documentation,
resource files, binaries of compiled extensions, ext. language source code,
and even binaries that instantiate a perl interpreter are ok, but not perl
libs.)


> 3. "I want to hide my source code from Evil Vicious Users!"  A
>consequence of #2, however we can make the "hiding" very, very
>cheesy and pun can "decompile".  But that's good enough for most.
>

No. One objectives of `par' is ``g) The archive file should be stored in a
format that can be easily created and inspected with widely available
external tools.'' (from alpha PDD). That means every user can access the
code inside the archive without problems. And this will be documented, so
that developpers can mess with their archives themselves.


> Unfortunately, this might hurt IndigoSTAR (makers of perl2exe) and the
> last thing I want to do with par is damage a Perl-friendly business.
> I'm not entirely sure how to approach them on this.
>

As I said before, #2 and #3 will not be hurt by `par'. However, `perlcc' in
Perl 6 will probably make building static executables easy and compiling to
bytecode will hide the source code, so I think `perl2exe' will probably and
unfortunately loose its mean, in Perl 6.

I actually agree that this is undesired (as you said, IndogoSTAR is
perl-friendly). But maybe there will still be people that want `perl2exe',
because even with perl2cc, bytecode and par all together, you won't be able
to compile this all into an executable with scripts and all static libraries
directly... And probably there will be still some Perl 5 applications that
benefit from it too.


>
> > 2: Is this really still language? If not, where should we be discussing
it?
>
> [EMAIL PROTECTED] anyone?
>

If someone cares to create this list? It should probably be
[EMAIL PROTECTED], because I believe the main focus is on shipping it with
Perl 6, althought I think we'll probably have a preliminary version on Perl
5.

- Branden

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Mon, Feb 12, 2001 at 01:14:58PM -0500, [EMAIL PROTECTED] wrote:
> On Mon, Feb 12, 2001 at 05:45:17PM +, Nicholas Clark wrote:
> > When I last tried it (over a year ago) running the 5.005 regression tests
> > with the standard libraries coming out of a zip file took about the same
> > time as running the regression tests with the standard libraries on disk.
> > 
> > [x86 BSD unix, fairly big machine, SCSI disks - something I'd expect to
> > be good at IO]
> 
> Alot of spare memory?  I'd suspect it probably just tossed everything
> into the cache.  Windows is fairly awful at doing that sort of thing.
> Also, did you decompress the libraries once at the start of the test
> and then throw them on disk?  Or did you decompress for each and every
> test script?

See
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1999-02/msg01379.html
for go three

In answers to questions
1: I can remember when plum.flirble.org had 196Mb, and when it got upgraded
   to 640Mb. Either way, I think that counts as "a lot of spare memory"
2: No, libraries didn't get thrown out to disk. No temporary files or child
   processes of any sort were used [no camels were harmed in the making of...]
   Include files got decompressed as needed. There was a single linear
   pass of the zip central directory, which was used to build a hash of
   files in that zip (which lasted for the duration of that perl
   interpreter). You look in the hash - if you find that file there, great
   [seek direct to correct point in zip file, validate local header, start
decompressing]. Else resume linear search of the central directory where
   you left off before, until you find the file you are interested in.
   If you hit the end of the central directory, mark the hash as complete
   [so as not to bother with any more scanning] and return not found

The only things that I know I should change are

a: It's not thread safe
b: It didn't store the last modified time of the zip, and chuck the cashed
   central directory and restart from the beginning if someone modified the
   zip file underneath it

> 1. "I don't want to use modules because the end-user might not have
> them installed"

yes, you can stick a zip file on the end of your perl script.

> 2. "My end-users might not have Perl installed"  Bundling a Perl
>interpretor with your program (until perlcc is viable)

yes, you can stick a zip file on the end of your perl interpreter.
Not sure if you get needless waste of memory like this
[I believe on Unix the pages of the file that aren't really the executable
don't ever come in from disk, so there's no waste of effort when you open
the file a second time to read it as a zip.  If there is, cunning tricks
with symbol tables to treat the zip "file" as a big void* blob in the
executable would be needed. This also is potentially something commercial]

> 3. "I want to hide my source code from Evil Vicious Users!"  A
>consequence of #2, however we can make the "hiding" very, very 
>cheesy and pun can "decompile".  But that's good enough for most.

can't do this

> Unfortunately, this might hurt IndigoSTAR (makers of perl2exe) and the
> last thing I want to do with par is damage a Perl-friendly business.
> I'm not entirely sure how to approach them on this.

they still have a selling point on number 3. And for practical purposes
they aren't vapourware. [IIRC the URL I quoted needed a perl built with
sfio. As I understand it, fine for MacPerl, which already defaults to sfio.
Not the default anywhere else. PerlIO in perl 5.7 will probably be the
default in perl 5.8, and I'm currently playing with that]

> > 2: Is this really still language? If not, where should we be discussing it?
> 
> [EMAIL PROTECTED] anyone?

"par" stood for what?

I'm assuming it's seen as a general (not just perl6) thing?

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Branden]

[EMAIL PROTECTED] wrote:
>
> I'm not planning on waiting for Perl 6 to start work on par, so Moore
> isn't with us.
>

Agreed, with the condition that we all make the specification for it
together, and it remains compatible with `par' that will be shipped with
Perl 6.

And I'll probably ask you to use another naming/extension, like pp5 (par for
perl 5), so that modules for both versions don't get mixed up (since they'll
be incompatible).

Other than that, I believe developping it for Perl 5 is a very good idea,
since it will let us see if we need another features, and make benchmarks
with all proposed formats. And probably `p52p6' can be used to translate it
to Perl 6 and include it in Perl 6 distribution, cutting much work (and
time) when Perl 6 is ready.

- Branden

Re: Auto-install (was autoloaded...)

[schwern]

On Mon, Feb 12, 2001 at 05:45:17PM +, Nicholas Clark wrote:
> When I last tried it (over a year ago) running the 5.005 regression tests
> with the standard libraries coming out of a zip file took about the same
> time as running the regression tests with the standard libraries on disk.
> 
> [x86 BSD unix, fairly big machine, SCSI disks - something I'd expect to
> be good at IO]

Alot of spare memory?  I'd suspect it probably just tossed everything
into the cache.  Windows is fairly awful at doing that sort of thing.
Also, did you decompress the libraries once at the start of the test
and then throw them on disk?  Or did you decompress for each and every
test script?


> 1: I don't see why we need to decide on actual format right now. Surely
>what we want to be able to do with it is more important?

We don't.  The format doesn't matter at all, it will be flexible.

What are we doing with it?  We are killing perl2exe.  The niches of:

1. "I don't want to use modules because the end-user might not have
them installed"

2. "My end-users might not have Perl installed"  Bundling a Perl
   interpretor with your program (until perlcc is viable)

3. "I want to hide my source code from Evil Vicious Users!"  A
   consequence of #2, however we can make the "hiding" very, very 
   cheesy and pun can "decompile".  But that's good enough for most.

Unfortunately, this might hurt IndigoSTAR (makers of perl2exe) and the
last thing I want to do with par is damage a Perl-friendly business.
I'm not entirely sure how to approach them on this.


> 2: Is this really still language? If not, where should we be discussing it?

[EMAIL PROTECTED] anyone?

Re: Auto-install (was autoloaded...)

[Branden]

[EMAIL PROTECTED] wrote:
> On Fri, Feb 09, 2001 at 06:17:34PM -0200, Branden wrote:
> > I put together a comparison table between par and rpm/jar.
>
> You forgot deb, which I'd *much* rather deal with than rpm (if only
> because I can point apt and dselect at CPAN).  You also forgot the "Is
> Vaporware?" category. ;)
>

I don't really see much _conceptual_ difference between rpm, deb, and the
other package formats used by Linux. I think all of them suffer from the
platform-specific problems, since I know they're not available/easy to
install/stable/solid on non-UNIX platforms. Porting them would probably be
much harder than building something that meets our needs from scratch.


>
> Its going to be tricky getting par working under MacPerl.  It can be
> done, but we're going to have to be very careful about what features
> we use.
>

I guess not. It will probably use only the `plain' features of underlying
archive format (no special data or info attached to files, only default
compression methods, ...). This means it will basically need only zlib
(which AFAIK is very portable, inclusive to Macs), and Archive::Zip that
only parses the file format, which is probably also portable. Filenames
(which are also a problem for Macs) can be translated by par/pun to a
standard naming. I don't know, are these the problems with MacPerl you're
talking about ?



>
> > | Supports binary, | Yes | Yes | No  |
> > | source and bytecode  | | |  (source?)  |
>
> par?  Binaries?  Bytecode?  Not until perlcc works, which won't be for
> years if we assume it won't work until Perl 6.
>

I see no problem, since this is perl6-language...

I really think par/pun are Perl 6 things. I realise module distribution with
Perl 5 is nasty, but I really don't think we can do something about it now.
IMO, if it's not shipped with Perl, most layman users won't have it, and
that defeats most of its purpose. Get it into Perl 5.8 (when it gets the
street) is valid to a certain point, but not every user would get it...



> > | Run from archive | Yes | No  | Yes |
>
> This is going to be tricky to make efficient,

I guess that's not meant to be efficient. I propose it so that:
1) You can try to run it to see if you like it before you install it
definitely.
2) One application can download an archive from Internet and run it inside
its own environment (s/application/browser/ and you have an applet).


> especially with tar.gz
> since we have to decompress the *whole* thing before we can pull
> individual files out of the archive.  I don't think zip has this
> problem.  Dunno about cpio or ar (which deb uses, I think).
>

That's why zip is my choice.


> I've been thinking about this... would dumping things into /tmp be
> considered cheating?

Probably not, but avoiding it would be appreciated (for the objectives
above -- consider a diskless machine).



>  Loading a Perl module from a filehandle might
> screw with .

As resource files can be attached to the archive, I think not allowing
__DATA__ wouldn't be such a big restriction. I don't know if we can find
other workarounds.


>  And then resources (such as sounds, images,
> etc...) will definately be expensive if we keep having to pull them
> from the archive every time they're used.
>

Actually, if you read a resource file more than once, you're paying the
price of it. Of course decompressing it more than once is more expensive,
but I think that's probably not such a big price to pay for it. As I said,
it doesn't need to be efficient, because installed ones will certainly be.

The `Resource' module that provides access to resouces in the archive could
also be told to cache the resources in memory/disk, so this problem would be
circumvented.




>
> > | Merge many archives  | Yes | No  | Yes |
> > | in one   | | | |
>
> ???  What does this mean?
>

Take some scripts and lots of .par (containing modules needed by the
scripts) and make all this directly into one .par, without needing for
having the .pars uncompressed somewhere.



> Sure, but automatically finding those dependencies will be difficiult.
> Closest I've yet seen is perl2exe, which often makes mistakes.  Java's
> static nature makes it fairly easy.  rpm and deb have to explicitly
> declare it, which they often get wrong.  Ditto CPAN modules.
>
> We might be able to throw something together which scans for the easy
> bits (ie. simple "use" and "require") recursively, and then warns if
> it sees any C in the code (or Module->require if
> UNIVERSAL::require is involved).
>

Exactly what I would propose.



>
> > | Signed archives  | No  | No  | Yes |
>
> Why not?  I was poking around inside a .jar yesterday and saw how they
> do it.  Its pretty simple.  It'll be especially simple if CPAN starts
> storing author's public keys.  This is something I'm going to retr

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Mon, Feb 12, 2001 at 12:08:12PM -0500, [EMAIL PROTECTED] wrote:
> On Sat, Feb 10, 2001 at 12:58:34AM +0100, Bart Lateur wrote:
> >  * On a currently normal Pentium of 500MHz, 64Mb, ungzipping and
> > untarring a .tgz archive of 250k (the ungzipped file itself is roughly
> > 1.5Mb) takes roughly 1 second. (ONE second!)
> 
> One second is too slow (for a Unix user it is, maybe not for a Windows

> I'm not planning on waiting for Perl 6 to start work on par, so Moore
> isn't with us.

I agree with the "too slow" opinion [mainly because I'm very impatient :-)]

When I last tried it (over a year ago) running the 5.005 regression tests
with the standard libraries coming out of a zip file took about the same
time as running the regression tests with the standard libraries on disk.

[x86 BSD unix, fairly big machine, SCSI disks - something I'd expect to
be good at IO]

The IO gain from having the libraries all in one smaller file rather than
scattered about a bit seemed to offset the CPU loss in having to actually
decompress files rather than just copy them.

Please don't take this as pro-zip anti tar

1: I don't see why we need to decide on actual format right now. Surely
   what we want to be able to do with it is more important?
2: Is this really still language? If not, where should we be discussing it?

Nicholas Clark

Re: Auto-install (was autoloaded...)

[schwern]

On Sat, Feb 10, 2001 at 12:58:34AM +0100, Bart Lateur wrote:
>  * On a currently normal Pentium of 500MHz, 64Mb, ungzipping and
> untarring a .tgz archive of 250k (the ungzipped file itself is roughly
> 1.5Mb) takes roughly 1 second. (ONE second!)

One second is too slow (for a Unix user it is, maybe not for a Windows
user ;P).  As in illustration, type "sleep 1" into your shell.  Then
add onto that the time spent actually running the program (especially
if its a Tk/Gtk program, which par will probably be used for
frequently).  Also, don't forgot the cost of doing checksums on the
code, as well as possibly signature checks.  Then add in that a
Pentium 500 is top-of-the-line (I'd like to know what your drive
performance is).

I'm not planning on waiting for Perl 6 to start work on par, so Moore
isn't with us.

Re: Auto-install (was autoloaded...)

[schwern]

On Fri, Feb 09, 2001 at 06:17:34PM -0200, Branden wrote:
> I put together a comparison table between par and rpm/jar.

You forgot deb, which I'd *much* rather deal with than rpm (if only
because I can point apt and dselect at CPAN).  You also forgot the "Is
Vaporware?" category. ;)


> | Available in a wide  | Yes | No  | Yes |
> | range of platforms   | | (Win32 +/-, | |
> |  | | MacOS, VMS) | |

Its going to be tricky getting par working under MacPerl.  It can be
done, but we're going to have to be very careful about what features
we use.


> | Supports binary, | Yes | Yes | No  |
> | source and bytecode  | | |  (source?)  |

par?  Binaries?  Bytecode?  Not until perlcc works, which won't be for
years if we assume it won't work until Perl 6.

> | Run from archive | Yes | No  | Yes |

This is going to be tricky to make efficient, especially with tar.gz
since we have to decompress the *whole* thing before we can pull
individual files out of the archive.  I don't think zip has this
problem.  Dunno about cpio or ar (which deb uses, I think).

I've been thinking about this... would dumping things into /tmp be
considered cheating?  Loading a Perl module from a filehandle might
screw with .  And then resources (such as sounds, images,
etc...) will definately be expensive if we keep having to pull them
from the archive every time they're used.


> | Merge many archives  | Yes | No  | Yes |
> | in one   | | | |

???  What does this mean?

> | Usable with external | Yes | No  | Yes |
> | tools (e.g. WinZip)  | | | |

I don't know about WinZip, but we can use a CPAN shell style "poke
around and see what you've got"

> | Dependencies of  | Yes | Yes | No  |
> | the archive  |  (included) | | |

Sure, but automatically finding those dependencies will be difficiult.
Closest I've yet seen is perl2exe, which often makes mistakes.  Java's
static nature makes it fairly easy.  rpm and deb have to explicitly
declare it, which they often get wrong.  Ditto CPAN modules.

We might be able to throw something together which scans for the easy
bits (ie. simple "use" and "require") recursively, and then warns if
it sees any C in the code (or Module->require if
UNIVERSAL::require is involved).


> | Signed archives  | No  | No  | Yes |

Why not?  I was poking around inside a .jar yesterday and saw how they
do it.  Its pretty simple.  It'll be especially simple if CPAN starts
storing author's public keys.  This is something I'm going to retrofit
onto the CPAN shell anyway (err, someday).

We'll just have to use something other than RSA most likely.

Re: Auto-install (was autoloaded...)

[Branden]

Bryan C. Warnock wrote:
> Is that '.tar and .zip' as in '.tar and .zip' or '.tar or .zip'?

.tar or .zip

> Aren't most tars still unindexed, requiring a full file scan anyway?

That was one I was not aware of... One more reason to use .zip!

Hey, .tgz people... Java's jar has used .zip as its format very successfully
until now. I really don't see a reason why we shouldn't do that. I agree
flexible is good, and I think that flexible would be easy if Archive::Tar,
Archive::Zip, Archive::* share the same interface. I actually can't say that
it's possible, because as this message is pointing, there are many
differences between the features that different archive formats offer, and I
can't see how they would offer the same functions... So I think we can start
with zip, and postpone the flexibility problem for when we define the
Archive::* implementations for Perl6, possibly stating that we'd like them
to have the same interface, so that we can use one or another with the same
code.

- Branden

Re: Auto-install (was autoloaded...)

[Branden]

Jarkko Hietaniemi wrote:
> On Mon, Feb 12, 2001 at 12:36:53PM -0300, Branden wrote:
> > The problem is that we cannot access individual files inside the archive
> > without decompressing the whole archive, what is possible with .tar (not
>
> I do not see a huge problem in decompressing the whole archive if needed.

Not a huge problem, indeed. With .zip would be ``no problem at all, since
the there's no need to decompress the whole archive''.

- Branden

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Fri, Feb 09, 2001 at 10:28:49AM -0200, Branden wrote:
> In http://www.w3.org/TR/NOTE-OSD.html#B they describe platform/cpu standard
> names, and we'll definetly need those for checking target architecture. Can
> we standardize upon those, or there's something missing? There's an issue

The info-zip home page has a list of things unzip runs on, which effectively
gives a very comprehensive list of OS types:

http://www.info-zip.org/pub/infozip/UnZip.html

the link to C-Kermit at the foot of the page reveals a large list of Unix
variants:

http://www.columbia.edu/kermit/unix.html

[The zip file "spec" is on infozip's pages:
http://www.info-zip.org/pub/infozip/doc/appnote-iz-latest.zip
the gzip file format is rfc1952 
the deflate compression method used by gzip is rfc1951

zip files can use a variety of compression methods - deflate is method 8.
As best I can tell only deflate and storing uncompressed (method 0) are
widely used nowadays.
In theory there's nothing stopping you putting a bzip2 stream (or whatever
arj or rar or any of the other archivers use) inside a zip archive if you
standardise on the compression method number.
Notice also the distinction - zip an archive format (put lots of files in
one file and take them out again) which allows entries to be compressed.
tar is just an archive format, gzip just a compression system.
how very unix - combine small tools to get a job done :-)
]

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Branden]

Bart Lateur wrote:
> On Fri, 9 Feb 2001 21:18:55 +, Nicholas Clark wrote:
>
>  * Archive::Tar is part of the Perl 5.6.0 distributions for Win32
> (Activestate and IndigoPerl)
>
>  * And I'd like the normal module distributions on CPAN to still work.
> Those are all .tar.gz.
>

I actually don't believe the modules already on CPAN will work directly,
since they'll probably have to go through p52p6 translation. And I don't see
any problem in supporting the old approach to installing modules, since it's
mainly manual and requires almost no support from the programs.

- Branden

Re: Auto-install (was autoloaded...)

[Bryan C. Warnock]

On Monday 12 February 2001 10:36, Branden wrote:
> The problem is that we cannot access individual files inside the archive
> without decompressing the whole archive, what is possible with .tar (not
> ..gz) and .zip. Then we can assign a perl filehandle to access a file from
> inside the archive and decompress it on-the-fly.

Is that '.tar and .zip' as in '.tar and .zip' or '.tar or .zip'?
Aren't most tars still unindexed, requiring a full file scan anyway?

-- 
Bryan C. Warnock
[EMAIL PROTECTED]

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Mon, Feb 12, 2001 at 12:36:53PM -0300, Branden wrote:
> 
> John Porter wrote:
> > Branden wrote:
> > >
> > > For example, with tgz it would be complex to deal
> > > with running without extracting,
> >
> > What?  tar -z not good enough for you?
> >
> 
> The problem is that we cannot access individual files inside the archive
> without decompressing the whole archive, what is possible with .tar (not

I do not see a huge problem in decompressing the whole archive if needed.

> .gz) and .zip. Then we can assign a perl filehandle to access a file from
> inside the archive and decompress it on-the-fly.
> 
> - Branden

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Branden]

John Porter wrote:
> Branden wrote:
> >
> > For example, with tgz it would be complex to deal
> > with running without extracting,
>
> What?  tar -z not good enough for you?
>

The problem is that we cannot access individual files inside the archive
without decompressing the whole archive, what is possible with .tar (not
.gz) and .zip. Then we can assign a perl filehandle to access a file from
inside the archive and decompress it on-the-fly.

- Branden

Re: Auto-install (was autoloaded...)

[James Mastros]

You should probably also take a look a Debian's packaging, the .deb.  

It consists of an ar archive containing three files: one for the magic
(named debian-binary, containing "2.0"), one for the filesystem image
(filesystem.tar.gz)

On Fri, Feb 09, 2001 at 06:17:34PM -0200, Branden wrote:
> | Platform independent | Yes | Yes | Yes |
Yes.

> | Available in a wide  | Yes | No  | Yes |
> | range of platforms   | | (Win32 +/-, | |
> |  | | MacOS, VMS) | |
No -- only debian, but that includes several HW archs, and both linux and
the hurd.  But source should be portable and abstracted decently.

> | Allow platform   | Yes | Yes | No  |
> | dependent deployment | | | |
Yes.

> | Supports binary, | Yes | Yes | No  |
> | source and bytecode  | | |  (source?)  |
Yes.  Source format is .dsc (metadata) + .tar.gz (upstream) + .patch.gz
(debian patches).  Keeping that would allow for many CPAN packages to be
used unmodified.  Not keeping it would allow for single-file distribution.

> | Install archive  | Yes | Yes | No  |
> | automatically| | |  (manually) |
Yes.

> | Uninstall and| Yes | Yes | No  |
> | upgrade archive  | | | |
Yes.

> | Install, uninstall   | No  | Yes | No  |
> | and upgrade scripts  | (possibly)  | | |
Yes.

> | Run from archive | Yes | No  | Yes |
No, but certianly possible.  (Replace the .tar.gz files with .zips.  Worse
compression but easyer to use individual files.  We could do .bz2.tar or
somesuch, but that's nastyer for others to deal with.)

> | Resources| Yes | Yes | Yes |
Yes, I think.  (Not certian what you mean by this.)

> | Documentation| Yes | Yes | No  |
Yes.

> | Supports various | Yes | No  | Yes |
> | modules per archive  | |(yes)| (packages)  |
No (one file, one package).  This could easily be changed, though.

> | Merge many archives  | Yes | No  | Yes |
> | in one   | | | |
No, but wouldn't be hard with small extention.

> | Usable with external | Yes | No  | Yes |
> | tools (e.g. WinZip)  | | | |
Yes, with a little pain (ar archive + .tar.gz files).

> | Dependencies of  | Yes | Yes | No  |
> | the archive  |  (included) | | |
Yes.  Complex dependencies supported (versions between A and B), and
support for autogeneration of dependencies in many cases.

> | Build archive from   | Yes | Yes | No  |
> | source tree  | | (external)  | |
Yes.

> | Could be bundled | Yes |  Probably   |  Maybe (if  |
> | with Perl 6? | | No  |  we bundle  |
> |  | |  (too big)  |  a JVM too) |
Yes.  (I think.)  (Binary of format-handling program is 110,288 -- a bit on
the big side, but not too bad.)

> | Signed archives  | No  | No  | Yes |
No.  (Source packages are signed, though.)  (At present, feature is planned
for future, and shouldn't be all that hard.)

-=- James Mastros
-- 
"All I really want is somebody to curl up with and pretend the world is a
safe place."
AIM: theorbtwo   homepage: http://www.rtweb.net/theorb/

Re: Auto-install (was autoloaded...)

[Andy Dougherty]

On Sat, 10 Feb 2001, Branden wrote:

> Another advantage I see on tar and gzip is that they are used by GNU, so I'm
> pretty sure there probably wouldn't be any licensing issues, and I'm not
> quite sure .zip doesn't use LZW, the same compression method of GIF...

Zip uses the same compress method as gzip.  In fact gzip's author,
Jean-loup Gailly, is also a prominent contributor to InfoZip -- the group
behind Zip and Unzip.

I don't see any licensing advantage to gzip over zip or vice-versa.

-- 
Andy Dougherty  [EMAIL PROTECTED]
Dept. of Physics
Lafayette College, Easton PA 18042

Re: Auto-install (was autoloaded...)

[Filipe Brandenburger]

Jarkko Hietaniemi wrote:
> On Fri, Feb 09, 2001 at 06:46:26PM -0200, Branden wrote:
> > Jarkko Hietaniemi wrote:
> > > Whatever we do I would much prefer being package format agnostic
> > > instead of tying ourselves too tightly with some single format.
> > Any ideas on how to do that? Without breaking requirements?
> There isn't a software problem another abstraction layer can't fix...
> Design and implement a package format independent API that has calls
> like %files = $package->getfiles(), $package->getfile($file),
> $package->gettotalsize(), $package->dependencies(),
> $package->install(), $package->uninstall(), and at lower levels
> (non-casual-user) calls like guessformat($file).  Yes, this would be
> harder than sticking with one format, but so much more flexible.
> An API and its implementation would be very beneficial also in
> installing other software than just Perl.


I agree that Archive::XYZ should all share the same interface (see threads
about interfaces), and `par' and `pun' tools could probably `plug' to one of
them and use them.

Only I'm relutant to add another 100KB to Perl distribution to include
Archive::Zip, Archive::Tar, Archive::Tgz, Archive::MacBin, Archive::Hqx,
Archive::Rpm. Not to mention maintain all these modules. Modules that ship
with Perl should be stable and high quality, and I don't think it's worth
having all them made and tested before releasing Perl 6.0.0. The less work
we have to ship it, the better it is.

Another issue is culture. Current modules are mostly all deployed in .tar.gz
format (see in CPAN if you can find some that aren't, there probably are,
but they are minorities). And not because .tar.gz is superior to others, but
because when MakeMaker was first made it had to package them someway.
Tar/Gzip got there probably because of convenience. Now we have the big
chance to choose one by features instead.




> > problems (like `oh! I don't have bzip2 and the developper only supplied
a
> > bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
> > whatever3 versions of the same thing only to satisfy users that use
> > different formats').
> And how is this different from 'oh! I don't have unzip'?

Zlib (handles .zip, right?) and Archive::Zip would be included with Perl
distribution. Doing it for one format is good, doing for all of them is
certainly trouble.


> > I think we should go for `standard' rather than `flexible' on this one.
> As I said on this I disagree.
> I have no problem with in the first implementation round doing just zip:
> what I abhor is the idea of hardwiring the zip assumption deeply intoour
design.

Won't be hardwired, only will be default. Other archive formats will be
supported by having the same interface for Archive::*.

- Branden

Re: Auto-install (was autoloaded...)

[Branden]

Actually, I'm starting to like .tgz. With .tgz you wouldn't be able to run
the program without extracting the archive, but if you store the archive as
.tar, there would be no problem. .tgz could be used for network
transmission, and the archive could either be installed or stored as a .tar
in a standard location. And after all, Archive::Tar is already there.
Another advantage I see on tar and gzip is that they are used by GNU, so I'm
pretty sure there probably wouldn't be any licensing issues, and I'm not
quite sure .zip doesn't use LZW, the same compression method of GIF...

I don't know if it would be nice or bad to provide Mac with .sea and .bin (I
think they also have .hqx, ???). Standardizing is still an issue here...

Anyway, if various archive formats are decided, there should be (either
bundled with Perl or provided as a CGI service so that the user could
request an archive in a determined format) an utility to convert a package
from one format to another. Otherwise, developer's life would be harder...

- Branden

Re: Auto-install (was autoloaded...)

[Bart Lateur]

On Fri, 9 Feb 2001 21:18:55 +, Nicholas Clark wrote:

>> What?  tar -z not good enough for you?
>
>I believe that that wognt work ugnless you have gnu tar.

As a Windows user, I should add:

 * Archive::Tar is part of the Perl 5.6.0 distributions for Win32
(Activestate and IndigoPerl)

 * On a currently normal Pentium of 500MHz, 64Mb, ungzipping and
untarring a .tgz archive of 250k (the ungzipped file itself is roughly
1.5Mb) takes roughly 1 second. (ONE second!) That is the 0.072 version
of Archive::Tar, which does in memory unarchiving only.

Needless to say, compressing is a lot slower.

 * And I'd like the normal module distributions on CPAN to still work.
Those are all .tar.gz.

-- 
Bart.

Re: Auto-install (was autoloaded...)

[abigail]

On Fri, Feb 09, 2001 at 09:22:13PM +, Nicholas Clark wrote:
> 
> Code to do unzip (yes, even including the whole of zlib just like gcc,
> xfree86 and several other things I can't remember offhand that irritate
> me as I have libz.so) is small enough to go in the perl core if needed.


Even after reading this twice, I still get the impression you think
both gcc and xfree86 would fit in the perl core snuggly.

Oh well, as long as you don't fit emacs in it; there's got to be some
limit.


Abigail

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Fri, Feb 09, 2001 at 03:25:42PM -0600, Jarkko Hietaniemi wrote:
> I assume the next logical thing to add would be MD5?

Yes, well before zip (IMHO). It would make CPAN.pm a happy bunny, and should
make the world (feel) more secure.

This is now really perl5 isn't it?

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 09:22:13PM +, Nicholas Clark wrote:
> On Fri, Feb 09, 2001 at 02:53:43PM -0600, Jarkko Hietaniemi wrote:
> > On Fri, Feb 09, 2001 at 06:46:26PM -0200, Branden wrote:
> > > problems (like `oh! I don't have bzip2 and the developper only supplied a
> > > bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
> > > whatever3 versions of the same thing only to satisfy users that use
> > > different formats').
> > 
> > And how is this different from 'oh! I don't have unzip'?
> 
> Code to do unzip (yes, even including the whole of zlib just like gcc,
> xfree86 and several other things I can't remember offhand that irritate
> me as I have libz.so) is small enough to go in the perl core if needed.

I assume the next logical thing to add would be MD5?

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 09:18:55PM +, Nicholas Clark wrote:
> On Fri, Feb 09, 2001 at 04:07:51PM -0500, John Porter wrote:
> > Branden wrote:
> > > 
> > > For example, with tgz it would be complex to deal
> > > with running without extracting, 
> > 
> > What?  tar -z not good enough for you?
> 
> I believe that that wognt work ugnless you have gnu tar.

So very true.  Good people, the world is not Linux is it was not VAX.
(one may have "gzip" in which case one can "gzip -dc foo.tgz|tar ...")

We must be flexible/robust/friendly.

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Fri, Feb 09, 2001 at 02:53:43PM -0600, Jarkko Hietaniemi wrote:
> On Fri, Feb 09, 2001 at 06:46:26PM -0200, Branden wrote:
> > problems (like `oh! I don't have bzip2 and the developper only supplied a
> > bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
> > whatever3 versions of the same thing only to satisfy users that use
> > different formats').
> 
> And how is this different from 'oh! I don't have unzip'?

Code to do unzip (yes, even including the whole of zlib just like gcc,
xfree86 and several other things I can't remember offhand that irritate
me as I have libz.so) is small enough to go in the perl core if needed.
It's smaller than Encode by a long way :-)
[or is that :-(]


> > I think we should go for `standard' rather than `flexible' on this one.
> 
> As I said on this I disagree.
> 
> I have no problem with in the first implementation round doing just zip:
> what I abhor is the idea of hardwiring the zip assumption deeply into
> our design.

I would agree - hardwiring is to be avoided.

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 09:42 AM 2/9/2001 +, Michael G Schwern wrote:
>On Thu, Feb 08, 2001 at 01:40:52PM -0500, Dan Sugalski wrote:
> > >Seperated documentation is no documentation.
> >
> > At some point things are going to get split out, unless you wedge the docs
> > into the actual program itself. (You were, after all, talking about config
> > files and XS modules, and those can't usefully stay inside the archive)
>
>True, but those things are essential to making the program work, and
>their loss will be dearly (and immediately) noted.  Documentation is
>the first thing to get ditched in any install process.  I was mostly
>thinking perldoc (yes, wedging the docs into the program) *plus* the
>normal man page generation and install.  That's how I normally do
>things.

So, then, do you want the bytecode format to have some sort of text segment 
in it so you can compile data files (and docs are data as much as anything 
else) into the output bytecode file? We can do that, I expect.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 11:32 AM 2/9/2001 -0200, Branden wrote:
>Nicholas Clark wrote:
> > > that I really don't know: in the same platform, different compilers
>generate
>  > > incompatible binaries? Because if this happens (and will still happen
>on
> > > Perl 6) the platform identification should be os/cpu/compiler. Perhaps
>each
> >
> > on perl 5 different configure options generate different binaries.
>
>Can this be standardized somehow? Perl 6 binary level compatibility, for
>extensions? In a way that distributing binary modules would be at least
>possible? It's already a big trouble to consider the various different
>os/cpu combinations, configure options is not feasible at all!

Perl, generally speaking, should have a mechanism to indicate compatibility 
based on various compilation issues. Given that this sort of thing impacts 
my platform of choice (VMS) more than others, it will.

>Compilers I
>can take, I don't actually expect gcc/MS VC++ generating compatible dlls,
>althought I think it could be possible...

I'm always amazed that people put up with this sort of nonsense. All 
compilers on a platform should generate compatible code. I don't really 
care whether it's COBOL, PL/I, ADA, C, Fortran, or BASIC.


Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 04:09:28PM -0500, John Porter wrote:
> Jarkko Hietaniemi wrote:
> > > 
> > > (for those of you who didn't get the reference)
> > 
> > Well, I certainly heard the reference before even hearing of Perl or Tom...
> 
> I only ever saw it with his name on it.

I believe the first part of it (adding layers) is an age-old
programming adage, the second part (not overdoing it) may well
be Tom's addition.

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Fri, Feb 09, 2001 at 04:07:51PM -0500, John Porter wrote:
> Branden wrote:
> > 
> > For example, with tgz it would be complex to deal
> > with running without extracting, 
> 
> What?  tar -z not good enough for you?

I believe that that wognt work ugnless you have gnu tar.

"Get New Utilities" as I've heard it called by the advocates.
(yes, I get irritated when I forget that BSD cp doesn't have a -a flag)
However, in our portability quest "Get New Utilities" is not good
[unless we package TomC's Perl Power Tools and just use those]

Nicholas Clark

Re: Auto-install (was autoloaded...)

[John Porter]

Jarkko Hietaniemi wrote:
> > 
> > (for those of you who didn't get the reference)
> 
> Well, I certainly heard the reference before even hearing of Perl or Tom...

I only ever saw it with his name on it.
So if he didn't coin it, then I think he "appropriated" it...

-- 
John Porter

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 04:05:54PM -0500, John Porter wrote:
> Jarkko Hietaniemi wrote:
> > 
> > There isn't a software problem another abstraction layer can't fix...
> 
> "...except the problem of too many layers of abstraction". tchrist
> 
> (for those of you who didn't get the reference)

Well, I certainly heard the reference before even hearing of Perl or Tom...

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[John Porter]

Branden wrote:
> 
> For example, with tgz it would be complex to deal
> with running without extracting, 

What?  tar -z not good enough for you?


> ... than have too much flexibility and end up with some distribution
> problems (like `oh! I don't have bzip2 and the developper only supplied a
> bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
> whatever3 versions of the same thing only to satisfy users that use
> different formats').

You way overstate the difficulty of this problem.
And you also way overstate the alleged downside of having a flexible
solution.


-- 
John Porter

Re: Auto-install (was autoloaded...)

[John Porter]

Jarkko Hietaniemi wrote:
> 
> There isn't a software problem another abstraction layer can't fix...

"...except the problem of too many layers of abstraction". tchrist

(for those of you who didn't get the reference)


-- 
John Porter

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 06:46:26PM -0200, Branden wrote:
> Jarkko Hietaniemi wrote:
> > Whatever we do I would much prefer being package format agnostic
> > instead of tying ourselves too tightly with some single format.
> >
> 
> Any ideas on how to do that? Without breaking requirements?

There isn't a software problem another abstraction layer can't fix...
Design and implement a package format independent API that has calls
like %files = $package->getfiles(), $package->getfile($file),
$package->gettotalsize(), $package->dependencies(),
$package->install(), $package->uninstall(), and at lower levels
(non-casual-user) calls like guessformat($file).  Yes, this would be
harder than sticking with one format, but so much more flexible.

An API and its implementation would be very beneficial also in
installing other software than just Perl.

> problems (like `oh! I don't have bzip2 and the developper only supplied a
> bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
> whatever3 versions of the same thing only to satisfy users that use
> different formats').

And how is this different from 'oh! I don't have unzip'?

> I think we should go for `standard' rather than `flexible' on this one.

As I said on this I disagree.

I have no problem with in the first implementation round doing just zip:
what I abhor is the idea of hardwiring the zip assumption deeply into
our design.

> Of course we still can change the file format, if someone isn't happy about
> zip. That's the advantage of discussing it before making it: it's not here
> for we to regret about it...
> 
> - Branden

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Branden]

Jarkko Hietaniemi wrote:
> Whatever we do I would much prefer being package format agnostic
> instead of tying ourselves too tightly with some single format.
>

Any ideas on how to do that? Without breaking requirements?

I actually agree that being able to use tgz is good, and better compression
like bzip2 would also be desired, but I think we couldn't use it if it would
lack features we want. For example, with tgz it would be complex to deal
with running without extracting, and bzip2 isn't handled with common, widely
available tools.

I think we'd better have one working and stablished (and stable) approach to
do it than have too much flexibility and end up with some distribution
problems (like `oh! I don't have bzip2 and the developper only supplied a
bzip2 version of the archive', or `oh! I'll have to do zip, tgz, bzip2,
whatever3 versions of the same thing only to satisfy users that use
different formats').

I think we should go for `standard' rather than `flexible' on this one.

Of course we still can change the file format, if someone isn't happy about
zip. That's the advantage of discussing it before making it: it's not here
for we to regret about it...

- Branden

Re: Auto-install (was autoloaded...)

[Jarkko Hietaniemi]

On Fri, Feb 09, 2001 at 06:17:34PM -0200, Branden wrote:
> 
> I had the time to do a research in Internet about rpm/jar. The correct URLs
> are:
> * http://www.rpm.org
> * http://java.sun.com/products/jdk/1.1/docs/guide/jar/
> 
> I found great utilitaries in http://www.rpm.org/software.html, we could
> probably steal some of them for `par'. I found out that most of them are
> written in Perl or otherwise Perl-related.
> * rpm2html / rpmfind: The equivalent of CPAN.pm?
> * rpm-cvs: Building a RPM from a CVS source tree.
> * AutoRpm: Update a system with rpms from any source.
> * rpmlint: A lint for rpms packages.
> * Perl-RPM: Attempts Perl bindings for RPM.
> * build-package: Build rpms from source packages.

There's also a tool called 'alien':

http://freshmeat.net/projects/alien/

> Alien converts between the rpm, deb, Stampede slp, and Slackware tgz
> file formats. If you want to use a package from another distribution
> than the one you have installed on your system, you can use alien to
> convert it to your preferred package format and install it.

Whatever we do I would much prefer being package format agnostic
instead of tying ourselves too tightly with some single format.

-- 
$jhi++; # http://www.iki.fi/jhi/
# There is this special biologist word we use for 'stable'.
# It is 'dead'. -- Jack Cohen

Re: Auto-install (was autoloaded...)

[Branden]

I had the time to do a research in Internet about rpm/jar. The correct URLs
are:
* http://www.rpm.org
* http://java.sun.com/products/jdk/1.1/docs/guide/jar/

I found great utilitaries in http://www.rpm.org/software.html, we could
probably steal some of them for `par'. I found out that most of them are
written in Perl or otherwise Perl-related.
* rpm2html / rpmfind: The equivalent of CPAN.pm?
* rpm-cvs: Building a RPM from a CVS source tree.
* AutoRpm: Update a system with rpms from any source.
* rpmlint: A lint for rpms packages.
* Perl-RPM: Attempts Perl bindings for RPM.
* build-package: Build rpms from source packages.


I put together a comparison table between par and rpm/jar. I would include
tgz here, but I didn't want to break the 70 column barrier and I think it
would probably be No for most features anyway... I intend to include the
table with the beta PDD.



   +-+-+-+
  Comparison table:| par | rpm | jar |
+--+-+-+-+
| Platform independent | Yes | Yes | Yes |
|  | | | |
| Available in a wide  | Yes | No  | Yes |
| range of platforms   | | (Win32 +/-, | |
|  | | MacOS, VMS) | |
|  | | | |
| Allow platform   | Yes | Yes | No  |
| dependent deployment | | | |
|  | | | |
| Supports binary, | Yes | Yes | No  |
| source and bytecode  | | |  (source?)  |
|  | | | |
| Install archive  | Yes | Yes | No  |
| automatically| | |  (manually) |
|  | | | |
| Uninstall and| Yes | Yes | No  |
| upgrade archive  | | | |
|  | | | |
| Install, uninstall   | No  | Yes | No  |
| and upgrade scripts  | (possibly)  | | |
|  | | | |
| Run from archive | Yes | No  | Yes |
|  | | | |
| Resources| Yes | Yes | Yes |
|  | | | |
| Documentation| Yes | Yes | No  |
|  | | | |
| Supports various | Yes | No  | Yes |
| modules per archive  | |(yes)| (packages)  |
|  | | | |
| Merge many archives  | Yes | No  | Yes |
| in one   | | | |
|  | | | |
| Usable with external | Yes | No  | Yes |
| tools (e.g. WinZip)  | | | |
|  | | | |
| Dependencies of  | Yes | Yes | No  |
| the archive  |  (included) | | |
|  | | | |
| Build archive from   | Yes | Yes | No  |
| source tree  | | (external)  | |
|  | | | |
| Could be bundled | Yes |  Probably   |  Maybe (if  |
| with Perl 6? | | No  |  we bundle  |
|  | |  (too big)  |  a JVM too) |
|  | | | |
| Signed archives  | No  | No  | Yes |
|  | | | |
|  | | | |
|  | | | |
|  | | | |
|  | | | |
|  | | | |
+--+-+-+-+


- Branden

Re: Auto-install (was autoloaded...)

[Branden]

This is the alpha version of the PDD about archives. I actually didn't have
the time to format it as a POD, and probably won't have the time to do it
until Monday, I don't even think I'll have time to check the lists on the
weekend. Nevertheless, I'm sending it on mail-message format for your
appreciation. I think it has most the information that was discussed here,
plus some thoughts I had.

I expect to collect some comments about it before writing the beta version,
this one in POD format. I'm starting to think this discussion should move to
another list, like -build, -stdlib. Maybe -source-control is related too. I
realise they're not active, but I actually think this is not a -language
issue. Anyway, I'm posting it here, if anyone thinks we should move it to
another list, just name the list and we move there.

Hope you like the text. Help to correct spell and grammar would be very
appreciated.

- Branden

 (cut here) ---



PDD: `par' -- The Perl Achive


1. Introduction

`par' stands by `Perl Archive'. It's a way to provide deploy and
installation of Perl programs/scripts/modules. It helps taking care of
module dependencies, which is mostly painful in Perl 5.



2. Motivation

When a programmer writes a Perl script (or module) and wants to share it
with other users, he probably uploads the code to CPAN, as a .tar.gz file or
something like that. Users are supposed to get the script via http or ftp,
untar it, and run it.

So far it's fine, but the problem begins when the script depends on modules
that aren't included in Perl's standard library, and are supposed to be
installed by users. The first headache starts when the user tries to run the
script and gets a message saying Perl can't find module X. He takes the time
to go to CPAN, download X, install it and try to run the script again, only
to find that the script also needed module Y or that X actually needed Z.
This process continues until there are no more needed modules or the user
gets tired of it and gives up running the script, whatever comes first.

This is a downside for developpers too, because they want to get around
this, and try to give the users installing instructions, telling them they
should install modules X, Y and Z to run the script. And since most layman
users don't care about installing modules, probably they'll also give up
running the script. The outcome is that only scripts that depend only on
standard modules achieve wide acceptance among layman users.




3. Goals


The proposal of this PDD is to define a standard way for developpers to
package their Perl code so that all needed dependencies are included in one
only file. And also define a standard way for users to install or run the
scripts distributed on this archive form.

The specification of `par' should satisfy the following properties:

a) It should allow distribution of Perl code in platform independent form.

b) It should allow distribution of Perl code and compiled C extensions in
platform dependent form.

c) It should allow distribution of Perl code in both bytecode and source
form, even both mixed in the same archive.

d) It should allow the user to install the contents of the archive. It
should also allow the user to uninstall an archive or upgrade to a new
version of it. It should allow the user to choose how the modules contained
on the archive will be installed in relation to the system modules. It
should allow the user to override the directories used for installation.

e) It should allow the user to run the scripts and use the modules contained
in the archive without requiring installation.

f) It should allow the inclusion of documentation and resource files
(images, sounds, dbms, text databases, ...) for all scripts and modules
included in the archive, and documentation for the archive itself.

g) The archive file should be stored in a format that can be easily created
and inspected with widely available external tools.

h) It should allow the developper to merge existing archives together with
files into new archives, so that the developper can package his scripts with
existing binary archives of depended modules.

i) There should be tools that help the developper to find dependencies of
his scripts, and to package his scripts along with modules it depends on an
archive.

j) The `par' mechanism should be bundled by default in Perl 6.0.0. Having it
added later would possibly cause problems with users of Perl 6.0.0 that
don't have it installed and would want to use archives. Not having anything
else on 6.0.0 isn't really that critical, because by using `par' it'll be
easy to install whatever missing module.





4. Proposed Implementation


The zip file format is proposed for packaging the files. It's independent of
platform and has no licensing constraints (must check that...). The reason
to choose zip instead of tar/gzip is that with zip one can access one file
without having to decompress the whole archive. Th

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Fri, Feb 09, 2001 at 10:28:49AM -0200, Branden wrote:
> Other important issue I don't know yet: Is there an Archive::Zip module for
> Perl? How cross-platform is it? Can we bundle it with Perl (licensing
> issues)? Is it stable? Will it give us the support we need (access to
> individual files in the archive, and the ability to `fake' them as real
> files through tied filehandles)?

I'm a bit more worried about Compress::Zlib, which Archive::Zip uses.
There are versions of that for pretty much every OS, including MacOS,
but its historically been a bit weird.  I remember there being a Very
Deep XS Bug that we couldn't quite dig out a while ago.  Or maybe it
was in Archive::Tar... one of the two.

Anyhow, I'm not worried about them much.


-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
Any sufficiently encapsulated hack is no longer a hack.

Re: Auto-install (was autoloaded...)

[Branden]

Nicholas Clark wrote:
> I take it "Lunix" is Linux.
> BSDi isn't FreeBSD, NetBSD or OpenBSD
> Nothing they list seems to be VMS
> Pace are still developing variants of Acorn's RISC OS to run set top boxes
> As I understood it there were about 39 variants of Unix ever, and they
seem
> to have 12 listed.
> [snip]
> The ommission of pa risc makes me wonder about quality of the rest of this
> document.

Clearly an article of Microsoft...

> > that I really don't know: in the same platform, different compilers
generate
 > > incompatible binaries? Because if this happens (and will still happen
on
> > Perl 6) the platform identification should be os/cpu/compiler. Perhaps
each
>
> on perl 5 different configure options generate different binaries.

Can this be standardized somehow? Perl 6 binary level compatibility, for
extensions? In a way that distributing binary modules would be at least
possible? It's already a big trouble to consider the various different
os/cpu combinations, configure options is not feasible at all! Compilers I
can take, I don't actually expect gcc/MS VC++ generating compatible dlls,
althought I think it could be possible...

> I don't know the answers to the other questions. However, I was intending
> to write a layer for perl 5.7 that can access zip entries via regular
> perl filehandles. (In theory this could let you run scripts direct out of
a
> zip without actually unpacking them anywhere)

This seems like an issue that will not only be used by `par', as I see it
brought up many times now and then. Probably should be encapsulated in a
generic module and be bundled in Perl 6's standard distribution.

> Layers of some form are needed in perl6 to cope with utf8 and other
> encodings, so I would expect the perl6 layer functionality can also be
> extended to zip manipulations.
>
> Nicholas Clark

AFAIK, Perl 6 will have a translation layer from bytes to the encoding, and
one of the encodings would be `binary', that would just pass the bytes
through. This is what should probably be used for zip. If you mean the
uncompress-on-the-fly thing I think it would be handled like source filters
are today. In Perl 5.7 you can even have a source filter that decompresses a
.gzipped script and runs it, by using a source filter. I believe zip could
be handled by this also in 5.7. Perl 6 will certainly improve this area very
much.

- Branden

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Fri, Feb 09, 2001 at 10:28:49AM -0200, Branden wrote:
> In http://www.w3.org/TR/NOTE-OSD.html#B they describe platform/cpu standard
> names, and we'll definetly need those for checking target architecture. Can
> we standardize upon those, or there's something missing? There's an issue

I take it "Lunix" is Linux.
BSDi isn't FreeBSD, NetBSD or OpenBSD
Nothing they list seems to be VMS
Pace are still developing variants of Acorn's RISC OS to run set top boxes
As I understood it there were about 39 variants of Unix ever, and they seem
to have 12 listed.

Processors are missing
pa risc (nice one, as HP-UX *is* in the OS list)
arm sh ia64 s390

and probably some more
[If I can run 'ls /usr/src/linux/arch/' why can't they? Hmm. Document
dates from 1997. Acorn was trading then]

The ommission of pa risc makes me wonder about quality of the rest of this
document.

> that I really don't know: in the same platform, different compilers generate
> incompatible binaries? Because if this happens (and will still happen on
> Perl 6) the platform identification should be os/cpu/compiler. Perhaps each

on perl 5 different configure options generate different binaries.

> Other important issue I don't know yet: Is there an Archive::Zip module for
> Perl? How cross-platform is it? Can we bundle it with Perl (licensing

http://www.cpan.org/modules/by-module/Archive/Archive-Zip-0.11.readme

> issues)? Is it stable? Will it give us the support we need (access to
> individual files in the archive, and the ability to `fake' them as real
> files through tied filehandles)?

I don't know the answers to the other questions. However, I was intending
to write a layer for perl 5.7 that can access zip entries via regular
perl filehandles. (In theory this could let you run scripts direct out of a
zip without actually unpacking them anywhere)

Layers of some form are needed in perl6 to cope with utf8 and other
encodings, so I would expect the perl6 layer functionality can also be
extended to zip manipulations.

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Branden]

Clayton Scott wrote:
>  PPM uses files containing a modified form of the Open Software
>  Distribution (OSD) specification for information about software
>  packages. These description files, which are written in Extensible
>  Markup Language (XML) code, are referred to as 'PPD' files.
>  Information about OSD can be found at the W3C web site (at the
>  time of this writing, http://www.w3.org/TR/NOTE-OSD.html). The
>  modifications to OSD used by PPM are documented in PPM::ppd.

Read it about OSD. I think it's good to use it for representing the
dependencies. It coulde be used by the `par' utility to build the archive.
It should probably be included in the archive as well, for checking the
dependencies and platform stuff. The OSD stuff is very `push' technology
oriented, which would match our definition of `automatic', what has been
described here as `evil' and `insecure', but using the DTD could provide
interoperability with other tools that support OSD. It's also
Java/Microsoft-oriented, so I'll see what PPM::ppd says about it to make it
usable by Perl.

The problem I see with PPM is that, AFAIK, it handles only singular modules,
it can't directly handle bundles of modules and also scripts. As it should
be possible, it probably wouldn't be easy and neither standardized. (Please
correct me if I'm wrong here.)

In http://www.w3.org/TR/NOTE-OSD.html#B they describe platform/cpu standard
names, and we'll definetly need those for checking target architecture. Can
we standardize upon those, or there's something missing? There's an issue
that I really don't know: in the same platform, different compilers generate
incompatible binaries? Because if this happens (and will still happen on
Perl 6) the platform identification should be os/cpu/compiler. Perhaps each
platform would have a list of `interchangeable' compilers, i.e. those that
produce compatible code and modules compiled with one can be used with a
Perl compiled with another.

BTW, I'm working on the PDD. Anyone has a suggestion on something to add?

Other important issue I don't know yet: Is there an Archive::Zip module for
Perl? How cross-platform is it? Can we bundle it with Perl (licensing
issues)? Is it stable? Will it give us the support we need (access to
individual files in the archive, and the ability to `fake' them as real
files through tied filehandles)?

- Branden

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 01:40:52PM -0500, Dan Sugalski wrote:
> >Seperated documentation is no documentation.
> 
> At some point things are going to get split out, unless you wedge the docs 
> into the actual program itself. (You were, after all, talking about config 
> files and XS modules, and those can't usefully stay inside the archive)

True, but those things are essential to making the program work, and
their loss will be dearly (and immediately) noted.  Documentation is
the first thing to get ditched in any install process.  I was mostly
thinking perldoc (yes, wedging the docs into the program) *plus* the
normal man page generation and install.  That's how I normally do
things.


-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
Only mindless violence can raise my spirits now!

Re: Auto-install (was autoloaded...)

[Bart Lateur]

On Thu, 8 Feb 2001 17:49:45 -0200, Branden wrote:

>I've never actually used PPM, only read about it in
>the web. I guess their file format is a disguised .tar.gz, right?

It's a combination of an XML file, file extension "PPD", which describes
the properties and dependencies, and platforms, and a normal .tar.gz
file. The latter can come straight from CPAN.

-- 
Bart.

Re: Auto-install (was autoloaded...)

[Bart Lateur]

On Thu, 8 Feb 2001 17:39:01 +, Nicholas Clark wrote:

>On Thu, Feb 08, 2001 at 12:26:59PM -0500, Dan Sugalski wrote:
>> (Including Archive::Tar as part of the base perl distribution's not 
>> inappropriate, assuming we can get permission. )

Since it's already part of the "standard distribution" for Win32, both
ActiveState and IndigoStar, as both use it in order to install packages,
I guess that that won't be much of a problem.

>Do we really want to use tar format (over say cpio) as tar rounds files
>up to 512 block boundaries, and has some arbitrary restrictions on filename
>lengths in the headers?

You don't need to use raw .tar. Archive::Tar works perfectly fine with
.tar.gz files. Compression effectively takes care of the padding with
nulls, because they get "compressed away". The compression is handled by
the Compress::Zlib module. So if there is any redistribution licensee
problem, I expect it to be there.

-- 
Bart.

Re: Auto-install (was autoloaded...)

[Clayton Scott]

Peter Scott wrote:
> 
> Eh?  I thought PPM was simply "perl -MCPAN -e install" for Windows users,
> pointed to a set of modules which have XS content that they'd had to fiddle
> with to port to Win32.

Not by far. It is a replacment for CPAN that builds and
 maintains its own local database of installed modules.


>From PPM docs:
 DESCRIPTION

 PPM is a group of functions intended to simplify the tasks of 
 locating, installing, upgrading and removing software 'packages'. It 
 can determine if the most recent version of a software package is 
 installed on a system, and can install or upgrade that package from 
 a local or remote host. 

 PPM uses files containing a modified form of the Open Software 
 Distribution (OSD) specification for information about software 
 packages. These description files, which are written in Extensible
 Markup Language (XML) code, are referred to as 'PPD' files. 
 Information about OSD can be found at the W3C web site (at the 
 time of this writing, http://www.w3.org/TR/NOTE-OSD.html). The
 modifications to OSD used by PPM are documented in PPM::ppd. 

 PPD files for packages are generated from POD files using the pod2ppd
command. 



Clayton

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 05:49 PM 2/8/2001 -0200, Branden wrote:
>Peter Scott wrote:
> > Eh?  I thought PPM was simply "perl -MCPAN -e install" for Windows users,
> > pointed to a set of modules which have XS content that they'd had to
>fiddle
> > with to port to Win32.
> >
>
>Sorry for the mistake... I've never actually used PPM, only read about it in
>the web. I guess their file format is a disguised .tar.gz, right?

Disguised .zip.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Branden]

Peter Scott wrote:
> Eh?  I thought PPM was simply "perl -MCPAN -e install" for Windows users,
> pointed to a set of modules which have XS content that they'd had to
fiddle
> with to port to Win32.
>

Sorry for the mistake... I've never actually used PPM, only read about it in
the web. I guess their file format is a disguised .tar.gz, right?

- Branden

Re: Auto-install (was autoloaded...)

[Peter Scott]

At 05:00 PM 2/8/01 -0200, Branden wrote:
>I wrote:
> > I think zip is the way to go! Is there any
> > platform/license or any other restricting issues we should care about zip?
> > Is it ported to all platforms Perl currently runs on? Is there a Perl
>module
> > for handling zips?
>
>Aren't we re-inventing the wheel here? It strikes me now that ActiveState's
>ActivePerl comes with PPM, or `Perl Package Manager'. AFAIK, it's only for
>downloading from ActiveState's site, and it only handles installing of
>individual modules (althought it checks dependencies and fetches needed
>modules as well, but it doesn't solve the problem for scripts/programs).
>
>Anyone of ActiveState there? Can't we adapt PPM so that it handles what's
>needed? Or is it too different from what we want? Does it use zip or
>tar/gzip or other?

Eh?  I thought PPM was simply "perl -MCPAN -e install" for Windows users, 
pointed to a set of modules which have XS content that they'd had to fiddle 
with to port to Win32.

--
Peter Scott
Pacific Systems Design Technologies

Re: Auto-install (was autoloaded...)

[Branden]

I'm writing a PDD on the subject, as suggested by Dan, and I intend to post
it by tomorrow. In the lack of a better list, I think we stick
with -language. If someone can suggest a better one, please do it.

- Branden

Re: Auto-install (was autoloaded...)

[Branden]

I wrote:
> I think zip is the way to go! Is there any
> platform/license or any other restricting issues we should care about zip?
> Is it ported to all platforms Perl currently runs on? Is there a Perl
module
> for handling zips?


Aren't we re-inventing the wheel here? It strikes me now that ActiveState's
ActivePerl comes with PPM, or `Perl Package Manager'. AFAIK, it's only for
downloading from ActiveState's site, and it only handles installing of
individual modules (althought it checks dependencies and fetches needed
modules as well, but it doesn't solve the problem for scripts/programs).

Anyone of ActiveState there? Can't we adapt PPM so that it handles what's
needed? Or is it too different from what we want? Does it use zip or
tar/gzip or other?

- Branden

Re: Auto-install (was autoloaded...)

[Branden]

Dan Sugalski wrote:
> At 05:58 PM 2/8/2001 +, Nicholas Clark wrote:
> >\zip's better in that it allows easy random access to a compressed file,
> >[without having to compress everything else first] but worse for the
> >same reason because you don't get as good a compression ratio by
> >compressing each file separately.
>
> I've seen it go both ways with compression, but I'm not sure that a few
> percent either way's a big deal. Packaging is more important than
> compression for this purpose anyway, I think.
>

Yes. Packaging is what's important. I actually expect to not have to install
the `par' and have perl6's magic filehandles decompress a file from the
package `on-the-fly'. I think zip is the way to go! Is there any
platform/license or any other restricting issues we should care about zip?
Is it ported to all platforms Perl currently runs on? Is there a Perl module
for handling zips?

- Branden

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 05:49 PM 2/8/2001 +, Michael G Schwern wrote:
>On Thu, Feb 08, 2001 at 12:31:25PM -0500, Dan Sugalski wrote:
> > Not unless you strip the bytecode. I want to optionally package the source
> > in the bytecode, since otherwise you can't do some optimizations after the
> > fact on the generated bytecode stream.
>
>Clever dog!

Altogether too much so, I expect. :) Reading the optimizing compiler books 
makes my head hurt, but there's a lot of interesting stuff in them.

> > > 2) You have to make provisions to distribute your documentation
> > >seperately.
> >
> > Presumably you'd package it up in the tar or zip archive containing the
> > fully-compiled program.
>
>Seperated documentation is no documentation.

At some point things are going to get split out, unless you wedge the docs 
into the actual program itself. (You were, after all, talking about config 
files and XS modules, and those can't usefully stay inside the archive)

> > > 5) Do YOU have a stable bytecode compiler??  I don't.
> >
> > For perl 6? No. Not yet.
>
>Is perlcc considered really stable and usable in 5.6?  Hmm, my little
>test would say no. :(

I shan't be going there for lack of time. It'd be nice if someone had the 
time to make it work.

> > I assumed that since you were discussing this on a perl 6 mailing list, 
> you
> > were talking about doing this with perl 6.
>
>What!  Me stay on topic?  HA!

Yeah, I know--what *was* I thinking?

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 05:58 PM 2/8/2001 +, Nicholas Clark wrote:
>On Thu, Feb 08, 2001 at 12:41:34PM -0500, Dan Sugalski wrote:
> > At 05:39 PM 2/8/2001 +, Nicholas Clark wrote:
>
> > >Do we really want to use tar format (over say cpio) as tar rounds files
> > >up to 512 block boundaries, and has some arbitrary restrictions on 
> filename
> > >lengths in the headers?
> >
> > Having the perl archives splittable by other available tools is a good
> > thing. Using the zip format's fine too--I don't much care either way.
>
>Yes, I agree. Hence cpio may not be great as tools to deal with it
>are much rarer

Yup, and finding them on non-unix platforms can be rather tricky, too. Zip 
and tar are probably the two biggies.

>\zip's better in that it allows easy random access to a compressed file,
>[without having to compress everything else first] but worse for the
>same reason because you don't get as good a compression ratio by
>compressing each file separately.

I've seen it go both ways with compression, but I'm not sure that a few 
percent either way's a big deal. Packaging is more important than 
compression for this purpose anyway, I think.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Thu, Feb 08, 2001 at 12:41:34PM -0500, Dan Sugalski wrote:
> At 05:39 PM 2/8/2001 +, Nicholas Clark wrote:

> >Do we really want to use tar format (over say cpio) as tar rounds files
> >up to 512 block boundaries, and has some arbitrary restrictions on filename
> >lengths in the headers?
> 
> Having the perl archives splittable by other available tools is a good 
> thing. Using the zip format's fine too--I don't much care either way. 

Yes, I agree. Hence cpio may not be great as tools to deal with it
are much rarer

> (Zip's better in some ways since you can encode extra info in the file 
> headers, but I don't know that we'll need it, nor if any platform besides 
> VMS uses it)

Acorn RISC OS zip tools use the extra info to store file metadata.
I think that the unix zip tools use the extra info field to store
create/access/modification times. There's a tagging format defined,
so a file can have multiple blocks of data in the extra info field
that programs that don't understand them treat as opaque.

zip's better in that it allows easy random access to a compressed file,
[without having to compress everything else first] but worse for the
same reason because you don't get as good a compression ratio by
compressing each file separately.

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 05:39:01PM +, Nicholas Clark wrote:
> Do we really want to use tar format (over say cpio) as tar rounds files
> up to 512 block boundaries, and has some arbitrary restrictions on filename
> lengths in the headers?

First cut will be tar.  Why?  Its simple, its common, and we have a
well-developed Perl module for it.  Later it can be changed to
anything we want.  Encapsulation++

-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
Maybe they hooked you up with one of those ass-making magazines.
-- brian d. foy as misheard by Michael G Schwern

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 12:31:25PM -0500, Dan Sugalski wrote:
> Not unless you strip the bytecode. I want to optionally package the source 
> in the bytecode, since otherwise you can't do some optimizations after the 
> fact on the generated bytecode stream.

Clever dog!


> > 2) You have to make provisions to distribute your documentation
> >seperately.
> 
> Presumably you'd package it up in the tar or zip archive containing the 
> fully-compiled program.

Seperated documentation is no documentation.


> > 5) Do YOU have a stable bytecode compiler??  I don't.
> 
> For perl 6? No. Not yet.

Is perlcc considered really stable and usable in 5.6?  Hmm, my little
test would say no. :(


> I assumed that since you were discussing this on a perl 6 mailing list, you 
> were talking about doing this with perl 6.

What!  Me stay on topic?  HA!

-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
BOFH excuse #189:

SCSI's too wide.

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 05:39 PM 2/8/2001 +, Nicholas Clark wrote:
>On Thu, Feb 08, 2001 at 12:26:59PM -0500, Dan Sugalski wrote:
> > This is an excellent reason, and one I hadn't considered. I withdraw any
> > objections. Care to put together a PDD on how it should be handled?
> > (Including Archive::Tar as part of the base perl distribution's not
> > inappropriate, assuming we can get permission. )
>
>Do we really want to use tar format (over say cpio) as tar rounds files
>up to 512 block boundaries, and has some arbitrary restrictions on filename
>lengths in the headers?

Having the perl archives splittable by other available tools is a good 
thing. Using the zip format's fine too--I don't much care either way. 
(Zip's better in some ways since you can encode extra info in the file 
headers, but I don't know that we'll need it, nor if any platform besides 
VMS uses it)

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Thu, Feb 08, 2001 at 12:26:59PM -0500, Dan Sugalski wrote:
> This is an excellent reason, and one I hadn't considered. I withdraw any 
> objections. Care to put together a PDD on how it should be handled? 
> (Including Archive::Tar as part of the base perl distribution's not 
> inappropriate, assuming we can get permission. )

Do we really want to use tar format (over say cpio) as tar rounds files
up to 512 block boundaries, and has some arbitrary restrictions on filename
lengths in the headers?

Nicholas Clark

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 02:43 PM 2/8/2001 -0200, Branden wrote:
>Dan Sugalski wrote:
> >
> > I'm not sure this is all necessary. Wouldn't we be reasonably better off
>if
> > we instead just shipped off bytecode compiled versions of the scripts?
> > Seems easier to ship that way than as an archive of stuff. (We can, if its
> > deemed useful, define the bytecode format in a way that allows us to
> > package up versions of modules that can be optionally loaded from the main
> > perl install instead)
> >
> > Seems simpler, and it also means you can, at the time the program is
> > initally compiled, crank up the optimization level a lot so you're handing
> > out the fastest version of your code you can.
> >
> > Dan
> >
>
>
>At a first glance, I really found that's much better. But I saw three small
>problems:
>1. updating the version of modules in the big-bloated-bytecode would
>potentially have to recompile the script and all other modules.

If the modules you're distributing change you need to repackage things up 
anyway. Not much difference here, really. (And we wouldn't necessarily want 
to automagically use a newer version of an installed module--that'd be 
rather unpleasant if we have another case like we had with GD)

>2. it doesn't work for scripts/modules deployed in source code format.

Why are you assuming the source can't be packaged up in the generated bytecode?

>3. it doesn't work for modules that use C extensions.

Definitely an issue, but packaging up a program with code that needs to be 
compiled on the target machine is problematic in a lot of ways anyway.

>Actually, I think the archive approach is more general, because it wouldn't
>have this kind of problems and would allow other resources to be deployed
>together with the code, like documentation, perhaps even text files and
>images used by a Perl/Tk application.

This is an excellent reason, and one I hadn't considered. I withdraw any 
objections. Care to put together a PDD on how it should be handled? 
(Including Archive::Tar as part of the base perl distribution's not 
inappropriate, assuming we can get permission. )

>Did I miss something here? Is it just me, or you also think this
>(deploy/install) is essential for a language be used by `layman-users', and
>not only JAPHs.

Generally speaking, I assume installs require a certain minimum competence 
on someone's part. The less want assumed on the part of the end user, the 
more is required on the part of the person packaging up the install. (It's 
been my experience that it's an inverse cube relationship)

Doing this portably is an interesting exercise. Doing it non-portably is a 
waste of time, since if you're going to be platform-specific you're better 
off using the platform install tools.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 11:52 AM 2/8/2001 +, Michael G Schwern wrote:
>On Thu, Feb 08, 2001 at 11:21:17AM -0500, Dan Sugalski wrote:
> > I'm not sure this is all necessary. Wouldn't we be reasonably better 
> off if
> > we instead just shipped off bytecode compiled versions of the scripts?
>
>Sure, except...
> 1) You lose your readable source code (discussions of B::Deparse as
>a viable alternative > /dev/null)

Not unless you strip the bytecode. I want to optionally package the source 
in the bytecode, since otherwise you can't do some optimizations after the 
fact on the generated bytecode stream.

> 2) You have to make provisions to distribute your documentation
>seperately.

Presumably you'd package it up in the tar or zip archive containing the 
fully-compiled program.

> 3) It makes it harder to bundle non-Perl things, like configuration
>files, images, sound files, etc...  If you want to send those along
>with the bytecode you windup needing a par-style utility anyway.

Once again, you can package it up in the tar or zip archive that you're 
distributing the program in.

> 4) What Brenden said

Some of what Brenden said isn't entirely applicable, though much of it is.

> 5) Do YOU have a stable bytecode compiler??  I don't.

For perl 6? No. Not yet.

>Perhaps it wasn't clear, I don't mean to have par as part of 6.0, I
>mean to have it out, like, maybe next month if I decide to work on it.

I assumed that since you were discussing this on a perl 6 mailing list, you 
were talking about doing this with perl 6.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 11:21:17AM -0500, Dan Sugalski wrote:
> I'm not sure this is all necessary. Wouldn't we be reasonably better off if 
> we instead just shipped off bytecode compiled versions of the scripts? 

Sure, except...
1) You lose your readable source code (discussions of B::Deparse as
   a viable alternative > /dev/null)
2) You have to make provisions to distribute your documentation
   seperately.
3) It makes it harder to bundle non-Perl things, like configuration
   files, images, sound files, etc...  If you want to send those along
   with the bytecode you windup needing a par-style utility anyway.
4) What Brenden said
5) Do YOU have a stable bytecode compiler??  I don't.

Perhaps it wasn't clear, I don't mean to have par as part of 6.0, I
mean to have it out, like, maybe next month if I decide to work on it.

-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
Kids - don't try this at--oh, hell, go ahead, give it a whirl...

Re: Auto-install (was autoloaded...)

[Branden]

Dan Sugalski wrote:
>
> I'm not sure this is all necessary. Wouldn't we be reasonably better off
if
> we instead just shipped off bytecode compiled versions of the scripts?
> Seems easier to ship that way than as an archive of stuff. (We can, if its
> deemed useful, define the bytecode format in a way that allows us to
> package up versions of modules that can be optionally loaded from the main
> perl install instead)
>
> Seems simpler, and it also means you can, at the time the program is
> initally compiled, crank up the optimization level a lot so you're handing
> out the fastest version of your code you can.
>
> Dan
>


At a first glance, I really found that's much better. But I saw three small
problems:
1. updating the version of modules in the big-bloated-bytecode would
potentially have to recompile the script and all other modules.
2. it doesn't work for scripts/modules deployed in source code format.
3. it doesn't work for modules that use C extensions.

Well, the first is not a big problem, since it's probably a very rare
situation and probably the script code would change a bit if modules'
versions change. The second also is not a big issue, since source code can
always be downloaded separately, and who wants the source code probably can
handle separate module downloading and installing.

Actually, I think the archive approach is more general, because it wouldn't
have this kind of problems and would allow other resources to be deployed
together with the code, like documentation, perhaps even text files and
images used by a Perl/Tk application.



Other thing that looks like a difference at first, is that the shipped
bytecode would run directly and the archive would have to be installed to
run. I think that's actually not true, as Perl 6 will probably use one of
its own's new magic filehandles to read the sources/bytecodes, and one of
that filehandled can be tied to decompress the archive on-the-fly and pass
the code to the interpreter.

And pure bytecode applications would be actually distributed on the
`.noarch' kind of par archive, to contrast the `.src' kind.

Did I miss something here? Is it just me, or you also think this
(deploy/install) is essential for a language be used by `layman-users', and
not only JAPHs.

- Branden

Re: Auto-install (was autoloaded...)

[Dan Sugalski]

At 01:44 PM 2/8/2001 -0200, Branden wrote:
>Michael G Schwern wrote:
> > On Thu, Feb 08, 2001 at 12:07:18PM -0200, Branden wrote:
> > > The issue is actually not auto-downloading modules and their
>prerequisites,
> > > but actually packaging several scripts and modules in one file, so as
>Java's
> > > jar do. I think supporting this would be neat.
> >
> > I thought about making a "par" utility.  It would basically do this:
> >
> > # for each module needed...
> > perl Makefile.PL PREFIX=foo LIB=foo/lib
> > make test
> > make install
> >
> > Then you just stick your program into foo/bin or something and tar it
> > all up and ship it off.  The "pun" utility (I couldn't resist) then
> > untars the thing and runs "perl -Ifoo/lib foo/bin/whatever.plx".
>
>
>That's what I was talking about.

I'm not sure this is all necessary. Wouldn't we be reasonably better off if 
we instead just shipped off bytecode compiled versions of the scripts? 
Seems easier to ship that way than as an archive of stuff. (We can, if its 
deemed useful, define the bytecode format in a way that allows us to 
package up versions of modules that can be optionally loaded from the main 
perl install instead)

Seems simpler, and it also means you can, at the time the program is 
initally compiled, crank up the optimization level a lot so you're handing 
out the fastest version of your code you can.

Dan

--"it's like this"---
Dan Sugalski  even samurai
[EMAIL PROTECTED] have teddy bears and even
  teddy bears get drunk

Re: Auto-install (was autoloaded...)

[Branden]

Michael G Schwern wrote:
> On Thu, Feb 08, 2001 at 12:07:18PM -0200, Branden wrote:
> > The issue is actually not auto-downloading modules and their
prerequisites,
> > but actually packaging several scripts and modules in one file, so as
Java's
> > jar do. I think supporting this would be neat.
>
> I thought about making a "par" utility.  It would basically do this:
>
> # for each module needed...
> perl Makefile.PL PREFIX=foo LIB=foo/lib
> make test
> make install
>
> Then you just stick your program into foo/bin or something and tar it
> all up and ship it off.  The "pun" utility (I couldn't resist) then
> untars the thing and runs "perl -Ifoo/lib foo/bin/whatever.plx".


That's what I was talking about.

Only have `par' copy the scripts to foo/bin itself and handle tar/gzip
(maybe using Perl modules internally, so that platform dependency issues are
isolated). Then it should save the file with a `.myarch.par' extension, to
signal it's a `Perl Archive' and it's intended to be used in the `MyArch'
architecture (replace `.myarch' with `.win32', `.linux-i386', ... and it
makes sense).

`pun' could then check the file extension and issue some warnings, like:
``You are trying to install a file for architecture `Linux-Intel386' on a
machine running `Windows32'.''

`par' should support deploying .src archives, to be built on the target
machine, supplying a Makefile or similar that would build all needed modules
at once. And `pun' should recognize them and actually start the make
processing (perhaps prompting the user for it).

`pun' should be able to install the modules on a directory specified when
creating the par (like foo/lib), on a user-specified directory (like
~/myperlscripts/foo-1.2/lib), or on the Perl's main modules directories,
where the modules would be shared. `pun' could yet check if modules shipped
with the par archive are already present on the system, and skip them (or
update the system's modules, if the shipped ones are newer and the user
grants permission to do that -- i.e. he trusts the par).



And, more importantly, `par' and `pun' must be shipped in Perl 6.0.0. That
was the big mistake in Java, they only introduced `jar' in 1.1, but then
Microsoft already used `cab's for that, and `jar' never got to work for
applets, since the two big browsers supported different things.

Shipping them would make everyone's life easier. Script developers, because
they could package their code and give simple instructions like ``download
the xyz-1.2.win32.par and run `pun xyz-1.2.win32.par', then run `perl
foo\bin\myscript.plx' to run the script''. And users, because they wouldn't
have to go into the issues of what are modules, how to get them and how to
build them. And specially Perl, because having easy ways of deploying and
installing scripts/programs would make much more developers make much more
scripts and tools available for a much more wider set of users.


>
> Any obvious flaws?  Poke me enough and I'll get around to doing it.
>

If there anyone points flaws, I help doing this thing. I think we should
only wait for the definition of how building modules will be in Perl 6,
since I don't expect it to use `make', but rather a Perl-defined tool.

- Branden

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 12:07:18PM -0200, Branden wrote:
> The issue is actually not auto-downloading modules and their prerequisites,
> but actually packaging several scripts and modules in one file, so as Java's
> jar do. I think supporting this would be neat.

I thought about making a "par" utility.  It would basically do this:

# for each module needed...
perl Makefile.PL PREFIX=foo LIB=foo/lib
make test
make install

Then you just stick your program into foo/bin or something and tar it
all up and ship it off.  The "pun" utility (I couldn't resist) then
untars the thing and runs "perl -Ifoo/lib foo/bin/whatever.plx".

Any obvious flaws?  Poke me enough and I'll get around to doing it.


> As to the question of security, if you download a script on a site that says
> it does XYZ and you actually trust the script does XYZ (trust in the sense
> that you *believe* it), I don't see why wouldn't you trust that the script
> would load modules that aren't harmful, either from CPAN or from another
> place.

Download Memoize from CPAN sometime and install it.  Make sure you're
sitting down.  All it takes is one joker, or one person to have a bad
day, or get a little too drunk one night near a computer.

We *can* automate security auditing of CPAN.  I know it can be done
because I've seen it done on smaller scales and it will happen.  If
you missed it, look at the CPANTS synopsis
http:[EMAIL PROTECTED]/msg00148.html

Its vapor yet, but its all within the realm of "solved problems".

-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
BOFH excuse #301:

appears to be a Slow/Narrow SCSI-0 Interface problem

Re: Auto-install (was autoloaded...)

[Branden]

Michael G Schwern wrote:
>
> Oddly enough, Perl does handle this... mostly.  The CPAN shell can
> automatically download and install prerequisites for modules, provided
> the module explicitly declares the prereqs.  Class::DBI ultimately
> needs something like 9 other CPAN modules, which would be a nightmare
> but for this feature.
>

The issue is actually not auto-downloading modules and their prerequisites,
but actually packaging several scripts and modules in one file, so as Java's
jar do. I think supporting this would be neat.

As to the question of security, if you download a script on a site that says
it does XYZ and you actually trust the script does XYZ (trust in the sense
that you *believe* it), I don't see why wouldn't you trust that the script
would load modules that aren't harmful, either from CPAN or from another
place.

And having to see the code before installing is not a proof of security at
all, since Perl is the king of obfuscating languages and (I expect) Perl 6
will be able to distribute code in byte-code form. So I don't actually see
how auto-loading of modules from Internet is so much more untrustful than
manual-loading of the same modules from Internet, if they are pretty
obfuscated or are in byte-code form. Only a sandbox or something like that
can assure security in either case.

- Branden

Re: Auto-install (was autoloaded...)

[Michael G Schwern]

On Thu, Feb 08, 2001 at 08:53:07AM -, Filipe Brandenburger wrote:
> Branden wrote:
> >When I download a module from Internet, say module Foo, then I install
> >it and try to use it, it promptly breaks when it tries to `use Bar'
> >and sees that Bar is not installed on my system. So I have to go on
> >to Internet again, find Bar, install it, so on, until I find Bar needs
> >Baz, or anything
> >like it.
> >
> >Well, I think this could be handled by Perl itself.

Oddly enough, Perl does handle this... mostly.  The CPAN shell can
automatically download and install prerequisites for modules, provided
the module explicitly declares the prereqs.  Class::DBI ultimately
needs something like 9 other CPAN modules, which would be a nightmare
but for this feature.

perl -MCPAN -e 'install Class::DBI'

Its only on the more recent versions of the CPAN shell (>1.48), but
since the shell whines about updating itself everytime you use it,
there's no reason not to have the latest.

I've taken advantage of this for local projects using the CPAN::Site
module and setting up a local repository for local code.

Now, the idea of perl automagically going out and downloading
untrusted code (all the code on CPAN is untrusted) from untrusted
sources (ergo, CPAN is untrusted) from untrusted scripts (if its a
core feature, any script can do it) makes my feet itch.  We can't do
this until there's a way to security audit CPAN... which is supposed
to be my job. :(

However, if you *really* want to do it, you can pretty easily.  Just
code up a function which tries to use the given module, otherwise it
fires up the CPAN shell and install it, then tries to use it again.
I've been meaning to do this for a while now.

Another problem is the fact that not everything on CPAN installs
easily.  Most does, but some require certain interaction and
configuration (LWP, Net::FTP, most any DBD driver...).  Those can't be
done automagically, but if we could do 80% then that's ok.


-- 

Michael G. Schwern   <[EMAIL PROTECTED]>http://www.pobox.com/~schwern/
If you have to shoot, shoot!  Don't talk.
-- Tuco, "The Good, The Bad And The Ugly"

Re: Auto-install (was autoloaded...)

[Nicholas Clark]

On Thu, Feb 08, 2001 at 08:53:07AM -, Filipe Brandenburger wrote:
> The solution I propose to this problem is borrowed (copied) from what Java 
> did in version 1.1 with jars (did wrong, of course), and somewhat like 
> RedHat's rpms. What I suggest is having a kind of archive that would be like 
> a container for all the code needed to run the thing, so that it can be 
> downloaded and installed all at once. Not .tgz, but a thing that perl could 
> recognize at his own's. A tool (`perlinstall', or something like that) 
> distributed with perl would take this package and extract scripts, modules, 
> ..., and install it in the right places, with minimal (ideal zero) user 
> choices.

> Comments?

If you don't need XS modules (or anything else that requires the services
of Makefile.PL or some other install type script on the end machine) is
is possible to put the necessary modules in a zip file, append the zip
file to the perl script after __END__
run zip -A to "adjust the self-extracting exe" and you have a single file
which is both a valid perl script (at the front) and a valid zip file
(at the end)

If you have a perl capable of reading @INC from zip files - I've produced 2
different patches for perl5 that could do this (1 using source filters,
1 using sfio) then you 

BEGIN {unshift $0, @INC}

and your versions of the modules are used if no others are found without
even needing to unzip anything.

problems
1: you can't rely on EOF on the DATA handle in main
2: your script has to be transfered in binary mode ftp, and not mangled
   by any web browsers when saved
3: you really need the new perlio implementation in 5.7 to make it work
   best in a "vanilla" perl
   I'm working towards being able to do this (see the experimental
   PerlIO::gzip on CPAN for gunzip in a perlio layer)
4: Can't do XS
5: not sure if it's a sick hack or an elegant hack

It is also possible to place the include zipfile on the end of the perl
executable and put $^X in @INC to make a 1 file perl distribution, but
I don't think anyone ever tried this

Nicholas Clark

Auto-install (was autoloaded...)

[Filipe Brandenburger]

Branden wrote:
>When I download a module from Internet, say module Foo, then I install
>it and try to use it, it promptly breaks when it tries to `use Bar'
>and sees that Bar is not installed on my system. So I have to go on
>to Internet again, find Bar, install it, so on, until I find Bar needs
>Baz, or anything
>like it.
>
>Well, I think this could be handled by Perl itself.
>This way, Perl could automatically fetch and install modules
>(supposing module install is not so hard as now, involving makefiles,
>and such...).



Well, I thought about it pretty much, and I finally realised what bothered 
me, and a (pretty reasonable) way to attenuate it a bit.

What bothers me is: Suppose I write a perl script to handle, say, currency 
conversion, or something like getting stock quotes from a variety of sites 
and building a spreadsheet table with them. That's typically something that 
would be pretty much used for `users', specially the ones that are *not* 
JAPH. Now suppose my script uses modules X, Y, and Z. X is in the standard 
library, but Y and Z are not. And Z uses module K, which isn't in the 
standard library either. Now suppose I want to make my script available at a 
web site for anyone who wants to download it and use it.

Now suppose a user (one that uses Windoze, didn't read the Camel, and barely 
can get to install Perl in binary form -- InstallShield is too complicated!) 
goes to the site, finds the script useful, and downloads it. The first time 
he'll want to try it, it will croak with a `module Y not found!'. Now, even 
if I put instructions on the web site about how to get required modules Y, 
Z, and K (which I really don't want to do for every little script I want to 
deploy), and even if the guy doesn't give up yet (which I'm sure he'll do), 
I'm certain he'll *NEVER* get the thing right, even if the modules are 
available in binary form, which I believe is not the case for most scripts.

The solution I propose to this problem is borrowed (copied) from what Java 
did in version 1.1 with jars (did wrong, of course), and somewhat like 
RedHat's rpms. What I suggest is having a kind of archive that would be like 
a container for all the code needed to run the thing, so that it can be 
downloaded and installed all at once. Not .tgz, but a thing that perl could 
recognize at his own's. A tool (`perlinstall', or something like that) 
distributed with perl would take this package and extract scripts, modules, 
..., and install it in the right places, with minimal (ideal zero) user 
choices.

Well, the thing with rpms is that they can be source, binaries, or platform 
independent, what is represented by having .src, .i386/.ppc/.alpha/..., or 
.noarch on the name. The same could be done with perl. The src would include 
the source for all needed modules, besides the scripts that the user is 
supposed to use. If any of the modules use XS (or its sucessor), it would be 
possible to build binary packages, that would take the burden out of the 
user to deal with C compilers and make (which is one thing I hope to see 
stripped off of module building, in Perl 6 -- since we have a MakeMaker, we 
could have a Maker that does it all! Advantages? Platform independence and 
no need for external tools but the C compiler).

Dependencies generators would be necessary as well, but I think there are 
some already.

Comments?

- Branden

_
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.