Re: Maven and repository@apache.org

[Niclas Hedhman]

On Tuesday 11 January 2005 12:18, Russell Gold wrote:
> Is there currently a policy for how long files are kept in the
> repository? It seems to me that old versions are not kept very long at
> ibiblio, if there are newer versions available.

I am not sure if you are talking about the Transit client or central 
repositories in general. So let's cover both.

Transit's CacheHandler is currently non-clearing, meaning all artifacts are 
held in the local file system cache until explicitly deleted, and AFAIK that 
is the case in Maven as well. We (the DPML folks) are planning to change that 
to a LRU algorithm at some point.


Central repositories should *never* remove artifacts, as this may have 
detrimental effects on applications. The general policy at ibiblio.org is the 
same, and I think ibiblio.org itself doesn't interfere. It is the people who 
have write access to ibiblio.org/maven that can possibly remove older 
versions, but I have personally not seen this happening. Old versions lacking 
is more often than not a matter of, only the recent ones have been added.

We (the DPML folks) are planning another ResourceHost type, which will request 
the location(s) of the artifacts from its main host, and then download 
directly from the source, such as SourceForge, tigris.org and others. We 
think this is a more proper way of resolving artifact resources.


Cheers
Niclas
-- 
+-//---+
|   http://www.dpml.net|
|  http://niclas.hedhman.org   |
+--//--+

Re: Maven and repository@apache.org

[Russell Gold]

Is there currently a policy for how long files are kept in the
repository? It seems to me that old versions are not kept very long at
ibiblio, if there are newer versions available.

Re: Maven and repository@apache.org

[Niclas Hedhman]

On Thursday 06 January 2005 09:21, Dain Sundstrom wrote:
> How do you get around the problem that it is very difficult to get a
> url handler installed?  Does this library work only when it is
> installed into the system class path?

That is corrrect.
The "issue" is in the java.net.URL handling, where the internal class (I think 
it was called URLClassPath) first tries to load the Handler with the callers 
Classloader (which is the Bootstrap CL since java.net.URL is the caller), and 
since that will fail for all custom ones, it then tries 
ClassLoader.getSystemClassLoader();

Unfortunately, so far Sun claims that the security issues are so unclear, that 
they do not intend to fix this.

I fully agree that this is less than an ideal situation, but messing with the 
default URLStreamHandlerFactory was not any better.

Internally we are working on a two layered approach, where the Handler is 
loaded by the System CL, but the entire workload is delegated down to 
reloadable parts. That is not ready yet.


Cheers
Niclas
---
Hard work pays off in the future, laziness pays off now.
 -  Steven Wright

+-//---+
|   http://www.dpml.net|
|  http://niclas.hedhman.org   |
+--//--+

Re: Maven and repository@apache.org

[Leo Simons]

Hi gang,

On 05-01-2005 09:39, "Nicola Ken Barozzi" <[EMAIL PROTECTED]> wrote:
> What relationship does the repository list have with Maven?
> What relationship does Maven have with the repository list?
> 
> Ass Brett Porter has written, and I agree:
> "
> 1) Maven PMC takes ownership for getting the repository right

IIRC [EMAIL PROTECTED] is actually a President's committee sort-of "hanging
off" infra@ that's been tasked with figuring all this stuff out. The reason
its not just the Maven PMC in the first place is that there's some other
people that have something to say as well (i.e. precisely the people listed
below). So I don't get why there's an ownership issue to reconsider. The
maven PMC peeps to work on this (Brett, Mark, anyone else who volunteers)
just get on the repository mailing list and JFDI.

> 2) Henk, myself (Maven PMC), Mark Diggory (if available), representative
> from interested Apache projects PMC (most likely someone from Ant) get
> together to sort out exactly what we think needs doing (we can use the
> repository list if appropriate)

Yep, sounds like a good idea.

> 3) suggest small steps to fix each problem rather than coming up with a
> killer solution that will never get implemented

Sounds like a good idea too. I'm guessing this is why I bailed out of that
mailing list in the first place; we were solving too big a problem at once.

> 4) can use the repository wiki for information based on what has already
> been discussed

And again, sounds like a good idea.

> I'd add that to use a common repository, there needs an implementation
> for all projects to use.

The [EMAIL PROTECTED] project was not tasked to deal with that. I think this
conflicts with #3.

Maven has support, ant will have support, magic (from the DPML guys) has
support, several ant-based scripts are around that provide support. Any tool
that can HTTP GET can trivially be made to get files from the repository.
For example I think I've done the basics in python, ruby, and bash+wget+grep
every now and then.


Cheers,


- Leo

Re: Maven and repository@apache.org

[Niclas Hedhman]

On Thursday 06 January 2005 18:57, Leo Simons wrote:

> Maven has support, ant will have support, magic (from the DPML guys) has
> support, several ant-based scripts are around that provide support. Any
> tool that can HTTP GET can trivially be made to get files from the
> repository. For example I think I've done the basics in python, ruby, and
> bash+wget+grep every now and then.

What "we" (the Dpml guys, as well as Gump guys) would really like to see from 
the Maven/Repository camp in the real short term, is 'placeholders' or just a 
list of the groups and artifacts for non-publishable resources/jars, since 
there is now a variety of different groups/artifactIds for javamail, jca, jdo 
and many others. Letting each project define this makes my local repo a real 
mess... :o(

Cheers
Niclas
-- 
---
Hard work pays off in the future, laziness pays off now.
 -  Steven Wright

+-//---+
|   http://www.dpml.net|
|  http://niclas.hedhman.org   |
+--//--+

Re: Maven and repository@apache.org

[Dain Sundstrom]

How do you get around the problem that it is very difficult to get a 
url handler installed?  Does this library work only when it is 
installed into the system class path?

-dain
--
Dain Sundstrom
Chief Architect
Gluecode Software
310.536.8355, ext. 26
On Jan 5, 2005, at 1:21 AM, Niclas Hedhman wrote:
On Wednesday 05 January 2005 16:39, Nicola Ken Barozzi wrote:
The Depot project SVN
is still there, ready to be used if/when needed by the Maven project.

From the Magic project we have spun off what we call Transit, which 
provides a
simple, flexible and powerful solution to artifact management and 
handling.

One of the most simple APIs one can imaging :o)
By putting the transit jar in system classpath, the most rudimentary 
use is;

  URL url = new URL( "artifact:jar:junit/junit#3.8.1" );
which will return a cached artifact, downloaded from configured 
resource hosts
out there.
The resource hosts can be of different types (i.e. not only the maven
organization), and the cache manager is also pluggable.
Downloads over https is supported, as well as server authorization, 
but we
still have not implemented certificate management and trust 
establishment,
but is in the pipeline.

Additional API levels support classloader management of generic 
application
plugins, or complete applications.

Interested?
Cheers
Niclas
--
---
The hardness of the butter is proportional to the softness of the 
bread.
 -  Steven Wright

+-//---+
|   http://www.dpml.net|
|  http://niclas.hedhman.org   |
+--//--+

Re: Maven and repository@apache.org

[Steve Loughran]

On Wed, 5 Jan 2005 17:21:03 +0800, Niclas Hedhman <[EMAIL PROTECTED]> wrote:
> On Wednesday 05 January 2005 16:39, Nicola Ken Barozzi wrote:
> > The Depot project SVN
> > is still there, ready to be used if/when needed by the Maven project.
> 
> From the Magic project we have spun off what we call Transit, which provides a
> simple, flexible and powerful solution to artifact management and handling.
> 
> One of the most simple APIs one can imaging :o)
> By putting the transit jar in system classpath, the most rudimentary use is;
> 
>   URL url = new URL( "artifact:jar:junit/junit#3.8.1" );
> 
> which will return a cached artifact, downloaded from configured resource hosts
> out there.
> The resource hosts can be of different types (i.e. not only the maven
> organization), and the cache manager is also pluggable.
> Downloads over https is supported, as well as server authorization, but we
> still have not implemented certificate management and trust establishment,
> but is in the pipeline.

I think for the base repository, simplicity is good. I'd like to see
what simple improvements we can do to the existing maven system,
performance and security being the ones that matter to me. I even
worry about XML manifests at the top of the tree, as the nice thing
about the maven layout today is that a downloaded repository exactly
matches the remote one; its easy to create a local mirror site.

That said, Transit appeals to me as a way of adding URLs to the
UrlClassloader and suddenly giving us access to those versions of
those files that are served by nearby machines. Which lets you provide
a declaration in your favourite deployment language of what JARs you
need, and then *wherever* you deploy, you get the stuff you want.

This interests me in my day job, which is deployment of complex
applications onto grid fabrics, working on the (LGPL) SmartFrog
project (http://smartfrog.org). We use the URL classloader with
security turned on to effect some of the dynamic code download stuff
in that system, stuff RMI needs (for code download it needs both ends
of the system to be downloading stuff from the same URL). Transit
could make a difference there for fault tolerant deployment,
distributed JUnit and other fun things I try to do.

If there was some work going on in this area, I could perhaps get
involved with daytime engineering hours. Plus with access to
PlanetLab, we have a good test harness.

-Steve

Re: Maven and repository@apache.org

[Steve Loughran]

On Thu, 6 Jan 2005 06:43:34 +1100, Brett Porter <[EMAIL PROTECTED]> wrote:
> On Wed, 5 Jan 2005 14:24:13 -0500, Noel J. Bergman <[EMAIL PROTECTED]> wrote:
> > Is Maven willing to provide suitable support for Ant to use it?  I just want
> > to make sure that this is not the Maven repository, but is THE repository.
> > Tool-agnostic.
> 
> Yes - I've already discussed this and Steve and Russell seem prepared
> to integrate it into their libraries task(s).
> 
> There shouldn't need to be anything special required for Ant to use it
> - its a plain Java API.

The sole requirement would be that we can build it with no
dependencies other than JREs 1.2+ and Xerces/Crimson. That way we can
ship the task built in to ant without any need for external libraries.

Re: Maven and repository@apache.org

[Niclas Hedhman]

On Wednesday 05 January 2005 16:39, Nicola Ken Barozzi wrote:
> The Depot project SVN
> is still there, ready to be used if/when needed by the Maven project.

>From the Magic project we have spun off what we call Transit, which provides a 
simple, flexible and powerful solution to artifact management and handling.

One of the most simple APIs one can imaging :o)
By putting the transit jar in system classpath, the most rudimentary use is;

  URL url = new URL( "artifact:jar:junit/junit#3.8.1" );

which will return a cached artifact, downloaded from configured resource hosts 
out there.
The resource hosts can be of different types (i.e. not only the maven 
organization), and the cache manager is also pluggable.
Downloads over https is supported, as well as server authorization, but we 
still have not implemented certificate management and trust establishment, 
but is in the pipeline.

Additional API levels support classloader management of generic application 
plugins, or complete applications.

Interested?

Cheers
Niclas
-- 
---
The hardness of the butter is proportional to the softness of the bread.
 -  Steven Wright

+-//---+
|   http://www.dpml.net|
|  http://niclas.hedhman.org   |
+--//--+

Re: Maven and repository@apache.org

[Brett Porter]

On Wed, 5 Jan 2005 14:24:13 -0500, Noel J. Bergman <[EMAIL PROTECTED]> wrote:
> Is Maven willing to provide suitable support for Ant to use it?  I just want
> to make sure that this is not the Maven repository, but is THE repository.
> Tool-agnostic.

Yes - I've already discussed this and Steve and Russell seem prepared
to integrate it into their libraries task(s).

There shouldn't need to be anything special required for Ant to use it
- its a plain Java API.

Cheers,
Brett

Re: Maven and repository@apache.org

[Brett Porter]

> Why not something like:
> 
>  
...
>  
>
>  
>  
>  
> 

In Maven 1.0, there is a maven.repo.remote property that allows you to
select what repositories to search on a per-project basis.

Maven 1.1 will allow this to be in the project.xml file, but is still
on a per-project basis. The user can always override this to ensure
only an internal repository is used.

I don't see the need to have this on a per-dependency basis.

But, this is getting OT :)

Cheers,
Brett

RE: Maven and repository@apache.org

[Noel J. Bergman]

> IMO we should [make] the Maven artifact handling better
> and used by all Apache and non-Apache projects.

Is Maven willing to provide suitable support for Ant to use it?  I just want
to make sure that this is not the Maven repository, but is THE repository.
Tool-agnostic.

--- Noel

Re: Maven and repository@apache.org

[Russell Gold]

On Wed, 05 Jan 2005 18:13:16 +0100, Nicola Ken Barozzi
<[EMAIL PROTECTED]> wrote:
> I mean, instead of:
> 
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> 
> Why not something like:
> 
>  
>  
>  
>  
>  
>  
>  
>  
>  
>
>  
>  
>  

Ah, I see. Yes, as it happens, both the ant task that Steve is working
on and the external ant task I wrote
 permit the
specification of a URL on a repository type, although the syntax is a
bit simpler than maven's. AFAIK maven does support this, using a
system property to specify the available repository URL(s).

> This is controversial, but it may be, and sun shows this in a similar
> way with the click-through license stuff, that some would not want to
> have their repo be part of a central DNS, and some users, especially in
> corporate environments, want to be able to direct users to different
> repos for different artifacts.

Which probably means safeguards on what is included in any public
repository. I note that some common Sun jars used to be in the ibiblio
repository and have been removed, presumably at Sun's behest. Possibly
each repository needs to have a licensing policy, only permitting the
inclusion of those with a particular class of license.  Some licenses,
such as the Apache license and MIT license could be used just about
anywhere. IBM's is more restrictive, and might be a reason to exclude
jars using it. The same is true for GPL and LGPL, all of which place
some restrictions on the companies using the artifacts.

I am considering recommending to my employer that we set up our own
repository, so that the legal folks can control which open-source
projects we use.

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Russell Gold wrote:
On Wed, 05 Jan 2005 17:04:17 +0100, Nicola Ken Barozzi 
<[EMAIL PROTECTED]> wrote:

I kinda disagree. Eclipse 'forces' you to enter the http of the 
repository from which you can get a plugin, and AFAIK it has not
become (yet?) unwieldy.
Plugins are not the same as dependencies. Each user selects a plugin 
typically *once* as part of a manual configuation of the tool. 
Dependencies are selected (and considered for download) *every* time 
an automated build is run.
:-?  Don't I write this down in a property file - pom?
Further, in an open-source world the builds are frequently run by
people who are not and have no access to those who defined the
builds. Downloading really needs to be easy and automatic.
This is not about downloading...
I mean, instead of:











Why not something like:









  




You see, directing users to repositories with diversely licensed
jars, is not something that is particularly nice, as you put
yourself in a position of being liable.
How so?
This is controversial, but it may be, and sun shows this in a similar
way with the click-through license stuff, that some would not want to
have their repo be part of a central DNS, and some users, especially in
corporate environments, want to be able to direct users to different
repos for different artifacts.
If we build a central "DNS", we would become possible target in case
someone does not want to be listed, or, vice versa, we could not provide
the repo service if someone has a repo but does not want to be listed.
--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

Re: Maven and repository@apache.org

[Russell Gold]

On Wed, 05 Jan 2005 17:04:17 +0100, Nicola Ken Barozzi
<[EMAIL PROTECTED]> wrote:
> I kinda disagree. Eclipse 'forces' you to enter the http of the
> repository from which you can get a plugin, and AFAIK it has not become
> (yet?) unwieldy.

Plugins are not the same as dependencies. Each user selects a plugin
typically *once* as part of a manual configuation of the tool.
Dependencies are selected (and considered for download) *every* time
an automated build is run.  Further, in an open-source world the
builds are frequently run by people who are not and have no access to
those who defined the builds. Downloading really needs to be easy and
automatic.

> You see, directing users to repositories with diversely licensed jars,
> is not something that is particularly nice, as you put yourself in a
> position of being liable.

How so?

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Tim O'Brien wrote:
 
From: Nicola Ken Barozzi [mailto:[EMAIL PROTECTED] 
...
This is something that is already present in the JJar[1] project.
Wouldn't it suffice to just tell the system in what 
repository to look for to get a certain artifact?
Sure, you could do this, but then you get into the problem of
management.  The early internet became unwieldy because every /etc/hosts
was explicit, then came DNS.
I kinda disagree. Eclipse 'forces' you to enter the http of the 
repository from which you can get a plugin, and AFAIK it has not become 
(yet?) unwieldy.

Consider ibiblio.org, if every artifact was associated with an
organization (apache.org ->
http://http://www.apache.org/dist/java-repository/), then you could
(optionally) configure your client to only download an artifact from an
authoritative server for that particular artifact.  You could use some
file like resolution.xml to just provide pointers to ASF managed
resources - say a version of the ASF maven repository at one of the
mirrors.  

This way there is never a need to create one centralized conglomeration
of artifacts.  You could have many maven repositories all containing
only a portion of the entire "artifact space".
I agree with this. I was just thinking of another way to do it.
You see, directing users to repositories with diversely licensed jars, 
is not something that is particularly nice, as you put yourself in a 
position of being liable.

If instead the user has to enter the URL of the repository that contains 
an artifact, this doesn't happen.

Also, if the central "DNS" goes down the system would halt
It would also still force anybody that wants to have something published 
to have an entry in that "DNS".

But still, this is becoming OT for the purpose of this discussion.
--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

RE: Maven and repository@apache.org

[Tim O'Brien]

 

> -Original Message-
> From: Nicola Ken Barozzi [mailto:[EMAIL PROTECTED] 
> Sent: Wednesday, January 05, 2005 8:36 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Maven and [EMAIL PROTECTED]
> 
> 
> Tim O'Brien wrote:
> > What about having an XML description of the contents of a 
> repository?  
> > 
> > Such a description could serve multiple purposes, it could 
> be used to 
> > enumerate known mirrors, it could be used to segment the 
> "namespace" - 
> > say we reach some agreement with Sun and all sun artifacts 
> fall into a 
> > namespace of sun-* and must be redirected to a Sun server, 
> etc.  This 
> > XML file could be used by tools that want to provide a list 
> of every 
> > possible artifact.
> > 
> > What if every Maven repository knew of every other Maven repository 
> > because they all shared a common resolution.xml file (think 
> /etc/hosts 
> > before the existence of DNS - hackish but it worked).
> 
> This is something that is already present in the JJar[1] project.
> 
> Wouldn't it suffice to just tell the system in what 
> repository to look for to get a certain artifact?
> 

Sure, you could do this, but then you get into the problem of
management.  The early internet became unwieldy because every /etc/hosts
was explicit, then came DNS.

Consider ibiblio.org, if every artifact was associated with an
organization (apache.org ->
http://http://www.apache.org/dist/java-repository/), then you could
(optionally) configure your client to only download an artifact from an
authoritative server for that particular artifact.  You could use some
file like resolution.xml to just provide pointers to ASF managed
resources - say a version of the ASF maven repository at one of the
mirrors.  

This way there is never a need to create one centralized conglomeration
of artifacts.  You could have many maven repositories all containing
only a portion of the entire "artifact space".

Tim

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Brett Porter wrote:
...
What interests me WRT repository is that it's possible for Ant users to
install an antlib that makes Wagon usable in Ant, and for Eclipse and
Netbeans users to be able to integrate it nicely in their artifact
stuff. Depot had an Eclipse pluin BTW.
Can you elaborate on how it was integrated into Eclipse and Netbeans?
I assume this would be when you add a dependency to a project - if so,
that's definitely something that has been talked about (the MevenIDE
project may have already done this - I'm not sure), and could use what
was in depot to do so.
http://krysalis.org/ruper/eclipse/
Note that it supports multiple repositories.
...
Out of interest, this is the proposed future repository layout for Maven:
http://docs.codehaus.org/pages/viewpage.action?pageId=8003
Any recommendation on how to move on from there, keeping in mind that
there are already two wikis with different specs and two different lists
([EMAIL PROTECTED] and this one)?
I don't think discussing the repository format is helpful at this
point - it is a big undertaking to change and I think there are other
priorities (security, mirroring and cleanup). That seems to be where
this list got bogged down last time, though what is there now looks
pretty much final. When it's not so late, I'll look at what is
actually different and whether this is even an issue.
The repository code can have different repository layouts, so if
necessary we can support different ones. Obviously, its preferable to
agree and be consistent.
Eventually it will make sense to cutover to a new layout - but we'll
want to ensure everything is in place first.
Big +1 to this. Write it in stone :-)
The extra features needed are Apache mirroring support and use of the
Sourceforge system without needing to recreate a repository, that Depot
somewhat supported.
That's interesting - being involved with a sourceforge project they
prefered to release to a maven repository and not use sourceforge :)
That's a good feature, though probably at the bottom of the list so far.
It permits project artifacts not to be put inside ibiblio, with the 
security and access issues that it brings.

I'm fine with the proposed layout, or any other reasonable alternative,
as it does not interfere with the above, being still http and file based.
Great.
--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Tim O'Brien wrote:
What about having an XML description of the contents of a repository?  

Such a description could serve multiple purposes, it could be used to
enumerate known mirrors, it could be used to segment the "namespace" -
say we reach some agreement with Sun and all sun artifacts fall into a
namespace of sun-* and must be redirected to a Sun server, etc.  This
XML file could be used by tools that want to provide a list of every
possible artifact.
What if every Maven repository knew of every other Maven repository
because they all shared a common resolution.xml file (think /etc/hosts
before the existence of DNS - hackish but it worked).  
This is something that is already present in the JJar[1] project.
Wouldn't it suffice to just tell the system in what repository to look 
for to get a certain artifact?

[1] http://jakarta.apache.org/commons/sandbox/jjar/
--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

RE: Maven and repository@apache.org

[Tim O'Brien]

What about having an XML description of the contents of a repository?  

Such a description could serve multiple purposes, it could be used to
enumerate known mirrors, it could be used to segment the "namespace" -
say we reach some agreement with Sun and all sun artifacts fall into a
namespace of sun-* and must be redirected to a Sun server, etc.  This
XML file could be used by tools that want to provide a list of every
possible artifact.

What if every Maven repository knew of every other Maven repository
because they all shared a common resolution.xml file (think /etc/hosts
before the existence of DNS - hackish but it worked).  

> -Original Message-
> From: Brett Porter [mailto:[EMAIL PROTECTED] 
> Sent: Wednesday, January 05, 2005 6:43 AM
> To: [EMAIL PROTECTED]; Steve Loughran
> Subject: Re: Maven and [EMAIL PROTECTED]
> 
> > I'll be the Ant rep.
> 
> Great, thanks.
> 
> > I am co-author of the (still stabilising) Ant  task; it'd
> 
> yeah, I've got to 50 mail threads sitting flagged in gmail to 
> read one day, as this is about the extent of what I know 
> about it :) (after you introduced it to repository@ last year)
> 
> > 1. security. this could be with MD5 checksums, or it could be with 
> > signed JARs.
> 
> MD5's aren't going to do much for security - they're mainly 
> for download integrity. checking and publishing ASC files is 
> a definite want I have, and that can be ramped up to the 
> level of security you need (there are obviously varying 
> levels of trust of the files and the KEYS themselves).
> 
> > JAR signing needs retrofitting to existing files, but has the 
> > advantage that JVMs integrate with it and you can do other tricks 
> > (like put http://ibiblio.org.../artifact.jar on the classpath with 
> > security turned on)
> 
> That I haven't looked into, but would also be a good, but 
> optional feature. I think this is more of a build feature 
> than a repository feature? In fact, I'm sure we already do 
> this for JNLP.
> 
> > 2. licenses. not just auto-download of .LICENSE files, but ideally 
> > some way to do click-through that even Sun are happy with.
> 
> Yeah, there's a low hundreds JIRA entry for that (ie OLD :) I 
> think even that wouldn't fly with Sun IIRC but it doesn't hurt to ask.
> 
> Should be easy to add hooks and allow a user to say "never 
> ask again for this license" to always accept ASL or 
> something, but still report the license on download.
> 
> Good ideas and reminders - keep them coming, and I'll put all 
> this together on the wiki tomorrow-ish.
> 
> Thanks,
> Brett
> 
> 

Re: Maven and repository@apache.org

[Steve Loughran]

On Wed, 5 Jan 2005 23:42:30 +1100, Brett Porter <[EMAIL PROTECTED]> wrote:
> > JAR signing needs retrofitting to existing files, but has
> > the advantage that JVMs integrate with it and you can do other tricks
> > (like put http://ibiblio.org.../artifact.jar on the classpath with
> > security turned on)
> 
> That I haven't looked into, but would also be a good, but optional
> feature. I think this is more of a build feature than a repository
> feature? In fact, I'm sure we already do this for JNLP.

yes, its a build feature. But if every jar was signed then you can
verify that it hasnt been tampered with, without having to verify MD5s
against those of a remote https site, etc etc. But it is
side-effecting on the jar.

> 
> > 2. licenses. not just auto-download of .LICENSE files, but ideally
> > some way to do click-through that even Sun are happy with.
> 
> Yeah, there's a low hundreds JIRA entry for that (ie OLD :) I think
> even that wouldn't fly with Sun IIRC but it doesn't hurt to ask.

I've been talking to Jesse Glick of the Netbeans team; they have some
public server with their own ant tasks to click-through licensing
every fetch -and provide a key for automated builds if you can justify
it.

What I'd like is

-license only appears if there is a change in the .LICENSE file
-in ant, the popup license would be managed so that IDEs, Cruise
control can do their own thing.
-you could register a set of licenses you always accept :
 "Apache,LGPL,Sun"
That'd need every license to be represented with a family and a
version, which means an XML file if I am not mistaken.

-steve

Re: Maven and repository@apache.org

[Brett Porter]

> I'll be the Ant rep.

Great, thanks.

> I am co-author of the (still stabilising) Ant  task; it'd

yeah, I've got to 50 mail threads sitting flagged in gmail to read one
day, as this is about the extent of what I know about it :) (after you
introduced it to repository@ last year)

> 1. security. this could be with MD5 checksums, or it could be with
> signed JARs. 

MD5's aren't going to do much for security - they're mainly for
download integrity. checking and publishing ASC files is a definite
want I have, and that can be ramped up to the level of security you
need (there are obviously varying levels of trust of the files and the
KEYS themselves).

> JAR signing needs retrofitting to existing files, but has
> the advantage that JVMs integrate with it and you can do other tricks
> (like put http://ibiblio.org.../artifact.jar on the classpath with
> security turned on)

That I haven't looked into, but would also be a good, but optional
feature. I think this is more of a build feature than a repository
feature? In fact, I'm sure we already do this for JNLP.

> 2. licenses. not just auto-download of .LICENSE files, but ideally
> some way to do click-through that even Sun are happy with. 

Yeah, there's a low hundreds JIRA entry for that (ie OLD :) I think
even that wouldn't fly with Sun IIRC but it doesn't hurt to ask.

Should be easy to add hooks and allow a user to say "never ask again
for this license" to always accept ASL or something, but still report
the license on download.

Good ideas and reminders - keep them coming, and I'll put all this
together on the wiki tomorrow-ish.

Thanks,
Brett

Re: Maven and repository@apache.org

[Steve Loughran]

On Wed, 05 Jan 2005 09:39:28 +0100, Nicola Ken Barozzi
<[EMAIL PROTECTED]> wrote:
> 2) Henk, myself (Maven PMC), Mark Diggory (if available), representative
> from interested Apache projects PMC (most likely someone from Ant) get
> together to sort out exactly what we think needs doing (we can use the
> repository list if appropriate)

I'll be the Ant rep. 

I am co-author of the (still stabilising) Ant  task; it'd
be nice if we can get the immediate improvements into the repository
when there is time to get the changes into ant before 1.7 ships (which
is not in the immediate future)

> 3) suggest small steps to fix each problem rather than coming up with a
> killer solution that will never get implemented

Agreed. 


I'm very interested in 

1. security. this could be with MD5 checksums, or it could be with
signed JARs. JAR signing needs retrofitting to existing files, but has
the advantage that JVMs integrate with it and you can do other tricks
(like put http://ibiblio.org.../artifact.jar on the classpath with
security turned on)

2. licenses. not just auto-download of .LICENSE files, but ideally
some way to do click-through that even Sun are happy with. I was
discussing the latter informally with someone at Sun recently -how
it'd be nice to be able to retrieve files from some Sun server with
the appropriate licenses appearing and needing accepting before the
jar files are decrypted. This would be similar to the licensing stuff
you get in SuSE Linux's package updater, where you have to accept the
licenses for some downloads.

(2) is clearly more complex, lower priority and really needs active
involvement from Sun.

-steve

Re: Maven and repository@apache.org

[Brett Porter]

putting repository@ back on the list - gmail tends to drop lists
unless you reply to all (hope this is what you intended).

> Seems a reasonable first step. However it seems a little fragile if a
> project changes artifact names or splits an existing artifact. e.g. if
> commons-jelly splits into jelly-api and jelly-impl, at least with the
> existing structure all artifacts of a given type are together and can
> be easily seen. Once the directory structure changes in the new
> layout, browsing will be difficult.
> 

I'm not sure what you mean: the existing one has commons-jelly-* JARS,
then commons-jelly-api-* and commons-jelly-impl-*. The new one would
have a commons-jelly subdirectory, a commons-jelly-api and
commons-jelly-impl subdirectory. They are all still side by side in
o/a/c/jelly/ ?

Still, an archival policy is a good idea so that commons-jelly can be
removed from here so people know what the current artifacts are (while
still being able to get the old ones).

I think there's merit to having 3 repositories: snapshot, archive and
current (current is not just the latest release, but all supported
releases). Equivalent to cvs.apache, archive.apache and www.apache
now.

Each can have different policies, but a common access technique.

- Brett

Re: Maven and repository@apache.org

[Brett Porter]

> Take the commons-jelly/ directories on ibiblio and place them in a
> similar structure as an example. 

org/apache/commons/jelly/commons-jelly/1.0/commons-jelly-1.0.jar[.md5|.asc], etc
org/apache/commons/jelly/commons-jelly-tags-ant/1.0/commons-jelly-1.0.jar
org/apache/commons/jelly/commons-jelly-tags-util/1.1/commons-jelly-1.1.jar

I guess there's a risk (especially if a project splits), that there be
a conflict between artifactId and groupId[n]. I've noted it in
confluence, and I'll look at how this differs to ASFRepository when
I'm not in need of sleep :)

Does this look ok?

> I think that document will need some more work.

Looking at it now, I agree it needs more examples, though I'm still
happy with the ideas.

- Brett

Re: Maven and repository@apache.org

[Dion Gillard]

On Wed, 5 Jan 2005 22:14:27 +1100, Brett Porter <[EMAIL PROTECTED]> wrote:
> Hi,
> 
> Firstly, can I check who is still actively involved here other than
> Nicola, Mark and myself?

I'm lurking.

> http://docs.codehaus.org/pages/viewpage.action?pageId=8003

> From what I remember, it takes into account all of the points raised
> on the ASFRepository wiki layout document.

Having had a quick read of that document it *seems* limited to
'groups' (the groupId array) with only one artifact, i.e. the
plexus-container example is a little simplistic.

Take the commons-jelly/ directories on ibiblio and place them in a
similar structure as an example. It's not clear from that document
what the 'version' would refer to for multiple unrelated artifacts or
how the 'groupId' tree would be structured, e.g.

Is the groupId org.apache.commons.jelly?
Is it org.apache.commons.jelly.tags for the tag libraries which are
produced by the same 'group'?

I think that document will need some more work.
-- 
http://www.multitask.com.au/people/dion/

Re: Maven and repository@apache.org

[Brett Porter]

> Ant is adding a task so that providers can plug into it, and Wagon can
> continue to be used in Ant.

Absolutely - all I meant was that Wagon maintaining its own Ant task
was probably not necessary given that. Wagon can plugin in to that.

> If we splinter implementations, the repository effort will always be
> half-baked, and acceptance will suffer.

I agree.

> What interests me WRT repository is that it's possible for Ant users to
> install an antlib that makes Wagon usable in Ant, and for Eclipse and
> Netbeans users to be able to integrate it nicely in their artifact
> stuff. Depot had an Eclipse pluin BTW.

Can you elaborate on how it was integrated into Eclipse and Netbeans?
I assume this would be when you add a dependency to a project - if so,
that's definitely something that has been talked about (the MevenIDE
project may have already done this - I'm not sure), and could use what
was in depot to do so.

> If this means that we also need a maven.jar in the classpath, I have no
> problem with it, as long as users are not forced to run maven itself to
> download the artifacts.

Right. It wouldn't even be maven.jar - just a smaller JAR that does
the repository and snapshot handling. It's definitely separate.

> > Out of interest, this is the proposed future repository layout for Maven:
> > http://docs.codehaus.org/pages/viewpage.action?pageId=8003
> 
> Any recommendation on how to move on from there, keeping in mind that
> there are already two wikis with different specs and two different lists
> ([EMAIL PROTECTED] and this one)?

I don't think discussing the repository format is helpful at this
point - it is a big undertaking to change and I think there are other
priorities (security, mirroring and cleanup). That seems to be where
this list got bogged down last time, though what is there now looks
pretty much final. When it's not so late, I'll look at what is
actually different and whether this is even an issue.

The repository code can have different repository layouts, so if
necessary we can support different ones. Obviously, its preferable to
agree and be consistent.

Eventually it will make sense to cutover to a new layout - but we'll
want to ensure everything is in place first.

> The extra features needed are Apache mirroring support and use of the
> Sourceforge system without needing to recreate a repository, that Depot
> somewhat supported.

That's interesting - being involved with a sourceforge project they
prefered to release to a maven repository and not use sourceforge :)

That's a good feature, though probably at the bottom of the list so far.

> I'm fine with the proposed layout, or any other reasonable alternative,
> as it does not interfere with the above, being still http and file based.

Great.

Cheers,
Brett

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Brett Porter wrote:
...
Wagon's scope is really only transport though - uploading and
downloading through a variety of providers with a common Java API
(there was also an Ant task at one point - though since Ant have
developed their own there is little point in updating the wagon one).
I disagree here.
Ant is adding a task so that providers can plug into it, and Wagon can 
continue to be used in Ant.

If we splinter implementations, the repository effort will always be 
half-baked, and acceptance will suffer.

MD5 hashing is handled, and I'd like to add PGP signing obviously.
However, the repository layout code is no longer in there. It is in a
separate component of Maven, so there can be some discussion on our
end about whether that really makes it part of the Wagon project, but
not the Wagon API. That's really administrative more than anything.
What interests me WRT repository is that it's possible for Ant users to 
install an antlib that makes Wagon usable in Ant, and for Eclipse and 
Netbeans users to be able to integrate it nicely in their artifact 
stuff. Depot had an Eclipse pluin BTW.

If this means that we also need a maven.jar in the classpath, I have no 
problem with it, as long as users are not forced to run maven itself to 
download the artifacts.

The Depot project SVN
is still there, ready to be used if/when needed by the Maven project.
I'll take a look, thanks. Is there any doco available on what depot's
advantages were, why it came into being and why it eventually closed?
http://incubator.apache.org/depot/
http://incubator.apache.org/depot/update/
http://incubator.apache.org/depot/version/
http://incubator.apache.org/projects/depot.html
(Update was caller Ruper)
Thus I would add that the [EMAIL PROTECTED] list and wikis should
support and point users to the Maven repository handling code.
Does this sound reasonable?
Sounds good to me.
I'll try and get my thoughts together and onto the wiki soon. I
noticed several requirements are missing, and the road map is probably
out of date.
Out of interest, this is the proposed future repository layout for Maven:
http://docs.codehaus.org/pages/viewpage.action?pageId=8003
Any recommendation on how to move on from there, keeping in mind that 
there are already two wikis with different specs and two different lists 
([EMAIL PROTECTED] and this one)?

From what I remember, it takes into account all of the points raised
on the ASFRepository wiki layout document.
The extra features needed are Apache mirroring support and use of the 
Sourceforge system without needing to recreate a repository, that Depot 
somewhat supported.

I'm fine with the proposed layout, or any other reasonable alternative, 
as it does not interfere with the above, being still http and file based.

--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

Re: Maven and repository@apache.org

[Brett Porter]

Hi,

Firstly, can I check who is still actively involved here other than
Nicola, Mark and myself?

Continuing on what was started in this thread:

> I'd add that to use a common repository, there needs an implementation
> for all projects to use. It's easy, correct and resonable to think that
> it should be under Maven. Is Wagon still active? 

Yes, Wagon is quite close to a public release. It's been stable for
over a year, but has not really had the force behind it to justify
separation from Maven.

Wagon's scope is really only transport though - uploading and
downloading through a variety of providers with a common Java API
(there was also an Ant task at one point - though since Ant have
developed their own there is little point in updating the wagon one).
MD5 hashing is handled, and I'd like to add PGP signing obviously.

However, the repository layout code is no longer in there. It is in a
separate component of Maven, so there can be some discussion on our
end about whether that really makes it part of the Wagon project, but
not the Wagon API. That's really administrative more than anything.

> The Depot project SVN
> is still there, ready to be used if/when needed by the Maven project.

I'll take a look, thanks. Is there any doco available on what depot's
advantages were, why it came into being and why it eventually closed?

> Thus I would add that the [EMAIL PROTECTED] list and wikis should
> support and point users to the Maven repository handling code.
> 
> Does this sound reasonable?

Sounds good to me.

I'll try and get my thoughts together and onto the wiki soon. I
noticed several requirements are missing, and the road map is probably
out of date.

Out of interest, this is the proposed future repository layout for Maven:
http://docs.codehaus.org/pages/viewpage.action?pageId=8003

>From what I remember, it takes into account all of the points raised
on the ASFRepository wiki layout document.

Cheers,
Brett

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Brett Porter wrote:
> Nicola Ken Barozzi wrote:
Ass Brett Porter has written, and I agree:
I hope the extra s was a slip up :)
Sorry :-/
--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-

Re: Maven and repository@apache.org

[Nicola Ken Barozzi]

Niclas Hedhman wrote:
On Wednesday 05 January 2005 16:39, Nicola Ken Barozzi wrote:
The Depot project SVN
is still there, ready to be used if/when needed by the Maven project.

From the Magic project we have spun off what we call Transit, which provides a 
simple, flexible and powerful solution to artifact management and handling.
...
Interested?
I'm sorry to say... no, I'm not.
I've already done the error of trying to spun off Depot, and I won't do 
it again. There is only one Apache project that has actually done 
something in the repository area and that's Maven.

IMO we should concentrate on that, and make the Maven artifact handling 
better and used by all Apache and non-Apache projects.

--
Nicola Ken Barozzi   [EMAIL PROTECTED]
- verba volant, scripta manent -
   (discussions get forgotten, just code remains)
-