Re: Rsync Emails
On Tue, 3 Aug 2004, Mark R. Diggory wrote: I'm curious, would others be interested in seeing the rsync emails from ibiblio for the rsync that runs every 4 hours on the login.ibibilio.org server? I could direct them to this list or possibly the maven devel list if its more appropriate. Are they verbose, or only in errors? This list seems a good place, unless overly verbose. regards Adam
Re: Rsync Emails
I could possibly tweek it to not send an email if there are no new files being moved. Please do. regards Adam
Re: Anywhere near concensus?
Done:http://cvs.apache.org/~ajack/repository/proposals/ I think of the group, I like repositoryFQDN/ best. For me, it is between that and group (oh yeah, I asked for that one. ;-) 'cos they are both of deterministic directory depth/layout. [BTW: If things get 'overloaded' we can do the traditional /fred - /f/r/fred w/o changing deterministic depth/layout.] That said, to me 'group' and FQDN don't need to be any different --- one is just saying 'for Java projects (or those w/ DN-like packages) the likely group name is the FQDN of the root package'. regards, Adam
Re: Anywhere near concensus?
IIRC, the repository structure used by Maven (http://www.ibiblio.org/maven/) has generated much discussion in the past, with the general concensus being that the flat structure: . didn't help artifact categorisation . made it difficult to navigate and locate artifacts Folk have resolved those issues with type and version sub-directories. What we have now isn't flat. It was an excellent first step, but I think it can be done better. Yes, me too, but we need a tight set of useful categorizations, and I'm not sure that 'sub-project' is one. Artefact id (plus the others) covers that. I think that things like 'sub-project' are just too fluid and too subjective to fix into the URI. [Gump has a different view of a project than Maven than ...] regards Adam
Re: Use of '/' in ???-specifier's
Not a criticism, but I'd prefer to know the requirements, before writing the tools. I know, I've been a huge advocate of that, but I'm starting to worry we are in analysis paralysis. Logical URIs are so virtual it is easy to miss practical implications. As such, I'd like to test the theory against the tool implementation in my head. That said, we need to revisit requirements, especially the tooling aspect of that. As Nick said having a tool know/display/process what is in a remote repostiory w/o metadata is my goal. As far as I can tell, maven doesn't do URI parsing. I don't know a lot about Gump, but if it wants to pull down the newest versions of jars, it can via the latest version tag [1]. Gump (like Maven) has metadata also, so could ask for a certain version of something. That said, we are talking about client side metadata with all of these, not in repository metadata. As such, this information is not in the repostory (per se) and so **not available to repository tools**. This is the key point. FWIIW: I *hate* maintaining version information in metadata (in CVS or not) 'cos I think it leads to stale links, and to too much duplication. I think I've said this (once or twice ;-) before though. BTW: I'd like the repository to work independent of client metadata, so we have a consistent repository however tools use it. Maybe we'll have a metadata.xml per directory that we can all agree upon, and extend, but that isn't what we have here and now. Avalon adds meta-data, which is supported through the statement in [2]: Projects should be able to deploy arbitrary artifacts to the repository, whether they be for end-users, or meta-data (e.g, maven's project.xml). Tools should ignore any artifact they don't understand. Yup, but I'd like to see some cross-project-type tools possible. I'd like tools to be able to work against all types of artefacts, built by Maven, by Gump, by whomever. regards Adam
Re: [proposal] URI Syntax - v0.2
Noel wrote: Adam, and how is said tool going to start in the first place? Without meta-data, there is a limit to what the tool can do. Basically, it would have to operate relative to the URL provided to it. My input here is primarily based on writting Ruper (http://www.krysalis.org/ruper), a tool that attempts exactly what I said. It is given links to repositories (local or remote), it read the repositories and allows queries into that repository based on attributes of the resources. It does this by parsing the URLs. You don't have to like the tool, I'm not trying to push the implementation, I'm just giving you experiences from that tool. It allows you to query what is there, query and capture oldest resources [and do a delete/clean], and download newest, etc. Some find such a tool useful, I'd like to believe that apache users (admins and external users) would find it useful. I don't care whose implementation gets used, I feel that these capabilities are so powerful that they ought consist of a minimum bar for apache. Sure, it isn't going to be a 100% generic tool for all cases, but apache is doing this for apache. Let the tools lead and the users (our own committers) can chose to follow. Once, along came a browser and sooner or later folks were converting their documents to HTML 'cos the benefits outweighed the resistence to change. I'm saying that we can't enforce things, but if we make the benefits sufficiently worthwhile transition easy folks then most folks will follow. Again, this tool works today on over 95% of the contents of the Maven repository without any spec. We could achieve this. A nice simple IDE plugin can update a project and download files with or w/o user intervention, e.g. http://www.krysalis.org/ruper/eclipse/index.html. Tim's URI schema supports your operations when combined with with a semantic layer, which can be implied or meta-data based. Aren't you saying that metadata can allow a remote tool to instrospect? Yes, I agree, this has nothing to do with an unparsable URI scheme. The URI scheme is generally fine, but if we aren't addressing metadata (almost impossible) why set back tools that mine metadta from URIs? It works today, why would we force a step backwards? [I sometimes feel the acadaemics of the URI Scheme Specifiecation are outweighing the practicalities of an implementation. I beleive in writting a specification first, but specifications get revised based upon real world experience. Tools are that experience.] For me, the strongest argument for tooling (other purely than saving admins effort) is download + verify (MD5/whatever). That does not require the kind of semantic your earlier operations require. The verification content can be relative to the URI provided to the tool. True, my bad, I go carried away with my argument, the tool I am familiar with, and my own dislike of stale software links. You could have the client tool be told the resource/URI by the user, and do the download/verification, yes. That said, I don't think it buys the user enough, they have to browse/locate stash the URI in some local config. I'd like to say go get me xerces from any repository it is in, but get me the latest, but I only want release not nightly/snapshot (e.g. http://www.krysalis.org/ruper/ant/reposync.htm). That to me, is useful. I don't mind being alone in my views, but I ask again -- if we don't set the bar higher than a one-way URI for download, why write a spec at all? I feel we have the potential to win big, and I'd like the ASF Repository to be a step forward towards these goals, not a step backward. regards Adam
Re: [proposal] common build version specifier - v0.1
Tim, I *love* your specifications, I really appreciate the clear/concise/explicit nature of them. I only wish you'd use Wiki not EyeBrowse as your persistent documentation tool. Wiki has versioning (so we can see older copies should we need to refer back) and such, and allows other to make (respectful) changes, and allows a view not cluttered throughout mail threads. I'm game to be your cut-n-paste-wallah if you really need one, but please (at least) refer to your proposals in the Wiki. regards Adam - Original Message - From: Tim Anderson [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 14, 2003 8:21 PM Subject: [proposal] common build version specifier - v0.1 Overview This proposal extends the URI Syntax proposal, v0.2: http://nagoya.apache.org/eyebrowse/[EMAIL PROTECTED]ms gNo=367 The key aims of this proposal are to: . formalise version-specifier for projects which provide formal and interim builds; . provide a set of best practices for such projects; and . enable tools to construct a URI to unambigously locate a particular project version using a set of known criteria URI Components == An absolute repository URI is written as follows: repository-uri = access-specifier / product-specifier / version-specifier / artifact-specifier This proposal defines version-specifier as follows: version-specifier = build build = formal-build | interim-build formal-build = [formal-build-designation /] version interim-build = interim-build-designation / version [ / interim-version ] interim-version = latest | MMDD [- HHMM [SS]] formal-build-designation = release | ... interim-build-designation = interim | nightly | snapshot | ... version = latest | *pchar (pchar is per http://www.ietf.org/rfc/rfc2396.txt) Build - Builds are separated into formal and interim builds. A formal build may be a final or milestone release. e.g: 1.0, release/1.0, 1.0-beta1, release/1.0-rc1 An interim build is an informal release, produced by a nightly build or an ad-hoc snapshot. e.g: nightly/1.0/20031113, snapshot/1.2beta1. Version --- Version is either latest or arbitrary, determined by the project or deployment tool. latest always refers to the latest version of a particular build, and may be supported using symbolic links, or via http redirection. E.g: http://repo.apache.org/apache/commons-cli/release/l.0/... http://repo.apache.org/apache/commons-cli/release/l.1/... http://repo.apache.org/apache/commons-cli/release/latest/... - symlink to ../1.1 Interim version --- The interim version is either a timestamp, or latest. latest always refers to the latest interim version and may be supported using symbolic links, or via http redirection. http://repo.apache.org/apache/commons-cli/nightly/1.0/20031112/... http://repo.apache.org/apache/commons-cli/nightly/1.0/20031113/... http://repo.apache.org/apache/commons-cli/nightly/1.0/latest/... - symlink to ../20031113 Rationale = Optional build designation for formal builds formal-build is defined as: formal-build = [formal-build-designation /] version formal-build-designation = release | ... The formal-build-designation is optional for those projects which don't produce interim builds, or don't wish to add another directory for formal releases. E.g: http://repo.apache.org/apache/commons-cli/l.0/... http://repo.apache.org/apache/commons-cli/l.1/... Mandatory version in interim builds --- interim-build is defined as: interim-build = interim-build-designation / version [ / interim-version ] interim-version = latest | MMDD [- HHMM [SS]] This enables support for multiple versions of builds, if there are two or more concurrent development streams. E.g, to support nightly builds of versions 1.0 and 2.0 of commons-cli: http://repo.apache.org/apache/commons-cli/nightly/1.0/20031112/... http://repo.apache.org/apache/commons-cli/nightly/1.0/20031113/... http://repo.apache.org/apache/commons-cli/nightly/1.0/latest/... - symlink to ../20031113 http://repo.apache.org/apache/commons-cli/nightly/2.0/20031112/... http://repo.apache.org/apache/commons-cli/nightly/2.0/20031113/... http://repo.apache.org/apache/commons-cli/nightly/2.0/latest/... - symlink to ../20031113 Build designation naming conventions formal-build-designation and interim-build-designation are defined as: formal-build-designation = release | ... interim-build-designation = interim | nightly | snapshot | ... In other words, tools may use release, interim etc, but may also extend them to define their own. Tool support Tools can unambigously locate
Parsable URI (Re: [proposal] URI Syntax - v0.2)
Noel wrote: You don't have to like the tool, I'm not trying to push the implementation I've never even seen the thing, and you are a priori assuming that I don't like it? No Neol, I'm not that emoition, I meant it dispassionately and without inference, maybe it just read differently. That was more 'one' doesn't have to like it. [I know this list has (in the past) slipped into implementation codebase factions, and I was hoping not to encourage that.] So perhaps I should've writen ... One doesn't have to like this tool/implementation, but the results are valuable at layer 1. It allows you to query what is there, query and capture oldest resources [and do a delete/clean], and download newest, etc. How does it know what OLDEST means? I see that Tim is trying to add some more structure, so maybe he's thinking that we can restrict the URI space so that a restricted notion of version assures an automatable concept of succession. Ruper parses all the attributes of the resources, including the version, and either do (pchar) string comparisons or (in versions case) structured comparisons. Much as there are a few different flavours of a versions they pretty much fall into a parsable pattern. Ruper (through Version) strictly parses the string in a number of different ways (known formats) until one matches. Again, the most important aspect of parsing the URI is knowing what is separated from what, that this pchar is a version, this pchar is a type (or whatever). If values can by groked within that, great, if not, it is still Some find such a tool useful, I'd like to believe that apache users (admins and external users) would find it useful. I don't disagree. I simply said that if you view the repository solution as a layer of specifications, the lowest layer can be a syntax that does not require semantics such as an automatable concept of succession. If we need that, we can add it either by a convention within the URI space, or by other means. We all agree to layers, but I am testing what are the minimum things we'll accept for layter one. I beleive that the repository needs to be 'tooling readable', hence the URI needs to be parse, the other aspects (can an attributed be fully groked) can come later. Again, I need to get to the wiki to put a proposal and pros/cons, I'll try next week. Absolutely. But that may require something more than the URI schema. :-) But if it doesn't have to, should it? I'm trying to determine what we ought will accept at the lowest level. I think clean up is important, I like the other aspects. I agree that much should be done via metadata (e.g. dependencies) however writting potentially shared/conflicting files to a repository is a scary step, and I'd like to see how much we can do with atomic artefacts. I feel we have the potential to win big, and I'd like the ASF Repository to be a step forward towards these goals, not a step backward. I agree. But one layer at a time. :-) Yes, and we are doing layer one -- without metadata, we still need to determine our minimum expectations. If URI is this contentious/involved, I could see metadata as being a long drawn out process one we don't agree on as a whole. Maybe this first layer is the hardest, but I'd like it to be the one giving the most rewards so we aren't all sitting waiting for metadata. regards, Adam
Tooling (was Version Specifier in Re: [proposal] URI Syntax - v0.2)
Its great idea to make Artifact Specifier to be opaque to give way to different languages, but I am not sure about the Version Specifier. Version Specifier can be considered as language independent and allowing different best practices in there would make the repository unordered and could confuse the users. I know of opinions on both sides. Some say we can't dictate, so best practices are the best we can expect even they'll be loose. Others say, we can't achieve conformity unless we try -- and that tools can't process totally unstructured opaque data. I think the questions become (building upon each other): 1) Ought the URI be uniquely (unambiguously) parsable [i.e. things such as your '-' not '/' proposal.] 2) Ought the URI be considered 'metadata' itself structured? 3) Is the goal for the repository to be tools processable [even without metadata]? I'll work something up in the Wiki TODOs section and transfer any pros/cons there. --- One last comment on tools-based verses human-based. We are discussing version in filename so it's version is identifiable once download from the repository. It occurred to me that we are likely assuming a dumb client (a human w/ browser perhaps) in that thought. Nothing is to stop tools downloading an unversioned filename and adding -[version] to the end as it writes it to disk. Since we can likely assume that many humans have bad habits of not doing verification checks of the contents of repositories that they download with a browser, ought we actually weight our cogitations towards tooling that can browse/select/download/verify. Note: I'm not talking implementation, nor any tool, and I'm definitely not saying disallow the human user use case, I'm just saying focus on tooling. IMHO tooling (built upon certain levels of repository consistency) make the repository@ venture more than just a re-organization of file systems, and allow us to scale this and save a lot of wasted human effort. It seems a critical goal to me. Thoughts? regards Adam
Re: URI/URL Syntax -- little nits to be aware of
Why do you want to parse strings which describe versions? So one can look at a repository of artefacts and select the best for the user automatically. Each user has a different view of best, some want latest (nightly/snapshot), some want latest release only. Having a repository with automated tools that require humans to understand the contents kinda breaks the automation... If you want to impose on anyone how they should version their artifacts? There is zero % of chance for that. No, but by being flexible to a given set of versioning schemes on can. Like I said, we are picking up most of Maven's repository w/o imposing anything, and it also works for other common version standards. regards Adam
Manage discussions to a conclusion...
I am trying to move contentious topics to the Wiki so we can reference them individually, and (over time) come to an agreement. How does this layout work for folks? I have started with one discussion topic: http://nagoya.apache.org/wiki/apachewiki.cgi?ASFRepository/ToDo If we can agree upon something like this we can mature it to allow counter proposals, and such. Thoughts? regards, Adam -- Experience Sybase Technology... http://www.try.sybase.com
Re: Processing Versions
[...] and repository possibly won't be limited to project hosted at ASF. This is why we need the Wiki, the e-mail archives are too hard to comsume. It was agreed a while ago that this is for Apache software, we aren't trying to solve this problem for everybody else. We need to stop going back over closed discussions to make simple progress. regards Adam
index.html (was Comments on URI Syntax
http://repo.apache.org/apache/commons-logging/1.0.3/index.html Clearly, this is only useful to users browsing the repository, and therefore makes no sense to include the version information. On index.html, wouldn't we discourage the use of this? Wouldn't we want the HTTP server to do a directory listing, so publishers wouldn't have to update it every time? Also, if we are to allow/support automatic remote publishing, not having to lock/update an index.html would be a bonus. Parsing HTML (simple or fancy) to find links that are children is pretty trivial, so it probably wouldn't matter (to clients who are going to have to do this anyway) but it probably matters to publishers. regards Adam
Disagreements/Agreements
I am almost sure that with WIKI we will have much more chaos. Wiki it's not a perfect techonology for maitaing disussions. -1 for using WIKI for dissusion I didn't mean solely, I think we need a mixture of both. I think e-mail threads are good (when tight) but a thread might end with the passionate folks in disagreement. Those I think we move to the Wiki for voting: http://nagoya.apache.org/wiki/apachewiki.cgi?ASFRepository/ToDo We need to record our votes, some simple pros/cons, and at a suitable point close the issue and move it to /Decisions. +1 for using WIKI for writing down the things which were agreed. We need this most certainly: http://nagoya.apache.org/wiki/apachewiki.cgi?ASFRepository/Decisions I am scouring the mails to this list to look for things I believe are agreed upon decisions. I'll most the non-controversial ones here. [If I make a mistake we can complete discussions on those aspects.] regards Adam
Re: URI Syntax was: Repository
Some artifacts don't like having the full version number. dll for example. I think the DLL name needs to be stable and thus would not have the full version info. For the dll example we can mandate that it has to be put in a versioned zip/tar.gzip If we continue to think 100% generically we'll never hold all the permutations in Wiki, let alone out heads, and we'll fluster ourselves paralysis. It is clear that Java has a set of requirements, a style, that do not fit other languages. We can't do it all. I feel we could/should unashamedly complete our thoughts on Java, then go and recruit some per-language specialists to chime in on their flavour. Maybe we'll have one repository 'class' with per language sub-classes. Let's mature what we can agree upon before we specialize determine what we can't. [If we have a Java repository separate from a C++ one, so be it, but N better than 0. Heck, separate might be easiest management anyway.] Just my two cents... regards, Adam
Re: URL Syntax parts
Things change, that is a fact that I'm just not sure we can generically pre-empt, nor try. Interestingly, Gump deals with a lot of the issues we are discussing here. The main difference is that Gump purely metadata based, and lives only in the 'now'. For changes it support aliasing. http://gump.covalent.com/log/cvsjars.html http://gump.covalent.com/log/bypackage.html BTW: Gump metadata management gets incredibly stale, incredibly quickly, if folks aren't nagged. I'm not proposing going that far w/ metadata, just realizing the similarities in situation... FWIIW: In Gump (and the repo it is producing) group = CVS module, not project. That could be changed. regards Adam - Original Message - From: Noel J. Bergman [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, October 30, 2003 5:37 PM Subject: RE: URL Syntax parts I would prefer to NOT have the TLP represented in the URI if possible. Projects can be promoted. The package name for James was always org.apache.james, and so did not change when the project was promoted. Point being that I'm not sure if the URI should reflect the ASF organization. A project name ought to be unique across the ASF. --- Noel
