Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Of course that content-based-storage system isn't a whole FS. That'll take
us back to where we were with some of the ClearCase modes of operaton where
all IO slowed down.  Or worse -  PVCS Dimensions in regular use. It's just
for read only pieces of our build-centric world.

-ph

On Thu, May 18, 2017 at 6:11 AM, Paul Hammant  wrote:

> You know, I think it would be cool if there were file systems that would
> implement a form of a 'link' as part of a content-based-storage mechanism
> like Git itself.  Meaning if I ran the above git-clone/jar command twice in
> two different directories it would only store the result once (assuming
> idempotence).  Hard and soft links are not it because they need a canonical
> version that the duplicates point to.  What you want is the thing stored
> once, and regardless of the order of deletion, the last deletion means it
> is truly gone.  The Mac's 'alias' feature is closer because it allows you
> to at least move the original/canonical.
>
> Kind of like String.intern(..) in Java.
>
> People are, of course, pushing the file system science in the direction of
> content-based (with dire/filenames a logical overlay).
>
> https://dev.arvados.org/projects/arvados/wiki/Keep
> https://github.com/ipfs/ipfs
> https://en.wikipedia.org/wiki/Content-addressable_storage
>
>
> - paul
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

You know, I think it would be cool if there were file systems that would
implement a form of a 'link' as part of a content-based-storage mechanism
like Git itself.  Meaning if I ran the above git-clone/jar command twice in
two different directories it would only store the result once (assuming
idempotence).  Hard and soft links are not it because they need a canonical
version that the duplicates point to.  What you want is the thing stored
once, and regardless of the order of deletion, the last deletion means it
is truly gone.  The Mac's 'alias' feature is closer because it allows you
to at least move the original/canonical.

Kind of like String.intern(..) in Java.

People are, of course, pushing the file system science in the direction of
content-based (with dire/filenames a logical overlay).

https://dev.arvados.org/projects/arvados/wiki/Keep
https://github.com/ipfs/ipfs
https://en.wikipedia.org/wiki/Content-addressable_storage


- paul

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

>
> I'm not sure I'm not mistaken for the read part: with the current repo
> format,
> one thing that is super easy is that each artifact version is just a path
> that
> you can use simultaneously in one build or in multiple builds
>

e.g.

build #1 using --classpath ~/.m2/repo/tw/xstream-1.4.3.jar
build #2 (concurrent or soon after) using --classpath
~/.m2/repo/tw/xstream-1.4.4.jar
other builds using one of those two (again) or more, concurrently or not.

Got it.


> With the proposed git repos, I don't get how read access to a tag is done
> without changing disk state, so you can access mutliple tags at the same
> time.
>

Without a *fanciful* change to java and javac, the toolchain is stilll
going to need to make
jars as we've always known them.

The toolchain for recreating jars (notwithstanding timestamps on class
files issues previously noted) from the blog entry:

git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
1.4.3
cd mc-xs-classes
rm -rf .git
jar cvfM ../xstream-1.4.3.jar .
cd ..
rm -rf mc-xs-classes

That might be faster for Java and the widely used JGit if it can be done in
memory. It took 3 secs for me, with 2 of that being the jar step.

On a development workstation, that style of use *exclusively* will mean you
can rebuild ~/.m2/repository as we know it today, jar by jar.
Specifically, you don't retain the jar .git folders per repo for things
pulled from from 'central at all.

Other people, optionally, could say then want to host their own .git/
centric cache of things pulled down from 'central.  That could be on a
network mount somehow, or (more realistically) an anemic Artifactory of
Nexus service. That could even be on their own box rather than for the
team, of they are happy to momentarily use *more* hard drive space than the
~/.m2/repository of today.

I think if had a local .git representation of things from 'central, I would
be deleting ~/.m2/repository every night when I wasn't using the machine. I
mean, in lieu of a more sophisticated solution.

Regards,

- Paul


>
>
> This "on read" question happens with "on write": but as you told the idea
> is
> more to consume releases from a central repository, and not on the desktop
> with SNAPSHOTs and so on, the write question is not really a question.
>
> Regards,
>
> Hervé
>
> >
> > - Paul
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Hervé BOUTEMY]

Le mercredi 17 mai 2017, 14:46:13 CEST Paul Hammant a écrit :
> > not to speak about read concurrency when one requires to use multiple
> > versions
> > of a lib. And of course, write concurrency is even harder.
> 
> I'm not following, dude.
:)

I'm not sure I'm not mistaken for the read part: with the current repo format, 
one thing that is super easy is that each artifact version is just a path that 
you can use simultaneously in one build or in multiple builds
With the proposed git repos, I don't get how read access to a tag is done 
without changing disk state, so you can access mutliple tags at the same time.


This "on read" question happens with "on write": but as you told the idea is 
more to consume releases from a central repository, and not on the desktop 
with SNAPSHOTs and so on, the write question is not really a question.

Regards,

Hervé

> 
> - Paul



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

And as I'm comfortable advocating for a few Git-centric alternates, GitHub
could be a hosting platform for jars of classes too.  I think I've
established that it is efficient for storing binary classes.

The https://github.com/paul-hammant/mc-xs-classes/releases page on the
mc-xs-classes I did for the blog entry, has downloads of tags. That's all
the classes you'd want and the manifest in a a zip. The trouble is GitHub
made a root directory which will break Java.

So I raised a feature request - https://cloud.githubusercontent.com/assets/
82182/26177670/9d0cc4b8-3b28-11e7-88e1-7e97f727623c.png

Don't worry, I'm 0 for 10 or so.

- Paul

On Wed, May 17, 2017 at 5:37 PM, Paul Hammant  wrote:

> The Maven-using developer community cares that the dependency downloader
> does its think that the uploader/deploy does too. Some new releases of
> those, and some back releases for the hard breaks (if any) going back in
> time - Maven1, Maven2 and whatever.
>
> Works well enough for Homebrew (https://github.com/homebrew - the core
> repo is at 125MB). Sure there's been teething troubles, and Ruby was the
> wrong choice for the DSL versus Python, but it is great really.
>
>
>
> On Wed, May 17, 2017 at 5:25 PM, Manfred Moser 
> wrote:
>
>> If you would run Central on git like that in a centralized manner you
>> would have to find someone that does that hosting for you and you would
>> have to get buy in from the community to use that - both extremely hard or
>> impossible.
>>
>> And if you dont do that but instead go with the distributed system you
>> end up with the registry model that I think just doesnt really work in the
>> real world.
>>
>> manfred
>>
>> Paul Hammant wrote on 2017-05-17 13:39:
>>
>> > Actually I'm proposing a predictable structure on 'central :
>> >
>> > g...@central.maven.org:
>> > maven2/> >
>> > (one minor fix versus previous description of the git:// location)
>> >
>> > Or for the three separate variant:
>> >
>> > g...@central.maven.org:
>> > maven2/> > g...@central.maven.org:
>> > maven2/> > g...@central.maven.org:
>> > maven2/> >
>> > More likely though (as you mentioned in your opening line) is the way
>> > homebrew works - you point at repos elsewhere, but control poms/shas
>> etc on
>> > 'central.
>> >
>> >
>> >
>> >
>> > On Wed, May 17, 2017 at 3:59 PM, Manfred Moser <
>> manf...@simpligility.com>
>> > wrote:
>> >
>> >> Having worked with repository managers and the implementation for
>> various
>> >> formats on Nexus for years I think such a format is a bit like Bower.
>> It is
>> >> a registry format that in turn points to git repositories that have the
>> >> content.
>> >>
>> >> From a corporate usage and implementation point of view this is a utter
>> >> nightmare since you would have to open up your systems to all those
>> >> different repositories and sites hosting them instead of just one.
>> >>
>> >> You also cant simply make a copy of the content or analyze it in the
>> way
>> >> it manifests as binaries.
>> >>
>> >> I am not sure what you are looking for as benefits but from my point of
>> >> view this is maybe a fun experiment but not something that will ever
>> take
>> >> off..
>> >>
>> >> Manfred
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
>> >> For additional commands, e-mail: dev-h...@maven.apache.org
>> >>
>> >>
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
>> For additional commands, e-mail: dev-h...@maven.apache.org
>>
>>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

The Maven-using developer community cares that the dependency downloader
does its think that the uploader/deploy does too. Some new releases of
those, and some back releases for the hard breaks (if any) going back in
time - Maven1, Maven2 and whatever.

Works well enough for Homebrew (https://github.com/homebrew - the core repo
is at 125MB). Sure there's been teething troubles, and Ruby was the wrong
choice for the DSL versus Python, but it is great really.



On Wed, May 17, 2017 at 5:25 PM, Manfred Moser 
wrote:

> If you would run Central on git like that in a centralized manner you
> would have to find someone that does that hosting for you and you would
> have to get buy in from the community to use that - both extremely hard or
> impossible.
>
> And if you dont do that but instead go with the distributed system you end
> up with the registry model that I think just doesnt really work in the real
> world.
>
> manfred
>
> Paul Hammant wrote on 2017-05-17 13:39:
>
> > Actually I'm proposing a predictable structure on 'central :
> >
> > g...@central.maven.org:
> > maven2/ >
> > (one minor fix versus previous description of the git:// location)
> >
> > Or for the three separate variant:
> >
> > g...@central.maven.org:
> > maven2/ > g...@central.maven.org:
> > maven2/ > g...@central.maven.org:
> > maven2/ >
> > More likely though (as you mentioned in your opening line) is the way
> > homebrew works - you point at repos elsewhere, but control poms/shas etc
> on
> > 'central.
> >
> >
> >
> >
> > On Wed, May 17, 2017 at 3:59 PM, Manfred Moser  >
> > wrote:
> >
> >> Having worked with repository managers and the implementation for
> various
> >> formats on Nexus for years I think such a format is a bit like Bower.
> It is
> >> a registry format that in turn points to git repositories that have the
> >> content.
> >>
> >> From a corporate usage and implementation point of view this is a utter
> >> nightmare since you would have to open up your systems to all those
> >> different repositories and sites hosting them instead of just one.
> >>
> >> You also cant simply make a copy of the content or analyze it in the way
> >> it manifests as binaries.
> >>
> >> I am not sure what you are looking for as benefits but from my point of
> >> view this is maybe a fun experiment but not something that will ever
> take
> >> off..
> >>
> >> Manfred
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> >> For additional commands, e-mail: dev-h...@maven.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Manfred Moser]

If you would run Central on git like that in a centralized manner you would 
have to find someone that does that hosting for you and you would have to get 
buy in from the community to use that - both extremely hard or impossible.

And if you dont do that but instead go with the distributed system you end up 
with the registry model that I think just doesnt really work in the real world.

manfred

Paul Hammant wrote on 2017-05-17 13:39:

> Actually I'm proposing a predictable structure on 'central :
> 
> g...@central.maven.org:
> maven2/ 
> (one minor fix versus previous description of the git:// location)
> 
> Or for the three separate variant:
> 
> g...@central.maven.org:
> maven2/ g...@central.maven.org:
> maven2/ g...@central.maven.org:
> maven2/ 
> More likely though (as you mentioned in your opening line) is the way
> homebrew works - you point at repos elsewhere, but control poms/shas etc on
> 'central.
> 
> 
> 
> 
> On Wed, May 17, 2017 at 3:59 PM, Manfred Moser 
> wrote:
> 
>> Having worked with repository managers and the implementation for various
>> formats on Nexus for years I think such a format is a bit like Bower. It is
>> a registry format that in turn points to git repositories that have the
>> content.
>>
>> From a corporate usage and implementation point of view this is a utter
>> nightmare since you would have to open up your systems to all those
>> different repositories and sites hosting them instead of just one.
>>
>> You also cant simply make a copy of the content or analyze it in the way
>> it manifests as binaries.
>>
>> I am not sure what you are looking for as benefits but from my point of
>> view this is maybe a fun experiment but not something that will ever take
>> off..
>>
>> Manfred
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
>> For additional commands, e-mail: dev-h...@maven.apache.org
>>
>>
> 


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Actually I'm proposing a predictable structure on 'central :

 g...@central.maven.org:
maven2/
wrote:

> Having worked with repository managers and the implementation for various
> formats on Nexus for years I think such a format is a bit like Bower. It is
> a registry format that in turn points to git repositories that have the
> content.
>
> From a corporate usage and implementation point of view this is a utter
> nightmare since you would have to open up your systems to all those
> different repositories and sites hosting them instead of just one.
>
> You also cant simply make a copy of the content or analyze it in the way
> it manifests as binaries.
>
> I am not sure what you are looking for as benefits but from my point of
> view this is maybe a fun experiment but not something that will ever take
> off..
>
> Manfred
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Manfred Moser]

Having worked with repository managers and the implementation for various 
formats on Nexus for years I think such a format is a bit like Bower. It is a 
registry format that in turn points to git repositories that have the content.

>From a corporate usage and implementation point of view this is a utter 
>nightmare since you would have to open up your systems to all those different 
>repositories and sites hosting them instead of just one. 

You also cant simply make a copy of the content or analyze it in the way it 
manifests as binaries. 

I am not sure what you are looking for as benefits but from my point of view 
this is maybe a fun experiment but not something that will ever take off.. 

Manfred

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

I'm "just" unzipping the source, and committing those sources. Sure, I
delete the previous set first, but I merge the *rm* set and *add* set into
one commit with an --amend ->

 # fn is xstream-1.4.3.jar and v is 1.4.3 for example

 git("rm", "-r", "*")

git("commit", "-m", v)
wget(root + url + "/" + v + "/" + fn)
unzip(fn)
rm(fn)
git("add", ".")
git("commit", "-m", v, "--amend")

On Wed, May 17, 2017 at 3:41 PM, Aldrin Leal  wrote:

> thats my point: the golang approach does no magic at all. It simply stores
> the source code and bases it on a convention. Just the files, and thats it.
>
> --
> -- Aldrin Leal,  / http://about.me/aldrinleal
>
> On Wed, May 17, 2017 at 2:38 PM, Paul Hammant  wrote:
>
> > Aldrin, The blog entry I wrote on saturday mulled classes, javadocs and
> > sources -
> > https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
> > (re
> > your "way more than" comment).
> >
> > The GH repo I linked you to earlier has all three in one repo (see the
> > branches drop-down) - https://github.com/paul-hammant/mc-xs-all
> >
> > - Paul
> >
> >
> >
> > On Wed, May 17, 2017 at 2:54 PM, Aldrin Leal  wrote:
> >
> > > I understand the approach is basically general, but maven artifacts are
> > way
> > > more than binary code (there's source and javadoc). I also understand
> its
> > > an interesting option for distribution.
> > >
> > > I really would like to see something close to what "go get" does. If
> not
> > > github and bitbucket (and go get includes git, hg and bzr among scms),
> it
> > > open the URL itself and resolve it into a(by means of HTML metadata).
> > >
> > > Just distributing the source allowing it to easily updates would be
> > > awesome, since it would allow less effort to create and distribute. Of
> > > course, we could have to limit the scope of what to do to avoid abuse.
> > >
> > >
> > > --
> > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > >
> > > On Wed, May 17, 2017 at 1:49 PM, Paul Hammant 
> wrote:
> > >
> > > > There is that, yes. And Git's general upper limits which are subject
> of
> > > "I
> > > > heard of a team that had a corruption at 2GB".  I've field tested Git
> > up
> > > to
> > > > 7GB for a git-svn-clone myself (a team considering saying bye bye to
> > > Svn),
> > > > but wouldn't put that live versus hive off history to a R/O repo, and
> > > start
> > > > over with HEAD of the old becoming the initial commit of the new.
> > > >
> > > > - Paul
> > > >
> > > > On Wed, May 17, 2017 at 2:40 PM, Aldrin Leal 
> > wrote:
> > > >
> > > > > Still, once github gets an outage, our repositories are basically
> > > > > 'left-padded' (taken offline)
> > > > >
> > > > > --
> > > > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > > > >
> > > > > On Wed, May 17, 2017 at 1:35 PM, Paul Hammant 
> > > wrote:
> > > > >
> > > > > > Aldrin - https://github.com/paul-hammant/mc-xs-all - no large
> > files
> > > > > added
> > > > > > to Git. Git makes 70% saving on bytes used ('bare' mode).
> > > > > >
> > > > > > On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal <
> ald...@leal.eng.br>
> > > > > wrote:
> > > > > >
> > > > > > > Just a friendly reminder that git is not optimized for large
> > files
> > > > (for
> > > > > > > this, they made git-lfs). Plus, when you do checkout a git
> repo,
> > > > > there's
> > > > > > no
> > > > > > > binary diffs - so if you've got plenty of releases, you'll be
> > > > wasting a
> > > > > > lot
> > > > > > > of space/time in terms of transmission and storage.
> > > > > > >
> > > > > > > --
> > > > > > > -- Aldrin Leal,  /
> > http://about.me/aldrinleal
> > > > > > >
> > > > > > > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant <
> p...@hammant.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > We can agree to differ on what I'm suggesting and what the
> > impact
> > > > of
> > > > > > that
> > > > > > > > would be :)
> > > > > > > >
> > > > > > > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox <
> bri...@infinity.nu
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Even more than redefining what Central does, you're
> > effectively
> > > > > > > > describing
> > > > > > > > > a new, unofficial java class packaging and distribution
> > > > mechanism.
> > > > > > This
> > > > > > > > > seems like it will violate signatures etc and make tracking
> > of
> > > > what
> > > > > > you
> > > > > > > > > actually have a nightmare.
> > > > > > > > >
> > > > > > > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> > > > > > herve.bout...@free.fr>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > this idea of putting everything in git is funny: not sure
> > > this
> > > > > will
> > > > > > > go
> > > > > > > > > very
> > > > > > > > > > far from this poc, but 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Robert,

>From the blog entry:


*Actually changing Maven Central to do this*
The maven ‘deploy’ workflow and plugin would invisibly do a commit to (or
create of) a dedicated Git repo up on central.

For XStream, a new deployment would not go into
http://central.maven.org/maven2/com/thoughtworks/xstream/xstream/ any more.
Instead they would go into (say) g...@central.maven.org:
maven2/com/thoughtworks/xstream/xstream.git

The maintainer for the group:artifact would not have to do anything
different to exist in the new deploy world, all the heavy lifting is done
in Maven Central and that future version of the deploy plugin. They would
have to upgrade their project to that deploy plugin, of course.

It is not just Maven Central. It would be Artifactory, Nexus, Gradle (etc)
technologies too.


I just put it on Github as that is an easy place to do look-here-see-bro
discussions.

:)

- Paul


On Wed, May 17, 2017 at 3:29 PM, Robert Scholte 
wrote:
>
> On Wed, 17 May 2017 20:41:02 +0200, Paul Hammant  wrote:
>
>> I would agree that it has the potential to be a new repository
>> implementation, Robert.
>> But I am not sure I follow your second sentence.
>
>
> So suppose I want to add xstream-1.4.9 as dependency to my project. How
should Maven know it has to go to
https://github.com/paul-hammant/mc-xs-classes?
> You need some kind of mapping, and with this structure you have to do it
for every git-stored dependency.
> The most straight-forward solution would be to add repositories (which
might fit in POM model 4.0.0), but I cannot imagine users want to do that
for every dependency.
>
> Robert
>
>
>
>> Maybe I do. There is one
>> Git repo per group/artifact. That's true for whether it is the principal
>> artifact you're after or a transitive dep.
>>
>> 1. For https://github.com/paul-hammant/mc-xs-all I have the sources,
>> classes, and javadoc as separate branches in one repo.
>>
>> 2. For https://github.com/paul-hammant/mc-xs-classes and
>> https://github.com/paul-hammant/mc-xs-javadocs and
>> https://github.com/paul-hammant/mc-xs-sources I have three Git repos per
>> group/artifact.
>>
>> #1 and #2 are alternate coices. #1 has poms in their own branch in that
Git
>> repo too - and they too are one-line retrievable from the remote.
>>
>> - Paul
>>
>> On Wed, May 17, 2017 at 1:27 PM, Robert Scholte 
>> wrote:
>>
>>> I consider this as a new repository implementation. But this also
implies
>>> that in your pom, for every dependency you have to add a
repository-entry
>>> as well, right?
>>>
>>> Robert
>>>
>>>
>>> On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal 
>>> wrote:
>>>
>>> Just a friendly reminder that git is not optimized for large files (for

 this, they made git-lfs). Plus, when you do checkout a git repo,
there's
 no
 binary diffs - so if you've got plenty of releases, you'll be wasting a
 lot
 of space/time in terms of transmission and storage.

 --
 -- Aldrin Leal,  / http://about.me/aldrinleal

 On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:

 We can agree to differ on what I'm suggesting and what the impact of
that
>
> would be :)
>
> On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:
>
> > Even more than redefining what Central does, you're effectively
> describing
> > a new, unofficial java class packaging and distribution mechanism.
This
> > seems like it will violate signatures etc and make tracking of what
you
> > actually have a nightmare.
> >
> > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
herve.bout...@free.fr>
> > wrote:
> >
> > > this idea of putting everything in git is funny: not sure this
will
> go
> > very
> > > far from this poc, but let's imagine...
> > >
> > > on classes branch, splitting the jar into individual .class has
IMHO
> a
> > big
> > > drawback: we loose original jar and its signature
> > >
> > > On the other branches, the current poc shows commits for versions
> that
> > are
> > > perfectly linear: if there are multiple branches that are
released in
> > > parallel, the commit won't be so clean. I don't know if this will
> have
> an
> > > impact on compression efficiency.
> > >
> > > Another issue with this idea: during development, with SNAPSHOTs,
the
> git
> > > repo
> > > will be polluted: this idea IMHO could only be valid for releases
> > >
> > > not to speak about read concurrency when one requires to use
multiple
> > > versions
> > > of a lib. And of course, write concurrency is even harder.
> > >
> > >
> > > Definitely, the idea is funny, but I don't see how this could go
very
> far
> > > than
> > > this funny idea (in addition to the complexity for implementing
this
> > > 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Aldrin Leal]

thats my point: the golang approach does no magic at all. It simply stores
the source code and bases it on a convention. Just the files, and thats it.

--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 2:38 PM, Paul Hammant  wrote:

> Aldrin, The blog entry I wrote on saturday mulled classes, javadocs and
> sources -
> https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
> (re
> your "way more than" comment).
>
> The GH repo I linked you to earlier has all three in one repo (see the
> branches drop-down) - https://github.com/paul-hammant/mc-xs-all
>
> - Paul
>
>
>
> On Wed, May 17, 2017 at 2:54 PM, Aldrin Leal  wrote:
>
> > I understand the approach is basically general, but maven artifacts are
> way
> > more than binary code (there's source and javadoc). I also understand its
> > an interesting option for distribution.
> >
> > I really would like to see something close to what "go get" does. If not
> > github and bitbucket (and go get includes git, hg and bzr among scms), it
> > open the URL itself and resolve it into a(by means of HTML metadata).
> >
> > Just distributing the source allowing it to easily updates would be
> > awesome, since it would allow less effort to create and distribute. Of
> > course, we could have to limit the scope of what to do to avoid abuse.
> >
> >
> > --
> > -- Aldrin Leal,  / http://about.me/aldrinleal
> >
> > On Wed, May 17, 2017 at 1:49 PM, Paul Hammant  wrote:
> >
> > > There is that, yes. And Git's general upper limits which are subject of
> > "I
> > > heard of a team that had a corruption at 2GB".  I've field tested Git
> up
> > to
> > > 7GB for a git-svn-clone myself (a team considering saying bye bye to
> > Svn),
> > > but wouldn't put that live versus hive off history to a R/O repo, and
> > start
> > > over with HEAD of the old becoming the initial commit of the new.
> > >
> > > - Paul
> > >
> > > On Wed, May 17, 2017 at 2:40 PM, Aldrin Leal 
> wrote:
> > >
> > > > Still, once github gets an outage, our repositories are basically
> > > > 'left-padded' (taken offline)
> > > >
> > > > --
> > > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > > >
> > > > On Wed, May 17, 2017 at 1:35 PM, Paul Hammant 
> > wrote:
> > > >
> > > > > Aldrin - https://github.com/paul-hammant/mc-xs-all - no large
> files
> > > > added
> > > > > to Git. Git makes 70% saving on bytes used ('bare' mode).
> > > > >
> > > > > On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal 
> > > > wrote:
> > > > >
> > > > > > Just a friendly reminder that git is not optimized for large
> files
> > > (for
> > > > > > this, they made git-lfs). Plus, when you do checkout a git repo,
> > > > there's
> > > > > no
> > > > > > binary diffs - so if you've got plenty of releases, you'll be
> > > wasting a
> > > > > lot
> > > > > > of space/time in terms of transmission and storage.
> > > > > >
> > > > > > --
> > > > > > -- Aldrin Leal,  /
> http://about.me/aldrinleal
> > > > > >
> > > > > > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant 
> > > > wrote:
> > > > > >
> > > > > > > We can agree to differ on what I'm suggesting and what the
> impact
> > > of
> > > > > that
> > > > > > > would be :)
> > > > > > >
> > > > > > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox  >
> > > > wrote:
> > > > > > >
> > > > > > > > Even more than redefining what Central does, you're
> effectively
> > > > > > > describing
> > > > > > > > a new, unofficial java class packaging and distribution
> > > mechanism.
> > > > > This
> > > > > > > > seems like it will violate signatures etc and make tracking
> of
> > > what
> > > > > you
> > > > > > > > actually have a nightmare.
> > > > > > > >
> > > > > > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> > > > > herve.bout...@free.fr>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > this idea of putting everything in git is funny: not sure
> > this
> > > > will
> > > > > > go
> > > > > > > > very
> > > > > > > > > far from this poc, but let's imagine...
> > > > > > > > >
> > > > > > > > > on classes branch, splitting the jar into individual .class
> > has
> > > > > IMHO
> > > > > > a
> > > > > > > > big
> > > > > > > > > drawback: we loose original jar and its signature
> > > > > > > > >
> > > > > > > > > On the other branches, the current poc shows commits for
> > > versions
> > > > > > that
> > > > > > > > are
> > > > > > > > > perfectly linear: if there are multiple branches that are
> > > > released
> > > > > in
> > > > > > > > > parallel, the commit won't be so clean. I don't know if
> this
> > > will
> > > > > > have
> > > > > > > an
> > > > > > > > > impact on compression efficiency.
> > > > > > > > >
> > > > > > > > > Another issue with this idea: during development, with
> > > SNAPSHOTs,
> > > > > the
> > > > 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Aldrin, The blog entry I wrote on saturday mulled classes, javadocs and
sources -
https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/ (re
your "way more than" comment).

The GH repo I linked you to earlier has all three in one repo (see the
branches drop-down) - https://github.com/paul-hammant/mc-xs-all

- Paul



On Wed, May 17, 2017 at 2:54 PM, Aldrin Leal  wrote:

> I understand the approach is basically general, but maven artifacts are way
> more than binary code (there's source and javadoc). I also understand its
> an interesting option for distribution.
>
> I really would like to see something close to what "go get" does. If not
> github and bitbucket (and go get includes git, hg and bzr among scms), it
> open the URL itself and resolve it into a(by means of HTML metadata).
>
> Just distributing the source allowing it to easily updates would be
> awesome, since it would allow less effort to create and distribute. Of
> course, we could have to limit the scope of what to do to avoid abuse.
>
>
> --
> -- Aldrin Leal,  / http://about.me/aldrinleal
>
> On Wed, May 17, 2017 at 1:49 PM, Paul Hammant  wrote:
>
> > There is that, yes. And Git's general upper limits which are subject of
> "I
> > heard of a team that had a corruption at 2GB".  I've field tested Git up
> to
> > 7GB for a git-svn-clone myself (a team considering saying bye bye to
> Svn),
> > but wouldn't put that live versus hive off history to a R/O repo, and
> start
> > over with HEAD of the old becoming the initial commit of the new.
> >
> > - Paul
> >
> > On Wed, May 17, 2017 at 2:40 PM, Aldrin Leal  wrote:
> >
> > > Still, once github gets an outage, our repositories are basically
> > > 'left-padded' (taken offline)
> > >
> > > --
> > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > >
> > > On Wed, May 17, 2017 at 1:35 PM, Paul Hammant 
> wrote:
> > >
> > > > Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files
> > > added
> > > > to Git. Git makes 70% saving on bytes used ('bare' mode).
> > > >
> > > > On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal 
> > > wrote:
> > > >
> > > > > Just a friendly reminder that git is not optimized for large files
> > (for
> > > > > this, they made git-lfs). Plus, when you do checkout a git repo,
> > > there's
> > > > no
> > > > > binary diffs - so if you've got plenty of releases, you'll be
> > wasting a
> > > > lot
> > > > > of space/time in terms of transmission and storage.
> > > > >
> > > > > --
> > > > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > > > >
> > > > > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant 
> > > wrote:
> > > > >
> > > > > > We can agree to differ on what I'm suggesting and what the impact
> > of
> > > > that
> > > > > > would be :)
> > > > > >
> > > > > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox 
> > > wrote:
> > > > > >
> > > > > > > Even more than redefining what Central does, you're effectively
> > > > > > describing
> > > > > > > a new, unofficial java class packaging and distribution
> > mechanism.
> > > > This
> > > > > > > seems like it will violate signatures etc and make tracking of
> > what
> > > > you
> > > > > > > actually have a nightmare.
> > > > > > >
> > > > > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> > > > herve.bout...@free.fr>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > this idea of putting everything in git is funny: not sure
> this
> > > will
> > > > > go
> > > > > > > very
> > > > > > > > far from this poc, but let's imagine...
> > > > > > > >
> > > > > > > > on classes branch, splitting the jar into individual .class
> has
> > > > IMHO
> > > > > a
> > > > > > > big
> > > > > > > > drawback: we loose original jar and its signature
> > > > > > > >
> > > > > > > > On the other branches, the current poc shows commits for
> > versions
> > > > > that
> > > > > > > are
> > > > > > > > perfectly linear: if there are multiple branches that are
> > > released
> > > > in
> > > > > > > > parallel, the commit won't be so clean. I don't know if this
> > will
> > > > > have
> > > > > > an
> > > > > > > > impact on compression efficiency.
> > > > > > > >
> > > > > > > > Another issue with this idea: during development, with
> > SNAPSHOTs,
> > > > the
> > > > > > git
> > > > > > > > repo
> > > > > > > > will be polluted: this idea IMHO could only be valid for
> > releases
> > > > > > > >
> > > > > > > > not to speak about read concurrency when one requires to use
> > > > multiple
> > > > > > > > versions
> > > > > > > > of a lib. And of course, write concurrency is even harder.
> > > > > > > >
> > > > > > > >
> > > > > > > > Definitely, the idea is funny, but I don't see how this could
> > go
> > > > very
> > > > > > far
> > > > > > > > than
> > > > > > > > this funny idea (in addition to the complexity for
> implementing

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Robert Scholte]

On Wed, 17 May 2017 20:41:02 +0200, Paul Hammant  wrote:


I would agree that it has the potential to be a new repository
implementation, Robert.
But I am not sure I follow your second sentence.


So suppose I want to add xstream-1.4.9 as dependency to my project. How  
should Maven know it has to go to  
https://github.com/paul-hammant/mc-xs-classes?
You need some kind of mapping, and with this structure you have to do it  
for every git-stored dependency.
The most straight-forward solution would be to add repositories (which  
might fit in POM model 4.0.0), but I cannot imagine users want to do that  
for every dependency.


Robert



Maybe I do. There is one
Git repo per group/artifact. That's true for whether it is the principal
artifact you're after or a transitive dep.

1. For https://github.com/paul-hammant/mc-xs-all I have the sources,
classes, and javadoc as separate branches in one repo.

2. For https://github.com/paul-hammant/mc-xs-classes and
https://github.com/paul-hammant/mc-xs-javadocs and
https://github.com/paul-hammant/mc-xs-sources I have three Git repos per
group/artifact.

#1 and #2 are alternate coices. #1 has poms in their own branch in that  
Git

repo too - and they too are one-line retrievable from the remote.

- Paul

On Wed, May 17, 2017 at 1:27 PM, Robert Scholte 
wrote:

I consider this as a new repository implementation. But this also  
implies
that in your pom, for every dependency you have to add a  
repository-entry

as well, right?

Robert


On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal 
wrote:

Just a friendly reminder that git is not optimized for large files (for
this, they made git-lfs). Plus, when you do checkout a git repo,  
there's

no
binary diffs - so if you've got plenty of releases, you'll be wasting a
lot
of space/time in terms of transmission and storage.

--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:

We can agree to differ on what I'm suggesting and what the impact of  
that

would be :)

On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:

> Even more than redefining what Central does, you're effectively
describing
> a new, unofficial java class packaging and distribution mechanism.  
This
> seems like it will violate signatures etc and make tracking of what  
you

> actually have a nightmare.
>
> On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY  


> wrote:
>
> > this idea of putting everything in git is funny: not sure this  
will

go
> very
> > far from this poc, but let's imagine...
> >
> > on classes branch, splitting the jar into individual .class has  
IMHO

a
> big
> > drawback: we loose original jar and its signature
> >
> > On the other branches, the current poc shows commits for versions
that
> are
> > perfectly linear: if there are multiple branches that are  
released in

> > parallel, the commit won't be so clean. I don't know if this will
have
an
> > impact on compression efficiency.
> >
> > Another issue with this idea: during development, with SNAPSHOTs,  
the

git
> > repo
> > will be polluted: this idea IMHO could only be valid for releases
> >
> > not to speak about read concurrency when one requires to use  
multiple

> > versions
> > of a lib. And of course, write concurrency is even harder.
> >
> >
> > Definitely, the idea is funny, but I don't see how this could go  
very

far
> > than
> > this funny idea (in addition to the complexity for implementing  
this

> > format in
> > tooling)
> >
> > Regards,
> >
> > Hervé
> >
> > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > One more repo:
> > >
> > > https://github.com/paul-hammant/mc-xs-all/
> > >
> > > One branch for each of classes, javadoc, sources, and poms
> > >
> > > 15 javadoc original versions: 24.1M
> > >
> > > 16 sources original versions: 4.9M
> > >
> > > 27 classes original versions: 8.4M
> > >
> > > Afterwards git work the bare .git folder is: 8.4M
> > >
> > > *77.5% saving on storage*
> > >
> > > Any artifact, *including the poms,* can be pulled down via a  
single

git
> > > command
> > >
> > > git clone https://github.com/paul-hammant/mc-xs-classes --depth  
1

> > --branch
> > > TAGNAME
> > >
> > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
classes-1.1,
> > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> classes-1.2.1,
> > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
classes-1.4.1,
> > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > classes-1.4.6,
> > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> javadoc-1.2.1,
> > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
javadoc-1.4.1,
> > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > javadoc-1.4.6,
> > > javadoc-1.4.7, javadoc-1.4.8, 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Aldrin Leal]

I understand the approach is basically general, but maven artifacts are way
more than binary code (there's source and javadoc). I also understand its
an interesting option for distribution.

I really would like to see something close to what "go get" does. If not
github and bitbucket (and go get includes git, hg and bzr among scms), it
open the URL itself and resolve it into a(by means of HTML metadata).

Just distributing the source allowing it to easily updates would be
awesome, since it would allow less effort to create and distribute. Of
course, we could have to limit the scope of what to do to avoid abuse.


--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 1:49 PM, Paul Hammant  wrote:

> There is that, yes. And Git's general upper limits which are subject of "I
> heard of a team that had a corruption at 2GB".  I've field tested Git up to
> 7GB for a git-svn-clone myself (a team considering saying bye bye to Svn),
> but wouldn't put that live versus hive off history to a R/O repo, and start
> over with HEAD of the old becoming the initial commit of the new.
>
> - Paul
>
> On Wed, May 17, 2017 at 2:40 PM, Aldrin Leal  wrote:
>
> > Still, once github gets an outage, our repositories are basically
> > 'left-padded' (taken offline)
> >
> > --
> > -- Aldrin Leal,  / http://about.me/aldrinleal
> >
> > On Wed, May 17, 2017 at 1:35 PM, Paul Hammant  wrote:
> >
> > > Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files
> > added
> > > to Git. Git makes 70% saving on bytes used ('bare' mode).
> > >
> > > On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal 
> > wrote:
> > >
> > > > Just a friendly reminder that git is not optimized for large files
> (for
> > > > this, they made git-lfs). Plus, when you do checkout a git repo,
> > there's
> > > no
> > > > binary diffs - so if you've got plenty of releases, you'll be
> wasting a
> > > lot
> > > > of space/time in terms of transmission and storage.
> > > >
> > > > --
> > > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > > >
> > > > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant 
> > wrote:
> > > >
> > > > > We can agree to differ on what I'm suggesting and what the impact
> of
> > > that
> > > > > would be :)
> > > > >
> > > > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox 
> > wrote:
> > > > >
> > > > > > Even more than redefining what Central does, you're effectively
> > > > > describing
> > > > > > a new, unofficial java class packaging and distribution
> mechanism.
> > > This
> > > > > > seems like it will violate signatures etc and make tracking of
> what
> > > you
> > > > > > actually have a nightmare.
> > > > > >
> > > > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> > > herve.bout...@free.fr>
> > > > > > wrote:
> > > > > >
> > > > > > > this idea of putting everything in git is funny: not sure this
> > will
> > > > go
> > > > > > very
> > > > > > > far from this poc, but let's imagine...
> > > > > > >
> > > > > > > on classes branch, splitting the jar into individual .class has
> > > IMHO
> > > > a
> > > > > > big
> > > > > > > drawback: we loose original jar and its signature
> > > > > > >
> > > > > > > On the other branches, the current poc shows commits for
> versions
> > > > that
> > > > > > are
> > > > > > > perfectly linear: if there are multiple branches that are
> > released
> > > in
> > > > > > > parallel, the commit won't be so clean. I don't know if this
> will
> > > > have
> > > > > an
> > > > > > > impact on compression efficiency.
> > > > > > >
> > > > > > > Another issue with this idea: during development, with
> SNAPSHOTs,
> > > the
> > > > > git
> > > > > > > repo
> > > > > > > will be polluted: this idea IMHO could only be valid for
> releases
> > > > > > >
> > > > > > > not to speak about read concurrency when one requires to use
> > > multiple
> > > > > > > versions
> > > > > > > of a lib. And of course, write concurrency is even harder.
> > > > > > >
> > > > > > >
> > > > > > > Definitely, the idea is funny, but I don't see how this could
> go
> > > very
> > > > > far
> > > > > > > than
> > > > > > > this funny idea (in addition to the complexity for implementing
> > > this
> > > > > > > format in
> > > > > > > tooling)
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Hervé
> > > > > > >
> > > > > > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > > > > > One more repo:
> > > > > > > >
> > > > > > > > https://github.com/paul-hammant/mc-xs-all/
> > > > > > > >
> > > > > > > > One branch for each of classes, javadoc, sources, and poms
> > > > > > > >
> > > > > > > > 15 javadoc original versions: 24.1M
> > > > > > > >
> > > > > > > > 16 sources original versions: 4.9M
> > > > > > > >
> > > > > > > > 27 classes original versions: 8.4M
> > > > > > > >
> > > > > > > > Afterwards git work the 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

There is that, yes. And Git's general upper limits which are subject of "I
heard of a team that had a corruption at 2GB".  I've field tested Git up to
7GB for a git-svn-clone myself (a team considering saying bye bye to Svn),
but wouldn't put that live versus hive off history to a R/O repo, and start
over with HEAD of the old becoming the initial commit of the new.

- Paul

On Wed, May 17, 2017 at 2:40 PM, Aldrin Leal  wrote:

> Still, once github gets an outage, our repositories are basically
> 'left-padded' (taken offline)
>
> --
> -- Aldrin Leal,  / http://about.me/aldrinleal
>
> On Wed, May 17, 2017 at 1:35 PM, Paul Hammant  wrote:
>
> > Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files
> added
> > to Git. Git makes 70% saving on bytes used ('bare' mode).
> >
> > On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal 
> wrote:
> >
> > > Just a friendly reminder that git is not optimized for large files (for
> > > this, they made git-lfs). Plus, when you do checkout a git repo,
> there's
> > no
> > > binary diffs - so if you've got plenty of releases, you'll be wasting a
> > lot
> > > of space/time in terms of transmission and storage.
> > >
> > > --
> > > -- Aldrin Leal,  / http://about.me/aldrinleal
> > >
> > > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant 
> wrote:
> > >
> > > > We can agree to differ on what I'm suggesting and what the impact of
> > that
> > > > would be :)
> > > >
> > > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox 
> wrote:
> > > >
> > > > > Even more than redefining what Central does, you're effectively
> > > > describing
> > > > > a new, unofficial java class packaging and distribution mechanism.
> > This
> > > > > seems like it will violate signatures etc and make tracking of what
> > you
> > > > > actually have a nightmare.
> > > > >
> > > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> > herve.bout...@free.fr>
> > > > > wrote:
> > > > >
> > > > > > this idea of putting everything in git is funny: not sure this
> will
> > > go
> > > > > very
> > > > > > far from this poc, but let's imagine...
> > > > > >
> > > > > > on classes branch, splitting the jar into individual .class has
> > IMHO
> > > a
> > > > > big
> > > > > > drawback: we loose original jar and its signature
> > > > > >
> > > > > > On the other branches, the current poc shows commits for versions
> > > that
> > > > > are
> > > > > > perfectly linear: if there are multiple branches that are
> released
> > in
> > > > > > parallel, the commit won't be so clean. I don't know if this will
> > > have
> > > > an
> > > > > > impact on compression efficiency.
> > > > > >
> > > > > > Another issue with this idea: during development, with SNAPSHOTs,
> > the
> > > > git
> > > > > > repo
> > > > > > will be polluted: this idea IMHO could only be valid for releases
> > > > > >
> > > > > > not to speak about read concurrency when one requires to use
> > multiple
> > > > > > versions
> > > > > > of a lib. And of course, write concurrency is even harder.
> > > > > >
> > > > > >
> > > > > > Definitely, the idea is funny, but I don't see how this could go
> > very
> > > > far
> > > > > > than
> > > > > > this funny idea (in addition to the complexity for implementing
> > this
> > > > > > format in
> > > > > > tooling)
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Hervé
> > > > > >
> > > > > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > > > > One more repo:
> > > > > > >
> > > > > > > https://github.com/paul-hammant/mc-xs-all/
> > > > > > >
> > > > > > > One branch for each of classes, javadoc, sources, and poms
> > > > > > >
> > > > > > > 15 javadoc original versions: 24.1M
> > > > > > >
> > > > > > > 16 sources original versions: 4.9M
> > > > > > >
> > > > > > > 27 classes original versions: 8.4M
> > > > > > >
> > > > > > > Afterwards git work the bare .git folder is: 8.4M
> > > > > > >
> > > > > > > *77.5% saving on storage*
> > > > > > >
> > > > > > > Any artifact, *including the poms,* can be pulled down via a
> > single
> > > > git
> > > > > > > command
> > > > > > >
> > > > > > > git clone https://github.com/paul-hammant/mc-xs-classes
> --depth
> > 1
> > > > > > --branch
> > > > > > > TAGNAME
> > > > > > >
> > > > > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3,
> classes-0.5,
> > > > > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
> > > classes-1.1,
> > > > > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> > > > > classes-1.2.1,
> > > > > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
> > > > classes-1.4.1,
> > > > > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > > > > > classes-1.4.6,
> > > > > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> > > > > javadoc-1.2.1,
> > > > > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
> > > > javadoc-1.4.1,

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Hervé,

on classes branch, splitting the jar into individual .class has IMHO a big
> drawback: we loose original jar and its signature
>

Agree. Git doesn't care about timestamps for classes in jars. Java doesn't
either, but SHA1 (etc) of the jar does.

Thus - the next iteration will reproduce even the timestamps of a resulting
jar.


>
> On the other branches, the current poc shows commits for versions that are
> perfectly linear: if there are multiple branches that are released in
> parallel, the commit won't be so clean. I don't know if this will have an
> impact on compression efficiency.
>

They can go in any random order (I tested) and Git achieves the same over
all saving.


>
> Another issue with this idea: during development, with SNAPSHOTs, the git
> repo
> will be polluted: this idea IMHO could only be valid for releases
>

That's true. I'm really only focussed on the bring-down-from-maven-central
cycle. Obviously I need an answer for the on-workstation workflow which
inserts into ~/.m2/repository/ too.


>
> not to speak about read concurrency when one requires to use multiple
> versions
> of a lib. And of course, write concurrency is even harder.
>

I'm not following, dude.

- Paul

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Aldrin Leal]

Still, once github gets an outage, our repositories are basically
'left-padded' (taken offline)

--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 1:35 PM, Paul Hammant  wrote:

> Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files added
> to Git. Git makes 70% saving on bytes used ('bare' mode).
>
> On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal  wrote:
>
> > Just a friendly reminder that git is not optimized for large files (for
> > this, they made git-lfs). Plus, when you do checkout a git repo, there's
> no
> > binary diffs - so if you've got plenty of releases, you'll be wasting a
> lot
> > of space/time in terms of transmission and storage.
> >
> > --
> > -- Aldrin Leal,  / http://about.me/aldrinleal
> >
> > On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:
> >
> > > We can agree to differ on what I'm suggesting and what the impact of
> that
> > > would be :)
> > >
> > > On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:
> > >
> > > > Even more than redefining what Central does, you're effectively
> > > describing
> > > > a new, unofficial java class packaging and distribution mechanism.
> This
> > > > seems like it will violate signatures etc and make tracking of what
> you
> > > > actually have a nightmare.
> > > >
> > > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY <
> herve.bout...@free.fr>
> > > > wrote:
> > > >
> > > > > this idea of putting everything in git is funny: not sure this will
> > go
> > > > very
> > > > > far from this poc, but let's imagine...
> > > > >
> > > > > on classes branch, splitting the jar into individual .class has
> IMHO
> > a
> > > > big
> > > > > drawback: we loose original jar and its signature
> > > > >
> > > > > On the other branches, the current poc shows commits for versions
> > that
> > > > are
> > > > > perfectly linear: if there are multiple branches that are released
> in
> > > > > parallel, the commit won't be so clean. I don't know if this will
> > have
> > > an
> > > > > impact on compression efficiency.
> > > > >
> > > > > Another issue with this idea: during development, with SNAPSHOTs,
> the
> > > git
> > > > > repo
> > > > > will be polluted: this idea IMHO could only be valid for releases
> > > > >
> > > > > not to speak about read concurrency when one requires to use
> multiple
> > > > > versions
> > > > > of a lib. And of course, write concurrency is even harder.
> > > > >
> > > > >
> > > > > Definitely, the idea is funny, but I don't see how this could go
> very
> > > far
> > > > > than
> > > > > this funny idea (in addition to the complexity for implementing
> this
> > > > > format in
> > > > > tooling)
> > > > >
> > > > > Regards,
> > > > >
> > > > > Hervé
> > > > >
> > > > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > > > One more repo:
> > > > > >
> > > > > > https://github.com/paul-hammant/mc-xs-all/
> > > > > >
> > > > > > One branch for each of classes, javadoc, sources, and poms
> > > > > >
> > > > > > 15 javadoc original versions: 24.1M
> > > > > >
> > > > > > 16 sources original versions: 4.9M
> > > > > >
> > > > > > 27 classes original versions: 8.4M
> > > > > >
> > > > > > Afterwards git work the bare .git folder is: 8.4M
> > > > > >
> > > > > > *77.5% saving on storage*
> > > > > >
> > > > > > Any artifact, *including the poms,* can be pulled down via a
> single
> > > git
> > > > > > command
> > > > > >
> > > > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth
> 1
> > > > > --branch
> > > > > > TAGNAME
> > > > > >
> > > > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
> > classes-1.1,
> > > > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> > > > classes-1.2.1,
> > > > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
> > > classes-1.4.1,
> > > > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > > > > classes-1.4.6,
> > > > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> > > > javadoc-1.2.1,
> > > > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
> > > javadoc-1.4.1,
> > > > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > > > > javadoc-1.4.6,
> > > > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > > > > pom-1.2.2,
> > > > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> > > > pom-1.4.4,
> > > > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,
> > sources-1.1.3,
> > > > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
> > > sources-1.3.1,
> > > > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> > > > sources-1.4.4,
> > > > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> > > > sources-1.4.9
> > > > > >
> > > > > >  - Paul
> > > > >
> > > > >
> > > > >
> > > > > 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

I would agree that it has the potential to be a new repository
implementation, Robert.
But I am not sure I follow your second sentence.  Maybe I do. There is one
Git repo per group/artifact. That's true for whether it is the principal
artifact you're after or a transitive dep.

1. For https://github.com/paul-hammant/mc-xs-all I have the sources,
classes, and javadoc as separate branches in one repo.

2. For https://github.com/paul-hammant/mc-xs-classes and
https://github.com/paul-hammant/mc-xs-javadocs and
https://github.com/paul-hammant/mc-xs-sources I have three Git repos per
group/artifact.

#1 and #2 are alternate coices. #1 has poms in their own branch in that Git
repo too - and they too are one-line retrievable from the remote.

- Paul

On Wed, May 17, 2017 at 1:27 PM, Robert Scholte 
wrote:

> I consider this as a new repository implementation. But this also implies
> that in your pom, for every dependency you have to add a repository-entry
> as well, right?
>
> Robert
>
>
> On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal 
> wrote:
>
> Just a friendly reminder that git is not optimized for large files (for
>> this, they made git-lfs). Plus, when you do checkout a git repo, there's
>> no
>> binary diffs - so if you've got plenty of releases, you'll be wasting a
>> lot
>> of space/time in terms of transmission and storage.
>>
>> --
>> -- Aldrin Leal,  / http://about.me/aldrinleal
>>
>> On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:
>>
>> We can agree to differ on what I'm suggesting and what the impact of that
>>> would be :)
>>>
>>> On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:
>>>
>>> > Even more than redefining what Central does, you're effectively
>>> describing
>>> > a new, unofficial java class packaging and distribution mechanism. This
>>> > seems like it will violate signatures etc and make tracking of what you
>>> > actually have a nightmare.
>>> >
>>> > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
>>> > wrote:
>>> >
>>> > > this idea of putting everything in git is funny: not sure this will
>>> go
>>> > very
>>> > > far from this poc, but let's imagine...
>>> > >
>>> > > on classes branch, splitting the jar into individual .class has IMHO
>>> a
>>> > big
>>> > > drawback: we loose original jar and its signature
>>> > >
>>> > > On the other branches, the current poc shows commits for versions
>>> that
>>> > are
>>> > > perfectly linear: if there are multiple branches that are released in
>>> > > parallel, the commit won't be so clean. I don't know if this will
>>> have
>>> an
>>> > > impact on compression efficiency.
>>> > >
>>> > > Another issue with this idea: during development, with SNAPSHOTs, the
>>> git
>>> > > repo
>>> > > will be polluted: this idea IMHO could only be valid for releases
>>> > >
>>> > > not to speak about read concurrency when one requires to use multiple
>>> > > versions
>>> > > of a lib. And of course, write concurrency is even harder.
>>> > >
>>> > >
>>> > > Definitely, the idea is funny, but I don't see how this could go very
>>> far
>>> > > than
>>> > > this funny idea (in addition to the complexity for implementing this
>>> > > format in
>>> > > tooling)
>>> > >
>>> > > Regards,
>>> > >
>>> > > Hervé
>>> > >
>>> > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
>>> > > > One more repo:
>>> > > >
>>> > > > https://github.com/paul-hammant/mc-xs-all/
>>> > > >
>>> > > > One branch for each of classes, javadoc, sources, and poms
>>> > > >
>>> > > > 15 javadoc original versions: 24.1M
>>> > > >
>>> > > > 16 sources original versions: 4.9M
>>> > > >
>>> > > > 27 classes original versions: 8.4M
>>> > > >
>>> > > > Afterwards git work the bare .git folder is: 8.4M
>>> > > >
>>> > > > *77.5% saving on storage*
>>> > > >
>>> > > > Any artifact, *including the poms,* can be pulled down via a single
>>> git
>>> > > > command
>>> > > >
>>> > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
>>> > > --branch
>>> > > > TAGNAME
>>> > > >
>>> > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
>>> > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
>>> classes-1.1,
>>> > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
>>> > classes-1.2.1,
>>> > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
>>> classes-1.4.1,
>>> > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
>>> > > classes-1.4.6,
>>> > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
>>> > javadoc-1.2.1,
>>> > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
>>> javadoc-1.4.1,
>>> > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
>>> > > javadoc-1.4.6,
>>> > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
>>> > > pom-1.2.2,
>>> > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
>>> > pom-1.4.4,
>>> > > > 

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Aldrin - https://github.com/paul-hammant/mc-xs-all - no large files added
to Git. Git makes 70% saving on bytes used ('bare' mode).

On Wed, May 17, 2017 at 11:10 AM, Aldrin Leal  wrote:

> Just a friendly reminder that git is not optimized for large files (for
> this, they made git-lfs). Plus, when you do checkout a git repo, there's no
> binary diffs - so if you've got plenty of releases, you'll be wasting a lot
> of space/time in terms of transmission and storage.
>
> --
> -- Aldrin Leal,  / http://about.me/aldrinleal
>
> On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:
>
> > We can agree to differ on what I'm suggesting and what the impact of that
> > would be :)
> >
> > On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:
> >
> > > Even more than redefining what Central does, you're effectively
> > describing
> > > a new, unofficial java class packaging and distribution mechanism. This
> > > seems like it will violate signatures etc and make tracking of what you
> > > actually have a nightmare.
> > >
> > > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
> > > wrote:
> > >
> > > > this idea of putting everything in git is funny: not sure this will
> go
> > > very
> > > > far from this poc, but let's imagine...
> > > >
> > > > on classes branch, splitting the jar into individual .class has IMHO
> a
> > > big
> > > > drawback: we loose original jar and its signature
> > > >
> > > > On the other branches, the current poc shows commits for versions
> that
> > > are
> > > > perfectly linear: if there are multiple branches that are released in
> > > > parallel, the commit won't be so clean. I don't know if this will
> have
> > an
> > > > impact on compression efficiency.
> > > >
> > > > Another issue with this idea: during development, with SNAPSHOTs, the
> > git
> > > > repo
> > > > will be polluted: this idea IMHO could only be valid for releases
> > > >
> > > > not to speak about read concurrency when one requires to use multiple
> > > > versions
> > > > of a lib. And of course, write concurrency is even harder.
> > > >
> > > >
> > > > Definitely, the idea is funny, but I don't see how this could go very
> > far
> > > > than
> > > > this funny idea (in addition to the complexity for implementing this
> > > > format in
> > > > tooling)
> > > >
> > > > Regards,
> > > >
> > > > Hervé
> > > >
> > > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > > One more repo:
> > > > >
> > > > > https://github.com/paul-hammant/mc-xs-all/
> > > > >
> > > > > One branch for each of classes, javadoc, sources, and poms
> > > > >
> > > > > 15 javadoc original versions: 24.1M
> > > > >
> > > > > 16 sources original versions: 4.9M
> > > > >
> > > > > 27 classes original versions: 8.4M
> > > > >
> > > > > Afterwards git work the bare .git folder is: 8.4M
> > > > >
> > > > > *77.5% saving on storage*
> > > > >
> > > > > Any artifact, *including the poms,* can be pulled down via a single
> > git
> > > > > command
> > > > >
> > > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> > > > --branch
> > > > > TAGNAME
> > > > >
> > > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,
> classes-1.1,
> > > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> > > classes-1.2.1,
> > > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
> > classes-1.4.1,
> > > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > > > classes-1.4.6,
> > > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> > > javadoc-1.2.1,
> > > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
> > javadoc-1.4.1,
> > > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > > > javadoc-1.4.6,
> > > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > > > pom-1.2.2,
> > > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> > > pom-1.4.4,
> > > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,
> sources-1.1.3,
> > > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
> > sources-1.3.1,
> > > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> > > sources-1.4.4,
> > > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> > > sources-1.4.9
> > > > >
> > > > >  - Paul
> > > >
> > > >
> > > >
> > > > 
> -
> > > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > > > For additional commands, e-mail: dev-h...@maven.apache.org
> > > >
> > > >
> > >
> >
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Robert Scholte]

I consider this as a new repository implementation. But this also implies  
that in your pom, for every dependency you have to add a repository-entry  
as well, right?


Robert

On Wed, 17 May 2017 17:10:49 +0200, Aldrin Leal  wrote:


Just a friendly reminder that git is not optimized for large files (for
this, they made git-lfs). Plus, when you do checkout a git repo, there's  
no
binary diffs - so if you've got plenty of releases, you'll be wasting a  
lot

of space/time in terms of transmission and storage.

--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:

We can agree to differ on what I'm suggesting and what the impact of  
that

would be :)

On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:

> Even more than redefining what Central does, you're effectively
describing
> a new, unofficial java class packaging and distribution mechanism.  
This
> seems like it will violate signatures etc and make tracking of what  
you

> actually have a nightmare.
>
> On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
> wrote:
>
> > this idea of putting everything in git is funny: not sure this will  
go

> very
> > far from this poc, but let's imagine...
> >
> > on classes branch, splitting the jar into individual .class has  
IMHO a

> big
> > drawback: we loose original jar and its signature
> >
> > On the other branches, the current poc shows commits for versions  
that

> are
> > perfectly linear: if there are multiple branches that are released  
in
> > parallel, the commit won't be so clean. I don't know if this will  
have

an
> > impact on compression efficiency.
> >
> > Another issue with this idea: during development, with SNAPSHOTs,  
the

git
> > repo
> > will be polluted: this idea IMHO could only be valid for releases
> >
> > not to speak about read concurrency when one requires to use  
multiple

> > versions
> > of a lib. And of course, write concurrency is even harder.
> >
> >
> > Definitely, the idea is funny, but I don't see how this could go  
very

far
> > than
> > this funny idea (in addition to the complexity for implementing this
> > format in
> > tooling)
> >
> > Regards,
> >
> > Hervé
> >
> > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > One more repo:
> > >
> > > https://github.com/paul-hammant/mc-xs-all/
> > >
> > > One branch for each of classes, javadoc, sources, and poms
> > >
> > > 15 javadoc original versions: 24.1M
> > >
> > > 16 sources original versions: 4.9M
> > >
> > > 27 classes original versions: 8.4M
> > >
> > > Afterwards git work the bare .git folder is: 8.4M
> > >
> > > *77.5% saving on storage*
> > >
> > > Any artifact, *including the poms,* can be pulled down via a  
single

git
> > > command
> > >
> > > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> > --branch
> > > TAGNAME
> > >
> > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2,  
classes-1.1,

> > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> classes-1.2.1,
> > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
classes-1.4.1,
> > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > classes-1.4.6,
> > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> javadoc-1.2.1,
> > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
javadoc-1.4.1,
> > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > javadoc-1.4.6,
> > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > pom-1.2.2,
> > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> pom-1.4.4,
> > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9,  
sources-1.1.3,

> > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
sources-1.3.1,
> > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> sources-1.4.4,
> > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> sources-1.4.9
> > >
> > >  - Paul
> >
> >
> >
> >  
-

> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > For additional commands, e-mail: dev-h...@maven.apache.org
> >
> >
>


-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Aldrin Leal]

Just a friendly reminder that git is not optimized for large files (for
this, they made git-lfs). Plus, when you do checkout a git repo, there's no
binary diffs - so if you've got plenty of releases, you'll be wasting a lot
of space/time in terms of transmission and storage.

--
-- Aldrin Leal,  / http://about.me/aldrinleal

On Wed, May 17, 2017 at 9:37 AM, Paul Hammant  wrote:

> We can agree to differ on what I'm suggesting and what the impact of that
> would be :)
>
> On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:
>
> > Even more than redefining what Central does, you're effectively
> describing
> > a new, unofficial java class packaging and distribution mechanism. This
> > seems like it will violate signatures etc and make tracking of what you
> > actually have a nightmare.
> >
> > On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
> > wrote:
> >
> > > this idea of putting everything in git is funny: not sure this will go
> > very
> > > far from this poc, but let's imagine...
> > >
> > > on classes branch, splitting the jar into individual .class has IMHO a
> > big
> > > drawback: we loose original jar and its signature
> > >
> > > On the other branches, the current poc shows commits for versions that
> > are
> > > perfectly linear: if there are multiple branches that are released in
> > > parallel, the commit won't be so clean. I don't know if this will have
> an
> > > impact on compression efficiency.
> > >
> > > Another issue with this idea: during development, with SNAPSHOTs, the
> git
> > > repo
> > > will be polluted: this idea IMHO could only be valid for releases
> > >
> > > not to speak about read concurrency when one requires to use multiple
> > > versions
> > > of a lib. And of course, write concurrency is even harder.
> > >
> > >
> > > Definitely, the idea is funny, but I don't see how this could go very
> far
> > > than
> > > this funny idea (in addition to the complexity for implementing this
> > > format in
> > > tooling)
> > >
> > > Regards,
> > >
> > > Hervé
> > >
> > > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > > One more repo:
> > > >
> > > > https://github.com/paul-hammant/mc-xs-all/
> > > >
> > > > One branch for each of classes, javadoc, sources, and poms
> > > >
> > > > 15 javadoc original versions: 24.1M
> > > >
> > > > 16 sources original versions: 4.9M
> > > >
> > > > 27 classes original versions: 8.4M
> > > >
> > > > Afterwards git work the bare .git folder is: 8.4M
> > > >
> > > > *77.5% saving on storage*
> > > >
> > > > Any artifact, *including the poms,* can be pulled down via a single
> git
> > > > command
> > > >
> > > > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> > > --branch
> > > > TAGNAME
> > > >
> > > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> > > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> > classes-1.2.1,
> > > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4,
> classes-1.4.1,
> > > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > > classes-1.4.6,
> > > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> > javadoc-1.2.1,
> > > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4,
> javadoc-1.4.1,
> > > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > > javadoc-1.4.6,
> > > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > > pom-1.2.2,
> > > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> > pom-1.4.4,
> > > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> > > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3,
> sources-1.3.1,
> > > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> > sources-1.4.4,
> > > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> > sources-1.4.9
> > > >
> > > >  - Paul
> > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > > For additional commands, e-mail: dev-h...@maven.apache.org
> > >
> > >
> >
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

We can agree to differ on what I'm suggesting and what the impact of that
would be :)

On Wed, May 17, 2017 at 8:59 AM, Brian Fox  wrote:

> Even more than redefining what Central does, you're effectively describing
> a new, unofficial java class packaging and distribution mechanism. This
> seems like it will violate signatures etc and make tracking of what you
> actually have a nightmare.
>
> On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
> wrote:
>
> > this idea of putting everything in git is funny: not sure this will go
> very
> > far from this poc, but let's imagine...
> >
> > on classes branch, splitting the jar into individual .class has IMHO a
> big
> > drawback: we loose original jar and its signature
> >
> > On the other branches, the current poc shows commits for versions that
> are
> > perfectly linear: if there are multiple branches that are released in
> > parallel, the commit won't be so clean. I don't know if this will have an
> > impact on compression efficiency.
> >
> > Another issue with this idea: during development, with SNAPSHOTs, the git
> > repo
> > will be polluted: this idea IMHO could only be valid for releases
> >
> > not to speak about read concurrency when one requires to use multiple
> > versions
> > of a lib. And of course, write concurrency is even harder.
> >
> >
> > Definitely, the idea is funny, but I don't see how this could go very far
> > than
> > this funny idea (in addition to the complexity for implementing this
> > format in
> > tooling)
> >
> > Regards,
> >
> > Hervé
> >
> > Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > > One more repo:
> > >
> > > https://github.com/paul-hammant/mc-xs-all/
> > >
> > > One branch for each of classes, javadoc, sources, and poms
> > >
> > > 15 javadoc original versions: 24.1M
> > >
> > > 16 sources original versions: 4.9M
> > >
> > > 27 classes original versions: 8.4M
> > >
> > > Afterwards git work the bare .git folder is: 8.4M
> > >
> > > *77.5% saving on storage*
> > >
> > > Any artifact, *including the poms,* can be pulled down via a single git
> > > command
> > >
> > > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> > --branch
> > > TAGNAME
> > >
> > > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> > > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2,
> classes-1.2.1,
> > > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
> > > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> > classes-1.4.6,
> > > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2,
> javadoc-1.2.1,
> > > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
> > > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> > javadoc-1.4.6,
> > > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> > pom-1.2.2,
> > > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3,
> pom-1.4.4,
> > > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> > > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
> > > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3,
> sources-1.4.4,
> > > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8,
> sources-1.4.9
> > >
> > >  - Paul
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> > For additional commands, e-mail: dev-h...@maven.apache.org
> >
> >
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Brian Fox]

Even more than redefining what Central does, you're effectively describing
a new, unofficial java class packaging and distribution mechanism. This
seems like it will violate signatures etc and make tracking of what you
actually have a nightmare.

On Tue, May 16, 2017 at 5:55 PM, Hervé BOUTEMY 
wrote:

> this idea of putting everything in git is funny: not sure this will go very
> far from this poc, but let's imagine...
>
> on classes branch, splitting the jar into individual .class has IMHO a big
> drawback: we loose original jar and its signature
>
> On the other branches, the current poc shows commits for versions that are
> perfectly linear: if there are multiple branches that are released in
> parallel, the commit won't be so clean. I don't know if this will have an
> impact on compression efficiency.
>
> Another issue with this idea: during development, with SNAPSHOTs, the git
> repo
> will be polluted: this idea IMHO could only be valid for releases
>
> not to speak about read concurrency when one requires to use multiple
> versions
> of a lib. And of course, write concurrency is even harder.
>
>
> Definitely, the idea is funny, but I don't see how this could go very far
> than
> this funny idea (in addition to the complexity for implementing this
> format in
> tooling)
>
> Regards,
>
> Hervé
>
> Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> > One more repo:
> >
> > https://github.com/paul-hammant/mc-xs-all/
> >
> > One branch for each of classes, javadoc, sources, and poms
> >
> > 15 javadoc original versions: 24.1M
> >
> > 16 sources original versions: 4.9M
> >
> > 27 classes original versions: 8.4M
> >
> > Afterwards git work the bare .git folder is: 8.4M
> >
> > *77.5% saving on storage*
> >
> > Any artifact, *including the poms,* can be pulled down via a single git
> > command
> >
> > git clone https://github.com/paul-hammant/mc-xs-classes --depth 1
> --branch
> > TAGNAME
> >
> > 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> > classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> > classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2, classes-1.2.1,
> > classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
> > classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5,
> classes-1.4.6,
> > classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2, javadoc-1.2.1,
> > javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
> > javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5,
> javadoc-1.4.6,
> > javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1,
> pom-1.2.2,
> > pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3, pom-1.4.4,
> > pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> > sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
> > sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3, sources-1.4.4,
> > sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8, sources-1.4.9
> >
> >  - Paul
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> For additional commands, e-mail: dev-h...@maven.apache.org
>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Hervé BOUTEMY]

this idea of putting everything in git is funny: not sure this will go very 
far from this poc, but let's imagine...

on classes branch, splitting the jar into individual .class has IMHO a big 
drawback: we loose original jar and its signature

On the other branches, the current poc shows commits for versions that are 
perfectly linear: if there are multiple branches that are released in 
parallel, the commit won't be so clean. I don't know if this will have an 
impact on compression efficiency.

Another issue with this idea: during development, with SNAPSHOTs, the git repo 
will be polluted: this idea IMHO could only be valid for releases

not to speak about read concurrency when one requires to use multiple versions 
of a lib. And of course, write concurrency is even harder.


Definitely, the idea is funny, but I don't see how this could go very far than 
this funny idea (in addition to the complexity for implementing this format in 
tooling)

Regards,

Hervé

Le lundi 15 mai 2017, 21:45:00 CEST Paul Hammant a écrit :
> One more repo:
> 
> https://github.com/paul-hammant/mc-xs-all/
> 
> One branch for each of classes, javadoc, sources, and poms
> 
> 15 javadoc original versions: 24.1M
> 
> 16 sources original versions: 4.9M
> 
> 27 classes original versions: 8.4M
> 
> Afterwards git work the bare .git folder is: 8.4M
> 
> *77.5% saving on storage*
> 
> Any artifact, *including the poms,* can be pulled down via a single git
> command
> 
> git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
> TAGNAME
> 
> 74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
> classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
> classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2, classes-1.2.1,
> classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
> classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5, classes-1.4.6,
> classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2, javadoc-1.2.1,
> javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
> javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5, javadoc-1.4.6,
> javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1, pom-1.2.2,
> pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3, pom-1.4.4,
> pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
> sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
> sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3, sources-1.4.4,
> sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8, sources-1.4.9
> 
>  - Paul



-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

One more repo:

https://github.com/paul-hammant/mc-xs-all/

One branch for each of classes, javadoc, sources, and poms

15 javadoc original versions: 24.1M

16 sources original versions: 4.9M

27 classes original versions: 8.4M

Afterwards git work the bare .git folder is: 8.4M

*77.5% saving on storage*

Any artifact, *including the poms,* can be pulled down via a single git
command

git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
TAGNAME

74 TAGNAMEs: classes-0.1, classes-0.2, classes-0.3, classes-0.5,
classes-0.6, classes-1.0, classes-1.0.1, classes-1.0.2, classes-1.1,
classes-1.1.1, classes-1.1.2, classes-1.1.3, classes-1.2, classes-1.2.1,
classes-1.2.2, classes-1.3, classes-1.3.1, classes-1.4, classes-1.4.1,
classes-1.4.2, classes-1.4.3, classes-1.4.4, classes-1.4.5, classes-1.4.6,
classes-1.4.7, classes-1.4.8, classes-1.4.9, javadoc-1.2, javadoc-1.2.1,
javadoc-1.2.2, javadoc-1.3, javadoc-1.3.1, javadoc-1.4, javadoc-1.4.1,
javadoc-1.4.2, javadoc-1.4.3, javadoc-1.4.4, javadoc-1.4.5, javadoc-1.4.6,
javadoc-1.4.7, javadoc-1.4.8, javadoc-1.4.9, pom-1.2, pom-1.2.1, pom-1.2.2,
pom-1.3, pom-1.3.1, pom-1.4, pom-1.4.1, pom-1.4.2, pom-1.4.3, pom-1.4.4,
pom-1.4.5, pom-1.4.6, pom-1.4.7, pom-1.4.8, pom-1.4.9, sources-1.1.3,
sources-1.2, sources-1.2.1, sources-1.2.2, sources-1.3, sources-1.3.1,
sources-1.4, sources-1.4.1, sources-1.4.2, sources-1.4.3, sources-1.4.4,
sources-1.4.5, sources-1.4.6, sources-1.4.7, sources-1.4.8, sources-1.4.9

 - Paul

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

There's one repo per thing that is versioned. All these are separate repos

g: com/thoughtworks/paranamer a: paranamer type: javadoc  is one repo
g: com/thoughtworks/paranamer a: paranamer type: sources  is one repo
g: com/thoughtworks/paranamer a: paranamer type: classes  is one repo
g: com/thoughtworks/xstream a: xstream type: javadoc  is one repo
g: com/thoughtworks/xstream a: xstream type: sources  is one repo
g: com/thoughtworks/xstream a: xstream type: classes  is one repo

Or you could do it in less repos, I'm sure, and Git will still make a
compression saving.

And despite what you said, Git manages to find some commonality in *binary
classes *to make a saving. XStream's regular JARs:

Total size for *27* original versions 8.4MB
The .git folder afterwards 2.4MB
Raw/bare storage space saving 71.4%
Your point about the git:// protocol being being more chatty that http://
is true. That's probably more CPU on the server side. It is definitely more
time. See here ...

Git takes 1s to do:

git clone https://github.com/paul-hammant/mc-xs-classes --depth 1 --branch
1.4.3 --bare


Wget takes 0.5s to do:

wget http://central.maven.org/maven2/com/thoughtworks/
xstream/xstream/1.4.3/xstream-1.4.3.jar


On pom files - they are within the base Git repos, yes.  You can't get them
from the server to the client in one Git operation, without bringing at
least a whole tag. Maybe there could be a tag just for the POM file to make
it addressable remotely, and checkoutable locally. I'll test that later
today.

- Paul


On Mon, May 15, 2017 at 4:14 AM, Stian Soiland-Reyes 
wrote:



> Interesting idea! All for experimenting with git!
>
> You did not take into account the protocol differences, fetching a single
> 8MB  jar from Maven Central is easy, while fetching 6 MB via multiple git
> HTTP resources (even assuming git packs there will be multiple http calls)
> is probably more expensive, particularly as a typical project may have 50
> dependencies.
>
> Would each version be a new git repo? The binary class files are not really
> suitable for "diffing" and would pretty sure be changed every release (even
> if the SRC is the exact same).
>
> It is possible in JAR to use jar200 packing, which can significantly reduce
> the compressed size. Perhsps that is relevant here. See
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/pack200.html
>
> Are the pom files within these git repositories or can they be prefetched
> separately like today to speed up transitive dependency loading?
>
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Stian Soiland-Reyes]

Interesting idea! All for experimenting with git!

You did not take into account the protocol differences, fetching a single
8MB  jar from Maven Central is easy, while fetching 6 MB via multiple git
HTTP resources (even assuming git packs there will be multiple http calls)
is probably more expensive, particularly as a typical project may have 50
dependencies.

Would each version be a new git repo? The binary class files are not really
suitable for "diffing" and would pretty sure be changed every release (even
if the SRC is the exact same).

It is possible in JAR to use jar200 packing, which can significantly reduce
the compressed size. Perhsps that is relevant here. See
http://docs.oracle.com/javase/7/docs/technotes/tools/share/pack200.html

Are the pom files within these git repositories or can they be prefetched
separately like today to speed up transitive dependency loading?



On 13 May 2017 8:04 pm, "Paul Hammant"  wrote:

> I was discussing this of the list today, and it may interest people on dev@
>
> https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
>
>  "Maven Central as multiple Git repositories"
>
> Enjoy,
>
> - Paul
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Aldrin Leal]

This is quite similar to what "go get" does to golang. I'd say its worth a
try

On May 14, 2017 09:28, "Paul Hammant"  wrote:

> Article updated to eliminate misunderstandings and talk about a different
> index for 'maven central' too.
>
> - ph
>
> On Sat, May 13, 2017 at 3:04 PM, Paul Hammant  wrote:
>
> > I was discussing this of the list today, and it may interest people on
> dev@
> >
> > https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
> >
> >  "Maven Central as multiple Git repositories"
> >
> > Enjoy,
> >
> > - Paul
> >
>

Re: Silly Saturday idea - If Maven Central were a bunch of Git repos

[Paul Hammant]

Article updated to eliminate misunderstandings and talk about a different
index for 'maven central' too.

- ph

On Sat, May 13, 2017 at 3:04 PM, Paul Hammant  wrote:

> I was discussing this of the list today, and it may interest people on dev@
>
> https://paulhammant.com/2017/05/13/maven-central-as-multiple-git-repos/
>
>  "Maven Central as multiple Git repositories"
>
> Enjoy,
>
> - Paul
>