Re: Proposal for migrating web content to git
Hi
Perfect! Thanks a lot for the hard/good work!!
--
Olivier
On Wed, 12 Dec 2018 at 07:02, Martin Stockhammer
wrote:
> Hi together,
>
> after my last request about the status on the JIRA ticket, the
> infrastructure guys changed the publish process from svn to git publish
> without further asking.
> So I will not ask for any votes, just raise your hand, if you have still
> concerns about it.
>
> I checked the publish process and it worked fine.
>
> Changes for the publish process (maven part, scripts, repository urls)
> are commited to master now.
>
>
> Some notes about the publish process:
> - archiva-web-content.git is like a mirror to the web site
> archiva.apache.org
>Anything that is pushed to the master branch on this will be
> available on the web site.
> - the archiva-web-content.git is fed by different maven configurations
> into subfolders:
>- archiva/archiva-doc
>- archiva/archiva-modules/...
>- archiva-site
>- archiva-redback-site
>- archiva-redback-core
>- archiva-redback-components/...
>
> - There are README files and scripts on the corresponding directories
>that tells how to build, check and deploy the site documents.
>
> I did not run a complete publish on all this components, just checked
> the publish process to a local repo. So the site has not changed by this
> migration.
>
>
> Regards
>
> Martin
>
>
> On 08.11.18 08:52, Martin wrote:
> > Hi,
> >
> > ok, so I will prepare the redback changes too.
> > And yes, I will provide some documentation, for sure.
> >
> > I will start the vote in parallel, in the hope that some other
> registered developers are listening here.
> >
> > Regards
> >
> > Martin
> >
> >
> >
> > Am Dienstag, 6. November 2018, 01:02:55 CET schrieb Olivier Lamy:
> >> Hi
> >> Looks good to me. Thanks for the hard work!
> >> Maybe add documentation somewhere?
> >> as a README file in the git repos (build/deploy site section)?
> >>
> >> On Mon, 5 Nov 2018 at 00:33, Martin wrote:
> >>
> >>> Hi together,
> >>>
> >>> sorry for the long mail, but it's a bit complicated ...
> >>>
> >>> The archiva webcontent repo (site-content) is the last one that still
> >>> resides on SVN.
> >>> I would like to move this one to git using the gitpubsub mechanism for
> >>> deployment to the HTTP server.
> >>>
> >>> There is a ticket at INFRA open already
> >>> https://issues.apache.org/jira/browse/INFRA-16387 to get more
> information
> >>> about
> >>> the migration process.
> >>>
> >>> This repo is still not migrated to git, because:
> >>> - The size is really a concern, because the old ref docs of previous
> >>> versions are stored there
> >>> - there are some specialities in the site generation process that are
> >>> bound to the svnpubsub and svn to avoid checking out
> >>>the whole repo
> >>> - The archiva-modules and archiva-docs site content is stored in a
> >>> subdirectory which is working fine with SVN but not so easy
> >>>with git
> >>> - redback site publish is using the same repository
> >>>
> >>> So for the migration, we have to make sure that:
> >>> 1.) the content of the git repo is the same as of the SVN repo
> >>> 2.) we are still able to publish site changes from our source tree to
> the
> >>> public archiva site:
> >>> - archiva-site -> The main site pages
> >>> - archiva/archiva-docs -> The user and admin documentation
> >>> - archiva/archiva-modules/... -> the reports and module
> specific
> >>> documentation
> >>> - redback -> The redback documentation
> >>> 3.) the publish process does not need to checkout the whole content
> >>> repository to publish one of the documentations above
> >>>
> >>> I think I have a proof of concept for these tasks (not for redback now,
> >>> but it will be the similar). You can check it out from the
> >>> Branch feature/site-git-migration
> >>>
> >>> for 1.) There is a migrated repo
> >>> https://gitbox.apache.org/repos/asf/archiva-web-content.git with the
> >>> current status of the SVN repository
> >>> for 2.)
> >>> I'm still using the maven-scm-publish-plugin which is
> capable of
> >>> using git repositories
> >>> The part with subdirectories and submodules is a bit tricky,
> >>> because you cannot specifiy a subdirectory of a git repository in the
> >>> publish url.
> >>> I change the path for the staging directory used by the site
> >>> plugin (e.g. to target/staging/docs/${project.version}) and reference
> the
> >>> staging
> >>> root directory in the scm-publish plugin.
> >>> For submodules we need a path to the archiva-modules project
> as
> >>> staging dir, therefore the new property entries in the submodules (if
> >>> somebody knows a
> >>> better solution, please tell me)
> >>> for 3.) I'm using git sparse checkouts which allows to checkout only
> >>> subtrees of a repository. The git repository index is still needed for
> each
> >>> site checkout, but
> >>> its about 86MB and that is
Re: Proposal for migrating web content to git
Hi together,
after my last request about the status on the JIRA ticket, the
infrastructure guys changed the publish process from svn to git publish
without further asking.
So I will not ask for any votes, just raise your hand, if you have still
concerns about it.
I checked the publish process and it worked fine.
Changes for the publish process (maven part, scripts, repository urls)
are commited to master now.
Some notes about the publish process:
- archiva-web-content.git is like a mirror to the web site
archiva.apache.org
Anything that is pushed to the master branch on this will be
available on the web site.
- the archiva-web-content.git is fed by different maven configurations
into subfolders:
- archiva/archiva-doc
- archiva/archiva-modules/...
- archiva-site
- archiva-redback-site
- archiva-redback-core
- archiva-redback-components/...
- There are README files and scripts on the corresponding directories
that tells how to build, check and deploy the site documents.
I did not run a complete publish on all this components, just checked
the publish process to a local repo. So the site has not changed by this
migration.
Regards
Martin
On 08.11.18 08:52, Martin wrote:
Hi,
ok, so I will prepare the redback changes too.
And yes, I will provide some documentation, for sure.
I will start the vote in parallel, in the hope that some other registered
developers are listening here.
Regards
Martin
Am Dienstag, 6. November 2018, 01:02:55 CET schrieb Olivier Lamy:
Hi
Looks good to me. Thanks for the hard work!
Maybe add documentation somewhere?
as a README file in the git repos (build/deploy site section)?
On Mon, 5 Nov 2018 at 00:33, Martin wrote:
Hi together,
sorry for the long mail, but it's a bit complicated ...
The archiva webcontent repo (site-content) is the last one that still
resides on SVN.
I would like to move this one to git using the gitpubsub mechanism for
deployment to the HTTP server.
There is a ticket at INFRA open already
https://issues.apache.org/jira/browse/INFRA-16387 to get more information
about
the migration process.
This repo is still not migrated to git, because:
- The size is really a concern, because the old ref docs of previous
versions are stored there
- there are some specialities in the site generation process that are
bound to the svnpubsub and svn to avoid checking out
the whole repo
- The archiva-modules and archiva-docs site content is stored in a
subdirectory which is working fine with SVN but not so easy
with git
- redback site publish is using the same repository
So for the migration, we have to make sure that:
1.) the content of the git repo is the same as of the SVN repo
2.) we are still able to publish site changes from our source tree to the
public archiva site:
- archiva-site -> The main site pages
- archiva/archiva-docs -> The user and admin documentation
- archiva/archiva-modules/... -> the reports and module specific
documentation
- redback -> The redback documentation
3.) the publish process does not need to checkout the whole content
repository to publish one of the documentations above
I think I have a proof of concept for these tasks (not for redback now,
but it will be the similar). You can check it out from the
Branch feature/site-git-migration
for 1.) There is a migrated repo
https://gitbox.apache.org/repos/asf/archiva-web-content.git with the
current status of the SVN repository
for 2.)
I'm still using the maven-scm-publish-plugin which is capable of
using git repositories
The part with subdirectories and submodules is a bit tricky,
because you cannot specifiy a subdirectory of a git repository in the
publish url.
I change the path for the staging directory used by the site
plugin (e.g. to target/staging/docs/${project.version}) and reference the
staging
root directory in the scm-publish plugin.
For submodules we need a path to the archiva-modules project as
staging dir, therefore the new property entries in the submodules (if
somebody knows a
better solution, please tell me)
for 3.) I'm using git sparse checkouts which allows to checkout only
subtrees of a repository. The git repository index is still needed for each
site checkout, but
its about 86MB and that is still manageable.
There is a shell script in the archiva-docs and archiva-modules
folder that runs the clone and sparse checkout and which is executed by a
exec plugin.
It is linux/bash only, if we need windows as well, I may work on
some platform independent way.
I did not checkin a valid publish url for the archiva-web-content.git
repository, to avoid littering it with too much testing commits, but you
can test it with a
local git clone (must be a bare clone):
git clone --bare
https://gitbox.apache.org/repos/asf/archiva-web-content.git
${HOME}/git/archiva-web-content.git
With this local clone you can test the publish
Re: Proposal for migrating web content to git
Hi
Looks good to me. Thanks for the hard work!
Maybe add documentation somewhere?
as a README file in the git repos (build/deploy site section)?
On Mon, 5 Nov 2018 at 00:33, Martin wrote:
> Hi together,
>
> sorry for the long mail, but it's a bit complicated ...
>
> The archiva webcontent repo (site-content) is the last one that still
> resides on SVN.
> I would like to move this one to git using the gitpubsub mechanism for
> deployment to the HTTP server.
>
> There is a ticket at INFRA open already
> https://issues.apache.org/jira/browse/INFRA-16387 to get more information
> about
> the migration process.
>
> This repo is still not migrated to git, because:
> - The size is really a concern, because the old ref docs of previous
> versions are stored there
> - there are some specialities in the site generation process that are
> bound to the svnpubsub and svn to avoid checking out
> the whole repo
> - The archiva-modules and archiva-docs site content is stored in a
> subdirectory which is working fine with SVN but not so easy
> with git
> - redback site publish is using the same repository
>
> So for the migration, we have to make sure that:
> 1.) the content of the git repo is the same as of the SVN repo
> 2.) we are still able to publish site changes from our source tree to the
> public archiva site:
> - archiva-site -> The main site pages
> - archiva/archiva-docs -> The user and admin documentation
> - archiva/archiva-modules/... -> the reports and module specific
> documentation
> - redback -> The redback documentation
> 3.) the publish process does not need to checkout the whole content
> repository to publish one of the documentations above
>
> I think I have a proof of concept for these tasks (not for redback now,
> but it will be the similar). You can check it out from the
> Branch feature/site-git-migration
>
> for 1.) There is a migrated repo
> https://gitbox.apache.org/repos/asf/archiva-web-content.git with the
> current status of the SVN repository
> for 2.)
> I'm still using the maven-scm-publish-plugin which is capable of
> using git repositories
> The part with subdirectories and submodules is a bit tricky,
> because you cannot specifiy a subdirectory of a git repository in the
> publish url.
> I change the path for the staging directory used by the site
> plugin (e.g. to target/staging/docs/${project.version}) and reference the
> staging
> root directory in the scm-publish plugin.
> For submodules we need a path to the archiva-modules project as
> staging dir, therefore the new property entries in the submodules (if
> somebody knows a
> better solution, please tell me)
> for 3.) I'm using git sparse checkouts which allows to checkout only
> subtrees of a repository. The git repository index is still needed for each
> site checkout, but
> its about 86MB and that is still manageable.
> There is a shell script in the archiva-docs and archiva-modules
> folder that runs the clone and sparse checkout and which is executed by a
> exec plugin.
> It is linux/bash only, if we need windows as well, I may work on
> some platform independent way.
>
>
> I did not checkin a valid publish url for the archiva-web-content.git
> repository, to avoid littering it with too much testing commits, but you
> can test it with a
> local git clone (must be a bare clone):
> git clone --bare
> https://gitbox.apache.org/repos/asf/archiva-web-content.git
> ${HOME}/git/archiva-web-content.git
>
> With this local clone you can test the publish process:
>
> archiva-site:
> ==
> - Checkout the branch feature/site-git-migration
> - Run
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> clean site
> # if you want to look at the staging area in target/staging
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> site:stage
> # Deploy:
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> site-deploy
> # You can verify the changes in your archiva-web-content.git clone
>
> archiva-docs:
> ==
> - Checkout the branch feature/site-git-migration of archiva repo
> - cd archiva-docs
> - Run
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> clean site
> # if you want to look at the staging area in target/staging
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> site:stage
> # Deploy:
> mvn
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> site-deploy
> # You can verify the changes in your archiva-web-content.git clone
>
>
> archiva-modules:
> =
> I reduced the generated reports to speedup the build process.
> - Checkout the branch feature/site-git-migration of archiva repo
> - Run
> cd archiva-modules
> ./deploySite.sh
> -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
> site-deploy
> # You can verify
Proposal for migrating web content to git
Hi together,
sorry for the long mail, but it's a bit complicated ...
The archiva webcontent repo (site-content) is the last one that still resides
on SVN.
I would like to move this one to git using the gitpubsub mechanism for
deployment to the HTTP server.
There is a ticket at INFRA open already
https://issues.apache.org/jira/browse/INFRA-16387 to get more information about
the migration process.
This repo is still not migrated to git, because:
- The size is really a concern, because the old ref docs of previous versions
are stored there
- there are some specialities in the site generation process that are bound to
the svnpubsub and svn to avoid checking out
the whole repo
- The archiva-modules and archiva-docs site content is stored in a subdirectory
which is working fine with SVN but not so easy
with git
- redback site publish is using the same repository
So for the migration, we have to make sure that:
1.) the content of the git repo is the same as of the SVN repo
2.) we are still able to publish site changes from our source tree to the
public archiva site:
- archiva-site -> The main site pages
- archiva/archiva-docs -> The user and admin documentation
- archiva/archiva-modules/... -> the reports and module specific
documentation
- redback -> The redback documentation
3.) the publish process does not need to checkout the whole content repository
to publish one of the documentations above
I think I have a proof of concept for these tasks (not for redback now, but it
will be the similar). You can check it out from the
Branch feature/site-git-migration
for 1.) There is a migrated repo
https://gitbox.apache.org/repos/asf/archiva-web-content.git with the current
status of the SVN repository
for 2.)
I'm still using the maven-scm-publish-plugin which is capable of using
git repositories
The part with subdirectories and submodules is a bit tricky, because
you cannot specifiy a subdirectory of a git repository in the publish url.
I change the path for the staging directory used by the site plugin
(e.g. to target/staging/docs/${project.version}) and reference the staging
root directory in the scm-publish plugin.
For submodules we need a path to the archiva-modules project as staging
dir, therefore the new property entries in the submodules (if somebody knows a
better solution, please tell me)
for 3.) I'm using git sparse checkouts which allows to checkout only subtrees
of a repository. The git repository index is still needed for each site
checkout, but
its about 86MB and that is still manageable.
There is a shell script in the archiva-docs and archiva-modules folder
that runs the clone and sparse checkout and which is executed by a exec plugin.
It is linux/bash only, if we need windows as well, I may work on some
platform independent way.
I did not checkin a valid publish url for the archiva-web-content.git
repository, to avoid littering it with too much testing commits, but you can
test it with a
local git clone (must be a bare clone):
git clone --bare https://gitbox.apache.org/repos/asf/archiva-web-content.git
${HOME}/git/archiva-web-content.git
With this local clone you can test the publish process:
archiva-site:
==
- Checkout the branch feature/site-git-migration
- Run
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
clean site
# if you want to look at the staging area in target/staging
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
site:stage
# Deploy:
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
site-deploy
# You can verify the changes in your archiva-web-content.git clone
archiva-docs:
==
- Checkout the branch feature/site-git-migration of archiva repo
- cd archiva-docs
- Run
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
clean site
# if you want to look at the staging area in target/staging
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
site:stage
# Deploy:
mvn -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
site-deploy
# You can verify the changes in your archiva-web-content.git clone
archiva-modules:
=
I reduced the generated reports to speedup the build process.
- Checkout the branch feature/site-git-migration of archiva repo
- Run
cd archiva-modules
./deploySite.sh
-DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git
site-deploy
# You can verify the changes in your archiva-web-content.git clone
So please try it out and tell me what you think about it, or if you have any
concerns or know something I have not considered yet.
We need a vote on this mailing list for the final migration, so would be
helpful to get some feedback before starting the voting process.
Greetings
Martin
