Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Vinícius Ferrão via xCAT-user]

As for https://xcat.org/files/xcat/repos/ which is the actual repositories 
(.deb and .rpm) it’s only 35GB.

# du -sh xcat
35G xcat

My plan is to keep it also public: http://mirror.versatushpc.com.br/xcat/

Regards.

On 24 May 2023, at 00:26, Russell Jones  wrote:

How large is the xcat repository? If there's any concern about opening up rsync 
causing a bandwidth or availability issue,  maybe the community could come 
together to host a few public mirrors with rsync access, and I'd be happy to be 
one.

I run "official" public mirrors for epel, fedora, rocky, and CentOS stream. I 
have the infrastructure to donate :-)

On Tue, May 23, 2023, 6:36 PM Vinícius Ferrão via xCAT-user 
mailto:xcat-user@lists.sourceforge.net>> wrote:
Hi Kilian,

reposync is great, mainly with Red Hat, since it’s the only way to mirror Red 
Hat distributions due to it’s “subscription manager” nature. I already use it 
for RHEL, but it’s not that great when you want to mirror multiple repositories 
and multiple versions. Also with EL8 there’s the would —download-metadata issue.

We do use it when it’s a stateful cluster with a single arch and single 
distribution, but my objetive here is to have a mirror, not a reposync one.

Also as I said on the last message, the storage system has standard storage 
tools, like rsync. Reposync is not available.

Regards.

> On 23 May 2023, at 20:06, Kilian Cavalotti 
> mailto:kilian.cavalotti.w...@gmail.com>> 
> wrote:
>
> Hi all,
>
> We routinely use reposync [1] to mirror the xcat-core and xcat-dep
> repositories, without rsync, and without having to re-download already
> downloaded packages.
> [2] has pointers on how to use it.
>
> Hope this helps!
>
> [1]: https://linux.die.net/man/1/reposync
> [2]: https://access.redhat.com/solutions/23016
>
> Cheers,
> --
> Kilian
>
> On Tue, May 23, 2023 at 12:51 PM Nathan A Besaw via xCAT-user
> mailto:xcat-user@lists.sourceforge.net>> 
> wrote:
>>
>> Hi Vinícius,
>>
>> Currently rsync is not available from xcat.org, but we can 
>> consider enabling it.
>>
>> How frequently do you plan to sync your local mirror from the 
>> xcat.org version?
>>
>> For the initial sync, I think you can use wget or curl to recursively 
>> download everything from https://xcat.org/files/xcat/ (or whatever 
>> directories are relevant to you).
>> If you are going to refresh your mirror infrequently (after every release?), 
>> using wget or curl may be sufficient. If you want to resync more regularly I 
>> would prefer to use a solution that includes incremental copying so you 
>> don't have to redownload ever file every time.
>>
>> 
>> From: Vinícius Ferrão via xCAT-user 
>> mailto:xcat-user@lists.sourceforge.net>>
>> Sent: Friday, May 19, 2023 1:02 PM
>> To: xCAT Users Mailing list 
>> mailto:xcat-user@lists.sourceforge.net>>
>> Cc: Vinícius Ferrão 
>> mailto:fer...@versatushpc.com.br>>
>> Subject: [EXTERNAL] [xcat-user] rsync on https://xcat.org/files/xcat/repos
>>
>> Hello, I would like to know if rsync is available on xCAT repository. I want 
>> to mirror it locally.
>>
>> I know that I can download the entire tarball from a given version, but I 
>> would like to use rsync to keep it updated.
>>
>> Thank you.
>>
>>
>>
>> ___
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>> ___
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
> --
> Kilian
>
>
> ___
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user


___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Russell Jones]

How large is the xcat repository? If there's any concern about opening up
rsync causing a bandwidth or availability issue,  maybe the community could
come together to host a few public mirrors with rsync access, and I'd be
happy to be one.

I run "official" public mirrors for epel, fedora, rocky, and CentOS stream.
I have the infrastructure to donate :-)

On Tue, May 23, 2023, 6:36 PM Vinícius Ferrão via xCAT-user <
xcat-user@lists.sourceforge.net> wrote:

> Hi Kilian,
>
> reposync is great, mainly with Red Hat, since it’s the only way to mirror
> Red Hat distributions due to it’s “subscription manager” nature. I already
> use it for RHEL, but it’s not that great when you want to mirror multiple
> repositories and multiple versions. Also with EL8 there’s the would
> —download-metadata issue.
>
> We do use it when it’s a stateful cluster with a single arch and single
> distribution, but my objetive here is to have a mirror, not a reposync one.
>
> Also as I said on the last message, the storage system has standard
> storage tools, like rsync. Reposync is not available.
>
> Regards.
>
> > On 23 May 2023, at 20:06, Kilian Cavalotti <
> kilian.cavalotti.w...@gmail.com> wrote:
> >
> > Hi all,
> >
> > We routinely use reposync [1] to mirror the xcat-core and xcat-dep
> > repositories, without rsync, and without having to re-download already
> > downloaded packages.
> > [2] has pointers on how to use it.
> >
> > Hope this helps!
> >
> > [1]: https://linux.die.net/man/1/reposync
> > [2]: https://access.redhat.com/solutions/23016
> >
> > Cheers,
> > --
> > Kilian
> >
> > On Tue, May 23, 2023 at 12:51 PM Nathan A Besaw via xCAT-user
> >  wrote:
> >>
> >> Hi Vinícius,
> >>
> >> Currently rsync is not available from xcat.org, but we can consider
> enabling it.
> >>
> >> How frequently do you plan to sync your local mirror from the xcat.org
> version?
> >>
> >> For the initial sync, I think you can use wget or curl to recursively
> download everything from https://xcat.org/files/xcat/ (or whatever
> directories are relevant to you).
> >> If you are going to refresh your mirror infrequently (after every
> release?), using wget or curl may be sufficient. If you want to resync more
> regularly I would prefer to use a solution that includes incremental
> copying so you don't have to redownload ever file every time.
> >>
> >> 
> >> From: Vinícius Ferrão via xCAT-user 
> >> Sent: Friday, May 19, 2023 1:02 PM
> >> To: xCAT Users Mailing list 
> >> Cc: Vinícius Ferrão 
> >> Subject: [EXTERNAL] [xcat-user] rsync on
> https://xcat.org/files/xcat/repos
> >>
> >> Hello, I would like to know if rsync is available on xCAT repository. I
> want to mirror it locally.
> >>
> >> I know that I can download the entire tarball from a given version, but
> I would like to use rsync to keep it updated.
> >>
> >> Thank you.
> >>
> >>
> >>
> >> ___
> >> xCAT-user mailing list
> >> xCAT-user@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/xcat-user
> >> ___
> >> xCAT-user mailing list
> >> xCAT-user@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/xcat-user
> >
> >
> >
> > --
> > Kilian
> >
> >
> > ___
> > xCAT-user mailing list
> > xCAT-user@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
> ___
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Vinícius Ferrão via xCAT-user]

Hi Kilian,

reposync is great, mainly with Red Hat, since it’s the only way to mirror Red 
Hat distributions due to it’s “subscription manager” nature. I already use it 
for RHEL, but it’s not that great when you want to mirror multiple repositories 
and multiple versions. Also with EL8 there’s the would —download-metadata issue.

We do use it when it’s a stateful cluster with a single arch and single 
distribution, but my objetive here is to have a mirror, not a reposync one.

Also as I said on the last message, the storage system has standard storage 
tools, like rsync. Reposync is not available.

Regards. 

> On 23 May 2023, at 20:06, Kilian Cavalotti  
> wrote:
> 
> Hi all,
> 
> We routinely use reposync [1] to mirror the xcat-core and xcat-dep
> repositories, without rsync, and without having to re-download already
> downloaded packages.
> [2] has pointers on how to use it.
> 
> Hope this helps!
> 
> [1]: https://linux.die.net/man/1/reposync
> [2]: https://access.redhat.com/solutions/23016
> 
> Cheers,
> --
> Kilian
> 
> On Tue, May 23, 2023 at 12:51 PM Nathan A Besaw via xCAT-user
>  wrote:
>> 
>> Hi Vinícius,
>> 
>> Currently rsync is not available from xcat.org, but we can consider enabling 
>> it.
>> 
>> How frequently do you plan to sync your local mirror from the xcat.org 
>> version?
>> 
>> For the initial sync, I think you can use wget or curl to recursively 
>> download everything from https://xcat.org/files/xcat/ (or whatever 
>> directories are relevant to you).
>> If you are going to refresh your mirror infrequently (after every release?), 
>> using wget or curl may be sufficient. If you want to resync more regularly I 
>> would prefer to use a solution that includes incremental copying so you 
>> don't have to redownload ever file every time.
>> 
>> 
>> From: Vinícius Ferrão via xCAT-user 
>> Sent: Friday, May 19, 2023 1:02 PM
>> To: xCAT Users Mailing list 
>> Cc: Vinícius Ferrão 
>> Subject: [EXTERNAL] [xcat-user] rsync on https://xcat.org/files/xcat/repos
>> 
>> Hello, I would like to know if rsync is available on xCAT repository. I want 
>> to mirror it locally.
>> 
>> I know that I can download the entire tarball from a given version, but I 
>> would like to use rsync to keep it updated.
>> 
>> Thank you.
>> 
>> 
>> 
>> ___
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>> ___
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
> 
> 
> 
> -- 
> Kilian
> 
> 
> ___
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user


___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Vinícius Ferrão via xCAT-user]

I’m considering to run the mirror command every day, three times a day, at 
hours: 2, 7, 21 (UTC-3 time).
The ideia is to be set and forget.

I’m building the repository right now, for the initial copy I’ve used lftp 
(http://lftp.yar.ru), which is painful slow and really bad with delta updates, 
and with my organization is bad, since my ideas is to run rsync directly on the 
storage (TrueNAS), with lftp I had to mount the /var/repos directory on other 
machine just to run lftp.

Files are already available here:

Index of /xcat/
mirror.versatushpc.com.br
[X]

rsync is definitely the best option to mirror due to delta sync as you said. 
Something like -aSH --delete --delete-delay --delay-updates would be good with 
rsync, considering that any symbolic link on xCAT side is relative, so it will 
not be a broken symlink after the mirror.

Thanks.

On 23 May 2023, at 16:50, Nathan A Besaw via xCAT-user 
 wrote:

Hi Vinícius,

Currently rsync is not available from xcat.org, but we can 
consider enabling it.

How frequently do you plan to sync your local mirror from the 
xcat.org version?

For the initial sync, I think you can use wget or curl to recursively download 
everything from https://xcat.org/files/xcat/ (or whatever directories are 
relevant to you).
If you are going to refresh your mirror infrequently (after every release?), 
using wget or curl may be sufficient. If you want to resync more regularly I 
would prefer to use a solution that includes incremental copying so you don't 
have to redownload ever file every time.


From: Vinícius Ferrão via xCAT-user 
Sent: Friday, May 19, 2023 1:02 PM
To: xCAT Users Mailing list 
Cc: Vinícius Ferrão 
Subject: [EXTERNAL] [xcat-user] rsync on https://xcat.org/files/xcat/repos

Hello, I would like to know if rsync is available on xCAT repository. I want to 
mirror it locally.

I know that I can download the entire tarball from a given version, but I would 
like to use rsync to keep it updated.

Thank you.



___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Kilian Cavalotti]

Hi all,

We routinely use reposync [1] to mirror the xcat-core and xcat-dep
repositories, without rsync, and without having to re-download already
downloaded packages.
[2] has pointers on how to use it.

Hope this helps!

[1]: https://linux.die.net/man/1/reposync
[2]: https://access.redhat.com/solutions/23016

Cheers,
--
Kilian

On Tue, May 23, 2023 at 12:51 PM Nathan A Besaw via xCAT-user
 wrote:
>
> Hi Vinícius,
>
> Currently rsync is not available from xcat.org, but we can consider enabling 
> it.
>
> How frequently do you plan to sync your local mirror from the xcat.org 
> version?
>
> For the initial sync, I think you can use wget or curl to recursively 
> download everything from https://xcat.org/files/xcat/ (or whatever 
> directories are relevant to you).
> If you are going to refresh your mirror infrequently (after every release?), 
> using wget or curl may be sufficient. If you want to resync more regularly I 
> would prefer to use a solution that includes incremental copying so you don't 
> have to redownload ever file every time.
>
> 
> From: Vinícius Ferrão via xCAT-user 
> Sent: Friday, May 19, 2023 1:02 PM
> To: xCAT Users Mailing list 
> Cc: Vinícius Ferrão 
> Subject: [EXTERNAL] [xcat-user] rsync on https://xcat.org/files/xcat/repos
>
> Hello, I would like to know if rsync is available on xCAT repository. I want 
> to mirror it locally.
>
> I know that I can download the entire tarball from a given version, but I 
> would like to use rsync to keep it updated.
>
> Thank you.
>
>
>
> ___
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
> ___
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user



-- 
Kilian


___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] rsync on https://xcat.org/files/xcat/repos

[Nathan A Besaw via xCAT-user]

Hi Vinícius,

Currently rsync is not available from xcat.org, but we can consider enabling it.

How frequently do you plan to sync your local mirror from the xcat.org version?

For the initial sync, I think you can use wget or curl to recursively download 
everything from https://xcat.org/files/xcat/ (or whatever directories are 
relevant to you).
If you are going to refresh your mirror infrequently (after every release?), 
using wget or curl may be sufficient. If you want to resync more regularly I 
would prefer to use a solution that includes incremental copying so you don't 
have to redownload ever file every time.


From: Vinícius Ferrão via xCAT-user 
Sent: Friday, May 19, 2023 1:02 PM
To: xCAT Users Mailing list 
Cc: Vinícius Ferrão 
Subject: [EXTERNAL] [xcat-user] rsync on https://xcat.org/files/xcat/repos

Hello, I would like to know if rsync is available on xCAT repository. I want to 
mirror it locally.

I know that I can download the entire tarball from a given version, but I would 
like to use rsync to keep it updated.

Thank you.



___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
___
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user