Re: FBR: add exclude to mirror crawler

[Adrian Reber]

Pushed and live on both MM crawlers.

Adrian

On Thu, Mar 08, 2018 at 04:37:41PM +0100, Adrian Reber wrote:
> The rsync based crawler has two excludes hardcoded:
> 
>  cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
> 
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
> 
>  rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
> 
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
> 
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg 
> b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>  
>  # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'
>  
>  ###
>  # Configuration options used by the crons
> 
> 
> This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
> to update the crawler's configuration?
> 
>   Adrian



> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org


Adrian

-- 
Adrian Reber http://lisas.de/~adrian/
Change in Earth's rotational speed


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: add exclude to mirror crawler

[Pierre-Yves Chibon]

On Thu, Mar 08, 2018 at 04:37:41PM +0100, Adrian Reber wrote:
> The rsync based crawler has two excludes hardcoded:
> 
>  cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
> 
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
> 
>  rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
> 
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
> 
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg 
> b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>  
>  # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'


+1 for me as well


Pierre


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: add exclude to mirror crawler

[Patrick Uiterwijk]

On Thu, Mar 8, 2018 at 4:37 PM, Adrian Reber  wrote:

> The rsync based crawler has two excludes hardcoded:
>
>  cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
>
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
>
>  rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
>
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
>
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>
>  # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400
> --exclude=lost+found'
>
>  ###
>  # Configuration options used by the crons
>
>
> This excludes 'lost+found' from being scanned on all hosts. Can I get two
> +1
> to update the crawler's configuration?
>

+1


>
> Adrian
>
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave@lists.
> fedoraproject.org
>
>
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: add exclude to mirror crawler

[Adrian Reber]

The rsync based crawler has two excludes hardcoded:

 cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"

I am just in discussions with a new mirror and the rsync based crawler
fails there with:

 rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)

To avoid this in this case and also on other mirrors I would like to
apply following patch:

diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg 
b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
index 2bdd60273..69c930b23 100644
--- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
+++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
@@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
 
 # Specify additional rsync parameters for the crawler
 # # --timeout 14400: abort rsync crawl after 4 hours
-CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
+CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'
 
 ###
 # Configuration options used by the crons


This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
to update the crawler's configuration?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org