[issue42447] robotsparser deny all with some rules

[Terry J. Reedy]

Change by Terry J. Reedy :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42447] robotsparser deny all with some rules

[Terry J. Reedy]

Terry J. Reedy  added the comment:

This issue tracker is for proposing changes to the github CPython repository, 
used to make python.org python releases.  Your post does not propose a change 
and does not demonstrate that there is a bug in current Python, which is 3.9.  
So my current opinion is that this issue should be closed as 'not a bug'.  
Questions about using Python should be directed to a user help forum, such as 
python.org python-list or stackoverflow.com or some equivalent in French.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42447] robotsparser deny all with some rules

[Net Offensive]

Net Offensive  added the comment:

Sorry if my message is not clear enough. I will ask my developer to come and 
complete the information if needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42447] robotsparser deny all with some rules

[Net Offensive]

New submission from Net Offensive :

Bonjour, 

Notre développeur a un soucis avec l'utilisation de cette librairie. Dans le 
cadre d'un projet SEO, nous souhaiterions scrapper les pages de notre réseau de 
site.

Nous avons essayé de tester avec l'un de nos site dont les pages se présentent 
comme ce guide sur le référencement : 
https://www.netoffensive.blog/referencement-naturel/

Elle ne sont pas détectées comme des pages à cause de leur forme en repertoire. 
A ton besoin forcément de créer des pages du type : page.ext ?

C'est pourtant un format utilisé sur Wordpress et d'autres CMS.

Merci

---

Hello, 

Our developer has a problem with the use of this library. As part of an SEO 
project, we would like to scramble the pages of our site network.

We tried to test with one of our site whose pages look like this SEO guide: 
https://www.netoffensive.blog/referencement-naturel/.

They are not detected as pages because of their directory shape. Do you 
necessarily need to create pages of the type: page.ext?

It is however a format used on Wordpress and other CMS.

Thanks

Translated with www.DeepL.com/Translator (free version)

--
components: Library (Lib)
messages: 381683
nosy: contact
priority: normal
severity: normal
status: open
title: robotsparser deny all with some rules
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com