[issue36207] robotsparser deny all with some rules

[STINNER Victor]

STINNER Victor  added the comment:

I removed two comments: none of the mentioned URL contains a "Disallow: ?" rule 
and the comments didn't add any value to this issue. It looks like regular spam 
(SEO).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg416847

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg416767

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[adiboo adib]

adiboo adib  added the comment:

Hi now it work on all my website https://www.matelesecretairemedicale.com/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[adiboo adib]

adiboo adib  added the comment:

I can't find a documentation about it, but all of the robots.txt checkers I 
find behave like this. You can test on this site: 
https://www.st-info.fr/robots.txt, I believe that this is how it's implemented 
now in most parsers ?

--
nosy: +adiboo67

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Irit Katriel]

Irit Katriel  added the comment:

I restored one non-spam message from the OP that was deleted.

Changing to enhancement because this is not a bug (i.e., deviation from 
documentation).

I don't know enough about this to have a view on whether this enhancement 
request should be accepted.

--
nosy: +iritkatriel
type: behavior -> enhancement
versions: +Python 3.11 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[wats0ns]

wats0ns  added the comment:

I can't find a documentation about it, but all of the robots.txt checkers I 
find behave like this. You can test on this site: 
http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented 
now in most parsers ?

--
nosy: +quentin-maire

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg402889

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Nico]

Nico  added the comment:

Had same problem today for my website (https://www.bonus4casino.fr/), following 
for a fix

--
nosy: +nico.bonefato

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg338298

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

STINNER Victor  added the comment:

I removed almost all messages of this issue since most of them looked list 
SPAM. I also blocked user accounts who posted SPAM. If it was a mistake, 
contact me.

This is the Python bug tracker, not a forum to ask questions how to use Python, 
or to report bugs in your website.

Multiple comments were written in French, whereas this bug tracker is in 
English.

I even hesitate to close the issue since it got too many SPAM comments.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg365770

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg370275

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg377058

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg377125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg374642

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg376032

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg366509

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg367546

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg374629

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg372112

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg378070

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg379615

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg379616

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg385859

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
Removed message: https://bugs.python.org/msg381443

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[STINNER Victor]

Change by STINNER Victor :


--
title: référencement naturel -> robotsparser deny all with some rules

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[jeanotlapin]

jeanotlapin  added the comment:

Il semblerait que le script continue d'afficher des erreurs et rencontre des 
bugs. Preuve en est puisque j'ai testé sur ce site d'auto hypnose 
https://www.lautohypnose.com/ en vain...

--
nosy: +jeanotlapin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[idee Animation Anniversaire]

idee Animation Anniversaire  added the 
comment:

idee animation anniversaire est une agence animation à Paris pour les 
prestations comme pour
Animation entreprise, animation arbre de Noël, animation anniversaire à 
domicile, animation centre de loisir avec spectacle magie, Spectacle 
maquillage, Spectacle Sculpture sur Ballon en Ile-de-France.
Vous organisez un anniversaire à domicile.
http://idee-animation-anniversaire.fr/prestations/

--
nosy: +ideeanimationanniversaire

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Nicolas]

Nicolas  added the comment:

Sorry, I meant https://www.meridigital.com

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Nicolas]

Nicolas  added the comment:

Seems like we have the same issue with http://meridigital.com/robots.txt

--
nosy: +nico702 -matthieuhemea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Matthieu hemea]

Matthieu hemea  added the comment:

Hi, 

Does anyone find the solution ?
It would help me for this one : https://www.hemea.com/fr/devis-travaux

--
nosy: +matthieuhemea -Jmgray47, Patrick Valibus 410 Gone, amiir.mascud, arnaud, 
calamina, jeanotlapin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[jeanotlapin]

jeanotlapin  added the comment:

Bonjour,

j'ai rencontré le même soucis. J'ai essayé de le faire fonctionner en vain. 
J'ai tenté de faire crawler par les robots la page de notre agence de 
communication à Montpellier 
https://www.monagencedecommunication.com/agence/montpellier/ mais cela n'a pas 
visiblement pas marché...

--
nosy: +jeanotlapin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[amiir mascud]

amiir mascud  added the comment:

Can you share the robot file that you are using for your website?
I am using the default robot file for my site https://meilleurdumoniteur.fr/

--
nosy: +amiir.mascud

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[calamina]

calamina  added the comment:

I have a problem with my robot.txt on https://www.sondage-remunere.com/

--
nosy: +calamina

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Arnaud LIPERINI-DIAZ]

Arnaud LIPERINI-DIAZ  added the comment:

Do you have documentation about robotParser ? The robot.txt of this website 
works fine : https://vauros.com/

--
nosy: +Arnaud LIPERINI-DIAZ

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[James Gray]

James Gray  added the comment:

Bonjour, je vois que nous ne sommes pas les seuls dans ce cas, nous avons 
besoin que les robots indexent nos pages html mais qu'elles n'indexent pas 
celles en /*.php$ ainsi que les ressources PC en PDF. Nous avons tenté en vain 
plusieurs solutions en passant par le robots.txt à la racine de notre domaine 
https://demolinux.org/ mais sans succès. Le RobotsParser ne prends pas ces 
règles en compte, merci.

--
nosy: +Jmgray47

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Patrick Valibus 410 Gone]

Patrick Valibus 410 Gone  added the comment:

Bonjour, nous n'avons pas réussi à le faire fonctionner. Nous l'avons utilisé 
dans le cadre d'un test seo car nous essayons e reproduire des alternatives à 
scrappy. Par exemple le robots devrait bine crawler la page de notre agence seo 
https://www.410-gone.fr/seo.html mais ne devrait pas accepter les pages 
finissant par /*.php$ et pourtant si malgré qu'elles soient bloquées en 
référencement dans notre robots.txt, merci.

--
nosy: +Patrick Valibus 410 Gone -Fred AYERS, artasca, cheryl.sabella, 
lagustais, mathias44, quentin-maire

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[mathias44]

mathias44  added the comment:

I can't display my robot.TXT. I want to ban robots 
https://ereputation-dereferencement.fr/

--
nosy: +mathias44

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Fred AYERS]

Fred AYERS  added the comment:

I tried this one http://gtxgamer.fr/robots.txt/;>http://gtxgamer.fr/robots.txt and it 
seems to work.

--
nosy: +Fred AYERS

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[asca]

asca  added the comment:

I thought it was going to work but apparently when I try 
https://www.actusite.fr/robots.txt, it doesn't

--
nosy: +artasca

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Rodriguez]

Rodriguez  added the comment:

I can't display my robot.TXT. I want to ban robots
 https://melwynn-rodriguez.fr/robots.txt

--
nosy: +lagustais

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[wats0ns]

wats0ns  added the comment:

I can't find a documentation about it, but all of the robots.txt checkers I 
find behave like this. You can test on this site: 
http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented 
now in most parsers ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[Cheryl Sabella]

Cheryl Sabella  added the comment:

Can you provide a link to documentation showing that "Disallow: ?" shouldn't be 
the same as deny all?  Thanks!

--
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36207] robotsparser deny all with some rules

[wats0ns]

New submission from wats0ns :

RobotsParser parse a "Disallow: ?" rule as a deny all, but this is a valid rule 
that should be interpreted as "Disallow: /?*" or "Disallow: /*?*"

--
components: Library (Lib)
messages: 337285
nosy: quentin-maire
priority: normal
severity: normal
status: open
title: robotsparser deny all with some rules
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com