I have checked the regex-urlfilter and by default I see this line:

# skip URLs containing certain characters as probable queries, etc.

In my case for a particular url I want to crawl a specific query, so wanted
to know what file would be the best to make changes to enable this.

Would it be regex-urlfilter or I also see a filters file suffix-urlfilter
and fast-urlfilter.

Would adding filters in any of the later two files would help.
Any idea why these filters are added, like what would be the potential

Also say if I add multiple filter plugins backed by these files, then how
url filtering works? Only those urls which pass all the plugins are
selected to be fetched or any of the plugin?


Reply via email to