Mingchun Zhao created CONNECTORS-1748:
-----------------------------------------
Summary: Add "User-Agent platform" option for crawling mobile
websites
Key: CONNECTORS-1748
URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
Project: ManifoldCF
Issue Type: New Feature
Reporter: Mingchun Zhao
Assignee: Mingchun Zhao
Hi [~daddywri] ,
Allow me propose a new feature for crawling mobile sites which restrict access
to content based on User-Agent header.
Actually, Our customer's mobile website was failing to crawl because access was
restricted based on whether the User-Agent request header includes the mobile
info. For this reason, we added the "User-Agent platform" option to the new
"Request Headers" tab on the web repository connector page so that this mobile
website doesn't fail to crawl.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
