I tried to restart the job dealing with www.duo.no on our test server, but it does not seem to touch the robots.txt file at all. That's the reason why it's able to continue. Both servers are set up to obey the rules of such files.
Erlend On 18.09.14 11:12, Erlend Garåsen wrote:
I'm facing the same problems with robot.txt files using RC1, so maybe this is another issue we have to fix. Can you please try to fetch the host below? For some odd reason, it seems that MCF on our test server can handle it. This is exactly the same that happened when I started MCF (referring to my previous post) after I had deployed RC1: 09-18-2014 11:02:14.400 robots parse https:www.duo.uio.no:443 ERRORS 0 3 Unknown robots.txt line: '====' No activity after this error. Here's the robots.txt file: https://www.duo.uio.no/robots.txt This is the content of manifoldcf.log after the startup: WARN 2014-09-18 11:02:14,401 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': '====' WARN 2014-09-18 11:02:14,401 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': ' The contents of this file are subject to the license and copyright' WARN 2014-09-18 11:02:14,402 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': ' detailed in the LICENSE and NOTICE files at the root of the source' WARN 2014-09-18 11:02:14,402 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': ' tree and available online at' WARN 2014-09-18 11:02:14,402 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': ' http://www.dspace.org/license/' WARN 2014-09-18 11:02:14,402 (Worker thread '19') - Web: Unknown robots.txt line from 'https:www.duo.uio.no:443': '====' E On 18.09.14 03:12, Karl Wright wrote:Please vote on whether to release Apache ManifoldCF 1.7.1, RC1. This release fixes a number of critical issues, as well as a number of user priorities, most notably: - A bad Zookeeper support issue, which made locking support fail when Zookeeper connections got lost and then restored; - The Alfresco connector, which was nonfunctional in both MCF 1.6 and 1.7; - Solr Cloud support, which had ceased working due to changes to SolrJ; - Non-null connector components caused failure; - PostgreSQL queries not performing well. The complete list of included fixes can be found at: https://svn.apache.org/repos/asf/manifoldcf/tags/release-1.7.1-RC1/CHANGES.txt The release candidate can be downloaded from: http://people.apache.org/~kwright/apache-manifoldcf-1.7.1 There is a tag at: https://svn.apache.org/repos/asf/manifoldcf/tags/release-1.7.1-RC1 Thanks, Karl
