Hi Karl,
Thanks for that I will try version 2.6. Whilst moving to MCF
2.6 I would potentially like to upgrade my backend PostgreSQL version from
9.3.5.
1) Do you have a recommendation for which PostgreSQL to use with MCF 2.6 e.g.
PostgreSQL 9.3.16 or PostgreSQL 9.6.2?
2) For a production system on a single server running a single MCF agents
process would you recommend the file based synchronisation locking or zookeeper
based synchronisation locking. With the file based synchronisation locking
mechanism I have sometimes seen errors of the form :
'D:\Apps\ManifoldCF\apache-manifoldcf-2.0.1\multiprocess-file-example\.\.\syncharea\475\708\lock-_POOLTARGET__OUTPUTCONNECTORPOOL_Solr
COLL1 osp_unstruct.lock' failed: Access is denied’ ( I have ensured that the
SYNCHAREA is not scanned by AV or Indexed by Windows Search and all MCF
processes run as the same user)
What could cause these errors?
Many Thanks,
Guy
From: Karl Wright [mailto:[email protected]]
Sent: 03 March 2017 17:35
To: [email protected]
Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
Hi Guy:
It is expected that sometimes database deadlock will develop, and the
transaction will need to be retried. There is code in MCF for doing this:
>>>>>>
if (sqlState != null && sqlState.equals("40001"))
return new
ManifoldCFException(message,e,ManifoldCFException.DATABASE_TRANSACTION_ABORT);
<<<<<<
I suspect that your version of MCF is old enough so that this particular error
and the associated retry are not taking place. Upgrading to 2.6 will
definitely help there.
Karl
On Fri, Mar 3, 2017 at 11:17 AM, Standen Guy
<[email protected]<mailto:[email protected]>> wrote:
Hi Karl,
Thanks for coming back so quickly. Unfortunately I wasn’t using
a JCIFS connection. One of the issues I was seeing was between a crawl of an
intranet site (no explicit throttling other than number of connections) and
scheduled crawl (every 5 mins) to a relational DB via JDBC connector again no
explicit throttling. To simplify things both jobs are using a NULL output
connection. Sometimes both the Web crawl and the JDBC connection can run
together but at other times 1 or both jobs will appear to lock up with just a
few active documents showing. When I get a lock up the mcf log contains errors
like:
“DEBUG 2017-03-03 15:28:20,466 (Worker thread '5') - Exception Database
exception: SQLException doing query (40001): ERROR: could not serialize access
due to read/write dependencies among transactions”
See the attached log extract for a little more detail.
Any view why this might be happening?
Best Regards,
Guy
From: Karl Wright [mailto:[email protected]<mailto:[email protected]>]
Sent: 03 March 2017 11:27
To: [email protected]<mailto:[email protected]>
Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
Hi Guy,
A issue with concurrent jobs is known for jobs sharing the same JCIFS
connection. Is that what you are using? This has nothing to do with the
version of Postgresql you are using; it has to do with what "bins" documents
are thought to come from. There has been a recent improvement for this issue,
which will be released in April. See
https://issues.apache.org/jira/browse/CONNECTORS-1364.
The current version of MCF (2.6) supports Solr 6.x.
Thanks,
Karl
On Fri, Mar 3, 2017 at 5:27 AM, Standen Guy
<[email protected]<mailto:[email protected]>> wrote:
Hi Karl,
I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and have had
some issues with multiple jobs running concurrently.
I am considering upgrading to MCF 2.6 and to a newer version of PostgreSQL.
Would you be able to advise which version of PostgreSQL I should consider using
with MCF 2.6 (e.g. PostgreSQL 9.3.16 or 9.6.2)
I am also considering upgrading from SOLR 4.10.3 to a newer version. The MCF
compatibility matrix mentions that compatibility has been tested to SOLR
version 4.5.1. Do you have any advice about compatibility with the newer
versions of SOLR e.g. 6.4.1.
Best Regards
Guy
Unless otherwise stated, this email has been sent from Fujitsu Services Limited
(registered in England No 96056); Fujitsu EMEA PLC (registered in England No
2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU
(EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of
Europe Limited (registered in England No. 4153469) both with registered offices
at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are
subject to a duty of confidence and may be privileged. Fujitsu does not
guarantee that this email has not been intercepted and amended or that it is
virus-free.
Unless otherwise stated, this email has been sent from Fujitsu Services Limited
(registered in England No 96056); Fujitsu EMEA PLC (registered in England No
2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU
(EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of
Europe Limited (registered in England No. 4153469) both with registered offices
at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are
subject to a duty of confidence and may be privileged. Fujitsu does not
guarantee that this email has not been intercepted and amended or that it is
virus-free.
Unless otherwise stated, this email has been sent from Fujitsu Services Limited
(registered in England No 96056); Fujitsu EMEA PLC (registered in England No
2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU
(EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of
Europe Limited (registered in England No. 4153469) both with registered offices
at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are
subject to a duty of confidence and may be privileged. Fujitsu does not
guarantee that this email has not been intercepted and amended or that it is
virus-free.
