Many Thanks Karl From: Karl Wright [mailto:[email protected]] Sent: 06 March 2017 09:24 To: [email protected] Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6
Hi Guy, (1) I have no experience with PostgresSQL versions beyond 9.3, but I doubt you would have problems. (2) If you are using multiple processes, even if there's only one agents process, you must use synchronization. I would recommend Zookeeper; file-system-based synchronization is deprecated. (3) Windows has many ways of interfering with file-based sync, including path-length issues. I have seen Windows fail to unlock files and need a reboot to release the lock. This is one reason why file-system-based locking is deprecated. Thanks, Karl On Mon, Mar 6, 2017 at 4:03 AM, Standen Guy <[email protected]<mailto:[email protected]>> wrote: Hi Karl, Thanks for that I will try version 2.6. Whilst moving to MCF 2.6 I would potentially like to upgrade my backend PostgreSQL version from 9.3.5. 1) Do you have a recommendation for which PostgreSQL to use with MCF 2.6 e.g. PostgreSQL 9.3.16 or PostgreSQL 9.6.2? 2) For a production system on a single server running a single MCF agents process would you recommend the file based synchronisation locking or zookeeper based synchronisation locking. With the file based synchronisation locking mechanism I have sometimes seen errors of the form : 'D:\Apps\ManifoldCF\apache-manifoldcf-2.0.1\multiprocess-file-example\.\.\syncharea\475\708\lock-_POOLTARGET__OUTPUTCONNECTORPOOL_Solr COLL1 osp_unstruct.lock' failed: Access is denied’ ( I have ensured that the SYNCHAREA is not scanned by AV or Indexed by Windows Search and all MCF processes run as the same user) What could cause these errors? Many Thanks, Guy From: Karl Wright [mailto:[email protected]<mailto:[email protected]>] Sent: 03 March 2017 17:35 To: [email protected]<mailto:[email protected]> Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6 Hi Guy: It is expected that sometimes database deadlock will develop, and the transaction will need to be retried. There is code in MCF for doing this: >>>>>> if (sqlState != null && sqlState.equals("40001")) return new ManifoldCFException(message,e,ManifoldCFException.DATABASE_TRANSACTION_ABORT); <<<<<< I suspect that your version of MCF is old enough so that this particular error and the associated retry are not taking place. Upgrading to 2.6 will definitely help there. Karl On Fri, Mar 3, 2017 at 11:17 AM, Standen Guy <[email protected]<mailto:[email protected]>> wrote: Hi Karl, Thanks for coming back so quickly. Unfortunately I wasn’t using a JCIFS connection. One of the issues I was seeing was between a crawl of an intranet site (no explicit throttling other than number of connections) and scheduled crawl (every 5 mins) to a relational DB via JDBC connector again no explicit throttling. To simplify things both jobs are using a NULL output connection. Sometimes both the Web crawl and the JDBC connection can run together but at other times 1 or both jobs will appear to lock up with just a few active documents showing. When I get a lock up the mcf log contains errors like: “DEBUG 2017-03-03 15:28:20,466 (Worker thread '5') - Exception Database exception: SQLException doing query (40001): ERROR: could not serialize access due to read/write dependencies among transactions” See the attached log extract for a little more detail. Any view why this might be happening? Best Regards, Guy From: Karl Wright [mailto:[email protected]<mailto:[email protected]>] Sent: 03 March 2017 11:27 To: [email protected]<mailto:[email protected]> Subject: Re: Advice on which PostgreSQL to use with ManifoldCF 2.6 Hi Guy, A issue with concurrent jobs is known for jobs sharing the same JCIFS connection. Is that what you are using? This has nothing to do with the version of Postgresql you are using; it has to do with what "bins" documents are thought to come from. There has been a recent improvement for this issue, which will be released in April. See https://issues.apache.org/jira/browse/CONNECTORS-1364. The current version of MCF (2.6) supports Solr 6.x. Thanks, Karl On Fri, Mar 3, 2017 at 5:27 AM, Standen Guy <[email protected]<mailto:[email protected]>> wrote: Hi Karl, I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and have had some issues with multiple jobs running concurrently. I am considering upgrading to MCF 2.6 and to a newer version of PostgreSQL. Would you be able to advise which version of PostgreSQL I should consider using with MCF 2.6 (e.g. PostgreSQL 9.3.16 or 9.6.2) I am also considering upgrading from SOLR 4.10.3 to a newer version. The MCF compatibility matrix mentions that compatibility has been tested to SOLR version 4.5.1. Do you have any advice about compatibility with the newer versions of SOLR e.g. 6.4.1. Best Regards Guy Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE. This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is virus-free. Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE. This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is virus-free. Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE. This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is virus-free. Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE. This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is virus-free.
