Yes sorry I wrongly described the 1/: the runtime exception happens in the
processDocument on the first and only document found in the seeding phase.
Here is the stack trace:
WARN 2022-03-22T13:07:03,761 (Worker thread '13') -
MCF|MCF-agent|apache.manifoldcf.connectors|JCIFS: Possibly transient exception
detected on attempt 1 while checking if file smb://localhost/test/ exists: null
jcifs.smb.SmbException: null
at jcifs.smb.SmbTreeImpl.waitForState(SmbTreeImpl.java:774)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:540)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465)
~[jcifs-ng-2.1.7.jar:?]
at
jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:559)
~[jcifs-ng-2.1.7.jar:?]
at jcifs.smb.SmbFile.exists(SmbFile.java:859) ~[jcifs-ng-2.1.7.jar:?]
at
com.francelabs.datafari.connectors.share.SharedDriveConnector.fileExists(SharedDriveConnector.java:2129)
[datafari-share-connector-6.0-dev-Community.jar:?]
at
com.francelabs.datafari.connectors.share.SharedDriveConnector.processDocuments(SharedDriveConnector.java:677)
[datafari-share-connector-6.0-dev-Community.jar:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
[mcf-pull-agent.jar:?]
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method) ~[?:?]
at java.lang.Object.wait(Object.java:328) ~[?:?]
at jcifs.smb.SmbTreeImpl.waitForState(SmbTreeImpl.java:771)
~[jcifs-ng-2.1.7.jar:?]
... 11 more
FATAL 2022-03-22T13:07:03,777 (Worker thread '13') -
MCF|MCF-agent|apache.manifoldcf.crawlerthreads|Error tossed: 'boolean
org.bouncycastle.asn1.ASN1ObjectIdentifier.equals(org.bouncycastle.asn1.ASN1Primitive)'
java.lang.NoSuchMethodError: 'boolean
org.bouncycastle.asn1.ASN1ObjectIdentifier.equals(org.bouncycastle.asn1.ASN1Primitive)'
at jcifs.spnego.NegTokenInit.parse(NegTokenInit.java:167) ~[?:?]
at jcifs.spnego.NegTokenInit.<init>(NegTokenInit.java:66) ~[?:?]
at
jcifs.smb.NtlmPasswordAuthenticator.createContext(NtlmPasswordAuthenticator.java:243)
~[?:?]
at jcifs.smb.SmbSessionImpl.createContext(SmbSessionImpl.java:706)
~[?:?]
at jcifs.smb.SmbSessionImpl.sessionSetupSMB2(SmbSessionImpl.java:544)
~[?:?]
at jcifs.smb.SmbSessionImpl.sessionSetup(SmbSessionImpl.java:491) ~[?:?]
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:369) ~[?:?]
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) ~[?:?]
at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) ~[?:?]
at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614)
~[?:?]
at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568)
~[?:?]
at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489)
~[?:?]
at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465)
~[?:?]
at
jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426)
~[?:?]
at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:559) ~[?:?]
at jcifs.smb.SmbFile.exists(SmbFile.java:859) ~[?:?]
at
com.francelabs.datafari.connectors.share.SharedDriveConnector.fileExists(SharedDriveConnector.java:2129)
~[?:?]
at
com.francelabs.datafari.connectors.share.SharedDriveConnector.processDocuments(SharedDriveConnector.java:677)
~[?:?]
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
[mcf-pull-agent.jar:?]
So after that FATAL exception, we end up in the 'wait' state of the
documentQueue
Julien
-----Message d'origine-----
De : Karl Wright <daddy...@gmail.com>
Envoyé : mercredi 23 mars 2022 16:32
À : dev <dev@manifoldcf.apache.org>
Objet : Re: WorkerThread runtime exceptions
Specifically, the stuffer thread is responsible for finding documents to
process and getting them to the worker threads via the internal queue that the
worker threads wait on. The stuffer thread uses a query to do this.
Either the query is not finding any documents, or the stuffer thread is down.
Probably it is the former, and the reason it is not finding any documents is
because the job is in the wrong state due to that runtime exception.
Can you describe what code is throwing that runtime exception? It would be
very helpful if you could provide a stack trace for it from the log.
Karl
On Wed, Mar 23, 2022 at 11:27 AM Karl Wright <daddy...@gmail.com> wrote:
> ' 1/ On the first and only one document of the seeding phase
> encountered, a runtime exception is triggered'
>
> The worker threads do not handle seeding. If a runtime exception
> takes place during seeding, no documents will be queued, and that is
> the problem. The state of the job must be incorrectly updated even
> though the seeding failed. OR the job's state is properly updated but
> the corresponding thread that is supposed to know when the job is
> completed (by looking at the job queue) doesn't properly trigger.
>
> The architecture of ManifoldCF has many threads that are individually
> responsible for transitioning the job state based on the jobqueue. If
> somehow the jobstate winds up not in the right state then those
> threads will not do the right thing.
>
> Karl
>
>
> On Wed, Mar 23, 2022 at 11:08 AM Julien Massiera <
> julien.massi...@francelabs.com> wrote:
>
>> Hi Karl,
>>
>> I had some time to investigate the problem I exposed in my first
>> mail, and here is the behavior I observed:
>>
>> 1/ On the first and only one document of the seeding phase
>> encountered, a runtime exception is triggered 2/ The runtime
>> exception is catched by the WorkerThread, logged, and the
>> WorkerThread stays alive (line 856 of the WorkerThread class) 3/ The
>> WorkerThread calls the getDocument method of its documentQueue (line
>> 121 of the WorkerThread class) 4/ The documentQueue ends in an
>> infinite 'wait' state because the queue size is 0 and the resetFlag
>> is false (lines 109 and 110 of the DocumentQueue class) 5/ Because of
>> the infinite 'wait' state of the documentQueue, the job stays freezed
>> on the 'running' state and it is impossible to stop it until the
>> Agent is restarted
>>
>> I don't know much about the WorkerThread and the DocumentQueue logic,
>> so from there, I really need your help to understand this behavior
>> and to figure out what can be done to prevent the job from hanging in
>> that case, which, I assume, can happen in other circumstances with
>> other repository connectors
>>
>> Regards,
>> Julien
>>
>> -----Message d'origine-----
>> De : Julien Massiera <julien.massi...@francelabs.com> Envoyé : jeudi
>> 24 février 2022 15:08 À : dev@manifoldcf.apache.org Objet : RE:
>> WorkerThread runtime exceptions
>>
>> Yes I understand
>>
>> -----Message d'origine-----
>> De : Karl Wright <daddy...@gmail.com> Envoyé : jeudi 24 février 2022
>> 14:59 À : dev <dev@manifoldcf.apache.org> Objet : Re: WorkerThread
>> runtime exceptions
>>
>> I'm currently completely consumed with upgrading dependencies for
>> Tika and CXF. This is a massive job and won't be done for probably
>> another week or two. Once that is done I can try to look into your concern.
>>
>> Karl
>>
>>
>> On Thu, Feb 24, 2022 at 8:47 AM Julien Massiera <
>> julien.massi...@francelabs.com> wrote:
>>
>> > Hi,
>> >
>> >
>> >
>> > I have faced a situation where the MCF agent was still up but was
>> > not doing anything after a runtime exception.
>> >
>> >
>> >
>> > My use case was the following :
>> > I have updated the libs used by a repository connector but forgot one.
>> > During doc processing, a runtime exception <
>> > java.lang.NoSuchMethodError > has been throwed because the sub
>> > dependency lib was not up to date and thus the method called was
>> > missing. The exception was catched by the WorkerThread and
>> > displayed < Error tossed: .. > but then nothing and the job stayed
>> > in running status and I was not able to abort it until I killed and
>> > I restarted the agent.
>> >
>> >
>> >
>> > The catching clause is located in the WorkerThread class at lines
>> 853-857.
>> > I
>> > know this is a particular case but I am not sure that the fact the
>> > agent hangs after this exception is a normal behavior and
>> > furthermore I can imagine that it can happen with other unkown runtime
>> > exceptions.
>> > Is there something we can do to avoid the agent to be hanging in
>> > those
>> cases ?
>> >
>> >
>> >
>> > Regards,
>> >
>> > Julien
>> >
>> >
>> >
>> >
>>
>>
>>
