Re: Manifold Job process isssue
SMB exceptions with jcifs in the trace tell us that JCIFS couldn't talk to your windows share server. That's all we can tell though. Karl On Mon, Nov 15, 2021 at 7:24 AM ritika jain wrote: > Hi, > > Raising the concern above again, to process only 60k of document (when > clock issue is fixed too), job process is not progressing , its being stuck > for like days. So had to restart the docker container every time for it to > process. > This time now we are getting this :- Timeout Exception. What we can be the > reason for it and how it can be fixed .? > ... 24 more > [Worker thread '23'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport40 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x,mid=4,wordCount=0,byteCount=86 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) > at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) > at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) > at > jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) > at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) > at > jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) > at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) > at jcifs.smb.SmbFile.length(SmbFile.java:1541) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileLength(SharedDriveConnector.java:2340) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector$ProcessDocumentsFilter.accept(SharedDriveConnector.java:4935) > at > jcifs.smb.SmbEnumerationUtil$ResourceFilterWrapper.accept(SmbEnumerationUtil.java:331) > at > jcifs.smb.FileEntryAdapterIterator.advance(FileEntryAdapterIterator.java:82) > at > jcifs.smb.FileEntryAdapterIterator.(FileEntryAdapterIterator.java:52) > at > jcifs.smb.DirFileEntryAdapterIterator.(DirFileEntryAdapterIterator.java:37) > at jcifs.smb.SmbEnumerationUtil.doEnum(SmbEnumerationUtil.java:223) > at > jcifs.smb.SmbEnumerationUtil.listFiles(SmbEnumerationUtil.java:279) > at jcifs.smb.SmbFile.listFiles(SmbFile.java:1273) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileListFiles(SharedDriveConnector.java:2380) > at > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:818) > at > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > [Worker thread '23'] WARN jcifs.smb.SmbTransportImpl - Disconnecting > transport while still in use Transport40[backup002.directory.intra/ > 136.231.158.172:445,state=5,signingEnforced=false,usage=5]: > [SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] > [Worker thread '23'] WARN jcifs.smb.SmbSessionImpl - Logging off session > while still in use > SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINPROJECTS,service=?,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] > [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed > jcifs.util.transport.RequestTimeoutException: Transport41 timedout waiting > for response to > command=SMB2_TREE_CONNECT,status=0,flags=0x,mid=4,wordCount=0,byteCount=80 > at > jcifs.util.transport.Transport.waitForResponses(Transport.java:365) > at jcifs.util.transport.Transport.sendrecv(Transport.java:232) > at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) > at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) > at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) > at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) > at > jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) > at > jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) > at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) > at > jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) > at
Re: Manifold Job process isssue
Hi, Raising the concern above again, to process only 60k of document (when clock issue is fixed too), job process is not progressing , its being stuck for like days. So had to restart the docker container every time for it to process. This time now we are getting this :- Timeout Exception. What we can be the reason for it and how it can be fixed .? ... 24 more [Worker thread '23'] WARN jcifs.util.transport.Transport - sendrecv failed jcifs.util.transport.RequestTimeoutException: Transport40 timedout waiting for response to command=SMB2_TREE_CONNECT,status=0,flags=0x,mid=4,wordCount=0,byteCount=86 at jcifs.util.transport.Transport.waitForResponses(Transport.java:365) at jcifs.util.transport.Transport.sendrecv(Transport.java:232) at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) at jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) at jcifs.smb.SmbFile.length(SmbFile.java:1541) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileLength(SharedDriveConnector.java:2340) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector$ProcessDocumentsFilter.accept(SharedDriveConnector.java:4935) at jcifs.smb.SmbEnumerationUtil$ResourceFilterWrapper.accept(SmbEnumerationUtil.java:331) at jcifs.smb.FileEntryAdapterIterator.advance(FileEntryAdapterIterator.java:82) at jcifs.smb.FileEntryAdapterIterator.(FileEntryAdapterIterator.java:52) at jcifs.smb.DirFileEntryAdapterIterator.(DirFileEntryAdapterIterator.java:37) at jcifs.smb.SmbEnumerationUtil.doEnum(SmbEnumerationUtil.java:223) at jcifs.smb.SmbEnumerationUtil.listFiles(SmbEnumerationUtil.java:279) at jcifs.smb.SmbFile.listFiles(SmbFile.java:1273) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileListFiles(SharedDriveConnector.java:2380) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:818) at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) [Worker thread '23'] WARN jcifs.smb.SmbTransportImpl - Disconnecting transport while still in use Transport40[backup002.directory.intra/ 136.231.158.172:445,state=5,signingEnforced=false,usage=5]: [SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]] [Worker thread '23'] WARN jcifs.smb.SmbSessionImpl - Logging off session while still in use SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINPROJECTS,service=?,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]] [Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed jcifs.util.transport.RequestTimeoutException: Transport41 timedout waiting for response to command=SMB2_TREE_CONNECT,status=0,flags=0x,mid=4,wordCount=0,byteCount=80 at jcifs.util.transport.Transport.waitForResponses(Transport.java:365) at jcifs.util.transport.Transport.sendrecv(Transport.java:232) at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021) at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409) at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347) at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611) at jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568) at jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489) at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465) at jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426) at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551) at jcifs.smb.SmbFile.exists(SmbFile.java:845) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2220) at org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610) at
Re: Manifold Job process isssue
One hour is quite a lot and will wreak havoc on the document queue. Karl On Tue, Nov 9, 2021 at 7:08 AM ritika jain wrote: > I have checked, there is only one hour time difference between docker > container and docker host > > On Tue, Nov 9, 2021 at 4:41 PM Karl Wright wrote: > >> If your docker image's clock is out of sync badly with the real world, >> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses >> that to manage throttling etc. I don't know if that is the correct >> explanation but it's the only thing I can think of. >> >> Karl >> >> >> On Tue, Nov 9, 2021 at 4:56 AM ritika jain >> wrote: >> >>> >>> Hi All, >>> >>> I am using window shares connector , manifoldcf 2.14 and ES as output. I >>> have configured a job to process 60k of documents, Also these documents are >>> new and do not have corresponding values in DB and ES index. >>> >>> So ideally it should process/Index the documents as soon as the job >>> starts. >>> But Manifoldcf does not process anything for many hours of job start >>> up.I have tried restarting the docker container as well. But it didn't help >>> much. Also logs only correspond to Long running queries. >>> >>> Why does the manifold behave like that? >>> >>> Thanks >>> Ritika >>> >>
Re: Manifold Job process isssue
I have checked, there is only one hour time difference between docker container and docker host On Tue, Nov 9, 2021 at 4:41 PM Karl Wright wrote: > If your docker image's clock is out of sync badly with the real world, > then System.currentTimeMillis() may give bogus values, and ManifoldCF uses > that to manage throttling etc. I don't know if that is the correct > explanation but it's the only thing I can think of. > > Karl > > > On Tue, Nov 9, 2021 at 4:56 AM ritika jain > wrote: > >> >> Hi All, >> >> I am using window shares connector , manifoldcf 2.14 and ES as output. I >> have configured a job to process 60k of documents, Also these documents are >> new and do not have corresponding values in DB and ES index. >> >> So ideally it should process/Index the documents as soon as the job >> starts. >> But Manifoldcf does not process anything for many hours of job start up.I >> have tried restarting the docker container as well. But it didn't help >> much. Also logs only correspond to Long running queries. >> >> Why does the manifold behave like that? >> >> Thanks >> Ritika >> >
Re: Manifold Job process isssue
If your docker image's clock is out of sync badly with the real world, then System.currentTimeMillis() may give bogus values, and ManifoldCF uses that to manage throttling etc. I don't know if that is the correct explanation but it's the only thing I can think of. Karl On Tue, Nov 9, 2021 at 4:56 AM ritika jain wrote: > > Hi All, > > I am using window shares connector , manifoldcf 2.14 and ES as output. I > have configured a job to process 60k of documents, Also these documents are > new and do not have corresponding values in DB and ES index. > > So ideally it should process/Index the documents as soon as the job starts. > But Manifoldcf does not process anything for many hours of job start up.I > have tried restarting the docker container as well. But it didn't help > much. Also logs only correspond to Long running queries. > > Why does the manifold behave like that? > > Thanks > Ritika >
