Hi,
Raising the concern above again, to process only 60k of document (when
clock issue is fixed too), job process is not progressing , its being stuck
for like days. So had to restart the docker container every time for it to
process.
This time now we are getting this :- Timeout Exception. What we can be the
reason for it and how it can be fixed .?
... 24 more
[Worker thread '23'] WARN jcifs.util.transport.Transport - sendrecv failed
jcifs.util.transport.RequestTimeoutException: Transport40 timedout waiting
for response to
command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=86
at
jcifs.util.transport.Transport.waitForResponses(Transport.java:365)
at jcifs.util.transport.Transport.sendrecv(Transport.java:232)
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021)
at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539)
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409)
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347)
at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611)
at
jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614)
at
jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568)
at
jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489)
at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465)
at
jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426)
at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551)
at jcifs.smb.SmbFile.length(SmbFile.java:1541)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileLength(SharedDriveConnector.java:2340)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector$ProcessDocumentsFilter.accept(SharedDriveConnector.java:4935)
at
jcifs.smb.SmbEnumerationUtil$ResourceFilterWrapper.accept(SmbEnumerationUtil.java:331)
at
jcifs.smb.FileEntryAdapterIterator.advance(FileEntryAdapterIterator.java:82)
at
jcifs.smb.FileEntryAdapterIterator.<init>(FileEntryAdapterIterator.java:52)
at
jcifs.smb.DirFileEntryAdapterIterator.<init>(DirFileEntryAdapterIterator.java:37)
at jcifs.smb.SmbEnumerationUtil.doEnum(SmbEnumerationUtil.java:223)
at
jcifs.smb.SmbEnumerationUtil.listFiles(SmbEnumerationUtil.java:279)
at jcifs.smb.SmbFile.listFiles(SmbFile.java:1273)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileListFiles(SharedDriveConnector.java:2380)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:818)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
[Worker thread '23'] WARN jcifs.smb.SmbTransportImpl - Disconnecting
transport while still in use Transport40[backup002.directory.intra/
136.231.158.172:445,state=5,signingEnforced=false,usage=5]:
[SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]]
[Worker thread '23'] WARN jcifs.smb.SmbSessionImpl - Logging off session
while still in use
SmbSession[credentials=svc_EScrawl,targetHost=backup002.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINPROJECTS,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]]
[Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed
jcifs.util.transport.RequestTimeoutException: Transport41 timedout waiting
for response to
command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=80
at
jcifs.util.transport.Transport.waitForResponses(Transport.java:365)
at jcifs.util.transport.Transport.sendrecv(Transport.java:232)
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1021)
at jcifs.smb.SmbTransportImpl.send(SmbTransportImpl.java:1539)
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:409)
at jcifs.smb.SmbSessionImpl.send(SmbSessionImpl.java:347)
at jcifs.smb.SmbTreeImpl.treeConnect(SmbTreeImpl.java:611)
at
jcifs.smb.SmbTreeConnection.connectTree(SmbTreeConnection.java:614)
at
jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:568)
at
jcifs.smb.SmbTreeConnection.connectHost(SmbTreeConnection.java:489)
at jcifs.smb.SmbTreeConnection.connect(SmbTreeConnection.java:465)
at
jcifs.smb.SmbTreeConnection.connectWrapException(SmbTreeConnection.java:426)
at jcifs.smb.SmbFile.ensureTreeConnected(SmbFile.java:551)
at jcifs.smb.SmbFile.exists(SmbFile.java:845)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2220)
at
org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.processDocuments(SharedDriveConnector.java:610)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
[Worker thread '10'] WARN jcifs.smb.SmbTransportImpl - Disconnecting
transport while still in use Transport41[homestore.directory.intra/
136.231.158.104:445,state=5,signingEnforced=false,usage=5]:
[SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=2,usage=3]]
[Worker thread '10'] WARN jcifs.smb.SmbSessionImpl - Logging off session
while still in use
SmbSession[credentials=svc_EScrawl,targetHost=homestore.directory.intra,targetDomain=null,uid=0,connectionState=3,usage=3]:[SmbTree[share=WINHOMES,service=?????,tid=-1,inDfs=false,inDomainDfs=false,connectionState=1,usage=1]]
[Worker thread '10'] WARN jcifs.util.transport.Transport - sendrecv failed
jcifs.util.transport.RequestTimeoutException: Transport44 timedout waiting
for response to
command=SMB2_TREE_CONNECT,status=0,flags=0x0000,mid=4,wordCount=0,byteCount=72
at
jcifs.util.transport.Transport.waitForResponses(Transport.java:365)
at jcifs.util.transport.Transport.sendrecv(Transport.java:232)
at jcifs.smb.SmbTransportImpl.sendrecv(SmbTransportImpl.java:1
On Tue, Nov 9, 2021 at 6:19 PM Karl Wright <daddy...@gmail.com> wrote:
> One hour is quite a lot and will wreak havoc on the document queue.
> Karl
>
>
> On Tue, Nov 9, 2021 at 7:08 AM ritika jain <ritikajain5...@gmail.com>
> wrote:
>
>> I have checked, there is only one hour time difference between docker
>> container and docker host
>>
>> On Tue, Nov 9, 2021 at 4:41 PM Karl Wright <daddy...@gmail.com> wrote:
>>
>>> If your docker image's clock is out of sync badly with the real world,
>>> then System.currentTimeMillis() may give bogus values, and ManifoldCF uses
>>> that to manage throttling etc. I don't know if that is the correct
>>> explanation but it's the only thing I can think of.
>>>
>>> Karl
>>>
>>>
>>> On Tue, Nov 9, 2021 at 4:56 AM ritika jain <ritikajain5...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi All,
>>>>
>>>> I am using window shares connector , manifoldcf 2.14 and ES as output.
>>>> I have configured a job to process 60k of documents, Also these documents
>>>> are new and do not have corresponding values in DB and ES index.
>>>>
>>>> So ideally it should process/Index the documents as soon as the job
>>>> starts.
>>>> But Manifoldcf does not process anything for many hours of job start
>>>> up.I have tried restarting the docker container as well. But it didn't help
>>>> much. Also logs only correspond to Long running queries.
>>>>
>>>> Why does the manifold behave like that?
>>>>
>>>> Thanks
>>>> Ritika
>>>>
>>>
