Hi Ameya, If reducing the number of simultaneous connections didn't help throughput, then frankly the only way to make things faster is to improve the performance of the CIFS drives you are crawling. You could try to purchase dedicated NetApp devices for those, if that hasn't already been done. Beyond that I have no solutions for you. See ManifoldCF In Action Chapter 1.
Karl On Wed, Jul 30, 2014 at 11:18 AM, Ameya Aware <[email protected]> wrote: > So how can improve the performance of crawling for shared drives? > > > Thanks, > Ameya > > > On Thu, Jul 24, 2014 at 2:53 PM, Karl Wright <[email protected]> wrote: > >> Hi Ameya, >> >> ManifoldCF can only crawl as fast as the underlying repository will >> support. The performance in the case of a network drive is obviously going >> to be slower than that of a local drive. >> >> Karl >> >> >> >> On Thu, Jul 24, 2014 at 2:49 PM, Ameya Aware <[email protected]> >> wrote: >> >>> Hi Karl, >>> >>> Currently document security is the not the main aim. >>> >>> I just want all the files to be crawled as quickly as possible. >>> >>> So do you want to say that my shared drive would never get crawled as >>> fast as my C drive? >>> >>> >>> On Thu, Jul 24, 2014 at 2:44 PM, Karl Wright <[email protected]> wrote: >>> >>>> Hi Ameya, >>>> >>>> The filesystem connector does not support document security. Are you >>>> sure this is what you want to do? >>>> >>>> Either way, the same logic applies, since it is the same underlying >>>> technology. >>>> >>>> Karl >>>> >>>> >>>> >>>> On Thu, Jul 24, 2014 at 2:40 PM, Ameya Aware <[email protected]> >>>> wrote: >>>> >>>>> i apologize for my mistake. >>>>> >>>>> I wanted to say that i am not using windows share connector. I am >>>>> using filesystem connector only where in i provide the location of shared >>>>> drive in repository paths. >>>>> >>>>> >>>>> On Thu, Jul 24, 2014 at 2:36 PM, Karl Wright <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Ameya, >>>>>> >>>>>> Yes, I figured that you would be using the share connector to crawl >>>>>> shared drives. That's what I am talking about, though. The share >>>>>> connector is only as fast as the server you are connected to, and there >>>>>> may >>>>>> be indirections also if the path contains DFS links. Regardless, though, >>>>>> Windows file servers are easily overloaded, in my experience, so try not >>>>>> to >>>>>> overload them unnecessarily. When a file server gets overloaded, it >>>>>> often >>>>>> fails with a timeout (which you can see in the simple history report, or >>>>>> in >>>>>> the manifoldcf log). That's a sign that you need to throttle more, and >>>>>> the >>>>>> best way to do that is to limit further the number of simultaneous >>>>>> connections. >>>>>> >>>>>> Thanks, >>>>>> Karl >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jul 24, 2014 at 2:32 PM, Ameya Aware <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Actually i am using windows share connector. I am using filesystem >>>>>>> connector only where in i provide the location of shared drive in >>>>>>> repository paths. >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 24, 2014 at 2:13 PM, Karl Wright <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Ameya, >>>>>>>> >>>>>>>> Windows shared drive access performance is not great, I'm afraid. >>>>>>>> However, it is possible that you are overwhelming the server you are >>>>>>>> talking to. Try reducing the maximum number of connections for the >>>>>>>> shared >>>>>>>> drive repository connection to between 2 and 5; that sometimes helps. >>>>>>>> >>>>>>>> Karl >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jul 24, 2014 at 2:11 PM, Ameya Aware <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> When i am crawling full of my C drive the job is running pretty >>>>>>>>> much faster. >>>>>>>>> >>>>>>>>> But at the same time when i m crawling documents kept on shared >>>>>>>>> drive, it is taking too much longer time than to crawl C drive. >>>>>>>>> >>>>>>>>> are documents at shared drive supposed to take to much time?? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ameya >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
