Please let me note that there are *tons* of errors you can get when crawling, from database errors to out-of-memory conditions to the actual ones you care about, namely errors accessing the repository. It is crucial that the connector code separate these errors into those that are fatal, those that can be retried, and those that indicate that the document should be skipped. It is simply not workable to try to insist that all errors are the same.
The difficulty comes in what the default behavior is for certain classes of errors that we've never seen before. I'm perfectly fine with trying to establish such a policy as you suggest in approach 1 for general classes of errors that are seen. But once again we need to catalog these and enumerate at least what classes these are. That's necessary on a connector-by-connector basis. The "brute force" approach of simply accepting all errors and continuing no matter what will not work, because really it's the same problem and the same bit of information you'd need to properly implement this. There's no shortcut I'm afraid. Please let me know which errors you are seeing and for which connector and let's work out how we handle them (or similar ones). Karl On Wed, Jun 5, 2019 at 10:41 AM Julien Massiera < [email protected]> wrote: > Hi Karl, > > I don't know for other MCF users, but we have many use cases where we > need to crawl several millions of documents from different kinds of > repositories. With those, we sometime have difficulties to manage issues > when crawl jobs suddenly stop because of problematic files that can only > be filtered to avoid the job to abort. > > From past discussions in the mailing list, I think that from your point > of view, it is preferable to stop a job when it encounters (or after > several failing retries) an unknown and/or unexpected issue in order to > be aware of this issue and fix it. > > Although I can understand your point of view, I do not think it > represents the exhaustivity of expected MCF behaviors in production. As > a matter of fact, we have encountered several times scenarios where > customers would prefer an approach where the crawl tries moving on, > while still giving us the possibility to investigate any file that may > have been skipped (One of the argument is that sometimes, jobs are > started on Friday evenings, and if it aborts during the weekend, we lost > at worse 60h of crawling before the admin can check the status of the job). > > Yet as of now, this is not feasible, as jobs end up aborting when > encountering non-clearly identified problematic files. > > We have brainstormed internally, and we have a proposal which we think > can satisfy both your view and ours, which we hope you consider as > satisfying : > > Whenever a job encounters an error that is not clearly identified : > 1. It immediately retries one time; > 2. If it succeeds, the crawl moves on as usual; > 3. If it fails, the job moves this document to the current end of the > processing pipeline, and crawls the remaining documents. It increments > the counter of tentative for this document to 2. > 4. When encountering this document again, the job tries again. If it > succeeds, the crawl moves on as usual. If it fails, it moves this > document to the current end of the processing pipeline, increment the > counter of 1, and doubles the delay between two tentatives. > 5. We iterate until the maximum number of tentatives of the crawl for > the problematic document has been reached. If it fails, abort the crawl. > With this behavior, a job is finally aborted on critical errors but at > least we will be able to crawl a maximum number of non problematic > documents till the failure. > > Another more "direct" approach, could be to simply have an optional > parameter for a job: a "skip errors" checkbox. This parameter would tell > a job to skip any encountered error. This is assuming we properly log > the errors in the log files and/or in the simple history, thus allowing > us to debug later on. > > We would gladly welcome your thoughts on these 2 approaches. > > Regards, > Julien >
