Hi, I am an IT-consultant located in Austria with experiences in doing workflows ( most in java using jbpm).
Here an idea for implementing an simple workflow engine to process new uploaded documents: An workflow consists of configurable actions. Which are processed by an workflow-handler. An action is an python class with has to have two function init and run. All the data generated within an action is stored in an context ( dict) . This context is passed to each action (init-function) so the data from the previous actions can be reused. In the example below the "fax cover sheet detection action" uses the text and the layout informations (hOCR) generated by "tesseract action" to classify which kind of document it is. The tagging action afterwards uses the classification to tag the document. Example workflow run for tiff (or other images). 3.1.1) run command line action: unpaper 3.1.2) run command line action: tesseract 3.1.2) run fax cover sheet detection action 3.1.3) run document tagging action Some Ideas for useful generic actions: - run command line program action (e.g. unpaper, tesseract, convert) - matching layout action (hOCR parser, to check if test is in an specific region of the document) - regular expression action ( parsing text for tagging document) - messaging action (notify an user or another system that an specific document has been imported) happy to get your feedback ! Matthias Am Donnerstag, 4. September 2014 00:48:15 UTC+2 schrieb Mathias Behrle: > > * Roberto Rosario: " Re: [Mayan EDMS: 816] Automatic upload from certain > staging folder" (Wed, 3 Sep 2014 11:47:52 -0700 (PDT)): > > > I like the barcode/qrcode idea very much, would allow for batch > scanning, > > for example several documents placed in a scanner with a document feeder > > and each document has a printed page with a barcode defining the > metadata > > kind of like FAX cover pages. Regional OCR is a must have feature and > > usually a defining feature of the commercial offerings, I don't know how > > accurate OCR a rectangle of text would but is there is a need for the > > feature let's do it. We need a way to let users mark/highlight the > fields > > they want scanned and entered as metadata. This would required some > design > > decisions (do we store the cursor's x and y positions of the square to > be > > scanned or the x and y % in relation to the current zoom level) and a > rich > > client w/ corresponding API endpoints to talk to the backend. > > > > On Wednesday, August 27, 2014 5:22:32 PM UTC-4, Mathias Behrle wrote: > > > > > > * Roberto Rosario: " Re: [Mayan EDMS: 761] Automatic upload from > certain > > > staging folder" (Wed, 30 Jul 2014 13:36:51 -0400): > > > > > > > This feature was actually started some time ago ( > > > > > > > > https://github.com/mayan-edms/mayan-edms/blob/master/mayan/apps/sources/models.py#L194) > > > > > > > > > but is not yet enabled because it depends on some scheduling update > that > > > > have not made it into the master branch. > > > > > > > > As for metadata, I came up with some ideas but none are implemented. > One > > > > was to let users set default metadata values as well as document > type > > > for > > > > each watch folder. Another idea was when a document is being > imported > > > from > > > > a watch folder to look for a file with the same name but with the > > > .metadata > > > > extension. No design decision has been reached yet so any ideas are > > > > welcomed. > > > > > > Both possibilities could have their individual use cases, for which > they > > > fit > > > best. The most flexible approach is the second. > > > > > > What I found when evaluating other DMS software: > > > > > > - Inclusion of some identifier on the document (could be a barcode, or > > > some > > > special formatted string, or...). This identifier must not > necessarily > > > be > > > fixed on the document, but could be the first page of a scan or some > > > paper > > > scanned together with the document. This method applies preferably > to > > > scanned > > > documents. > > > > > > > > I like the barcode/qrcode idea very much, would allow for batch > scanning, > > for example several documents placed in a scanner with a document feeder > > and each document has a printed page with a barcode defining the > metadata > > kind of like FAX cover pages. > > Yes, the comparison with the Fax cover page hits the mark. > > Question: > When batch scanning, how to determine the beginning and the end of a > batch? > Will each document require a 'cover page' or can such a cover page be > valid for > several documents? Perhaps the number of documents could be included on > the > cover page, but this would always require a new cover page per batch. > > > > - Rather straightforward is a sort of recognition, where templates can > be > > > defined containing regions formatted in an individual way. E.g. if > you > > > have a > > > supplier with his custom invoice format displaying the invoice > number, > > > date, > > > amount at fixed places, they could be used on such a template and > the > > > software > > > can check, if the document contains such a region. > > > > > > > > > Regional OCR is a must have feature and usually a defining feature of > the > > commercial offerings, I don't know how accurate OCRing a rectangle of > text > > would be but if there is a need for the feature let's do it. I see some > > requirements, we need a way to let users mark/highlight the fields they > > want scanned and entered as metadata. This would require some design > > decisions (do we store the cursor's x and y positions of the square to > be > > scanned or the x and y % in relation to the current zoom level) > > The more agnostic of the zoom level, the better. So I would think x and y > in > relation to X and Y (where X and Y are the dimensions of the whole page). > > > and a rich client w/ corresponding API endpoints to talk to the backend. > > Do you mean, a separate client is needed for that purpose? > > > > Perhaps this could be used slightly modified but simpler by defining > > > string patterns, that could be matched on the OCR result. So at last > > > repeating patterns could be used to extract metadata. > > > > > > > > > In any case I would find useful a document queue containing documents > > > already processed (OCR available), but still to be completed with > > > metadata. So > > > to speak an inversion of the current workflow (where metadata are > defined > > > first). > > > As already discussed in > https://github.com/mayan-edms/mayan-edms/issues/9) > > > I > > > think it would be best for the manual completion of metadata to have a > > > view of > > > the document together with its OCR data available directly on the > metadata > > > form. > > > > > > > > > > > > > I am imagining a staging folder, from which the documents are > processed > > > immediately. If after the initial processing no metadata are available > > > for the document, it is added to the postprocessing queue. When > finally > > > (manually) processed, those documents are removed from the queue. > > > > > > There should be some configuration options: > > > - Which metadata are required to be filled for a document to be able > to > > > leave > > > the queue? > > > - Should only documents missing the required metadata be added to the > > > queue or > > > just all (if postprocessing control for all processed documents is > > > desired)? > > > > > > > > I'm still wrapping my head around the post processing queue, do we > create a > > post processing attached or detached from the watch folder? > > Hopefully I understand your question. My vision is: > > - process all documents in the watch folder: > i.e. > - do OCR > - include the document in the database > - add the document to the post processing queue > - delete the document from the watch folder > > so that all documents are already available in the system, but should/can > be > post-processed to control/add metadata. > > So IIUC the question, the queue is independent from the watch folder. > > Probably we should also add an option when uploading via API, if the > document > should be added to this queue. > > Nice would be a configuration option to skip inclusion in the > post-processing > queue, if certain metadata are registered (e.g. specific metadata fields > contain values). > > > -- > > Mathias Behrle > PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6 > -- --- You received this message because you are subscribed to the Google Groups "Mayan EDMS" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
