[OPEN-ILS-DEV] overdue and pre-due notice config
I'm beginning work on courtesy (predue) notices and a re-architecture of the existing overdue notice script for better templating and configuration. Attached is a sample configuration section (from opensrf.xml) I created for setting these up. I wanted to work backwards from the configuration to the code to make sure the requirements are accurately represented. This configuration is only for email notices and generating a local XML file (for sending to a vendor, creating local mail notices, etc.). This is not the full-blown notication system (telephony, etc.) described in http://open-ils.org/dokuwiki/doku.php?id=scratchpad:notification_systems=notices. However, I'd like to think that this is at least a step in that direction. For predue notices, there was a discussion of basing the notice date on a percentage of the total circulation duration. I toyed with that some and decided in the end to model predue notices on duration date ranges instead. It's just easier on the eyes. I'm open to discussion on that one, though. For 1.4.0, the plan is to have locale-aware templates, built with Perl's Template Toolkit. 1.4.X will introduce per-org-unit templates. Suggestions and comments on the configuration welcome. -b -- Bill Erickson | VP, Software Development Integration | Equinox Software, Inc. / The Evergreen Experts | phone: 877-OPEN-ILS (673-6457) | email: [EMAIL PROTECTED] | web: http://esilibrary.com notifications !-- global mail server settings -- smtp_serverlocalhost/smtp_server sender_address[EMAIL PROTECTED]/sender_address !-- Overdue notices -- overdue !-- optionally, you can define a sender address per notice type -- sender_address[EMAIL PROTECTED]/sender_address !-- The system can generate an XML file of overdue notices. This is the directory where they are stored. Files are named overdue.-MM-DD.xml -- notice_dir/openils/var/data/overdue/notice_dir notice !-- Notify at 7 days overdue -- notify_interval7 days/notify_interval !-- Options include always, noemail, and never. 'noemail' means a notice will be appended to the notice file only if the patron has no valid email address. -- file_appendnoemail/file_append !-- do we attempt email notification? -- email_notifytrue/email_notify !-- notice template file -- template/openils/var/data/templates/overdue_7day/template /notice notice notify_interval14 days/notify_interval file_appendalways/file_append email_notifytrue/email_notify template/openils/var/data/templates/overdue_14day/template /notice /overdue !-- Courtesy notices -- predue notice_dir/openils/var/data/predue/notice_dir notice !-- All circulations that circulate between 5 and 13 days -- circ_duration_range5 days,13 days/circ_duration_range !-- notify at 1 day before the due date -- notify_interval1 day/notify_interval file_appendfalse/file_append email_notifytrue/email_notify template/openils/var/data/templates/predue_1day/template /notice notice !-- All circulations that circulate for 14 or more days -- circ_duration_range14 days/circ_duration_range notify_interval2 day/notify_interval file_appendfalse/file_append email_notifytrue/email_notify template/openils/var/data/templates/predue_2day/template /notice /predue held_item_ready notice email_notifytrue/email_notify template/openils/var/data/templates/held_item_ready/template /notice /held_item_ready /notifications
Re: [OPEN-ILS-DEV] Adding opensrf-perl.pl to the autotools build in OpenSRF
Applied with thanks. 2008/8/6 Kevin Beswick [EMAIL PROTECTED]: Here is a patch to add the new opensrf-perl.pl script to the autotools build for OpenSRF. By the way, for future reference, what is the difference between dist_bin_SCRIPTS and bin_SCRIPTS? -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] overdue and pre-due notice config
Hi Bill: 2008/8/6 Bill Erickson [EMAIL PROTECTED]: I'm beginning work on courtesy (predue) notices and a re-architecture of the existing overdue notice script for better templating and configuration. Excellent! Attached is a sample configuration section (from opensrf.xml) I created for setting these up. I wanted to work backwards from the configuration to the code to make sure the requirements are accurately represented. This configuration is only for email notices and generating a local XML file (for sending to a vendor, creating local mail notices, etc.). This is not the full-blown notication system (telephony, etc.) described in http://open-ils.org/dokuwiki/doku.php?id=scratchpad:notification_systems=notices. However, I'd like to think that this is at least a step in that direction. For predue notices, there was a discussion of basing the notice date on a percentage of the total circulation duration. I toyed with that some and decided in the end to model predue notices on duration date ranges instead. It's just easier on the eyes. I'm open to discussion on that one, though. I think you've made the right choice, although I have some suggestions for the duration range configuration syntax (below). For 1.4.0, the plan is to have locale-aware templates, built with Perl's Template Toolkit. 1.4.X will introduce per-org-unit templates. Excellent! Suggestions and comments on the configuration welcome. Some general comments: It's not explicit, but I suspect this could be used/abused for items like laptops that might have a four hour circulation period, so that you could send an email 30 mins in advance to remind the user that it's due real soon now. For the pre-overdues, it would be nice to take advantage of the system's knowledge of the circ rule in play to offer a link to a renewal page (if applicable for each item) and specify the fines that will be accrued. I'm also hoping that this will group items together in some way, rather than generating one email per item. Specific comment on the configuration syntax: At first glance, I didn't find the circ_duration_range syntax particularly intuitive. 5 days, 13 days looked like an OR set of items, and 14 days on its own looked like, well, just 14 days, not 14 days or more. At the risk of making the XML even more verbose, but perhaps more intuitive, I would propose something more like the following: !-- rather than 5 days, 13 days notation -- circ_duration_range from5 days/from to13 days/to /circ_duration_range !-- for items with a circ duration of exactly 28 days, set from == to -- circ_duration_range from28 days/from to28 days/to /circ_duration_range !-- if to element is missing, then the range extends to the end of time -- circ_duration_range from14 days/from /circ_duration_range !-- if from element is missing, then the range extends from the start (0 mins) to to -- circ_duration_range to5 days/to /circ_duration_range !-- and if circ_duration_range holds neither from nor to, then it encompasses all circ_duration ranges -- circ_duration_range/circ_duration_range -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] overdue and pre-due notice config
On Wed, 06 Aug 2008 13:00:48 -0400, Dan Scott [EMAIL PROTECTED] wrote: Hi Bill: 2008/8/6 Bill Erickson [EMAIL PROTECTED]: [snip] Some general comments: It's not explicit, but I suspect this could be used/abused for items like laptops that might have a four hour circulation period, so that you could send an email 30 mins in advance to remind the user that it's due real soon now. That should be feasible. I'm thinking we need a command-line param to the notifier which tells it to generate notices for circs that have a notify_interval of = 1 day. That way, CRON can be set up to run the notifier every hour, but once a day it will set the switch to generate notices for = 1 day notify_interval items. In other words, if I'm supposed to get a notice today about an item due tomorrow, I won't get a notice every hour. For the pre-overdues, it would be nice to take advantage of the system's knowledge of the circ rule in play to offer a link to a renewal page (if applicable for each item) and specify the fines that will be accrued. I like that. I'm also hoping that this will group items together in some way, rather than generating one email per item. The current behavior for overdues and planned behavior for this next batch of code is to generate one email per type of notice. E.g. 1 email for all items 7 days overdue, 1 email for all items 14 days overdue, etc. This allows for separate templates for each type. Is this grouped enough or do you think putting all notices into a single email is crucial? Specific comment on the configuration syntax: At first glance, I didn't find the circ_duration_range syntax particularly intuitive. 5 days, 13 days looked like an OR set of items, and 14 days on its own looked like, well, just 14 days, not 14 days or more. At the risk of making the XML even more verbose, but perhaps more intuitive, I would propose something more like the following: !-- rather than 5 days, 13 days notation -- circ_duration_range from5 days/from to13 days/to /circ_duration_range !-- for items with a circ duration of exactly 28 days, set from == to -- circ_duration_range from28 days/from to28 days/to /circ_duration_range !-- if to element is missing, then the range extends to the end of time -- circ_duration_range from14 days/from /circ_duration_range !-- if from element is missing, then the range extends from the start (0 mins) to to -- circ_duration_range to5 days/to /circ_duration_range !-- and if circ_duration_range holds neither from nor to, then it encompasses all circ_duration ranges -- circ_duration_range/circ_duration_range That works for me. Thanks, Dan! -b -- Bill Erickson | VP, Software Development Integration | Equinox Software, Inc. / The Evergreen Experts | phone: 877-OPEN-ILS (673-6457) | email: [EMAIL PROTECTED] | web: http://esilibrary.com
[OPEN-ILS-DEV] Large Bibliographic Imports
I have about 960 000 bibliographic records I need to import into an Evergreen system. The database server is dual quad-core Xeons with 24GB of RAM. Currently, I've split the bibliographic records into 8 batches of ~120K records each, did the marc_bre/direct_ingest/parellel_pg_loader dance, but one of those files has been chugging along in psql now for more than 16 hours. How long should I expect these files to take? Would more smaller files load more quickly in terms of total time for the same full recordset? I notice that the insert into metabib.full_rec seems to be taking by far the longest. It does have more records than any of the other pieces to import, but the time taken still seems disproportionate. I notice that metabib.full_rec has this trigger -- zzz_update_materialized_simple_record_tgr AFTER INSERT OR DELETE OR UPDATE ON metabib.full_rec FOR EACH ROW EXECUTE PROCEDURE reporter.simple_rec_sync(). Is the COPY INTO calling this trigger every time I copy in a new record? If so, can I remove to trigger to defer this update, and do it en masse afterward? Would it be quicker? Just looking for any tips I can use to increase the loading speed of huge-ish datasets. Cheers, Brandon == Brandon W. Uhlman, Systems Consultant Public Library Services Branch Ministry of Education Government of British Columbia 850-605 Robson Street Vancouver, BC V6B 5J3 Phone: (604) 660-2972 E-mail: [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: [OPEN-ILS-DEV] Large Bibliographic Imports
On Wed, Aug 6, 2008 at 4:30 PM, Brandon W. Uhlman [EMAIL PROTECTED] wrote: I have about 960 000 bibliographic records I need to import into an Evergreen system. The database server is dual quad-core Xeons with 24GB of RAM. Currently, I've split the bibliographic records into 8 batches of ~120K records each, did the marc_bre/direct_ingest/parellel_pg_loader dance, but one of those files has been chugging along in psql now for more than 16 hours. How long should I expect these files to take? Would more smaller files load more quickly in terms of total time for the same full recordset? I notice that the insert into metabib.full_rec seems to be taking by far the longest. It does have more records than any of the other pieces to import, but the time taken still seems disproportionate. I notice that metabib.full_rec has this trigger -- zzz_update_materialized_simple_record_tgr AFTER INSERT OR DELETE OR UPDATE ON metabib.full_rec FOR EACH ROW EXECUTE PROCEDURE reporter.simple_rec_sync(). Is the COPY INTO calling this trigger every time I copy in a new record? If so, can I remove to trigger to defer this update, and do it en masse afterward? Would it be quicker? Just looking for any tips I can use to increase the loading speed of huge-ish datasets. There is a new materialized view (a of 1.2.2) based on the trigger you noted. However, as of 1.2.2.1 there are management functions to dis/enable that when needed. So, inside the transaction started in the main pg_loader sql output file and before the first COPY statement, you'll want to add: SELECT reporter.disable_materialized_simple_record_trigger(); and then at the end, before you commit, add: SELECT reporter.enable_materialized_simple_record_trigger(); The latter will take a few minutes (perhaps 30) but not the 16 hours you're suffering through right now. -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: [EMAIL PROTECTED] | web: http://www.esilibrary.com
Re: [OPEN-ILS-DEV] Large Bibliographic Imports
Hey Brandon: The full text indexes are absolutely the key - check out this thread from July 2nd: http://list.georgialibraries.org/pipermail/open-ils-dev/2008-July/003265.html - I think it addresses your questions for the most part. And yeah, as Mike notes, we really should document that in the appropriate section of the wiki. Especially as I'm about to embark on a refresh of our several-million records :0 Dan 2008/8/6 Brandon W. Uhlman [EMAIL PROTECTED]: I have about 960 000 bibliographic records I need to import into an Evergreen system. The database server is dual quad-core Xeons with 24GB of RAM. Currently, I've split the bibliographic records into 8 batches of ~120K records each, did the marc_bre/direct_ingest/parellel_pg_loader dance, but one of those files has been chugging along in psql now for more than 16 hours. How long should I expect these files to take? Would more smaller files load more quickly in terms of total time for the same full recordset? I notice that the insert into metabib.full_rec seems to be taking by far the longest. It does have more records than any of the other pieces to import, but the time taken still seems disproportionate. I notice that metabib.full_rec has this trigger -- zzz_update_materialized_simple_record_tgr AFTER INSERT OR DELETE OR UPDATE ON metabib.full_rec FOR EACH ROW EXECUTE PROCEDURE reporter.simple_rec_sync(). Is the COPY INTO calling this trigger every time I copy in a new record? If so, can I remove to trigger to defer this update, and do it en masse afterward? Would it be quicker? Just looking for any tips I can use to increase the loading speed of huge-ish datasets. Cheers, Brandon == Brandon W. Uhlman, Systems Consultant Public Library Services Branch Ministry of Education Government of British Columbia 850-605 Robson Street Vancouver, BC V6B 5J3 Phone: (604) 660-2972 E-mail: [EMAIL PROTECTED] [EMAIL PROTECTED] -- Dan Scott Laurentian University
Re: [OPEN-ILS-DEV] Large Bibliographic Imports
Thanks, Dan (and also Mike). Great tip! I think documenting this is a good piece, for sure. Is there any reason we also wouldn't want to include it in the default SQL generated by pg_loader/parallel_pg_loader? If we're concerned about it automatically being called without checking the data, we could include it as a comment in pg_loader_output.sql, just we currently do the commit, as a visual reminder. ~B Quoting Dan Scott [EMAIL PROTECTED]: Hey Brandon: The full text indexes are absolutely the key - check out this thread from July 2nd: http://list.georgialibraries.org/pipermail/open-ils-dev/2008-July/003265.html - I think it addresses your questions for the most part. And yeah, as Mike notes, we really should document that in the appropriate section of the wiki. Especially as I'm about to embark on a refresh of our several-million records :0 Dan 2008/8/6 Brandon W. Uhlman [EMAIL PROTECTED]: I have about 960 000 bibliographic records I need to import into an Evergreen system. The database server is dual quad-core Xeons with 24GB of RAM. Currently, I've split the bibliographic records into 8 batches of ~120K records each, did the marc_bre/direct_ingest/parellel_pg_loader dance, but one of those files has been chugging along in psql now for more than 16 hours. How long should I expect these files to take? Would more smaller files load more quickly in terms of total time for the same full recordset? I notice that the insert into metabib.full_rec seems to be taking by far the longest. It does have more records than any of the other pieces to import, but the time taken still seems disproportionate. I notice that metabib.full_rec has this trigger -- zzz_update_materialized_simple_record_tgr AFTER INSERT OR DELETE OR UPDATE ON metabib.full_rec FOR EACH ROW EXECUTE PROCEDURE reporter.simple_rec_sync(). Is the COPY INTO calling this trigger every time I copy in a new record? If so, can I remove to trigger to defer this update, and do it en masse afterward? Would it be quicker? Just looking for any tips I can use to increase the loading speed of huge-ish datasets. Cheers, Brandon == Brandon W. Uhlman, Systems Consultant Public Library Services Branch Ministry of Education Government of British Columbia 850-605 Robson Street Vancouver, BC V6B 5J3 Phone: (604) 660-2972 E-mail: [EMAIL PROTECTED] [EMAIL PROTECTED] -- Dan Scott Laurentian University == Brandon W. Uhlman, Systems Consultant Public Library Services Branch Ministry of Education Government of British Columbia 605 Robson Street, 5th Floor Vancouver, BC V6B 5J3 Phone: (604) 660-2972 E-mail: [EMAIL PROTECTED] [EMAIL PROTECTED]
