On Fri, Sep 10, 2010 at 11:03 PM, Thomas Berezansky <[email protected]> wrote:
I believe I have completed work on this particular project. For those
interested here but not following launchpad, I submitted a patch there:
https://bugs.launchpad.net/evergreen/+bug/635463
Thanks, Thomas.
Right off the bat I want to make it clear that because this is a big
feature addition, and as it stands contains large schema changes
(which I'd also like to discuss more -- there are parts that surprised
me), I can't see this getting into 2.0.
With that out of the way, here are some points that jumped out at me
looking at the patch (relevant sections pasted in-line), in no
particular order:
------------------------------------------------
** It looks like the circ mod test for inclusion now falls through **
@@ -182,15 +188,11 @@
CASE WHEN m.usr_age_upper_bound IS NOT NULL
THEN 0.5 ELSE 0 END DESC LOOP
IF current_mp.circ_modifier IS NOT NULL THEN
- CONTINUE WHEN current_mp.circ_modifier <>
item_object.circ_modifier OR item_object.circ_modifier IS NULL;
+ CONTINUE WHEN current_mp.circ_modifier <>
item_object.circ_modifier;
END IF;
IF current_mp.marc_type IS NOT NULL THEN
- IF item_object.circ_as_type IS NOT NULL THEN
- CONTINUE WHEN current_mp.marc_type <>
item_object.circ_as_type;
- ELSE
- CONTINUE WHEN current_mp.marc_type <>
rec_descriptor.item_type;
- END IF;
+ CONTINUE WHEN current_mp.marc_type <>
COALESCE(item_object.circ_as_type, rec_descriptor.item_type);
END IF;
IF current_mp.marc_form IS NOT NULL THEN
This wasn't discussed, and seems surprising to me. If the potential
matchpoint has a circ mod and the object either has none or it's is
different from what the matchpoint wants, then we should move on.
However, I like the COALESCE optimization of the marc_type bit! :)
It may be that you intended to simplify the test without changing the
semantics, but because of how NULL acts in a <> comparison (or any
comparison for that matter) it changed the behavior. Consider this
simplified example:
=# select case when 1 <> null then true else false end;
case
------
f
(1 row)
The equivalent of the above (the change in your patch) would /not/
trigger the CONTINUE (which it should), where as the equivalent of the
below (the original code) does, obviously.
=# select case when 1 <> null or null is null then true else false end;
case
------
t
(1 row)
------------------------------------------------
** Concern about conditional ordering and logic when breaking out of
the <<usergroup>> loop **
+ EXIT usergroup WHEN full_depth = false
+ AND matchpoint.circulate IS NOT NULL
+ AND matchpoint.duration_rule IS NOT NULL
+ AND matchpoint.recurring_fine_rule IS NOT NULL
+ AND matchpoint.max_fine_rule IS NOT NULL
+ AND hold_ratio_fallthrough OR (
+ matchpoint.total_copy_hold_ratio IS NOT NULL
+ AND
matchpoint.available_copy_hold_ratio IS NOT NULL
+ );
The OR'd parts there really need to be grouped together. But I'm also
concerned that it's not doing what is intended. IIUC, we want to
force looking for non-NULL ratios when hold_ratio_fallthrough is
enabled, else we just take what we have when we've filled in the four
required bits. That WHEN clause reads to me like, "we're done if: we
have all required parts AND ( either we have ratios OR we need to find
ratios )". Perhaps:
EXIT usergroup WHEN (
full_depth = false
AND matchpoint.circulate IS NOT NULL
AND matchpoint.duration_rule IS NOT NULL
AND matchpoint.recurring_fine_rule IS NOT NULL
AND matchpoint.max_fine_rule IS NOT NULL
) AND (
NOT hold_ratio_fallthrough
OR (matchpoint.total_copy_hold_ratio IS NOT NULL AND
matchpoint.available_copy_hold_ratio IS NOT NULL)
);
------------------------------------------------
** Result set coming from action.find_circ_matrix_matchpoint **
- RETURN matchpoint;
+ -- This is a successful matchpoint finding IF we have the
circulate button AND the rules
+ IF matchpoint.circulate IS NOT NULL AND matchpoint.duration_rule
IS NOT NULL AND matchpoint.recurring_fine_rule IS NOT NULL AND
matchpoint.max_fine_rule IS NOT NULL THEN
+ result.success = true;
+ result.matchpoint = matchpoint;
+ ELSE
+ result.success = false;
+ END IF;
+ -- For debug purposes, include the full considered row list regardless.
+ result.buildrows = row_list;
+ RETURN result;
The fact that we failed to find a complete constructed matchpoint is
certainly a failure case insofar as the database call is concerned.
However, we shouldn't simply fail to work. If there is no ultimate
"outer" matchpoint with all the information we need, the we need to
specify that somewhere. Either OU settings or global settings are
candidates for ultimate fall-back values. Another option, which
requires more magic in configuration but will require no changes to
the backend code you've presented, is to make a specific matchpoint
(with an id of, perhaps, 0) special and require it be tied to the
top-most OU, the top-most user group, and force it to have values for
the four fields we need.
------------------------------------------------
** Refactoring that complicates the code **
@@ -388,21 +435,16 @@
RETURN NEXT result;
END IF;
- -- Fail if the total copy-hold ratio is too low
- IF circ_test.total_copy_hold_ratio IS NOT NULL THEN
+ -- Fail if the total/available copy-hold ratio is too low
+ IF circ_test.total_copy_hold_ratio IS NOT NULL OR
circ_test.available_copy_hold_ratio IS NOT NULL THEN
SELECT INTO hold_ratio * FROM
action.copy_related_hold_stats(match_item);
- IF hold_ratio.total_copy_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio < circ_test.total_copy_hold_ratio THEN
+ IF circ_test.total_copy_hold_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio IS NOT NULL AND
hold_ratio.total_copy_ratio < circ_test.total_copy_hold_ratio THEN
result.fail_part :=
'config.circ_matrix_test.total_copy_hold_ratio';
result.success := FALSE;
done := TRUE;
RETURN NEXT result;
END IF;
- END IF;
-
- -- Fail if the available copy-hold ratio is too low
- IF circ_test.available_copy_hold_ratio IS NOT NULL THEN
- SELECT INTO hold_ratio * FROM
action.copy_related_hold_stats(match_item);
- IF hold_ratio.available_copy_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio < circ_test.available_copy_hold_ratio
THEN
+ IF circ_test.available_copy_hold_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio IS NOT NULL AND
hold_ratio.available_copy_ratio < circ_test.available_copy_hold_ratio
THEN
result.fail_part :=
'config.circ_matrix_test.available_copy_hold_ratio';
result.success := FALSE;
done := TRUE;
I don't see any functional change here, but it's certainly more
complicated to read and therefore maintain. I'd see this go back to
two blocks.
------------------------------------------------
** No changes to Circulate.pm included **
Certainly, because the return type of action.item_user_circ_test
changed, the Perl will need to lean how to use it. A simple diff
oversight, I'm sure.
------------------------------------------------
** No upgrade script **
That'll be critical to applying the change to trunk.
---------------------------------
Finally, will you describe your testing methodology? Did you set up
uninherited (fully filled) matchpoints on old and new DBs and confirm
that they work the same? Do you have some sample inherited
matchpoints that show the different features of the new design,
including the switch for ratio inheritance?
Note, not everything I mention above needs to have a complete
resolution before this goes into trunk. Getting it in to trunk will
mean more folks can poke at it, so I don't want to stall that any
longer than we must, but at least some discussion (if not significant
code changes) are warranted before commit, IMO.
Thanks again, Thomas. I'm very excited about this!
--miker
Thomas Berezansky
Merrimack Valley Library Consortium
Quoting Mike Rylander <[email protected]>:
There as been a /lot/ of discussion surrounding this functionality on
IRC over the last week, just to let everyone here know that it's not
being ignored.
More below.
On Tue, Sep 7, 2010 at 4:29 PM, Thomas Berezansky <[email protected]> wrote:
My previous email included a patch that was, unfortunately, just the tip
of
the iceberg, so to speak. I have since figured out a lot of what needs to
be
done to do this on the backend.
However, in looking at what needs to be done, I came up with several
questions as to how to handle things.
Currently the circulation flag, duration rule, recurring fine rule, and
max
fine rule are on my list of things to have "fall through".
I am unsure as to whether or not to have the total copy hold ratio or the
available copy hold ratio values fall through or not. They are not
"endpoint" data, but at the same time may be considered to be an
important
part of the checks.
In that regard, I see three logical options:
1 - Don't have them fall through. If they aren't set on the first
matchpoint
found, they aren't set.
2 - Have them fall through until we run out of rules or they are filled,
like the four values above.
3 - Have them fall through, but only until we have filled the other four
values. Once the other four values are filled we officially stop checking
them.
I am partial to 1 or 3 in this case.
After discussion, option 3 seems sane and has gained general
consensus. One point I brought up was, how do you tell the system
that you do /not/ want to use an inherited test value and also don't
want to supply a value? IOW, what if you don't want to test a
copy-hold ratio and yet some broader rule does set a value for one of
these tests?
The answer is that a value of 0 (zero) acts the same as "don't test"
since all possible values will pass.
The other main issue I am seeing right now is the circ mod test. In
particular, the current system uses the matchpoint ID for the check. As
my
changes make it so that there is more than one matchpoint ID possibly
involved there are several options:
1 - Negate the check when there is more than one matchpoint used in the
final result
2 - Use only the first matchpoint found for the check
3 - Have it use all matchpoints that were considered for the check (were
valid and before we filled in our values)
4 - Have it use all matchpoints that contributed to the check (were
valid,
before we filled in our values, AND set at least one value)
I am partial to 3 in this case, as that would allow for creating a
matchpoint that would be specifically for triggering that limit, without
changing any of the resulting rules.
This question has not been addressed yet. I think that without
broader input, and in fact outside requests, we need to sit on this
one. I understand the argument for the flexibility of 3, but these
matchpoints and rulesets can already be very complicated, and the
potential for SAAD is high. Consider:
* matchpoint 1 (closest match) supplies all but max fine rule, and
has a circ-mod test for "book" that limits the user to 10 items out
* matchpoint 2 (fall-through by group) supplies the max fine rule,
and also has a circ-mod test for "book", but sets the limit at 5 items
out
Which do we honor? And, like the question about copy-hold ratios, how
do we say "do NOT test per-circ-mod items out at all"? There are no
magic values here, and the benefit of flexibility IMO is outweighed by
the complexity of getting it right from a user perspective.
Any thoughts?
All that said, I think this is some awesome work. Thanks, Thomas!
--miker
Thomas Berezansky
Merrimack Valley Library Consortium
Quoting Thomas Berezansky <[email protected]>:
Theory:
Make the things returned (circulate, duration_rule,
recurring_fine_rule,
max_fine_rule) "fall through" when set to NULL.
The attached patch attempts to make this happen on the back end, but
provides no front end interface changes for configuring it.
IT HAS NOT BEEN TESTED, mainly because I don't want to screw with our
test system right now when others may be trying to test existing
functionality.
It also adds in the ability to pass in one more (optional) boolean
parameter to the function to return the entire list of rules used to
create the final result, intended for "debug/test" front end
functionality
to show what rules were considered in the fall-through checking.
Pros:
You can override a subset of those fields with a specific rule while
allowing broader rules to fill in the holes.
This may result in less duplication of information across rules, making
things easier to maintain.
Thus, this may result in less rules in general, and thus less
processing
time on sorting them overall.
Cons:
Manually figuring out the specifics of what will happen will take more
time/effort.
Changing a single rule may have a greater unintended effect on other
rules.
Staff would need training for when to have a rule fall through and when
to set it.
More time to return from the DB for any rule that is "falling through"
to
broader rules.
Examples for the following org tree:
CONS
-SYSA
--LIBC
--LIBD
-SYSB
--LIBE
--LIBF
Implementing the following "business" rules:
At the CONS level:
By default, everything circulates, uses DFLT_DUR duration, DFLT_RFINE
recurring fine, and DFLT_MFINE max fine.
Circ Modifier "book" uses the duration BOOK_DUR
Reference flagged materials don't circulate
At the SYSA level there are no special rules.
At the SYSB level the max fine should be SYSB_MFINE.
At the LIBC level the recurring fine is LIBC_RFINE
At the LIBD level circ modifier "book" uses the DFLT_DUR duration
instead
of "BOOK_DUR"
At the LIBE level reference flagged materials circulate.
At the LIBF level there are no special rules.
The current method would require the following circ rules to implement
those business rules:
CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
CONS NULL NULL TRUE DFLT_DUR DFLT_RFINE
DFLT_MFINE
CONS NULL TRUE FALSE DFLT_DUR DFLT_RFINE
DFLT_MFINE
CONS book NULL TRUE BOOK_DUR DFLT_RFINE
DFLT_MFINE
CONS book TRUE FALSE BOOK_DUR DFLT_RFINE
DFLT_MFINE
SYSB NULL NULL TRUE DFLT_DUR DFLT_RFINE
SYSB_MFINE
SYSB NULL TRUE FALSE DFLT_DUR DFLT_RFINE
SYSB_MFINE
SYSB book NULL TRUE BOOK_DUR DFLT_RFINE
SYSB_MFINE
SYSB book TRUE FALSE BOOK_DUR DFLT_RFINE
SYSB_MFINE
LIBC NULL NULL TRUE DFLT_DUR LIBC_RFINE
DFLT_MFINE
LIBC NULL TRUE FALSE DFLT_DUR LIBC_RFINE
DFLT_MFINE
LIBC book NULL TRUE BOOK_DUR LIBC_RFINE
DFLT_MFINE
LIBC book TRUE FALSE BOOK_DUR LIBC_RFINE
DFLT_MFINE
LIBD book NULL TRUE DFLT_DUR DFLT_RFINE
DFLT_MFINE
LIBD book TRUE FALSE DFLT_DUR DFLT_RFINE
DFLT_MFINE
LIBE NULL NULL TRUE DFLT_DUR DFLT_RFINE
SYSB_MFINE
LIBE book NULL TRUE BOOK_DUR DFLT_RFINE
SYSB_MFINE
16 circ rules total.
The new method would require the following circ rules to implement
those
business rules:
CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
CONS NULL NULL TRUE DFLT_DUR DFLT_RFINE
DFLT_MFINE
CONS book NULL NULL BOOK_DUR NULL NULL
CONS NULL TRUE FALSE NULL NULL NULL
SYSB NULL NULL NULL NULL NULL
SYSB_MFINE
LIBC NULL NULL NULL NULL LIBC_RFINE NULL
LIBD book NULL NULL DFLT_DUR NULL NULL
LIBE NULL TRUE TRUE NULL NULL NULL
7 circ rules total.
Starting with the above, lets assume that SYSA wants to change their
recurring fine to SYSA_RFINE.
LIBC's recurring fine is to be unchanged.
The current method requires the following changes:
ADD the following entries:
CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
SYSA NULL NULL TRUE DFLT_DUR SYSA_RFINE
DFLT_MFINE
SYSA NULL TRUE FALSE DFLT_DUR SYSA_RFINE
DFLT_MFINE
SYSA book NULL TRUE BOOK_DUR SYSA_RFINE
DFLT_MFINE
SYSA book TRUE FALSE BOOK_DUR SYSA_RFINE
DFLT_MFINE
UPDATE the LIBD entries:
CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
LIBD book NULL TRUE DFLT_DUR SYSA_RFINE
DFLT_MFINE
LIBD book TRUE FALSE DFLT_DUR SYSA_RFINE
DFLT_MFINE
4 rules added, 2 changed, total is now 20 rules.
The new method would require the following changes:
ADD the following entry:
CIRC_LIB CIRC_MOD REFERENCE CIRC? DURATION_RULE RECURRING_FINE MAX_FINE
SYSA NULL NULL NULL NULL SYSA_RFINE NULL
1 rule added, 0 changed, total is now 8 rules.
Thomas Berezansky
Merrimack Valley Library Consortium
--
Mike Rylander
| VP, Research and Design
| Equinox Software, Inc. / The Evergreen Experts
| phone: 1-877-OPEN-ILS (673-6457)
| email: [email protected]
| web: http://www.esilibrary.com
--
Mike Rylander
| VP, Research and Design
| Equinox Software, Inc. / The Evergreen Experts
| phone: 1-877-OPEN-ILS (673-6457)
| email: [email protected]
| web: http://www.esilibrary.com
