On Jan 9, 2008, at 11:14 AM, Matt Accola wrote:
I have upgraded to 7.0p2 and am still seeing the creeping memory.
Well, as I said, this is a feature, not a bug.
I have tried the following...
1. Added the following statement to my .clp file:
(set-node-index-hash 1)
"1" is a little extreme -- a small prime like "13" would be a good
choice. Based on the below, I'd say you put this at the end of your
script. It sets a value that's used while rules are being compiled,
so it has to be called *before* a rule is parsed to have any effect
on that rule.
2. Changed the jess.TokenTree#clear() method to allow for debugging:
When I run my Rete execution in a loop I see the m_tokens array is
steadily increasing and the number of non-null values increasing.
The length of m_tokens, like the number of non-null TokenVector
objects, will eventually level off. It's being adjusted to account
for the "load factor" of the memory. The Rete memories are kind of
like hash tables, and if the individual buckets get too full, the
number of buckets is increased.
The only way to prevent this is to change the Jess source code as you
suggest below. When I put in the call to Arrays.fill() then the
memory
stops growing.
Right.
I am very hesitant to change the Jess source code as we use several
different usage patterns, some are transaction-oriented, some are more
knowledge-based.
Well, it's really a trivial change; basically what it does is put the
Rete memories back in their initial state before each run, so every
run starts off just like the first. You'll save a little memory -- to
the extent that the runs differ from one another -- at the expense of
doing more allocations.
I guess I don't understand the value of retaining the older tokens.
Not older tokens: space to store tokens. Check out the source code
to java.util.ArrayList. In particular, look at the "add()", "remove
()" and "clear()" methods. Note that add() can grow the internal
array used by ArrayList, but remove() *does not shrink it*. Nor does
clear(). Once the array grows, it doesn't shrink, on the theory that
if it grew to some size N in the past, it's probably going to grow
back to N in the future -- so why bother to delete and reallocate a
smaller array, only to turn around and delete and reallocate a larger
array later?
That's *exactly* what's going on here, albeit with a more complex
data structure.
Some questions I still have:
* What is the value of the retained tokens?
* Isn't the algorithm for releasing the tokens somewhat arbitrary?
I hope I cleared these two up. It's basically a bet -- borne out by
experiment -- that it will be faster to hang onto those arrays rather
than reallocate them later.
* How can I test for the upper limit on the array size you mention?
Well, running for a long time, basically. I actually forgot that the
memories do re-hashing these days; they didn't use to. That means
that technically there *is* no upper limit on the number of
TokenVectors; but in practice, the number will be limited by the
number of Tokens that are stored during any one run.
* Why does my setting for set-node-index-hash-value have little or no
effect?
Try putting it at the top of the program.
I am just trying to understand the behavior better and avoid
modifications to the source code if at all possible.
We could add the alternative behavior I described to the official
codebase as a selectable alternative. You'd be right in thinking that
putting a time/space tradeoff like this in the user's hands would be
a good thing.
If you can guide
me on a way to keep the memory down within the current Jess source
code
I would be grateful.
That's really another matter altogether. If you're having problems
with total memory usage, then we might want to look at redesigning
one or more rules that are generating too many partial matches.
Modifying the rules might greatly reduce the number of Tokens that
need to be stored in these memories, thus reducing the number and
size of TokenVectors that have to be created.
Matt
-----Original Message-----
From: Ernest Friedman-Hill [mailto:[EMAIL PROTECTED]
Sent: Tuesday, January 08, 2008 3:21 PM
To: Matt Accola
Subject: Re: JESS: TokenTree memory leak?
On Jan 8, 2008, at 4:06 PM, Matt Accola wrote:
We should have upgraded the library long ago, it has just fallen
lower on priorities. I am trying that now but need to do a
regression.
We are calling the Rete engine repetitively in our application and
at the end of each evaluation we call the reset() method to return
the engine to empty state, ready for the next call. Is there any
way to force all memory to be reclaimed and override this memory
retention you described?
There's no built-in way, but it would be an easy mod. In jess/
TokenTree.java, there's a method like
final void clear() {
for (int i=0; i< m_hash; i++)
if (m_tokens[i] != null)
m_tokens[i].clear();
m_size = 0;
}
You could just change it to something like
final void clear() {
java.util.Arrays.fill(m_tokens, null);
m_size = 0;
}
Note that there's a performance tradeoff here; the next run will very
likely turn around and re-allocate many of these same objects. Jess
is lazy-initializing the contents of that m_tokens array, one element
at a time; when all the m_tokens arrays of all the TokenTree objects
are non-null, that's the upper limit I mentioned. Which of the
possible array elements is populated on any one run is going to
depend entirely on the data for that run.
Confidentiality Notice:
**********************************************
This E-mail and any attachments thereto, are intended only for use
by the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended
recipient of this E-mail, you are hereby notified any
dissemination, distribution or copying of this E-mail, and any
attachments thereto, is strictly prohibited. If you receive this E-
mail in error, please immediately notify me by reply E-mail or
telephone at (218) 723-7887 and permanently delete the original and
any copy of this E-mail, and any printout thereof.
---------------------------------------------------------
Ernest Friedman-Hill
Informatics & Decision Sciences Phone: (925) 294-2154
Sandia National Labs FAX: (925) 294-2234
PO Box 969, MS 9012 [EMAIL PROTECTED]
Livermore, CA 94550 http://www.jessrules.com
Confidentiality Notice:
**********************************************
This E-mail and any attachments thereto, are intended only for use
by the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended
recipient of this E-mail, you are hereby notified any
dissemination, distribution or copying of this E-mail, and any
attachments thereto, is strictly prohibited. If you receive this E-
mail in error, please immediately notify me by reply E-mail or
telephone at (218) 723-7887 and permanently delete the original and
any copy of this E-mail, and any printout thereof.
--------------------------------------------------------------------
To unsubscribe, send the words 'unsubscribe jess-users
[EMAIL PROTECTED]'
in the BODY of a message to [EMAIL PROTECTED], NOT to the list
(use your own address!) List problems? Notify owner-jess-
[EMAIL PROTECTED]
--------------------------------------------------------------------
---------------------------------------------------------
Ernest Friedman-Hill
Informatics & Decision Sciences Phone: (925) 294-2154
Sandia National Labs FAX: (925) 294-2234
PO Box 969, MS 9012 [EMAIL PROTECTED]
Livermore, CA 94550 http://www.jessrules.com
--------------------------------------------------------------------
To unsubscribe, send the words 'unsubscribe jess-users [EMAIL PROTECTED]'
in the BODY of a message to [EMAIL PROTECTED], NOT to the list
(use your own address!) List problems? Notify [EMAIL PROTECTED]
--------------------------------------------------------------------
