On 10.05.2017 21:38, Rick McGuire wrote:
>
> On Wed, May 10, 2017 at 2:55 PM, Rony <[email protected]
> <mailto:[email protected]>>
> wrote:
>
> Ok, what do you think then are possible causes for these crashes? How
> would one be ableshould
> one proceed to systematically debug this?
>
>
> I gave you suggestions several emails ago that you have completely ignored.
This is surprising as I have not ignored one single suggestion of yours!
As ooRexx is a black-box because of a missing specifications that define all
aspects of it, in case
of an exception that has interdependent causes it is extremely difficult and
extremely
time-consuming to debug such a crash.
Here is a brief log of what you have suggested and advised over the past two
and a half months with
respect to the ooRexx crash I reported in February:
* "Well, at first glance, it appears that the method is being passed an
invalid argument on
the SendMessageArray() call. This could be a corrupted value or maybe a
stale address of an
object that has already been garbage collected because it was not
properly protected. That's
probably where you should start looking." (2017-02-19, checked all of
bsf4oorexx-code over
again)
* "No, once you have passed the arguments into via SendMessageArray(),
the arguments are
automatically protected from garbage collection. However,
SendMessageArray() does not make a
copy of that array, but rather just uses the array directly. If you
make a call out to your
code and it reuses the array for another callback into Rexx, then you
will have clobbered
the original arguments. If the problem is occurring on the second call
to arg(arg()), then
it is likely something on the bsfrexx code is overwriting the argument
array. " (2017-02-19,
checked all of bsf4oorexx-code over again)
* "No, they should be protected as long as the context that allocated
them is active, unless
you've used ReleaseLocalReference() to indicate you are finished with
them. This sounds like
an overwrite problem. One thing you can try. Rather than use multiple
calls to arg(arg()),
use one call and assign the value to a local variable. If the problem
still occurs, then it
is a garbage collection problem. If the problem goes away, it is likely
an overwrite. "
(2017-02-20, followed your advice)
* "It looks like the directory object is getting garbage collected
prematurely. Does your code
use ReleaseLocalReference() on that object after it is allocated?"
(2017-02-21, checking out)
* "This looks like a classic GC problem, likely unrelated to the
arg(arg()) issue. I don't see
anything obvious from the stack trace that could be causing the error.
Debugging this
requires some pretty sophisticated usage of the debugger. How
repeatable is the error? These
types of problems are much easier to trouble shoot if they fail the
same way with repeated
runs." (2017-02-21)
* "Ok, the way I would debug this is to launch the program in the
debugger, hit F5 to run to
the exception. The problem is occurring because an invalid object
reference has gotten
stored in another object. This frequently occurs because there are some
windows where a
garbage collection even can occur between the time an object is created
and the reference
gets protected. This is a simple fix if we can find the window.
Fortunately, with thie stack trace you have here, we already have this
information. The
object holding the bad reference is a RexxActivation instance. So,
launch the program in the
debugger, if F5 to allow it to run until the exception, then inspect
the stack frame for
RexxActivation::live() to see what line is being executed when the
exception occurs. This
should give us a good hit for where to look next." (2017-02-21)
* "Ok, I was wrong. This is the original issue once again. An object in
the SendMessage()
arguments is a bad reference, so we're back to the original two
possibilities of either part
of the argument array was over written or the object was garbage
collected before it was
stored in the array.
So, inspect the top stack frame and not the value of the "this"
variable. That's the bad
reference. This *should* be the directly object allocated in the
bsfrexx code. Hopefully
this value is still in a local variable in the stackframe that calls
SendMessageArray(). If
the values are different, then this is a memory overwrite. If they are
the same, then the
object has been garbage collected and we need to figure out why."
(2017-02-21)
* "And the variable of interest is the variable "this". That is the bad
reference value. "
(2017-02-21)
* "It's not that simple. The original code had some attempts at this with
a special debug
version that attempted to detect any attempts at storing an invalid
reference into an
object. It was never all that reliable at detecting the problems and
the code involved was
never kept up-to-date with changes to memory management so it was
removed. Good debuggers
and debugging skills were a much better path to figuring out the
causes." (2017-02-21)
* "A memory overwrite was suggested as a possibility in every one of
them, so yeah, this could
be what was happening. " (2017-02-22)
* "The flurry of AttachThreads() and DetachThreads() might be the cause
of the GC issues. When
a DetachThread() occurs, the GC protections on any objects returned by
call on that context
are lost." (2017-03-14)
* "This error only is issued for an out of memory error, not for stack
space." (2017-04-26)
* "The memory manager will force any pending uninits to be run before it
gives up and issues
the out of resources error messages. I really suspect something in the
way your code is
creating the proxies and tracking them is preventing the objects from
being identified as
dead objects, so these objects will just keep accumulating. The more
live objects locked in
memory, the longer a GC cycle will take because there is a larger live
set to mark."
(2017-04-27)
* "Running the garbage collector will not force the uninit methods to be
run. When the garbage
collector runs, any newly dead objects with uninit methods are marked
as being ready to have
the uninit method run. At some later time, when the interpreter is at a
safe point, then the
objects with pending uninit methods will be processed. These are major
boundaries such as
return from a method or call or thread termination.
Since you are creating class objects, things can get delayed even
further since the class
object will not become "dead" until all of its instances are first
"dead". It then gets
moved to the uninit queue for processing." (2017-04-29)
* "I doubt this causes much of the delay, since these objects don't get
moved to the uninit
queue until the next time it is necessary to run the garbage collector.
It is highly
unlikely that more than a few objects would be affected by this, but
you might give it a
try and see what happens." (2017-04-29)
* "I want to verify something first. In this scenario, Rexx code makes a
call to a Java method
(i.e., the parser) which then makes a lot of callbacks all on the same
thread? If so, then I
think I might see where the problem could be. The code that handles
nested thread attaches
(that is, attaching to a thread that already exists) appears to hold on
to any allocated
objects until it returns to the original exit point (in your case, the
initial call to the
parser). This may take a little while to figure out the details of a
fix, but I at least
know where to start. You might want to open a bug indicating there
appears to be an
AttachThread()/DetachThread() memory leak." (2017-04-30)
* "The location of the problem appears to be the same as the
multi-threaded problem you
reported earlier. It is occurring because a bad argument to a method
invocation is getting
marked during garbage collection. You can used the debugger to poke
around and figure out
what this RexxActivation actually is (name of method or routine, the
package it is in,
etc.). That might be useful, but I'm starting to think there's
something else going on. Try
applying this patch and see if it makes a difference." (2017-05-04)
* "This is the same problem." (2017-05-04)
* "New version of the previous patch. This makes the termination an
little more consistent."
(2017-05-04, 14:42 MET)
That patch that you were able to come up with has fixed the memory leak
problem and almost
all garbage collector related crashes that I started to report in
February! Looking at the
patch it is impressive to see what needed to be changed to fix the
memory leaks and related
crashes.
* "Sorry, I'm not interested in getting that involved with this again. I
will answer specific
questions about how the code works, but that's the limit. I will NOT
get involved in
directly debugging these issues." (2017-05-10, after offering the
application that still
crashes ooRexx at the end and reporting that the 64-bit debug version
surfaces a crash at a
different location)
There is one (probably different) problem left, that needs to be tracked down
to learn about the
causes in order to become able to devise a fix.
As ooRexx has no documented specifications (except for its implementation) this
is extremely
difficult to debug for anyone else. Hence the reason why it is so important
that you lend your hand
for debugging such cases!
[N.B. for a user of a black box it is 1000-times as effortful to come up with
stripped down examples
that would cause the same exception in the black box. Testing literally
hundreds of variants, going
through the own - complex - code paths over and over in order to see whether
there is something that
is done wrongly outside of the black box. Believe me I have literally invested
all nights and all
weekends since February, any free minute I could get from my time-consuming day
job.]
> Just pasting stack traces from the crash and trying to throw the problem over
> the wall for someone
> else to solve is not a good approach to take. So far, you've ignored all of
> my previous attempts
> to educate you on what is going on. You need to show that you are making an
> effort and ask
> SPECIFIC questions about things.
Oh, I have been trying to be as specific as it has been possible for me! Yet,
ooRexx specifications
being a black-box with you as the implementor being the only one in the know,
it is next to
impossible to come up with more "specific" questions in such a situation,
believe it or not.
E.g. looking at the "this" variable in the Activation stack frame with
plentiful of (nested) members
of most I do not know what their purpose is and how they intertwine with the
rest of the (runtime)
system, it is next to impossible to infer anything meaningful without a
knowledge what this all
supposed to be (in order to become able to assess what variables look crippled
and inferring then
why that became possible in the first place).
> I have given you rather extensive explanations about garbage
> collection/uninit/etc. in the past,
> but you are showing no sign that you actually bothered reading any of them.
Over the last years I have marked and collected any explanation of yours.
Unfortunately, they do not explain or shed any lights on the problem(s) at
hand. For instance, there
were no clues that would have allowed me to become able to understand the
problem at hand and come
up with the patch that fixed the memory leak problem (and also practically all
memory-related
problems).
There is at least one more problem that leads to a crash of ooRexx in this use
case. Therefore I
think it to be important to put all efforts into solving it one way or the
other.
However, I can understand looking at the time-frame debugging this until coming
up with a patch that
fixes almost everything, that one gets frustrated, if still the fix was not
fixing everything.
If there was a documented specification, we all could help much more to track
down such crashes and
maybe even become able to come up with efficient patches to fix them.
That is the reason why I think that at this stage (with these kind of errors)
it is important to
start analyzing ooRexx' implementation from the "specification view angle" to
infer its
specifications and document them for the community. And if you want to help the
community behind
this project, it would be great if you could start e.g. in blog-form to give
sketches of the
architecture and behaviour and interdependencies, such that over time it
becomes possible for the
community members to ask those "specific" questions that you wish us to ask.
---rony
>
>
>
> My problem currently is that I have no conception of the overall
> architecture (what would it
> be from a bird eye's view?), nor from the dynamics (what happens exactly
> at startup, when a
> Rexx instance gets created and when terminated, how do the garbage
> collector and uninit
> mechanism work - strategies, pre-/postconditions they apply?).
>
> Maybe this (interpreter documentation) is something the devlopers should
> start immediately
> while working on bugs/crashes, before adding any new functionalities to
> the interpreter?
>
> ---rony
>
> Rony G. Flatscher (mobil/e)
>
> Am 10.05.2017 um 19:41 schrieb Rick McGuire <[email protected]
> <mailto:[email protected]>>:
>
>> Sorry, I'm not interested in getting that involved with this again. I
>> will answer specific
>> questions about how the code works, but that's the limit. I will NOT get
>> involved in directly
>> debugging these issues.
>>
>> Rick
>>
>>
>>
>> On Wed, May 10, 2017 at 1:33 PM, Rony G. Flatscher
>> <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Rick, would it help, if I created a zip-archive to download with
>> directions how to start
>> the Rexx
>> application (using JavaFX)? If so, please let me know which bitness
>> you would need for the
>> bsf4oorexx.dll (pre-requisite is that you have Java 1.8 installed).
>> Also, I can make the
>> 32- and
>> 64-debug versions of ooRexx (from today's trunk with your fix for
>> the memory leak applied)
>> available, either as an NSIS installer or as a zip-archive of the
>> bin-directories.
>>
>> This might allow you to better debug the cause with Visual Studio or
>> the like. It is
>> strange that
>> this application causes different errors it seems, depending on the
>> bitness of the debug
>> version of
>> ooRexx.
>>
>> ---rony
>>
>> P.S.: All other JavaFX samples work flawlessly with the patched
>> version.
>>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Oorexx-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/oorexx-devel
