Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
Zoran> but I can sync this to SF. If you would like to give it a Zoran> a second try, and report problems you get, I may be of help. Yes please. -- Jeff Davis http://xarg.net mailto:[EMAIL PROTECTED] -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 19:16, Andrew Piskorski wrote: > On Fri, Feb 25, 2005 at 12:58:45PM -0500, Dossy Shiobara wrote: > > > Zoran's "ttrace" extension might help here, but "ttrace" doesn't give > > you the performance boost of Tcl's parsing and byte compiling on first > > execution, and subsequent executions are just calls into the bytecode. > > I'm not sure what you mean there. I haven't looked at the code, but > my understanding is that ttrace uses the Tcl "unknown" proc to look up > and copy procs from a server-wide store into the local interp, when > each proc is first called. Thus, you take a performance hit on the > first call of each proc, but from that point onward calling that proc > works exactly the same way as it would in stock AOLserver without > ttrace. Is that correct? > This is correct. And, in my experience, I have not noticed any speed penalties. OK, others may get different metrics (depending on the app). What you get is: about 1/3 of the memory footprint and much faster thread creation. To startup one thread/interp fully loaded, we need about 2 sec on a 2.4 GHZ Linux. With ttrace it is about 50 msec. Still a lot of time but it is all Tcl now. If rewritten in C, this could be even less. > ttrace looks like a win-win situation to me. If you want certain > procs pre-loaded into the interp, it should be feasible to add some > sort of ttrace hook to greedily install those procs into every new > interp. And if you used that hook for ALL procs, every single one, > then you'd basically be back to exactly how stock AOLserver behaves > without ttrace, no? ttrace is already doing this: ttrace::preloadregister procedures to be preloaded always Those will be loaded always and other will be loaded by the "unknown" first time they are referenced. Zoran -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 19:31, Jeff Davis wrote: > Speaking of ttrace, is the sourceforge code the latest and greatest? > > I attempted to get it to work with openacs but a few things we do > seemed to give it indigestion (some proc renames and such). I have > been meaning to try again but have not gotten to it. > > I think it would be a tremendous win for us since we do things like load > all the test procs and create procs only used at install time etc. > And I would really like to get it working. > I have it working for about a year with very good results. Admitently, I have not hit all the posibilities and there might still be problems there allright. The SF version is little bit neglected lately (I have made some changes privately in order to better support XOTcl introspection) but I can sync this to SF. If you would like to give it a a second try, and report problems you get, I may be of help. Zoran -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
Speaking of ttrace, is the sourceforge code the latest and greatest? I attempted to get it to work with openacs but a few things we do seemed to give it indigestion (some proc renames and such). I have been meaning to try again but have not gotten to it. I think it would be a tremendous win for us since we do things like load all the test procs and create procs only used at install time etc. And I would really like to get it working. -- Jeff Davis http://xarg.net mailto:[EMAIL PROTECTED] -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
> > The real garbage-collection is a tough issue. Hence not tackled by the > > AS yet. So, what's here (ns_cleanup) works but not realy universally. > > Jeff Hobbs has mentioned that Tcl interp cloning work on one > of the Tcl 8.3.x branches several times now. Jeff, how fast > is that stuff, or how fast could it be made? Could it > conceivably be fast enough to make it practical to simply > dump and re-clone the WHOLE Tcl interp for each and every > AOLserver connection? Yes, I would think so. The work was done in porting Tcl to specialized routers where each incoming call would create another interp to respond. The Tcl_CloneInterp worked all down at the C level, even copying bytecodes. I would think that is the best possible solution, but it isn't the easiest. The 8.3 branch that has it was designed to work in a single-threaded environment. Other tweaks were made to save memory (this was for a router, after all), that you would not want. Fortunately those should be orthogonal, but making the whole Tcl_CloneInterp robust for the HEAD would take some time I imagine (at least a few weeks). Jeff -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Fri, Feb 25, 2005 at 12:58:45PM -0500, Dossy Shiobara wrote: > Zoran's "ttrace" extension might help here, but "ttrace" doesn't give > you the performance boost of Tcl's parsing and byte compiling on first > execution, and subsequent executions are just calls into the bytecode. I'm not sure what you mean there. I haven't looked at the code, but my understanding is that ttrace uses the Tcl "unknown" proc to look up and copy procs from a server-wide store into the local interp, when each proc is first called. Thus, you take a performance hit on the first call of each proc, but from that point onward calling that proc works exactly the same way as it would in stock AOLserver without ttrace. Is that correct? ttrace looks like a win-win situation to me. If you want certain procs pre-loaded into the interp, it should be feasible to add some sort of ttrace hook to greedily install those procs into every new interp. And if you used that hook for ALL procs, every single one, then you'd basically be back to exactly how stock AOLserver behaves without ttrace, no? -- Andrew Piskorski <[EMAIL PROTECTED]> http://www.piskorski.com/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 18:44, Dossy Shiobara wrote:
> Great! I suggested this a while back and you can read Rolf's responses
> in the thread. See if he's changed his mind since then.
>
As already mentioned in this thread, a kind of dom introspection
would actually be much cleaner and more powerful solution:
foreach doc [dom documents] {
$doc delete
}
kind of thing. Same what you'd do with
foreach chan [file channels] {
close $chan
}
Hm?
--
With kind regards,
Mit besten Gruessen,
Meilleures Salutations,
___
Zoran Vasiljevic, Vice President, CTO
E-mail: mailto:[EMAIL PROTECTED]
Web:http://www.archiware.com
Phone: +49 89 319 084 98
Fax:+49 89 319 084 99
S-mail: ArchiWare(R) GmbH
Gutenbergstr. 3
D-85716 Munich, Germany
___
PresSTORE(R) - Backup Synchronize Archive
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Fri, Feb 25, 2005 at 09:44:59AM -0800, Jeff Hobbs wrote: > Eh? Tcl interpreters are pretty lightweight, in terms of cost > to create and kill. This is the timing on Windows with 8.4.9 > That is 3 milliseconds to create and delete the interpreter. > > When AOLserver initializes a new thread, how much of that > > time is spent in setting up the Tcl interp (and all its > > procs)? Am I correct in assuming that it is the vast majority? > > Now this of course is something different. The init of the > thread, and the way AOLServer does it, is where you will find a > good chunk of time taken up. When you say "init of the thread", what do you mean exactly? Copying all the procs to the Tcl interpreter? (I would call that part of initializing the interp, not the thread.) Or something else? Hm, if all the overhead is in copying the procs to the interp, then maybe Zoran's ttrace, besides its other advantages, would also let you cheaply destroy and re-create Tcl interps at will in AOLserver? -- Andrew Piskorski <[EMAIL PROTECTED]> http://www.piskorski.com/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 18:42, Andrew Piskorski wrote: > On Fri, Feb 25, 2005 at 06:08:05PM +0100, Zoran Vasiljevic wrote: > > > The real garbage-collection is a tough issue. Hence not tackled by > > the AS yet. So, what's here (ns_cleanup) works but not realy universally. > > Jeff Hobbs has mentioned that Tcl interp cloning work on one of the > Tcl 8.3.x branches several times now. Jeff, how fast is that stuff, > or how fast could it be made? Could it conceivably be fast enough to > make it practical to simply dump and re-clone the WHOLE Tcl interp for > each and every AOLserver connection? I do not think this is thread-safe. I recall looking in that code some time ago... > Both, I think. And if only one of the two, the private namespace > looks more important, as that will prevent the user from > unintentionally smashing over important Tcl procs like "string". Lets see what we can do about it... -- With kind regards, Mit besten Gruessen, Meilleures Salutations, ___ Zoran Vasiljevic, Vice President, CTO E-mail: mailto:[EMAIL PROTECTED] Web:http://www.archiware.com Phone: +49 89 319 084 98 Fax:+49 89 319 084 99 S-mail: ArchiWare(R) GmbH Gutenbergstr. 3 D-85716 Munich, Germany ___ PresSTORE(R) - Backup Synchronize Archive -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Jeff Hobbs <[EMAIL PROTECTED]> wrote: > Eh? Tcl interpreters are pretty lightweight, in terms of cost > to create and kill. This is the timing on Windows with 8.4.9 > on a P4/2.8: ... > > That is 3 milliseconds to create and delete the interpreter. That doesn't include all of the Tcl procs that AOLserver loads into the interp. before it's actually ready to process a request, though. Zoran's "ttrace" extension might help here, but "ttrace" doesn't give you the performance boost of Tcl's parsing and byte compiling on first execution, and subsequent executions are just calls into the bytecode. > > When AOLserver initializes a new thread, how much of that > > time is spent in setting up the Tcl interp (and all its > > procs)? Am I correct in assuming that it is the vast majority? > > Now this of course is something different. The init of the > thread, and the way AOLServer does it, is where you will find a > good chunk of time taken up. Exactly. So, for AOLserver to continuously tear down and recreate Tcl interps is a huge time-suck. -- Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 07:27, Dossy Shiobara wrote:
> On 2005.02.25, Andrew Piskorski <[EMAIL PROTECTED]> wrote:
> > Moving all tDOM auto-created procs into their own namespace would also
> > make solving the memory leak problem above easier, no? Just remove
> > everything in that namespace when the conn ends.
>
> Absolutely. Right now we do linear searches (read: SLOW AS CRAP) using
> stuff like this:
>
> foreach proc [info proc domDoc*] {
> rename proc {}
> }
>
I'm using namespaces so cleanup is easy. Don't know how you would apply it to
your code, but creating a namespace for your commands and at the same time
registering a callback is the easiest solution. For example:
# ns_atclose wrapper. Must invoke once per connection
# to remove data from memory.
proc ::datastore::closeStore { store } {
if {![info exists ::datastore::store::${store}::atClose]} {
ns_atclose ::datastore::atClose $store
namespace eval ::datastore::store::${store} { set atClose 1 }
}
}
# callback procedure, invoked once conn closes or
# at end of scheduled procedure exec:
proc ::datastore::atClose { store } {
if {[namespace exists ::datastore::store::${store}]} {
namespace delete ::datastore::store::${store}
log Notice "Closed store (memory delete) $store"
return $store
}
}
Not shown is the fact that closeStore is called when you create a datastore.
Any procs put into the namespace after this will be deleted regardless of
what happens to the conn.
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 18:22, Andrew Piskorski wrote:
> On Fri, Feb 25, 2005 at 11:30:13AM -0500, Dossy Shiobara wrote:
>
> > So, we DO clean up channels, and Tcl global variables in the top-level
> > ("::") namespace, but not namespace variables -- those tend to be used
> > differently than Tcl globals, so they generally DO need to persist
> > between connections within an interp.
>
> Why? Why would ever you want namespace vars to persist in the Tcl
> interp for that one particular thread? As opposed to merely
> tolerating such persistence?
>
> > Yes, if you want to enforce full clean-up by dumping the Tcl interp
> > after processing each request, you could set connsperthread == 1,
> > but then you'd lose the
>
> Yuck. Tcl interps are often (maybe always?) FAR from light weight, so
> destroying and re-creating them constantly simply in order to re-set a
> bit of state seems like massive overkill.
>
> When AOLserver initializes a new thread, how much of that time is
> spent in setting up the Tcl interp (and all its procs)? Am I correct
> in assuming that it is the vast majority?
>
>
What Dossy is rightly saying is: you wouldn't want to drop
Tcl interp for each conn. This is correct and this is how
AS is doing things already.
The issue here is a "leak" of Tcl commands, namespaces, vars
etc, left in the interp after the connection processing.
This is *not* easy to handle. Therefore nobody really tackled
this issue. So, you do the best you can: leave the interp
until the conn *thread* goes down the drain, which will eventually
never happen and hence you have de-facto leaks(s). Or invent
a ns_cleanup thing which does *part* of the job.
So, this issue is about those resource "leaks". Not the real
memory leaks you would track with Purify or such. Rather
the "resource" leaks; things created but never destroyed.
tdom is not different than any Tcl extension. It will create
"things" for you and you should be responsible for destroying
"things" when not needed any more. Now, the life would be easier with
kindof [tdom cleanup] because it would do the right thing for
you with one shot. I agree with Dossy on this point.
But, thinking about this again... [tdom cleanup] would be
nice and short, yet pretty inflexible. I'd rather make something
like introspection in form of:
dom documents
dom commands
or such which will report all resources generated by tdom
package. Then you could decide what you'd like to do with them.
Some similar to [file channels] from Tcl.
Zoran
--
With kind regards,
Mit besten Gruessen,
Meilleures Salutations,
___
Zoran Vasiljevic, Vice President, CTO
E-mail: mailto:[EMAIL PROTECTED]
Web:http://www.archiware.com
Phone: +49 89 319 084 98
Fax:+49 89 319 084 99
S-mail: ArchiWare(R) GmbH
Gutenbergstr. 3
D-85716 Munich, Germany
___
PresSTORE(R) - Backup Synchronize Archive
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
Andrew Piskorski wrote:
> On Fri, Feb 25, 2005 at 11:30:13AM -0500, Dossy Shiobara wrote:
> > Yes, if you want to enforce full clean-up by dumping the Tcl interp
> > after processing each request, you could set connsperthread == 1, but
> > then you'd lose the
>
> Yuck. Tcl interps are often (maybe always?) FAR from light
> weight, so destroying and re-creating them constantly simply
> in order to re-set a bit of state seems like massive overkill.
Eh? Tcl interpreters are pretty lightweight, in terms of cost
to create and kill. This is the timing on Windows with 8.4.9
on a P4/2.8:
(Tcl) 49 % time {interp delete [interp create a]} 100
3370 microseconds per iteration
And because no self-respecting server would run on anything less
than an AMD64 running unix/linux, those timings are:
% time {interp delete [interp create a]} 100
2756 microseconds per iteration
That is 3 milliseconds to create and delete the interpreter.
> When AOLserver initializes a new thread, how much of that
> time is spent in setting up the Tcl interp (and all its
> procs)? Am I correct in assuming that it is the vast majority?
Now this of course is something different. The init of the
thread, and the way AOLServer does it, is where you will find a
good chunk of time taken up.
Jeff Hobbs, The Tcl Guy
http://www.ActiveState.com/, a division of Sophos
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > All resources in this interpreter not handled by the programmer > *or* by ns_cleanup will stay until this interp gets dumped. It will > never get dumped if the connection thread never exits. And it will > never exit if connsperthread == 0. Is this what we are talking about? Yes. And, for performance reasons, you WANT to set connsperthread == 0, which is the default. > > No, but tDOM could provide a way of letting AOLserver clean up > > tDOM-created objects. Currently, there is NO fail-proof way of doing > > the cleanup, which is why I asked for a [tdom cleanup] where tDOM could > > track what objects it's created and then clean them up when asked to. > > OK. This is clear. The [dom cleanup] kind of thing would be > feasible and practical. No doubt about it. I will see with > Rolf to get it in the CVS. Great! I suggested this a while back and you can read Rolf's responses in the thread. See if he's changed his mind since then. http://groups.yahoo.com/group/tdom/message/922 | > Basically, what we need is a list to be maintained in what is | > known as "connection-local storage" in AOLserver terminology of | > all the proc's created by tDOM, so that at the end of the | > connection, we can get a list of them all and then [rename] them | > away or Tcl_DeleteCommand() from C. This implies that | > everywhere these proc's are created, we would need to add them | > to the list maintained in CLS (conn-local storage), and when the | > proc delete traces execute, they would have to be removed. A | > simple Tcl_HashTable should work just fine for this. | > | > I'm going to try and find some time to implement this and submit | > a patch, but in the meantime -- do you have any objections to | > this approach? | | I'm not convinced, so far. Mostly because, I don't see the | problem, as described above. But feel free, to come up with a | concrete application scenario, to convince me. > > That's it. The suggestion is that by default tDOM should be creating > > the commands in a sub-namespace of "tdom" to avoid it polluting the > > caller's namespace. This would also make automated cleanup of > > tDOM-created objects possible. > > Well, this is not difficult to make. The only thing is how to > assure backward compatibility? I will discuss this with Rolf > and we will see what we can do about it... Backward-compatibility will be the tough one. appendFromScript could evaluate the script within the sub-namespace, but then calling procs that are not in that namespace may get messy. Or at least, ones in their own namespaces outside the top-level :: namespace, at least -- whatever isn't in Tcl's normal search path for procs. > So, basically, either [tdom cleanup] or tdom:: private namespace > or both... -- Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Fri, Feb 25, 2005 at 06:08:05PM +0100, Zoran Vasiljevic wrote: > The real garbage-collection is a tough issue. Hence not tackled by > the AS yet. So, what's here (ns_cleanup) works but not realy universally. Jeff Hobbs has mentioned that Tcl interp cloning work on one of the Tcl 8.3.x branches several times now. Jeff, how fast is that stuff, or how fast could it be made? Could it conceivably be fast enough to make it practical to simply dump and re-clone the WHOLE Tcl interp for each and every AOLserver connection? > So, basically, either [tdom cleanup] or tdom:: private namespace > or both... Both, I think. And if only one of the two, the private namespace looks more important, as that will prevent the user from unintentionally smashing over important Tcl procs like "string". -- Andrew Piskorski <[EMAIL PROTECTED]> http://www.piskorski.com/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Andrew Piskorski <[EMAIL PROTECTED]> wrote: > When AOLserver initializes a new thread, how much of that time is > spent in setting up the Tcl interp (and all its procs)? Am I correct > in assuming that it is the vast majority? Without actually profiling the code and measuring, my gut agrees with you. -- Dossy -- Dossy Shiobara mail: [EMAIL PROTECTED] Panoptic Computer Network web: http://www.panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Fri, Feb 25, 2005 at 11:30:13AM -0500, Dossy Shiobara wrote:
> So, we DO clean up channels, and Tcl global variables in the top-level
> ("::") namespace, but not namespace variables -- those tend to be used
> differently than Tcl globals, so they generally DO need to persist
> between connections within an interp.
Why? Why would ever you want namespace vars to persist in the Tcl
interp for that one particular thread? As opposed to merely
tolerating such persistence?
> Yes, if you want to enforce full clean-up by dumping the Tcl interp
> after processing each request, you could set connsperthread == 1,
> but then you'd lose the
Yuck. Tcl interps are often (maybe always?) FAR from light weight, so
destroying and re-creating them constantly simply in order to re-set a
bit of state seems like massive overkill.
When AOLserver initializes a new thread, how much of that time is
spent in setting up the Tcl interp (and all its procs)? Am I correct
in assuming that it is the vast majority?
It is easy to demonstrate that repeatedly destroying and re-creating
AOLserver conn threads absolutely hoses OpenACS performance. However,
much of that is probably because OpenACS does various sorts of caching
PREDICATED on the assumption that threads and Tcl interps stick around
for a long time. E.g., templating system Tcl/ADP pairs are compiled
to Tcl code, and stored per-thread.
--
Andrew Piskorski <[EMAIL PROTECTED]>
http://www.piskorski.com/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 17:30, Dossy Shiobara wrote:
> nscp 1> info body ns_cleanup
This is a half-baked approach still arround from the non-namespaced
Tcl age. Attempt to clean up what you can... Sure something
not really deserving the garbage-collection name but works
sometimes... I recall several threads on the aolserver list
asking for a better garbage collector...
> So, we DO clean up channels, and Tcl global variables in the top-level
> ("::") namespace, but not namespace variables -- those tend to be used
> differently than Tcl globals, so they generally DO need to persist
> between connections within an interp.
They do not. It is just practically unfeasible (performace wise) to remember
all interp resources before the processing and afterwards walk arround and
see what has changed (and how) and then rectify this. This is the reason.
The real garbage-collection is a tough issue. Hence not tackled by
the AS yet. So, what's here (ns_cleanup) works but not realy universally.
>
> AOLserver really wins by reusing interps in a pool, which is why the
> default of "connsperthread == 0" is important. Yes, if you want to
> enforce full clean-up by dumping the Tcl interp after processing each
> request, you could set connsperthread == 1, but then you'd lose the
> performance gain of having already "warmed up" Tcl interps in a pool
> waiting to be used. And, as I said, we recently discovered a memory
> leak situation when setting connsperthread != 0.
Ah, I think we both speak about the same thing, but use different
notation. While the connection thread lives, it holds Tcl interpreter.
All resources in this interpreter not handled by the programmer
*or* by ns_cleanup will stay until this interp gets dumped. It will
never get dumped if the connection thread never exits. And it will
never exit if connsperthread == 0. Is this what we are talking about?
> No, but tDOM could provide a way of letting AOLserver clean up
> tDOM-created objects. Currently, there is NO fail-proof way of doing
> the cleanup, which is why I asked for a [tdom cleanup] where tDOM could
> track what objects it's created and then clean them up when asked to.
OK. This is clear. The [dom cleanup] kind of thing would be
feasible and practical. No doubt about it. I will see with
Rolf to get it in the CVS.
>
> Do it with "string" and not "junk". Why should it behave differently
> based on the proc name you choose? This is a bug.
You asked for it:
lexxsrv:nscp 1> rename string ""
lexxsrv:nscp 2> string
Connection closed by foreign host.
Is this a bug? The thing is: "unknown" is using Tcl string command
and you get weird output when renaming some Tcl built-in commands.
So, no [string} command in hash-table triggers unknown, which uses
[string] which is not there: bang: you blow the stack (AS) or get
this:
[EMAIL PROTECTED]:~/rel-1-5-0-branch/dev/lexxapps> tclsh
% rename string ""
% string
self-referential recursion in "unknown" for command "string"
>
> That's it. The suggestion is that by default tDOM should be creating
> the commands in a sub-namespace of "tdom" to avoid it polluting the
> caller's namespace. This would also make automated cleanup of
> tDOM-created objects possible.
Well, this is not difficult to make. The only thing is how to
assure backward compatibility? I will discuss this with Rolf
and we will see what we can do about it...
So, basically, either [tdom cleanup] or tdom:: private namespace
or both...
--
With kind regards,
Mit besten Gruessen,
Meilleures Salutations,
___
Zoran Vasiljevic, Vice President, CTO
E-mail: mailto:[EMAIL PROTECTED]
Web:http://www.archiware.com
Phone: +49 89 319 084 98
Fax:+49 89 319 084 99
S-mail: ArchiWare(R) GmbH
Gutenbergstr. 3
D-85716 Munich, Germany
___
PresSTORE(R) - Backup Synchronize Archive
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote:
> If you, with every page, create new Tcl commands, yes, they will
> stay in memory (interp) and never get garbage collected until this
> interp is dumped. But if the connection thread is sticky, then
> this will not happen, of course. I wouldn't call this a leak, though.
> This is true for any other Tcl resource: channels, namespaced
> variables, namespaced procedures etc... The AS garbage-collection
> mechanism on those things is poor and incomplete (admit: problem
> is not easy to solve w/o programmer involvement).
nscp 1> info body ns_cleanup
ns_cleanupchans; # close files
ns_cleanupvars; # dump globals
ns_set cleanup; # free sets
ns_http cleanup; # abort any http request
ns_ictl cleanup; # internal cleanup (e.g,. Ns_TclRegisterDefer's)
I asked for a [tdom cleanup] so we could add it to AOLserver's
[ns_cleanup] backin September 2004:
http://groups.yahoo.com/group/tdom/message/918
So, we DO clean up channels, and Tcl global variables in the top-level
("::") namespace, but not namespace variables -- those tend to be used
differently than Tcl globals, so they generally DO need to persist
between connections within an interp.
AOLserver really wins by reusing interps in a pool, which is why the
default of "connsperthread == 0" is important. Yes, if you want to
enforce full clean-up by dumping the Tcl interp after processing each
request, you could set connsperthread == 1, but then you'd lose the
performance gain of having already "warmed up" Tcl interps in a pool
waiting to be used. And, as I said, we recently discovered a memory
leak situation when setting connsperthread != 0.
> Hence: this is not a tdom leak. This is the AS garbage-collector
> issue and cannot be workaround'ed by the tdom implementation.
No, but tDOM could provide a way of letting AOLserver clean up
tDOM-created objects. Currently, there is NO fail-proof way of doing
the cleanup, which is why I asked for a [tdom cleanup] where tDOM could
track what objects it's created and then clean them up when asked to.
> lexxsrv:nscp 7> dom createNodeCmd elementNode junk
> ::junk
> lexxsrv:nscp 8> junk
> called outside domNode context
> lexxsrv:nscp 9> rename junk ""
>
> lexxsrv:nscp 10> junk
> invalid command name "junk"
>
> Where is the problem?
Do it with "string" and not "junk". Why should it behave differently
based on the proc name you choose? This is a bug.
> Is this what you're looking for:
>
> lexxsrv:nscp 12> namespace eval junk {dom createNodeCmd elementNode string}
> ::junk::string
> lexxsrv:nscp 13> string
> wrong # args: should be "string option arg ?arg ...?"
> lexxsrv:nscp 14> ::junk::string
> called outside domNode context
>
> Or, is it something else I'm missing here?
That's it. The suggestion is that by default tDOM should be creating
the commands in a sub-namespace of "tdom" to avoid it polluting the
caller's namespace. This would also make automated cleanup of
tDOM-created objects possible.
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 15:28, Dossy Shiobara wrote:
> On 2005.02.25, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote:
> >
> > All procs are dumped by Tcl interpreter being dumped when the
> > connection thread exits.
>
> This is not true, unless you set connsperthread != 0, which we've
> recently discovered also has a big memory leak issue which I'm still
> tracking down. But, under normal operation (where connsperthread == 0),
> only the global variables get dumped when the conn thread exits. Tcl
> procs stay behind, which is why tDOM documents (which create Tcl procs)
> linger and "leak" memory if they're not explicitly deleted.
You're saying that the Tcl interp still lives when a connection
thread exits? This is not true. When the connection thread exits,
the associated interp (if any) is dumped.
What you rather mean is: when the *connection* exits (closes).
This is something entirely different...
If you, with every page, create new Tcl commands, yes, they will
stay in memory (interp) and never get garbage collected until this
interp is dumped. But if the connection thread is sticky, then
this will not happen, of course. I wouldn't call this a leak, though.
This is true for any other Tcl resource: channels, namespaced
variables, namespaced procedures etc... The AS garbage-collection
mechanism on those things is poor and incomplete (admit: problem
is not easy to solve w/o programmer involvement).
Hence: this is not a tdom leak. This is the AS garbage-collector
issue and cannot be workaround'ed by the tdom implementation.
> Perhaps what I meant was to illustrate the fact that tDOM can indeed
> clobber Tcl procs, any Tcl proc. It doesn't create these commands in a
> ::dom sub-namespace or somewhere safe -- this is just important to know.
> It surprised me; maybe it's surprised other people, or will surprise
> other people in the future.
Well, admit, this fact should be stated more clearly in the docs.
> Well, I was expecting this:
>
> $ tclsh
> % proc foo {} {}
> % foo
> % rename foo {}
> % foo
> invalid command name "foo"
>
> I wasn't expecting this:
>
> self-referential recursion in "unknown" for command "foo"
>
> Is this not a bug? Why can I not safely "delete" new tDOM-created procs
> by renaming them to the empty string like I can with ordinary Tcl procs?
lexxsrv:nscp 7> dom createNodeCmd elementNode junk
::junk
lexxsrv:nscp 8> junk
called outside domNode context
lexxsrv:nscp 9> rename junk ""
lexxsrv:nscp 10> junk
invalid command name "junk"
Where is the problem?
> I actually didn't want to. I just wanted to be able to use the
> "appendFromScript" feature of tDOM but my document has a node called
> "" so I had to do:
>
> $doc createNodeCmd elementNode string
>
> I did NOT want it to clobber the Tcl ::string proc. Because of this,
> I could NOT use the createNodeCmd/appendFromString. I *could* have
> worked around it with something like this, I suppose:
Is this what you're looking for:
lexxsrv:nscp 12> namespace eval junk {dom createNodeCmd elementNode string}
::junk::string
lexxsrv:nscp 13> string
wrong # args: should be "string option arg ?arg ...?"
lexxsrv:nscp 14> ::junk::string
called outside domNode context
Or, is it something else I'm missing here?
--
With kind regards,
Mit besten Gruessen,
Meilleures Salutations,
___
Zoran Vasiljevic, Vice President, CTO
E-mail: mailto:[EMAIL PROTECTED]
Web:http://www.archiware.com
Phone: +49 89 319 084 98
Fax:+49 89 319 084 99
S-mail: ArchiWare(R) GmbH
Gutenbergstr. 3
D-85716 Munich, Germany
___
PresSTORE(R) - Backup Synchronize Archive
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Andrew Piskorski <[EMAIL PROTECTED]> wrote:
> Moving all tDOM auto-created procs into their own namespace would also
> make solving the memory leak problem above easier, no? Just remove
> everything in that namespace when the conn ends.
Absolutely. Right now we do linear searches (read: SLOW AS CRAP) using
stuff like this:
foreach proc [info proc domDoc*] {
rename proc {}
}
But, since the proc name is user-specified when using createNodeCmd, you
can't use this trick to find them all. Having *ALL* tDOM-created procs
(both for domDocs and domNodes, and createNodeCmd, etc.) in a ::tdom
namespace would be terrific. It'd make the necessary cleanup trivially
easy.
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Fri, Feb 25, 2005 at 09:28:20AM -0500, Dossy Shiobara wrote: > But, under normal operation (where connsperthread == 0), only the > global variables get dumped when the conn thread exits. Tcl procs > stay behind, which is why tDOM documents (which create Tcl procs) > linger and "leak" memory if they're not explicitly deleted. Hm. Does tdom keep any sort of track of what procs it has created? If so, then at the end of each connection we just need to iterate over that list of procs and free them all, much as for database handles. > Perhaps what I meant was to illustrate the fact that tDOM can indeed > clobber Tcl procs, any Tcl proc. It doesn't create these commands in a > ::dom sub-namespace or somewhere safe -- this is just important to know. > It surprised me; maybe it's surprised other people, or will surprise > other people in the future. I haven't used tDOM at all, but let me see if I understand this: tDOM is at least semi-implicitly creating Tcl procs for the user, but it stuffs them into the global namespace rather than a tDOM namespace? Why, is there some good reason for that? What do the tDOM maintainers have to say abou this? Moving all tDOM auto-created procs into their own namespace would also make solving the memory leak problem above easier, no? Just remove everything in that namespace when the conn ends. -- Andrew Piskorski <[EMAIL PROTECTED]> http://www.piskorski.com/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On 2005.02.25, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote:
>
> All procs are dumped by Tcl interpreter being dumped when the
> connection thread exits.
This is not true, unless you set connsperthread != 0, which we've
recently discovered also has a big memory leak issue which I'm still
tracking down. But, under normal operation (where connsperthread == 0),
only the global variables get dumped when the conn thread exits. Tcl
procs stay behind, which is why tDOM documents (which create Tcl procs)
linger and "leak" memory if they're not explicitly deleted.
> > Another scary side effect of this:
> >
> > % dom createNodeCmd elementNode string
> > ::string
> >
> > % string is integer abcd
> > called outside domNode context
>
> This is not scary. If you do:
>
> proc string {} {
> return [string]
> }
>
> it will also junk your string command. The createNodeCmd just
> creates another Tcl command. If you happen to name it as some
> already existent Tcl command, it is your decision. Tcl will
> also not forbid you renaming existing commands, or?
Perhaps what I meant was to illustrate the fact that tDOM can indeed
clobber Tcl procs, any Tcl proc. It doesn't create these commands in a
::dom sub-namespace or somewhere safe -- this is just important to know.
It surprised me; maybe it's surprised other people, or will surprise
other people in the future.
> > OK, so where did the real Tcl [string] proc go? Poof, gone. This is
> > also a neat side-effect:
> >
> > % rename ::string {}
> > % string is integer abcd
> > self-referential recursion in "unknown" for command "string"
> >
> > Huh? Weird.
>
> Weird is what you're expecting. Please be aware that createNodeCmd
> creates a special Tcl command which can be only used from within
> the node context as the above error message correctly says:
>
> % string is integer abcd
> called outside domNode context
Well, I was expecting this:
$ tclsh
% proc foo {} {}
% foo
% rename foo {}
% foo
invalid command name "foo"
I wasn't expecting this:
self-referential recursion in "unknown" for command "foo"
Is this not a bug? Why can I not safely "delete" new tDOM-created procs
by renaming them to the empty string like I can with ordinary Tcl procs?
> > Anyhow -- I guess this a big caveat or warning to everyone using tDOM
> > with AOLserver if you're using [dom createNodeCmd] -- this way is
> > fraught with peril. Use at your own risk.
>
> This is not a big caveat and nobody should be warned. This is
> absolutely OK. You wanted to scratch/redefine the Tcl proc.
I actually didn't want to. I just wanted to be able to use the
"appendFromScript" feature of tDOM but my document has a node called
"" so I had to do:
$doc createNodeCmd elementNode string
I did NOT want it to clobber the Tcl ::string proc. Because of this,
I could NOT use the createNodeCmd/appendFromString. I *could* have
worked around it with something like this, I suppose:
namespace eval ::dom {
$doc createNodeCmd elementNode string
$doc createNodeCmd textNode text
$root appendFromScript {
string {text foo}
string {text bar}
}
}
Again, I think it's important to tell folks to use a sub-namespace if
they want to avoid clobbering procs with the same name. I didn't know
to do this originally, and I didn't find this technique mentioned in
either of these places:
http://www.tdom.org/dom.html
http://wiki.tcl.tk/8984
> Hence you must live with the side-effects. You kave the knife.
> How you use it, this is in your responsibility.
The knife is unnecessarily sharp but doesn't cut any better because of
it.
-- Dossy
--
Dossy Shiobara mail: [EMAIL PROTECTED]
Panoptic Computer Network web: http://www.panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tDOM "createNodeCmd" leading to leaks?
On Friday 25 February 2005 05:12, Dossy Shiobara wrote:
> I started playing with the tDOM "createNodeCmd" and "appendFromScript"
> idiom in AOLserver, and I think it (may?) be a cause for memory leakage.
I cannot confirm this. We are using it for about 5 years in
very demanding app and there is no memory leaks with tdom as far
as I know. I would have noticed this already years ago.
>
> In particular, the [dom createNodeCmd] thing creates an actual Tcl proc.
> AOLserver, at the end of request processing, doesn't "clean up" new
> procs that were defined as part of the request. This would include
> these new procs defined using [dom createNodeCmd].
All procs are dumped by Tcl interpreter being dumped when the
connection thread exits.
> Another scary side effect of this:
>
> % dom createNodeCmd elementNode string
> ::string
>
> % string is integer abcd
> called outside domNode context
This is not scary. If you do:
proc string {} {
return [string]
}
it will also junk your string command. The createNodeCmd just
creates another Tcl command. If you happen to name it as some
already existent Tcl command, it is your decision. Tcl will
also not forbid you renaming existing commands, or?
>
> OK, so where did the real Tcl [string] proc go? Poof, gone. This is
> also a neat side-effect:
>
> % rename ::string {}
> % string is integer abcd
> self-referential recursion in "unknown" for command "string"
>
> Huh? Weird.
Weird is what you're expecting. Please be aware that createNodeCmd
creates a special Tcl command which can be only used from within
the node context as the above error message correctly says:
% string is integer abcd
called outside domNode context
>
> Anyhow -- I guess this a big caveat or warning to everyone using tDOM
> with AOLserver if you're using [dom createNodeCmd] -- this way is
> fraught with peril. Use at your own risk.
This is not a big caveat and nobody should be warned. This is
absolutely OK. You wanted to scratch/redefine the Tcl proc.
Hence you must live with the side-effects. You kave the knife.
How you use it, this is in your responsibility.
--
With kind regards,
Mit besten Gruessen,
Meilleures Salutations,
___
Zoran Vasiljevic, Vice President, CTO
E-mail: mailto:[EMAIL PROTECTED]
Web:http://www.archiware.com
Phone: +49 89 319 084 98
Fax:+49 89 319 084 99
S-mail: ArchiWare(R) GmbH
Gutenbergstr. 3
D-85716 Munich, Germany
___
PresSTORE(R) - Backup Synchronize Archive
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
