Re: Novice Profiling Question.
I had this problem for about seven years. It had to be tackled, but until just now I had more pressing problems revolving around getting my application to work. I feared learning the profiler and solving this problem. I thought the process would be a distraction that I didn't want to deal with; I shouldn't have - the profiler is a doddle, and it is now indeed proving to me that I have solved my problem. I see memory being consumed and released like waves lapping a shore. All the tips given from this group were informative. and they were given without the ego one finds on StackOverflow. Good kudos for the NetBeans community. Owen. On Sun, 25 Nov 2018 at 14:18, Owen Thomas wrote: > I'd just like to let you know that I solved my CPU load and socket > stability problems by removing a call I was making to System.gc() I put in > when I was trying to solve the problem earlier. I've been running my > application now for about 30 minutes, and the profiler shows that no > instances survive one gc. My Ubuntu system monitor shows no significant > memory consumption so far. > > Your tip was golden. Thanks. > > On Sun, 25 Nov 2018 at 11:37, Owen Thomas > wrote: > >> Hello Leo. >> >> On Sun, 25 Nov 2018 at 08:29, Leo Donahue wrote: >> >>> Maybe this helps? >>> >>> SER10-J >>> >>> https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=88487787 >>> >>> >> Still early days yet, but I think a heart-felt thank you might be in >> order here. >> >> Resetting the output stream after writing may have released transmitters >> that remain cached after writing. Doing this appears to have made the >> transmitters eligible of garbage collection. >> >> The number of "Live Objects" rarely goes above 4 and the "Live Bytes" >> column rarely goes into three figures for any sample displayed in the >> objects view on the profiler. I still see the number of "Allocated Objects" >> appears to rise without limit, and so I'm not sure what this column is >> displaying: is it the total number of objects created during the time that >> the view has been sampling? >> >> I notice that my CPU's have become considerably more active (all four of >> them now run at about 80% instead of at about 30%), but I put this down to >> the extra load being generated by the increased activity of the garbage >> collector. I also notice a significant increase in the amount of "socket >> resetting" required by my application. Again, I believe this is due to the >> garbage collectors staking the CPU's away from my processes which allow >> only a small window of delay before closing the current socket and creating >> a new socket to replace it; this is a designed intent, and one that may add >> even more load onto the CPU's. >> >> I do still see memory rising slowly on the system monitor, and so I >> deduce that I haven't eliminated all my memory leak problems with this >> apparent fix... pity. >> >> The main thing is that this appears to be progress that wouldn't have >> happened without your tip. Perhaps, however, I still have more questions to >> ask. >> >> Thank you, >> >> Owen. >> >>
Re: Novice Profiling Question.
I'd just like to let you know that I solved my CPU load and socket stability problems by removing a call I was making to System.gc() I put in when I was trying to solve the problem earlier. I've been running my application now for about 30 minutes, and the profiler shows that no instances survive one gc. My Ubuntu system monitor shows no significant memory consumption so far. Your tip was golden. Thanks. On Sun, 25 Nov 2018 at 11:37, Owen Thomas wrote: > Hello Leo. > > On Sun, 25 Nov 2018 at 08:29, Leo Donahue wrote: > >> Maybe this helps? >> >> SER10-J >> >> https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=88487787 >> >> > Still early days yet, but I think a heart-felt thank you might be in order > here. > > Resetting the output stream after writing may have released transmitters > that remain cached after writing. Doing this appears to have made the > transmitters eligible of garbage collection. > > The number of "Live Objects" rarely goes above 4 and the "Live Bytes" > column rarely goes into three figures for any sample displayed in the > objects view on the profiler. I still see the number of "Allocated Objects" > appears to rise without limit, and so I'm not sure what this column is > displaying: is it the total number of objects created during the time that > the view has been sampling? > > I notice that my CPU's have become considerably more active (all four of > them now run at about 80% instead of at about 30%), but I put this down to > the extra load being generated by the increased activity of the garbage > collector. I also notice a significant increase in the amount of "socket > resetting" required by my application. Again, I believe this is due to the > garbage collectors staking the CPU's away from my processes which allow > only a small window of delay before closing the current socket and creating > a new socket to replace it; this is a designed intent, and one that may add > even more load onto the CPU's. > > I do still see memory rising slowly on the system monitor, and so I deduce > that I haven't eliminated all my memory leak problems with this apparent > fix... pity. > > The main thing is that this appears to be progress that wouldn't have > happened without your tip. Perhaps, however, I still have more questions to > ask. > > Thank you, > > Owen. > >
Re: Novice Profiling Question.
Hello Leo. On Sun, 25 Nov 2018 at 08:29, Leo Donahue wrote: > Maybe this helps? > > SER10-J > > https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=88487787 > > Still early days yet, but I think a heart-felt thank you might be in order here. Resetting the output stream after writing may have released transmitters that remain cached after writing. Doing this appears to have made the transmitters eligible of garbage collection. The number of "Live Objects" rarely goes above 4 and the "Live Bytes" column rarely goes into three figures for any sample displayed in the objects view on the profiler. I still see the number of "Allocated Objects" appears to rise without limit, and so I'm not sure what this column is displaying: is it the total number of objects created during the time that the view has been sampling? I notice that my CPU's have become considerably more active (all four of them now run at about 80% instead of at about 30%), but I put this down to the extra load being generated by the increased activity of the garbage collector. I also notice a significant increase in the amount of "socket resetting" required by my application. Again, I believe this is due to the garbage collectors staking the CPU's away from my processes which allow only a small window of delay before closing the current socket and creating a new socket to replace it; this is a designed intent, and one that may add even more load onto the CPU's. I do still see memory rising slowly on the system monitor, and so I deduce that I haven't eliminated all my memory leak problems with this apparent fix... pity. The main thing is that this appears to be progress that wouldn't have happened without your tip. Perhaps, however, I still have more questions to ask. Thank you, Owen.
Re: Novice Profiling Question.
Maybe this helps? SER10-J https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=88487787 On Sat, Nov 24, 2018, 04:41 Owen Thomas The profiler appears to work when I select the view display only objects > of the single class I'm interested in. > > Summarily, my application is composed of about 600 source files. About 80% > code for exceptions; ~130 files code for the application logic itself. It > is written entirely in Java SE. The running instance being profiled > communicates with another instance of the same application. The two > instances exchange serialised objects called transmitters. Transmitters are > swapped between instances about every 88 milliseconds; two transmitters > (one created and the other de-serialised) come into existence at a rate of > about 11 hertz. The transmitters, once created and exchanged, have no > further use and should be destroyed. They are not, and the two running > instances can consume all available physical memory if left running for > long enough. > > If you can make a program do something like that, you might be able to > replicate the problem I have been having with the profiler. I may even have > pointed you to some problem with my use of the profiler in what I've > disclosed above. Let me know you suspect that this may have happened. > > I know I could perhaps be more helpful, but I don't want to jump down a > rabbit hole trying to debug a profiler when what I want to do is to find > out why my application is leaking memory. Additionally, my code is > proprietary, and I'm certainly not about to hand out working copies to play > with. I'm sure you understand, and I'm sorry I can't offer anything better > than this. NetBeans has been good to me, and so I hope this is of some help > for the community in some way. > > > On Sat, 24 Nov 2018 at 20:34, Geertjan Wielenga > wrote: > >> No one is going to be able to help you unless you can provide the >> application together with steps to reproduce your problems or if you can >> show those problems occurring with a sample application that comes with >> NetBeans, e.g., the Anagram Game. >> >> Gj >> >> On Sat, Nov 24, 2018 at 10:16 AM Owen Thomas >> wrote: >> >>> Looks like I'm getting confused and dismayed with my attempts to run the >>> profiler. Any help from any NetBeans user would be appreciated. >>> >>> Thanks, >>> >>> Owen. >>> >>> -- Forwarded message - >>> From: Owen Thomas >>> Date: Sat, 24 Nov 2018 at 20:11 >>> Subject: Re: Novice Profiling Question. >>> To: >>> >>> >>> This is strange... >>> >>> When I start my application from the profiler with the profile view set >>> to Objects, I see the objects that make up my application. I can switch the >>> view to Telemetry, and the telemetry view appears. But when I switch the >>> view back to Objects, nothing appears. >>> >>> Instead, I get the following in my application console output: >>> >>> Profiler Agent Error: Exception when handling command from client: >>> java.io.IOException: JFluid wire protocol error: code does not >>> correspond to command or response. Value -1 >>> Profiler Agent: Connection with agent closed >>> >>> I ensured the application was completely compiled by selecting Clean and >>> Build from the projects menu before I ran it. I'm finding this perhaps a >>> little confusing... >>> >>> >>> On Sat, 24 Nov 2018 at 17:40, Vic Ricker wrote: >>> >>>> Did you check the objects being created? Do they interact with >>>> anything else? (passing references to themselves, or their methods, maybe >>>> lambdas? to other things that can't be garbage collected?) Doesn't the >>>> profiler tell what's holding the references? >>>> On 11/24/18 12:34 AM, Owen Thomas wrote: >>>> >>>> Thanks for the tip. I have just tried it. Unfortunately (a bit >>>> surprisingly too) it hasn't worked. I call gc every 10 seconds. The number >>>> of objects I see still seems to grow without limit. >>>> >>>> On Sat, 24 Nov 2018 at 15:47, Vic Ricker wrote: >>>> >>>>> Did you try forcing garbage collection with System.gc()? Those >>>>> allocations may not be leaked. It may be that the GC hasn't got to >>>>> them >>>>> yet. >>>>> >>>>> >>>>> On 11/23/18 11:19 PM, Owen Thomas wrote: >>>>> > Hello NetBeans Users. >>
Re: Novice Profiling Question.
The profiler appears to work when I select the view display only objects of the single class I'm interested in. Summarily, my application is composed of about 600 source files. About 80% code for exceptions; ~130 files code for the application logic itself. It is written entirely in Java SE. The running instance being profiled communicates with another instance of the same application. The two instances exchange serialised objects called transmitters. Transmitters are swapped between instances about every 88 milliseconds; two transmitters (one created and the other de-serialised) come into existence at a rate of about 11 hertz. The transmitters, once created and exchanged, have no further use and should be destroyed. They are not, and the two running instances can consume all available physical memory if left running for long enough. If you can make a program do something like that, you might be able to replicate the problem I have been having with the profiler. I may even have pointed you to some problem with my use of the profiler in what I've disclosed above. Let me know you suspect that this may have happened. I know I could perhaps be more helpful, but I don't want to jump down a rabbit hole trying to debug a profiler when what I want to do is to find out why my application is leaking memory. Additionally, my code is proprietary, and I'm certainly not about to hand out working copies to play with. I'm sure you understand, and I'm sorry I can't offer anything better than this. NetBeans has been good to me, and so I hope this is of some help for the community in some way. On Sat, 24 Nov 2018 at 20:34, Geertjan Wielenga wrote: > No one is going to be able to help you unless you can provide the > application together with steps to reproduce your problems or if you can > show those problems occurring with a sample application that comes with > NetBeans, e.g., the Anagram Game. > > Gj > > On Sat, Nov 24, 2018 at 10:16 AM Owen Thomas > wrote: > >> Looks like I'm getting confused and dismayed with my attempts to run the >> profiler. Any help from any NetBeans user would be appreciated. >> >> Thanks, >> >> Owen. >> >> -- Forwarded message - >> From: Owen Thomas >> Date: Sat, 24 Nov 2018 at 20:11 >> Subject: Re: Novice Profiling Question. >> To: >> >> >> This is strange... >> >> When I start my application from the profiler with the profile view set >> to Objects, I see the objects that make up my application. I can switch the >> view to Telemetry, and the telemetry view appears. But when I switch the >> view back to Objects, nothing appears. >> >> Instead, I get the following in my application console output: >> >> Profiler Agent Error: Exception when handling command from client: >> java.io.IOException: JFluid wire protocol error: code does not correspond >> to command or response. Value -1 >> Profiler Agent: Connection with agent closed >> >> I ensured the application was completely compiled by selecting Clean and >> Build from the projects menu before I ran it. I'm finding this perhaps a >> little confusing... >> >> >> On Sat, 24 Nov 2018 at 17:40, Vic Ricker wrote: >> >>> Did you check the objects being created? Do they interact with anything >>> else? (passing references to themselves, or their methods, maybe lambdas? >>> to other things that can't be garbage collected?) Doesn't the profiler >>> tell what's holding the references? >>> On 11/24/18 12:34 AM, Owen Thomas wrote: >>> >>> Thanks for the tip. I have just tried it. Unfortunately (a bit >>> surprisingly too) it hasn't worked. I call gc every 10 seconds. The number >>> of objects I see still seems to grow without limit. >>> >>> On Sat, 24 Nov 2018 at 15:47, Vic Ricker wrote: >>> >>>> Did you try forcing garbage collection with System.gc()? Those >>>> allocations may not be leaked. It may be that the GC hasn't got to >>>> them >>>> yet. >>>> >>>> >>>> On 11/23/18 11:19 PM, Owen Thomas wrote: >>>> > Hello NetBeans Users. >>>> > >>>> > An application that I am developing seems to be slowly leaking >>>> memory, >>>> > and I have been given some advice that makes me think that the best >>>> > way to get to the bottom of this problem is by running an instance of >>>> > my application in the profile. >>>> > >>>> > I can do that, and as just as I thought, I am seeing particular type >>>> > types of objects being created but not being destroyed by the garbage >>>> > collector. I've walked though my code many times, and as far as I can >>>> > see, never do I assign the object to a variable that has any scope >>>> > other than method scope. I can't see why they wouldn't be gc'd, but >>>> > indeed this is not happening, and I am wondering if this is because I >>>> > am setting these objects to a variable that persists beyond method >>>> scope. >>>> > >>>> > My question (I suppose it's very simple), is this: how do I get the >>>> > references to objects I see in my profiler? >>>> >>>
Re: Novice Profiling Question.
No one is going to be able to help you unless you can provide the application together with steps to reproduce your problems or if you can show those problems occurring with a sample application that comes with NetBeans, e.g., the Anagram Game. Gj On Sat, Nov 24, 2018 at 10:16 AM Owen Thomas wrote: > Looks like I'm getting confused and dismayed with my attempts to run the > profiler. Any help from any NetBeans user would be appreciated. > > Thanks, > > Owen. > > -- Forwarded message - > From: Owen Thomas > Date: Sat, 24 Nov 2018 at 20:11 > Subject: Re: Novice Profiling Question. > To: > > > This is strange... > > When I start my application from the profiler with the profile view set to > Objects, I see the objects that make up my application. I can switch the > view to Telemetry, and the telemetry view appears. But when I switch the > view back to Objects, nothing appears. > > Instead, I get the following in my application console output: > > Profiler Agent Error: Exception when handling command from client: > java.io.IOException: JFluid wire protocol error: code does not correspond > to command or response. Value -1 > Profiler Agent: Connection with agent closed > > I ensured the application was completely compiled by selecting Clean and > Build from the projects menu before I ran it. I'm finding this perhaps a > little confusing... > > > On Sat, 24 Nov 2018 at 17:40, Vic Ricker wrote: > >> Did you check the objects being created? Do they interact with anything >> else? (passing references to themselves, or their methods, maybe lambdas? >> to other things that can't be garbage collected?) Doesn't the profiler >> tell what's holding the references? >> On 11/24/18 12:34 AM, Owen Thomas wrote: >> >> Thanks for the tip. I have just tried it. Unfortunately (a bit >> surprisingly too) it hasn't worked. I call gc every 10 seconds. The number >> of objects I see still seems to grow without limit. >> >> On Sat, 24 Nov 2018 at 15:47, Vic Ricker wrote: >> >>> Did you try forcing garbage collection with System.gc()? Those >>> allocations may not be leaked. It may be that the GC hasn't got to them >>> yet. >>> >>> >>> On 11/23/18 11:19 PM, Owen Thomas wrote: >>> > Hello NetBeans Users. >>> > >>> > An application that I am developing seems to be slowly leaking memory, >>> > and I have been given some advice that makes me think that the best >>> > way to get to the bottom of this problem is by running an instance of >>> > my application in the profile. >>> > >>> > I can do that, and as just as I thought, I am seeing particular type >>> > types of objects being created but not being destroyed by the garbage >>> > collector. I've walked though my code many times, and as far as I can >>> > see, never do I assign the object to a variable that has any scope >>> > other than method scope. I can't see why they wouldn't be gc'd, but >>> > indeed this is not happening, and I am wondering if this is because I >>> > am setting these objects to a variable that persists beyond method >>> scope. >>> > >>> > My question (I suppose it's very simple), is this: how do I get the >>> > references to objects I see in my profiler? >>> >>
