Here's some additional info about one of the failure modes I'm seeing. I kept cranking up the memory limits until I now have the limits at default =4192 max=16388. Some channels are handled ok by this tool now.
Others fail pretty quickly while clearing the first target channel. One thing I see that concerns me are processes I see running Java that don't have the increased memory allocations. Like this one: root 3595 3349 6 14:32 ? 00:00:13 /usr/bin/java -Dibm.dst.compatibility=true -Xms1024m -Xmx4096m -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client64/lib:/usr/lib/oracle/11.2/client/lib -classpath /usr/share/java/tanukiwrapper.jar:/usr/share/rhn/classes:/usr/share/java/struts.jar:/usr/share/java/jfreechart.jar:/usr/share/java/jpam.jar:/usr/share/java/javamail.jar:/usr/share/java/axis/axis-ant.jar:/usr/share/java/quartz.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-beanutils.jar:/usr/share/java/ojdbc14.jar:/usr/share/java/jta.jar:/usr/share/java/concurrent.jar:/usr/share/rhn/lib/spacewalk-asm.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-collections.jar:/usr/share/java/taglibs-standard.jar:/usr/share/java/axis/axis.jar:/usr/share/java/xalan-j2.jar:/usr/share/java/commons-validator.jar:/usr/share/java/asm/asm-attrs.jar:/usr/share/java/jaf.jar:/usr/share/java/jdom.jar:/usr/share/java/oro.jar:/usr/share/rhn/lib/rhn.jar:/usr/share/java/redstone-xmlrpc.jar:/usr/share/java/oscache.jar:/usr/share/java/log4j.jar:/usr/share/java/wsdl4j.jar:/usr/share/java/jcommon.jar:/usr/share/java/commons-el.jar:/usr/share/java/taglibs-core.jar:/usr/share/java/commons-lang.jar:/usr/share/java/commons-digester.jar:/usr/share/java/jasper5-runtime.jar:/usr/share/java/jspapi.jar:/usr/share/java/c3p0.jar:/usr/share/java/sitemesh.jar:/usr/share/java/jasper5-compiler.jar:/usr/share/java/axis/saaj.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-discovery.jar:/usr/share/java/xml-commons-apis.jar:/usr/share/java/axis/jaxrpc.jar:/usr/share/java/commons-cli.jar:/usr/share/java/bcel.jar:/usr/share/java/antlr.jar:/usr/share/java/xerces-j2.jar:/usr/share/java/hibernate3.jar:/usr/share/java/cglib.jar:/usr/share/java/dom4j.jar:/usr/share/rhn/lib/java-branding.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/slf4j/jcl.jar:/usr/share/java/simple-core.jar:/usr/share/java/commons-dbcp.jar:/usr/share/java/commons-pool.jar:/usr/share/java/quartz-oracle.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/mchange-commons.jar:/usr/share/java/commons-io.jar -Dwrapper.key=Ny4YUY7QmPOi432B -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=3349 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=3 com.redhat.rhn.taskomatic.core.TaskomaticDaemon What is the configuration file that controls the memory allocation for this process? I already modified /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf: # Initial Java Heap Size (in MB) wrapper.java.initmemory=4196 # Maximum Java Heap Size (in MB) wrapper.java.maxmemory=16388 If the process I show is supposed to be controlled by this file, the memory allocation specified is being ignored, or applied in some way that doesn't match with the process shown. : [email protected] [mailto:[email protected]] On Behalf Of Boyd, Robert Sent: Friday, January 29, 2016 12:27 PM To: [email protected] Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle Thanks Avi, I tried what you suggested and got some minor improvement -- the tool runs longer before things blow up. I tried increasing the memory limits even more -- but that doesn't seem to make any difference. It's still eventually blowing up the same way, and the logs you suggested looking at don't show anything interesting around the time of the failures. Any other suggestions? I could keep increasing the memory limits even farther, but I'm not sure that's a good idea. Thanks, Robert -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Avi Miller Sent: Thursday, January 28, 2016 6:12 PM To: [email protected] Subject: Re: [Spacewalk-list] ? Failures Cloning Channels with spacewalk-manage-channel-lifecycle Hey, > On 29 Jan 2016, at 9:56 AM, Boyd, Robert <[email protected]> wrote: > > I have tried poking at log files, but I haven’t found where there seems to be > an indication of a problem. Can someone point me to a prioritized list of > logs I should be looking at to find the nature of what’s dying on the back > end? I suspect there is a java resource failure or some other error and > would like to track it down and fix it rather than resort to manual/scripted > channel by channel cloning( which might fail in the same way). You may want to increase the memory available to Tomcat and Taskomatic: http://docs.oracle.com/cd/E37670_01/E58680/html/sw22-issues-memory.html This feels like a Tomcat issue to me, the logs are in /var/log/tomcat6 on the Spacewalk server. The Taskomatic logs are in /var/log/rhn/rhn_taskomatic_daemon.log Cheers, Avi -- Oracle <http://www.oracle.com> Avi Miller | Product Management Director | +61 (3) 8616 3496 Oracle Linux and Virtualization 417 St Kilda Road, Melbourne, Victoria 3004 Australia _______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list
