Sai Vamsi,
> Am 08.12.2023 um 06:43 schrieb Bodavula, Sai Vamsi Mohan Krishna (TR > Technology) <saivamsimohankrishna.bodav...@thomsonreuters.com.INVALID>: > > Hey Christopher., > Greetings of the day. > > > 1. > Might I have confused you with posting the arguments directly ., Yeah as i > just shared you the annotations with comments , to state you the stuff i am > using., But in my deployment ., I am using them in catalina opts., and trying > to call them from values.yaml., which looks like this : > > javaVMMemoryArgument: "-Xms2048M -Xmx10240M -XX:+UseStringDeduplication > -XX:+UseContainerSupport -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.local.only=false > -Dcom.sun.management.jmxremote.port=49151 > -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local > " > > and I am referring the word "javaVMMemoryArgument" from values yaml and > calling it in Catalina_opts, so that it would fetch all these > annotations as mentioned above, during the deployment. This is my deployment > part., where I am referring to the above values from values.yaml > env: > - name: CATALINA_OPTS > value: {{ .Values.deployment.javaVMMemoryArgument }} > > I compare with my CATALINA_OPTS (and I noticed that some of my options were still in JAVA_OPTS boooh!) : export CATALINA_OPTS="${CATALINA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${HOSTNAME} -Dcom.sun.management.jmxremote.local.only=false" I see all my options in root@tomcat:/usr/local/tomcat# ps -aef | less UID PID PPID C STIME TTY TIME CMD tomcat 1 0 71 10:42 ? 00:04:02 /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/opt/apache-tomcat.base/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -agentlib:jdwp=transport=dt_socket,address=0.0.0.0:8000,server=y,suspend=n -XX:NativeMemoryTracking=summary -Dhostname=<hostname> -Djava.awt.headless=true -Djavax.net.ssl.trustStore=/opt/apache-tomcat.base/conf/ssl/cacerts.jks -Xlog:gc:/opt/apache-tomcat.base/logs/gc.log -Djava.security.egd=file:/dev/urandom -Dsun.net.inetaddr.ttl=60 -Djava.library.path=/usr/local/tomcat/native-jni-lib -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djdk.tls.server.enableStatusRequestExtension=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=10001 -Dcom.sun.management.jmxremote.rmi.port=10002 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<hostname> -Dcom.sun.management.jmxremote.local.only=false -javaagent:/opt/apache-tomcat.base/bin/jmx_prometheus_javaagent-0.12.0.jar=8080:/opt/apache-tomcat.base/bin/tomcat.yaml -Djava.net.debug=ssl -XX:+UnlockDiagnosticVMOptions -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/apache-tomcat.base -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/opt/apache-tomcat.base/temp org.apache.catalina.startup.Bootstrap start Ports are opened root@tomcat:/usr/local/tomcat# netstat -tulpn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name ... tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN - ... tcp 2 0 0.0.0.0:8443 0.0.0.0:* LISTEN - ... jconsole <hostname>:10001 > JMX connect success > > > > 1. > Coming to Process., I have searched for Java process that listens on my > mentioned port ie., 49151, but none of the process is listening to that > process. > > I even tried with > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep LISTEN > tcp6 0 0 :::10109 :::* LISTEN > 1/java > tcp6 0 0 :::9109 :::* LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > more > Active Internet connections (only servers) > Proto Recv-Q Send-Q Local Address Foreign Address State > PID/Program name > tcp6 0 0 :::10109 :::* LISTEN > 1/java > tcp6 0 0 :::9109 :::* LISTEN > 1/java > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep ':443'netstat -tulpn | grep ':443'^C > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn | > grep ':49151' > root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# > > which confirms me that , any of the process is being listening on the port > 49151. > You should at least see some error when starting Tomcat. Anything in the logs or on the console? Peter > 2. > I would like to request you to suggest me with a better approach ., where i > am missing anything in this process! > > > > > > > Thanks & Regards, > -- > > SAI VAMSI .B > Senior DevOps Engineer > > > ________________________________ > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Thursday, December 7, 2023 23:33 > To: users@tomcat.apache.org <users@tomcat.apache.org> > Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues. > > Sai Vamsi, > > On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote: >> Hello Christopher. >> Thanks for the response! >> >> Let me walk you through the Environment., >> I have a cluster and I am installing multiple microservices in the cluster. >> And one of the microservices I am integrating the JMX on of the >> microservices., >> and the mentioned IP of the pod is being changed dynamically after each >> deployment . >> >> Here are the annotations that I am using in my deployment: >> - name: JAVA_OPTS >> # value: >- >> # -Dcom.sun.management.jmxremote >> # -Dcom.sun.management.jmxremote.authenticate=false >> # -Dcom.sun.management.jmxremote.ssl=false >> # -Dcom.sun.management.jmxremote.local.only=false >> # -Dcom.sun.management.jmxremote.port=49151 >> # -Dcom.sun.management.jmxremote.rmi.port=49151 >> # >> -Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local > > If that's YAML, then all of the above is comments. Those lines do nothing. > >> as i said , I have implemented these in three scenarios., Catalina_opts / >> Java_opts / Java_Tool_opts > > You should definitely use CATALINA_OPTS and none of the other things you > mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS > don't affect Tomcat at all are useless for you. JAVA_OPTS affects all > Tomcat-related scripts, and you'll find that even "shutdown" causes an > error... because it's trying to initialize JMX when shutting-down, which > is again useless. > >> I have tried deploying catalina_ots and Java_opts as in the case of above >> mentioned. >> I also tried other way, mentioning all these in the values.yaml file ( I am >> using Helm Deployment to upgrade my deployments) and referring these >> annotations as , and mentioning them in the Value >> env: >> - name: CATALINA_OPTS >> value: {{ .Values.deployment.javaVMMemoryArgument }} >> >> >> getting netstat output from my pod: >> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn >> Active Internet connections (only servers) >> Proto Recv-Q Send-Q Local Address Foreign Address State >> PID/Program name >> tcp6 0 0 :::10109 :::* LISTEN >> 1/java >> tcp6 0 0 :::9109 :::* LISTEN >> 1/java >> >> root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep >> tomcat >> root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep >> --color=auto tomcat > > Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word > 'tomcat' does not appear in a normal JVM launch command for Tomcat. > >> right now , I am passing my args with Catalina_opts in my deployment >> and I am not able to get any tomcat process pid while i check with >> netstat -apn , I can see multiple Java Process but not single tomcat process >> >> My Jmx port is being configured as a service., so that it could be >> bounded with Tcp protocol externally to the container., and allow all >> tcp connections. >> >> Hope I am able to provide you some detailed information., Please let >> me know., if I can help you with anything in further! > > I think maybe you are both configuring your YAML file incorrectly and > also looking for the process incorrectly. > > -chris > >> ________________________________ >> From: Christopher Schultz <ch...@christopherschultz.net> >> Sent: Wednesday, December 6, 2023 20:40 >> To: users@tomcat.apache.org <users@tomcat.apache.org> >> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues. >> >> Sai Vamsi, >> >> On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote: >>> I have added my annotations in Catalina_opts ., and I am not able to see >>> any Java Process with the JMX, neither the port is being listened. >>> It came back to previous stage where Annotations are not on the JVM ., >>> Came back to previous stage, where connection refused error., as follows. >>> >>> Could not initialize instance: tomcat-10.75.12.136-49151: >>> java.util.concurrent.ExecutionException: java.io.IOException: Failed to >>> retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root >>> exception is java.rmi.ConnectException: Connection refused to host: >>> 10.75.12.136; nested exception is: >>> java.net.ConnectException: Connection refused (Connection refused)] >>> at >>> java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) >>> at >>> java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) >>> at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953) >>> at org.datadog.jmxfetch.App$5.invoke(App.java:927) >>> at >>> org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63) >>> at org.datadog.jmxfetch.App.init(App.java:917) >>> at org.datadog.jmxfetch.App.start(App.java:413) >>> at org.datadog.jmxfetch.App.run(App.java:173) >>> at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57) >>> at org.datadog.jmxfetch.App.main(App.java:91) >> >> Right: the DataDog server is unable to connect to your appication via >> JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running, >> and that DataDog isn't running on that server. It's on some other IP, right? >> >>> Here we were not able to find any process , to confirm that atleast >>> port is being listened to. >> What is your exact CATALINA_OPTS value (masking any secrets that may be >> in there). >> >> Is Tomcat definitely running? Can you prove that with a "ps" listing, >> for example? That will also confirm it's running with the proper >> CATALINA_OPTS settings. >> >> Can you run "netstat" and show all of the ports your Tomcat process is >> listening to? >> >> I'm assuming that your Tomcat process is being hosted in some kind of >> container. Is the IP address 10.75.12.136 the IP of the container, or >> the IP of the host which hosts the container? Is your JMX port >> configured to be visible external to the container itself, or only >> internally to the container? >> >> -chris >> >> >> >>> Thanks & Regards, >>> -- >>> >>> SAI VAMSI .B >>> Senior DevOps Engineer >>> >>> ________________________________ >>> From: Christopher Schultz <ch...@christopherschultz.net> >>> Sent: Tuesday, December 5, 2023 19:19 >>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues. >>> >>> Sai Vamsa, >>> >>> On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote: >>>> Thanks for the response.. >>>> >>>> as you suggested., I tried adding my annotations in CATALINA_OPTS., >>>> Instead of Java_Opts and Java_tool_options., >>>> >>>> But its not even working out., Java process is not getting created., and >>>> port is not bound to any process. >>>> Let me know if I am missing anything. >>> >>> Okay. Can you please summarize the current situation? I think you've >>> made a few changes, etc. so it would help to know the current state. >>> >>> Stop Tomcat and any other related services, delete all appropriate log >>> files, etc. and start Tomcat (or attempt to). Then post any output that >>> is relevant, and if you get things like "address in use" then please >>> post the details about which process id owns that port and the full >>> command-line that launched that process. >>> >>> Thanks, >>> -chris >>> >>>> ________________________________ >>>> From: Christopher Schultz <ch...@christopherschultz.net> >>>> Sent: Monday, December 4, 2023 23:22 >>>> To: users@tomcat.apache.org <users@tomcat.apache.org> >>>> Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues. >>>> >>>> Sai Vamsi, >>>> >>>> On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote: >>>>> Firstly thanks for adding a point me in asking me to check, if the >>>>> annotations are reflecting in the Java process, which opened me a door to >>>>> add the concerned annotations in correct place., by adding in >>>>> java_tool_options in stead of Java_opts. >>>> >>>> You will probably want to use CATALINA_OPTS instead of any of the other >>>> ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat. >>>> You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS >>>> any time it invokes a JVM. For example, running bind/digest.sh doesn't >>>> need to have the JMX subsystem starting-up and trying to grab a port. >>>> >>>> JAVA_TOOL_OPTS is an environment variable used by JVM-launching >>>> processes, like jps for example... >>>> >>>>> yeah they are reflecting and creating a Java Process. >>>>> but I am facing a problem here., while i am checking JSP, Thats : the >>>>> port i am using here to enable JMX is been opening a process with the >>>>> mentioned port and at the same time shows port is being used.: >>>>> >>>>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps >>>>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M >>>>> -XX:+UseStringDeduplication -XX:+UseContainerSupport >>>>> -Dcom.sun.management.jmxremote >>>>> -Dcom.sun.management.jmxremote.authenticate=false >>>>> -Dcom.sun.management.jmxremote.ssl=false >>>>> -Dcom.sun.management.jmxremote.local.only=false >>>>> -Dcom.sun.management.jmxremote.port=49151 >>>>> -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local >>>>> -javaagent:/datadog-lib/dd-java-agent.jar >>>>> -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh >>>>> -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log >>>>> Error: Exception thrown by the agent : java.rmi.server.ExportException: >>>>> Port already in use: 49151; nested exception is: >>>>> java.net.BindException: Address already in use (Bind failed) >>>> >>>> Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind >>>> to your port which is aready bound by your Tomcat process. >>>> >>>> The solution is to use only CATALINA_OPTS to set these options. >>>> >>>> -chris >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >