Sai Vamsi,
On 12/6/23 11:04, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
Hello Christopher.
Thanks for the response!
Let me walk you through the Environment.,
I have a cluster and I am installing multiple microservices in the cluster. And
one of the microservices I am integrating the JMX on of the microservices.,
and the mentioned IP of the pod is being changed dynamically after each
deployment .
Here are the annotations that I am using in my deployment:
- name: JAVA_OPTS
# value: >-
# -Dcom.sun.management.jmxremote
# -Dcom.sun.management.jmxremote.authenticate=false
# -Dcom.sun.management.jmxremote.ssl=false
# -Dcom.sun.management.jmxremote.local.only=false
# -Dcom.sun.management.jmxremote.port=49151
# -Dcom.sun.management.jmxremote.rmi.port=49151
#
-Djava.rmi.server.hostname=lab1workflow4scalsvc2zus1-service.hqm-lab1.svc.cluster.local
If that's YAML, then all of the above is comments. Those lines do nothing.
as i said , I have implemented these in three scenarios., Catalina_opts /
Java_opts / Java_Tool_opts
You should definitely use CATALINA_OPTS and none of the other things you
mention, here. Specifically because of the port issues. JAVA_TOOL_OPTS
don't affect Tomcat at all are useless for you. JAVA_OPTS affects all
Tomcat-related scripts, and you'll find that even "shutdown" causes an
error... because it's trying to initialize JMX when shutting-down, which
is again useless.
I have tried deploying catalina_ots and Java_opts as in the case of above
mentioned.
I also tried other way, mentioning all these in the values.yaml file ( I am
using Helm Deployment to upgrade my deployments) and referring these
annotations as , and mentioning them in the Value
env:
- name: CATALINA_OPTS
value: {{ .Values.deployment.javaVMMemoryArgument }}
getting netstat output from my pod:
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp6 0 0 :::10109 :::* LISTEN
1/java
tcp6 0 0 :::9109 :::* LISTEN
1/java
root@lab1workflow4scalsvc2zus1-deployment-fd64ff775-cwzn6:/# ps aux | grep
tomcat
root 563 0.0 0.0 6328 748 pts/0 S+ 15:55 0:00 grep
--color=auto tomcat
Don't look for 'tomcat'. Look for 'java'. Or 'catalina'. The word
'tomcat' does not appear in a normal JVM launch command for Tomcat.
right now , I am passing my args with Catalina_opts in my deployment
and I am not able to get any tomcat process pid while i check with
netstat -apn , I can see multiple Java Process but not single tomcat process
My Jmx port is being configured as a service., so that it could be
bounded with Tcp protocol externally to the container., and allow all
tcp connections.
Hope I am able to provide you some detailed information., Please let
me know., if I can help you with anything in further!
I think maybe you are both configuring your YAML file incorrectly and
also looking for the process incorrectly.
-chris
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, December 6, 2023 20:40
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/5/23 08:57, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
I have added my annotations in Catalina_opts ., and I am not able to see any
Java Process with the JMX, neither the port is being listened.
It came back to previous stage where Annotations are not on the JVM .,
Came back to previous stage, where connection refused error., as follows.
Could not initialize instance: tomcat-10.75.12.136-49151:
java.util.concurrent.ExecutionException: java.io.IOException: Failed to
retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root
exception is java.rmi.ConnectException: Connection refused to host:
10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
at org.datadog.jmxfetch.App$5.invoke(App.java:927)
at
org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
at org.datadog.jmxfetch.App.init(App.java:917)
at org.datadog.jmxfetch.App.start(App.java:413)
at org.datadog.jmxfetch.App.run(App.java:173)
at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
at org.datadog.jmxfetch.App.main(App.java:91)
Right: the DataDog server is unable to connect to your appication via
JMX. I'm assuming that 10.75.12.136 is the IP where Tomcat is running,
and that DataDog isn't running on that server. It's on some other IP, right?
Here we were not able to find any process , to confirm that atleast
port is being listened to.
What is your exact CATALINA_OPTS value (masking any secrets that may be
in there).
Is Tomcat definitely running? Can you prove that with a "ps" listing,
for example? That will also confirm it's running with the proper
CATALINA_OPTS settings.
Can you run "netstat" and show all of the ports your Tomcat process is
listening to?
I'm assuming that your Tomcat process is being hosted in some kind of
container. Is the IP address 10.75.12.136 the IP of the container, or
the IP of the host which hosts the container? Is your JMX port
configured to be visible external to the container itself, or only
internally to the container?
-chris
Thanks & Regards,
--
SAI VAMSI .B
Senior DevOps Engineer
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Tuesday, December 5, 2023 19:19
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsa,
On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
Thanks for the response..
as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of
Java_Opts and Java_tool_options.,
But its not even working out., Java process is not getting created., and port
is not bound to any process.
Let me know if I am missing anything.
Okay. Can you please summarize the current situation? I think you've
made a few changes, etc. so it would help to know the current state.
Stop Tomcat and any other related services, delete all appropriate log
files, etc. and start Tomcat (or attempt to). Then post any output that
is relevant, and if you get things like "address in use" then please
post the details about which process id owns that port and the full
command-line that launched that process.
Thanks,
-chris
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, December 4, 2023 23:22
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.
Sai Vamsi,
On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
Firstly thanks for adding a point me in asking me to check, if the annotations
are reflecting in the Java process, which opened me a door to add the concerned
annotations in correct place., by adding in java_tool_options in stead of
Java_opts.
You will probably want to use CATALINA_OPTS instead of any of the other
ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
any time it invokes a JVM. For example, running bind/digest.sh doesn't
need to have the JMX subsystem starting-up and trying to grab a port.
JAVA_TOOL_OPTS is an environment variable used by JVM-launching
processes, like jps for example...
yeah they are reflecting and creating a Java Process.
but I am facing a problem here., while i am checking JSP, Thats : the port i
am using here to enable JMX is been opening a process with the mentioned port
and at the same time shows port is being used.:
root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication
-XX:+UseContainerSupport -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=49151
-Djava.rmi.server.hostname=tomcat.default.svc.cluster.local
-javaagent:/datadog-lib/dd-java-agent.jar
-XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh
-XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port
already in use: 49151; nested exception is:
java.net.BindException: Address already in use (Bind failed)
Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
to your port which is aready bound by your Tomcat process.
The solution is to use only CATALINA_OPTS to set these options.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
