[
https://issues.apache.org/jira/browse/SAMZA-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585724#comment-14585724
]
József Márton Jung commented on SAMZA-670:
------------------------------------------
Hi guys,
Thanks for the replies and suggestions. I implemented the writing of the JMX
address to coordinator stream by extending the container to host mapping. Now
it looks like this:
{noformat}
{"host":"127.0.1.1","source":"SamzaContainer-0","values":{"jmx-tunneling-address":"service:jmx:rmi://ubuntu:42448/jndi/rmi://ubuntu:42447/jmxrmi","ip":"127.0.1.1","jmx-address":"service:jmx:rmi://localhost:42448/jndi/rmi://localhost:42447/jmxrmi"},"username":"jjung","timestamp":1434360265426}
{noformat}
But I have difficulties reading the most recent message. For example: when the
Samza app start for the first time, {{LocalityManager.readContainerLocality()}}
reads nothing. Then the second time, when it should read the new address, it
reads the older address instead, and so on.
I don't know what can be the reason for it, because
{{LocalityManager.writeContainerToHostMapping}} writes the values to
coordinator stream as it is expected. I verified that by looking into Kafka
stream:
{noformat}
deploy/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
__samza_coordinator_my_stream_1 --from-begin
{noformat}
Now here is my example case for the problem above:
I started and stopped my Samza app 3 times, and started it for the 4th time, so
the application is running. When I look into Kafka stream directly, I see the
following:
{noformat}
{"host":"127.0.1.1","source":"SamzaContainer-0","values":{"jmx-tunneling-address":"service:jmx:rmi://ubuntu:53430/jndi/rmi://ubuntu:53429/jmxrmi","ip":"127.0.1.1","jmx-address":"service:jmx:rmi://localhost:53430/jndi/rmi://localhost:53429/jmxrmi"},"username":"jjung","timestamp":1434357937551}
...
...
{"host":"127.0.1.1","source":"SamzaContainer-0","values":{"jmx-tunneling-address":"service:jmx:rmi://ubuntu:39749/jndi/rmi://ubuntu:39748/jmxrmi","ip":"127.0.1.1","jmx-address":"service:jmx:rmi://localhost:39749/jndi/rmi://localhost:39748/jmxrmi"},"username":"jjung","timestamp":1434358645918}
...
...
{"host":"127.0.1.1","source":"SamzaContainer-0","values":{"jmx-tunneling-address":"service:jmx:rmi://ubuntu:58197/jndi/rmi://ubuntu:58196/jmxrmi","ip":"127.0.1.1","jmx-address":"service:jmx:rmi://localhost:58197/jndi/rmi://localhost:58196/jmxrmi"},"username":"jjung","timestamp":1434359423479}
...
...
{"host":"127.0.1.1","source":"SamzaContainer-0","values":{"jmx-tunneling-address":"service:jmx:rmi://ubuntu:42448/jndi/rmi://ubuntu:42447/jmxrmi","ip":"127.0.1.1","jmx-address":"service:jmx:rmi://localhost:42448/jndi/rmi://localhost:42447/jmxrmi"},"username":"jjung","timestamp":1434360265426}
{noformat}
This is correct, the latest JMX address is the valid one. But when I look into
application master logs, the call which reads container locality reads the
following:
{noformat}
2015-06-15 11:24:24 SamzaAppMasterTaskManager [INFO] Started container ID 0
2015-06-15 11:24:24 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2015-06-15 11:24:24 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2015-06-15 11:24:24 LocalityManager [INFO] Read locality for container 0:
{jmx-tunneling-address=service:jmx:rmi://ubuntu:58197/jndi/rmi://ubuntu:58196/jmxrmi,
ip=127.0.1.1,
jmx-address=service:jmx:rmi://localhost:58197/jndi/rmi://localhost:58196/jmxrmi}
2015-06-15 11:24:24 LocalityManager [INFO] Read locality for container 0:
{jmx-tunneling-address=service:jmx:rmi://ubuntu:39749/jndi/rmi://ubuntu:39748/jmxrmi,
ip=127.0.1.1,
jmx-address=service:jmx:rmi://localhost:39749/jndi/rmi://localhost:39748/jmxrmi}
2015-06-15 11:24:24 LocalityManager [INFO] Read locality for container 0:
{jmx-tunneling-address=service:jmx:rmi://ubuntu:53430/jndi/rmi://ubuntu:53429/jmxrmi,
ip=127.0.1.1,
jmx-address=service:jmx:rmi://localhost:53430/jndi/rmi://localhost:53429/jmxrmi}
{noformat}
The latest address which is in the coordinator stream is not there. And I have
no idea why.
Do you have any idea how can this happen?
> Allow easier access to JMX port
> -------------------------------
>
> Key: SAMZA-670
> URL: https://issues.apache.org/jira/browse/SAMZA-670
> Project: Samza
> Issue Type: Improvement
> Components: container
> Affects Versions: 0.9.0
> Reporter: Naveen Somasundaram
> Assignee: József Márton Jung
> Fix For: 0.10.0
>
>
> The current way we expose JMX is throw logs. The process for figuring this
> out is (To debug a particular partition):
> 1. Figure out the Partition to container mapping through the YARM AM or logs.
> 2. Once we know the container, access the container logs and figure out the
> JMX port.
> 3. Connect to the machine using the JMX port.
> We should ideally expose it through some REST service or make it available
> through the coordinator stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
