julio666 opened a new issue #7304: URL: https://github.com/apache/pulsar/issues/7304
**The bug** On Kubernetes, we run a streaming spark application which uses pulsar-spark connector. As the spark application runs, it does a ton of requests on `/admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId` ending up to the broker managing the topic (topic not being filled by any source during that time): ``` [...] 14:20:02.563 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.583 [pulsar-web-42-4] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 14:20:02.591 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.610 [pulsar-web-42-1] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 14:20:02.618 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.637 [pulsar-web-42-8] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 14:20:02.645 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.664 [pulsar-web-42-7] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/KDECOLEAPP_ANONYMISE/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 14:20:02.672 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.692 [pulsar-web-42-4] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 14:20:02.700 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/lastMessageId HTTP/1.1" 200 186 "-" "Pulsar-Java-v2.4.2" 4 14:20:02.719 [pulsar-web-42-6] INFO org.eclipse.jetty.server.RequestLog - 10.42.7.18 - - [18/Jun/2020:14:20:02 +0000] "GET /admin/v2/persistent/public/default/TOPIC_NAME/partitions HTTP/1.1" 200 16 "-" "Pulsar-Java-v2.4.2" 3 [...] ``` And so on for a certain time. As the requests are going on, we can see a memory leak on grafana graph on the concerned broker (Direct Memory):  And when the limit is reached, the broker is killed by Kubernetes for an OOM. **To Reproduce** Steps to reproduce the behavior: 1. Deploy Pulsar with the helm chart with default values.yaml concerning `requests` 2. Run a streaming spark application feeding on a topic of Pulsar 3. Let it run 4. See error **Expected behavior** The broker should not go out of memory. **Versions:** - Pulsar 2.5.2 - Spark 2.4.5 in case you are familiar with Spark issues ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
