2018-04-14 11:21:17 UTC - Igor Zubchenok: Hello Pulsars! If I create 10000 subscriptions in parallel with `admin.persistentTopics().createSubscriptionAsync` method, it takes 16 minutes to complete on a standalone cluster.
How can I improve performance of subscription creation? ---- 2018-04-14 13:58:08 UTC - Vasily Yanov: @Vasily Yanov has joined the channel ---- 2018-04-14 14:11:44 UTC - Vasily Yanov: Hi there! When I try to create subscription with ```curl -X PUT \ <http://prod-pulsar.cluster:8080/admin/persistent/server-eagle/prod-pulsar-cluster-1/session_init/4bde2ee8-e366-47e0-a0ff-808f39873dc7/subscription/sender> \ -H 'accept: application/json' \ -H 'cache-control: no-cache' \ -H 'content-type: application/json' \ -H 'postman-token: 70eda244-7d06-13e4-6d7c-a9a1fa6bdf4e' \ -H 'user-agent: Jersey/2.23.2 (HttpUrlConnection 1.8.0_121)' \ -d '{"adminRoles":["admins"],"allowedClusters":["prod-pulsar-cluster-1"]}'``` I got and error 500 from the server. Cluster consists of 3 nodes. In logs I see: ```Apr 14 14:06:28 localhost broker[27675]: 2018-04-14 14:06:28,523 - WARN - [pulsar-web-61-5:PersistentTopics@1077] - [null] [<persistent://server-eagle/prod-pulsar-cluster-1/session_init/4bde2ee8-e366-47e0-a0ff-808f39873dc7>] Failed to create subscription sender1233 at message id -1:-1 :-1 Apr 14 14:06:28 localhost broker[27675]: javax.ws.rs.WebApplicationException: HTTP 307 Temporary Redirect Apr 14 14:06:28 localhost broker[27675]: #011at org.apache.pulsar.broker.web.PulsarWebResource.validateDestinationOwnership(PulsarWebResource.java:503) Apr 14 14:06:28 localhost broker[27675]: #011at org.apache.pulsar.broker.admin.PersistentTopics.validateAdminOperationOnDestination(PersistentTopics.java:286) Apr 14 14:06:28 localhost broker[27675]: #011at org.apache.pulsar.broker.admin.PersistentTopics.createSubscription(PersistentTopics.java:1061)``` any ideas why? Same request to the test cluster with 1 node returns possitive result. ---- 2018-04-14 16:12:03 UTC - Igor Zubchenok: I have such issue with 3 nodes pulsar cluster while using PulsarAdmin original java client (`org.apache.pulsar:pulsar-client-admin-original:1.22.0-incubating`). ``` org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: Some error occourred on the server at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:162) at org.apache.pulsar.client.admin.internal.PersistentTopicsImpl.createSubscription(PersistentTopicsImpl.java:628) at com.zubchenok.PulsarTest.n(PulsarTest.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1032) at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819) at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92) at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:228) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:448) at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:332) at org.apache.pulsar.client.admin.internal.PersistentTopicsImpl.createSubscription(PersistentTopicsImpl.java:626) ... 23 more ``` Investigated that the reason of the issue is a redirect to master pulsar node, which seems is not handled properly. And it is very suspicious that I get `500 Internal Server Error`. But using `org.apache.pulsar:pulsar-client-admin:1.22.0-incubating` is not possible due to #1409 issue. BTW I'm not sure it is handled. ---- <https://github.com/apache/incubator-pulsar/issues/1409> ---- 2018-04-15 04:51:42 UTC - Matteo Merli: > If I create 10000 subscriptions in parallel with `admin.persistentTopics().createSubscriptionAsync` method, it takes 16 minutes to complete on a standalone cluster. > How can I improve performance of subscription creation? ---- 2018-04-15 04:52:43 UTC - Matteo Merli: @Igor Zubchenok Are you waiting for the future to complete at each step? I should be *very much* faster than that if you issue all the operations and then wait for all the futures to complete ---- 2018-04-15 04:54:21 UTC - Matteo Merli: > Hi there! When I try to create subscription with > …. > -d ‘{“adminRoles”:[“admins”],“allowedClusters”:[“prod-pulsar-cluster-1"]}’ @Vasily Yanov It looks like you’re passing the body that was for a different call (likely creating a new tenant) instead of the payload for the create subscription ----
