2019-06-07 11:12:19 UTC - Ilya Lavrovsky: @Ilya Lavrovsky has joined the channel ---- 2019-06-07 11:36:32 UTC - Alexandre DUVAL: Hi, IIRC persistents topics are persisted only if they had a consumer. Can we make them persist without a first subscription? ---- 2019-06-07 14:12:37 UTC - Yuwei Jiang: Hi @Ali Ahmed - From grafana dashboard: local publish throughput is around 1kB/s; local delivery throughput is around 1.4kB/s ---- 2019-06-07 15:09:16 UTC - Vladimir Puzakov: @Vladimir Puzakov has joined the channel ---- 2019-06-07 15:19:07 UTC - Sijie Guo: all the data will always be persisted first for persistent topics. a topic will be deleted when it becomes inactive (e.g. no retention, no subscriptions and etc).
you can either: a) disable `brokerDeleteInactiveTopicsEnabled=false` in conf/broker.conf b) set a retention for the topic. ---- 2019-06-07 15:52:26 UTC - Kendall Magesh-Davis: In case anyone is looking here later, I wasn’t able to identify an absolute source for this problem. I replicated the error message (but with many others) by force-deleting the EBS volume out from under the PV. Resolved ZK issue by tearing down and redeploying *only the Zookeeper statefulset* Hopefully someone finds a more specific reason for this problem and a more elegant solution. Hope this helps. :thumbsup: ---- 2019-06-07 17:15:22 UTC - Raghu Velagala: @Raghu Velagala has joined the channel ---- 2019-06-07 17:31:50 UTC - Matteo Merli: @Kendall Magesh-Davis :+1: thanks for the update ---- 2019-06-07 17:52:08 UTC - Addison Higham: hrm... so I have worked through the details of pulsar authz/authn and think I understand it all... but I am realizing it seems like there might be a bit of a gap where I want to allow a role to manage a single namespace (such as retention or creating ACLs for topics in that namespace) and not to be able to impact other namespaces in the same tenant ---- 2019-06-07 17:52:50 UTC - durga: Hi Guys - We are running into a strange problem with Pulsar. Basically when we deploy `pulsar` on a 15GB node in a k8s cluster and run our smoke tests, everything works fine. But when we run the same system on a 60GB node in a k8s cluster then things don’t work. I had problems in past with products when memory size was decreased but it is rare where product has problem because more memory is allocated to it :slightly_smiling_face: . I am sure we are doing something wrong in the memory configuration settings. When we run on a 15 GB node in a k8s cluster, following were the JVM settings we have used and the workload (i.e., smoke tests) run fine. ``` zookeeper: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem -Dzookeeper.forceSync=no bookkeeper: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest broker: -Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem proxy: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g ``` Now if we run the same workload with pulsar on a bigger node (60GB RAM) with following configuration, it fails. ``` zookeeper: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem -Dzookeeper.forceSync=no bookkeeper: -Xms6g -Xmx6g -XX:MaxDirectMemorySize=6g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest broker: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem proxy: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g ``` I appreciate very much if anyone can provide some pointers on what we could be doing wrong. CC @Yuwei Jiang ---- 2019-06-07 17:54:23 UTC - Addison Higham: Am I missing something? or is that not currently covered by the existing ACL scheme? Looking at the AuthorizationProvider interface it seems to not be covered ---- 2019-06-07 17:54:35 UTC - Matteo Merli: Correct, It’s not currently possible to do that ---- 2019-06-07 17:55:03 UTC - Matteo Merli: The tenant admin right now is allowed to administrate all the namespaces for a given tenant ---- 2019-06-07 17:56:02 UTC - Addison Higham: which makes sense, I wonder if you could piggy back on top of the ACL framework and have another action of "manage" that could allow for management of all settings related to a topic or namespace? ---- 2019-06-07 17:56:05 UTC - Matteo Merli: I guess this could be extended by having an “admin” action permission that can be granted on a namespace level (rather than tenant) ---- 2019-06-07 17:56:52 UTC - Matteo Merli: Yes, right now there are 3 possible ACLs actions: produce, consume, functions. We could extend with admin as well ---- 2019-06-07 17:57:47 UTC - Addison Higham: seems reasonable... I don't have need of it this minute, but if I get some time in the next little bit I may research that more and see if it would make sense to make a PIP out of it ---- 2019-06-07 17:57:59 UTC - Matteo Merli: :+1: ---- 2019-06-07 18:08:39 UTC - Joe Francis: Be very clear on how this will be, dealing with child granted but parent revoked is a rathole ---- 2019-06-07 18:09:55 UTC - Matteo Merli: why? you will have 2 different principals that may admin the namespace. One for the entire tenant and the other for the namespace. ---- 2019-06-07 18:12:50 UTC - Joe Francis: Yeah, but the right of the second principal is granted by the first one. ---- 2019-06-07 18:13:03 UTC - Matteo Merli: Yes ---- 2019-06-07 18:14:22 UTC - Matteo Merli: But the same happens for “produce” permission, the tenant admin will grant permission to “user-a” to publish. Now, if that specific admin will get its permission revoked, the “user-a” will still be able to publish.. unless someone removes that permission explicitely ---- 2019-06-07 18:14:22 UTC - Joe Francis: So what will you do if parent revokes and child grants? ---- 2019-06-07 18:16:21 UTC - Joe Francis: Changing admin privs is not the issue - the issue is if you have conflicting grants. ---- 2019-06-07 18:16:52 UTC - Matteo Merli: the grants are always “addictive”. I don’t see how they can conflict ---- 2019-06-07 18:17:58 UTC - Joe Francis: Namespace admin can grant, and tenant admin can revoke - what's the net outcome? ---- 2019-06-07 18:18:18 UTC - Joe Francis: Or vice-versa ---- 2019-06-07 18:19:10 UTC - Matteo Merli: permission to publish? that will result in no permission. the ACLs for namespace are written in 1 single place. Now you’ll have 2 principals that can change that, but the info is not duplicated ---- 2019-06-07 18:21:25 UTC - Joe Francis: Sure, that is how I would prefer it. But be clear about this, that there is no hierarchical permissions ---- 2019-06-07 18:21:38 UTC - Matteo Merli: :+1: ---- 2019-06-07 19:10:56 UTC - Sijie Guo: what errors did you see? ---- 2019-06-07 22:17:40 UTC - Alexandre DUVAL: oh okay :slightly_smiling_face:, thanks ---- 2019-06-08 05:02:33 UTC - Yuwei Jiang: Hi @Sijie Guo - The segment of the log files containing the errors are below ---- 2019-06-08 05:04:14 UTC - Yuwei Jiang: bookkeeper log ----
