2020-01-06 09:23:55 UTC - Grant McNulty: @Grant McNulty has joined the channel ---- 2020-01-06 10:42:14 UTC - Walter: How to check pulsar is up and running in initcontainer in kubernetes ---- 2020-01-06 12:06:53 UTC - Sijie Guo: 1. for zookeeper you can use `ruok` command to check. 2. for bookkeeper you can use `bookkeeper shell simpletest` 3. for brokers you can use the healthcheck endpoint exposed at 8080. You can check the helm chart to see how we did for each components. ---- 2020-01-06 12:08:25 UTC - Sijie Guo: • for zookeeper, you can backup the snapshots and txn log. • for bookkeeper, there is no tools yet. you can try to backup the volumes used for bookies. Alternatively, you can consider during tiered storage feature to offload the data to cloud storage like S3. It serves as a backup mechanism. ---- 2020-01-06 12:14:31 UTC - Sijie Guo: @Sijie Guo set the channel topic: - Pulsar user survey: <https://bit.ly/2Qtrrnf> - First Pulsar Summit is planned for April 2020 in San Francisco. CFP and pre-registration are open: <https://pulsar.apache.org/blog/2019/12/18/Pulsar-summit-cfp/> - Pulsar release 2.4.2 released! ---- 2020-01-06 13:00:23 UTC - Andreas Sittler: @Andreas Sittler has joined the channel ---- 2020-01-06 14:05:19 UTC - Tilden: thanks @Sijie Guo will try it out ---- 2020-01-06 14:13:56 UTC - Roman Popenov: Happy 2020 everyone! Hope everyone had good holidays! I am usually using the pulsar admin cli to manage the tenants/namespaces/policies, but I am also curious as to how use the web-interface to manage the cluster. I am running Pulsar on an EKS cluster in AWS, and I have a pulsar-manager pod running. I am trying to create a new enviroment, but I am not exactly sure what will the *Service URL* be ---- 2020-01-06 14:20:45 UTC - juraj: the proxy or broker k8s service's URL (or one of the node's IP if u r using NodePort) + :8080 i think ---- 2020-01-06 14:23:59 UTC - Roman Popenov: Yeah, for some reason, I wasn’t able to configure to expose the proxy as LoadBalancer and running it as NodePort ---- 2020-01-06 14:24:48 UTC - Roman Popenov: Proxy_Public_IP:NodePort does seem to work ---- 2020-01-06 14:26:16 UTC - juraj: ```proxy: ... service: annotations: {} type: NodePort ports: # http api proxy: - name: http port: 8080 nodePort: 30003 protocol: TCP # pulsar api proxy - name: pulsar port: 6650 nodePort: 30004 protocol: TCP``` ---- 2020-01-06 14:26:48 UTC - Roman Popenov: Type is still type: `NodePort` ---- 2020-01-06 14:27:11 UTC - Roman Popenov: I wanted to expose the proxy as LoadBalancer or have a LoadBalancer pointing to the Proxy pod ---- 2020-01-06 14:27:41 UTC - juraj: you do that by installing taefik in the cluster and specifying the ingress for that service ---- 2020-01-06 14:28:55 UTC - Roman Popenov: Yeah, I figured that I would need some kind of ingress controller… I was just hoping that the proxy can be leveraged as one ---- 2020-01-06 14:29:29 UTC - Roman Popenov: And exposing it through a loadbalancer that would simply redirect the traffick to it ---- 2020-01-06 14:29:38 UTC - juraj: you don't need load balancer for proxy, that's one extra indirection, the proxy is already balanced by k8s if you have more instances. you just expose it as nodeport and give your clients the list of those nodes. ---- 2020-01-06 14:30:02 UTC - juraj: that's how i do it anyway ---- 2020-01-06 14:30:04 UTC - Roman Popenov: I wouldn’t want to give node ports ---- 2020-01-06 14:30:08 UTC - juraj: i use loadbalancer for the dashboard only ---- 2020-01-06 14:32:26 UTC - juraj: you should be able to use the proxy as LoadBalancer type with the proper annotation it should create the AWS ELB ---- 2020-01-06 14:33:46 UTC - Roman Popenov: Yes, for some reason, things were breaking when I was setting proxy as `LoadBalancer` type ---- 2020-01-06 14:33:56 UTC - juraj: <https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1>
<https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws> eyes : Roman Popenov ---- 2020-01-06 14:38:46 UTC - Roman Popenov: I will need to get on top of that, thanks ---- 2020-01-06 14:40:10 UTC - Roman Popenov: So I was able to set up the Pulsar Admin UI, but I don’t have any data displayed about the clusters when I navigate to the `Cluster` tab ---- 2020-01-06 14:51:34 UTC - Olav Groenaas Gjerde: I have the same issue, not investigated why it is like that yet though ---- 2020-01-06 15:01:01 UTC - juraj: did u guys create any tenants + namespaces ---- 2020-01-06 15:12:20 UTC - Roman Popenov: Through CLI I have, but in the UI, I would need to specify a cluster to add a tenant, but the cluster shows `No Data` in the UI ---- 2020-01-06 15:21:34 UTC - tuteng: Which admin ui to use? ---- 2020-01-06 15:23:18 UTC - tuteng: Please provide the project url, if possible. ---- 2020-01-06 15:23:27 UTC - Roman Popenov: <http://EXTERNAL-IP:9527/#/> ---- 2020-01-06 15:23:53 UTC - Sijie Guo: if you running a pulsar-manager pod within the k8s cluster, you can just use `http://<pulsar-proxy-job-name>:8080` +1 : Roman Popenov ---- 2020-01-06 15:24:08 UTC - Roman Popenov: I am just using the pulsar-manager from the heml chart ---- 2020-01-06 15:24:10 UTC - Sijie Guo: since they are in the same network ---- 2020-01-06 15:24:49 UTC - Sijie Guo: you just need to expose the pulsar-manager port through a load balancer or node port. so you can use the UI out side of the k8s cluster. ---- 2020-01-06 15:24:51 UTC - tuteng: Please check pulsar-manager.log or use command <https://github.com/apache/pulsar-manager/issues/214#issuecomment-560313826> ---- 2020-01-06 15:25:06 UTC - Roman Popenov: Let me try with the job name now ---- 2020-01-06 15:26:16 UTC - tuteng: Can you show me your helm file? ---- 2020-01-06 15:28:15 UTC - Roman Popenov: Here are the contents of the helm file ---- 2020-01-06 15:30:33 UTC - Roman Popenov: clusters list: ```root@lolling-molly-pulsar-broker-7844b89746-66dzp:/pulsar# ./bin/pulsar-admin clusters list lolling-molly-pulsar``` and ```root@lolling-molly-pulsar-broker-7844b89746-66dzp:/pulsar# ./bin/pulsar-admin clusters get lolling-molly-pulsar { "serviceUrl" : "<http://10.0.0.42:8080>", "brokerServiceUrl" : "<pulsar://10.0.0.42:6650>" }``` ---- 2020-01-06 15:32:39 UTC - Roman Popenov: That worked! ---- 2020-01-06 15:33:16 UTC - Roman Popenov: I am still unable to see the clusters for some reason: <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1578321610290600> ---- 2020-01-06 15:45:10 UTC - Sijie Guo: is 10.0.0.42 a pod ip or a node ip? ---- 2020-01-06 15:46:10 UTC - Roman Popenov: Pod IP ---- 2020-01-06 15:55:47 UTC - Sijie Guo: is this a broker? is it reachable from pulsar-manager pod? did you see any errors in pulsar-manager pod log? ---- 2020-01-06 15:56:49 UTC - Roman Popenov: Yes, this is the IP of one of the brokers ---- 2020-01-06 15:56:54 UTC - Roman Popenov: Let me check the logs of the broker ---- 2020-01-06 16:04:38 UTC - Roman Popenov: There is nothing obvious in the logs of the broker ---- 2020-01-06 16:04:46 UTC - Roman Popenov: What should I look in the logs of the manager? ---- 2020-01-06 16:04:55 UTC - Roman Popenov: Also, I can ping the brokers: ```/pulsar-manager # ping 10.0.0.42:6650 PING 10.0.0.42:6650 (10.0.0.42): 56 data bytes 64 bytes from 10.0.0.42: seq=0 ttl=253 time=1.191 ms 64 bytes from 10.0.0.42: seq=1 ttl=253 time=1.183 ms ^C --- 10.0.0.42:6650 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 1.183/1.187/1.191 ms /pulsar-manager # ping 10.0.0.42:8080 PING 10.0.0.42:8080 (10.0.0.42): 56 data bytes 64 bytes from 10.0.0.42: seq=0 ttl=253 time=1.203 ms 64 bytes from 10.0.0.42: seq=1 ttl=253 time=1.256 ms ^C --- 10.0.0.42:8080 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 1.203/1.229/1.256 m``` ---- 2020-01-06 16:05:31 UTC - Sijie Guo: Can you share the screenshots of Pulsar Manager? ---- 2020-01-06 16:05:41 UTC - Roman Popenov: Logs? ---- 2020-01-06 16:09:32 UTC - Sijie Guo: • the screenshots for the Pulsar Manager UI. • the logs if possible ---- 2020-01-06 16:11:10 UTC - Roman Popenov: Here is the UI image: ---- 2020-01-06 16:12:01 UTC - Roman Popenov: Actually there are errors in the log of the manager: ---- 2020-01-06 16:12:01 UTC - Roman Popenov: ```16:11:37.279 [pulsar-external-web-4-6] WARN org.eclipse.jetty.server.HttpChannel - /admin/v2/clusters java.lang.IllegalArgumentException: Negative initial size: -1 at <http://java.io|java.io>.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:74) ~[?:1.8.0_232] at org.apache.pulsar.proxy.server.AdminProxyHandler$ReplayableProxyContentProvider.<init>(AdminProxyHandler.java:150) ~[org.apache.pulsar-pulsar-proxy-2.4.2.jar:2.4.2] at org.apache.pulsar.proxy.server.AdminProxyHandler.proxyRequestContent(AdminProxyHandler.java:199) ~[org.apache.pulsar-pulsar-proxy-2.4.2.jar:2.4.2] at org.eclipse.jetty.proxy.ProxyServlet.service(ProxyServlet.java:114) ~[org.eclipse.jetty-jetty-proxy-9.4.20.v20190813.jar:9.4.20.v20190813] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482) ~[org.eclipse.jetty-jetty-servlet-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.Server.handle(Server.java:494) ~[org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268) [org.eclipse.jetty-jetty-server-9.4.20.v20190813.jar:9.4.20.v20190813] at <http://org.eclipse.jetty.io|org.eclipse.jetty.io>.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813] at <http://org.eclipse.jetty.io|org.eclipse.jetty.io>.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813] at <http://org.eclipse.jetty.io|org.eclipse.jetty.io>.ChannelEndPoint$2.run(ChannelEndPoint.java:117) [org.eclipse.jetty-jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) [org.eclipse.jetty-jetty-util-9.4.20.v20190813.jar:9.4.20.v20190813] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]``` ---- 2020-01-06 16:14:03 UTC - Roman Popenov: No errors in the broker or anywhere else, just that `Negative initial size: -1` in the logs of the proxy ```16:13:37.338 [pulsar-external-web-4-8] WARN org.eclipse.jetty.server.HttpChannel - /admin/v2/clusters java.lang.IllegalArgumentException: Negative initial size: -1``` ---- 2020-01-06 16:15:26 UTC - Sijie Guo: is this error in proxy or pulsar-manager? ---- 2020-01-06 16:15:40 UTC - Roman Popenov: Proxy ---- 2020-01-06 16:16:11 UTC - Sijie Guo: oh i see. ---- 2020-01-06 16:17:27 UTC - Sijie Guo: Can you to the environment list (the top-right corn)? Change the environment to use “http://<broker-job-name>:8080”. ---- 2020-01-06 16:17:46 UTC - Roman Popenov: Manager logs seem to be clean: ```Starting PostGreSQL Server The files belonging to this database system will be owned by user "pulsar". This user must also own the server process. The database cluster will be initialized with locales COLLATE: C CTYPE: C.UTF-8 MESSAGES: C MONETARY: C NUMERIC: C TIME: C The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. creating directory /data/postgresql ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default timezone ... UTC selecting dynamic shared memory implementation ... posix creating configuration files ... ok running bootstrap script ... ok sh: locale: not found 2019-12-24 15:41:33.035 UTC [20] WARNING: no usable system locales were found performing post-bootstrap initialization ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. syncing data to disk ... ok Success. waiting for server to start....2019-12-24 15:41:35.734 UTC [24] LOG: listening on IPv4 address "127.0.0.1", port 5432 2019-12-24 15:41:35.735 UTC [24] LOG: could not bind IPv6 address "::1": Address not available 2019-12-24 15:41:35.735 UTC [24] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry. 2019-12-24 15:41:35.737 UTC [24] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" 2019-12-24 15:41:35.740 UTC [24] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" 2019-12-24 15:41:35.751 UTC [25] LOG: database system was shut down at 2019-12-24 15:41:35 UTC 2019-12-24 15:41:35.756 UTC [24] LOG: database system is ready to accept connections done server started ALTER ROLE CREATE DATABASE GRANT You are now connected to database "pulsar_manager" as user "pulsar". CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE Starting Pulsar Manager Front end Starting Pulsar Manager Back end Start servie no enable JWT. 2019-12-24 15:41:37,756 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2019-12-24 15:41:37,762 INFO supervisord started with pid 41 2019-12-24 15:41:38,952 INFO spawned: 'pulsar-manager-backend' with pid 45 2019-12-24 15:41:40,053 INFO success: pulsar-manager-backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)``` ---- 2020-01-06 16:17:59 UTC - Sijie Guo: I am not sure how the exception was thrown yet. but let’s try to use the broker service url as the environment url. ---- 2020-01-06 16:36:52 UTC - juraj: 2.4.2, trying to view a topic message in dashboard fails silently (w/ error 500), anybody has a success w/ that? ---- 2020-01-06 16:50:24 UTC - Sijie Guo: did you see errors in pulsar-dashboard or brokers? ---- 2020-01-06 16:55:15 UTC - Roman Popenov: Sorry, I had to step out ---- 2020-01-06 16:55:31 UTC - Roman Popenov: > I am not sure how the exception was thrown yet. but let’s try to use the broker service url as the environment url. Could you walk me through? ---- 2020-01-06 16:57:34 UTC - juraj: on the broker i just see: ```16:56:55.159 [pulsar-web-30-6] INFO org.eclipse.jetty.server.RequestLog - 192.168.21.190 - - [06/Jan/2020:16:56:55 +0000] "GET //admin/v2/persistent/cb/development/topicname/subscription/arch/position/1 HTTP/1.1" 404 408 "-" "python-requests/2.21.0" 10``` ---- 2020-01-06 16:58:48 UTC - juraj: in browser dev console i see `Status Code: 500 Internal Server Error` ---- 2020-01-06 16:59:00 UTC - juraj: nothing in the dashboard log ---- 2020-01-06 17:02:16 UTC - Roman Popenov: Is there a REST endpoint somewhere to ingest data to Pulsar? ---- 2020-01-06 17:02:50 UTC - Roman Popenov: Or I would need to build on top of the WebSocket API? ---- 2020-01-06 17:18:02 UTC - Lukas Chripko: @Lukas Chripko has joined the channel ---- 2020-01-06 17:18:21 UTC - Sijie Guo: No rest endpoint yet. websocket is the closest option. thanks : Roman Popenov ---- 2020-01-06 17:19:00 UTC - Sijie Guo: Ok. Can you file an issue about this? @tuteng can help looking into it in his morning. ---- 2020-01-06 17:19:02 UTC - Roman Popenov: Time to give back to the community :smile: +1 : Sijie Guo ---- 2020-01-06 17:21:24 UTC - juraj: the dashboard image is actually 2.4.1 as there was another error with 2.4.2 dashboard which completely prevented it from running ---- 2020-01-06 17:24:32 UTC - juraj: <https://github.com/apache/pulsar/issues/6002> ---- 2020-01-06 17:24:59 UTC - juraj: is there a way to use the web api to get message IDs on a topic/subscriptions? i think i have a problem with batch acknowledgements and want to debug it ---- 2020-01-06 17:26:30 UTC - Roman Popenov: Or do you mean to use <http://10.0.0.42:8080/> when I set up the enviroment in the UI? ---- 2020-01-06 17:27:08 UTC - Roman Popenov: Yes, that works ---- 2020-01-06 17:30:01 UTC - Roman Popenov: Using <http://NodeIP>:NodePort/ or <http://Proxy_Job:8080/> doesn’t work ---- 2020-01-06 17:40:51 UTC - Roman Popenov: Partition is `-1` could that be the cause? ---- 2020-01-06 17:54:32 UTC - Sijie Guo: partition is -1 (disabled) means it is a non-partitioned topic. ---- 2020-01-06 17:54:52 UTC - Sijie Guo: but I think the UI isn’t rendered corretly. ---- 2020-01-06 17:55:17 UTC - Sijie Guo: I was suggesting you using `<http://Broker_Job:8080>` man-facepalming : Roman Popenov ---- 2020-01-06 17:55:45 UTC - Sijie Guo: I think <http://NodeIP>:NodePort might not be working well. ---- 2020-01-06 17:56:17 UTC - Roman Popenov: Trying <http://Broker_Job:8080> ---- 2020-01-06 17:56:29 UTC - Sijie Guo: you can use `topics peek` command +1 : juraj ---- 2020-01-06 18:22:59 UTC - Ming: @Roman Popenov We built the first version of this product before holidays to ingest data via an endpoint and push to registered webhook interface. <https://github.com/kafkaesque-io/pulsar-beam/> ---- 2020-01-06 18:25:41 UTC - Ming: In fact, our end-to-end CI uses GCP cloud function to receive push-ed events, process them, reply back to Pulsar topic (as sink source) via our server. I hope this is something can be helpful. ---- 2020-01-06 18:37:32 UTC - Roman Popenov: I will take a look, thank you :smile: ---- 2020-01-06 20:27:15 UTC - Guilherme Perinazzo: Anyone know if the twitter firehose connector can use the powertrack api? ---- 2020-01-06 20:41:05 UTC - Roman Popenov: Is there a way for Pulsar IO or function to monitor an S3 bucket for new files? ---- 2020-01-06 20:47:31 UTC - Roman Popenov: Or an integration with *Amazon CloudWatch Events* of sorts? ---- 2020-01-06 21:22:54 UTC - Roman Popenov: Also another question about the message size, if the data that is being sent is a String/JSON string, is the Max Data Size 5 MB or does the limit only applies for binary data (bytes)? What is the maximum number of chars? ---- 2020-01-06 22:16:25 UTC - Mathieu Druart: Hi ! Does somebody already succeded in using a Schema as input of a pulsar Function (rather than a simple String) ? I try to use a topic with a Avro Schema as the input of my Function, but I always have this exception when I send a message in the topic : `java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to xxx.MyPOJO`, any ideas ? Thank you ! ---- 2020-01-06 23:48:01 UTC - Eron Wright: @Eron Wright has joined the channel ---- 2020-01-06 23:59:08 UTC - Sijie Guo: i think the schema might not have the right avro namespace name. similar to <https://github.com/apache/pulsar/issues/5503> ---- 2020-01-06 23:59:41 UTC - Sijie Guo: it would be great to provide all the steps on how to reproduce your issue. ---- 2020-01-07 00:09:20 UTC - Mathieu Druart: @Sijie Guo I will try find the exact steps I did, but mainly I created the topic like that : `Producer<MyPOJO> producer = pulsarClient.newProducer(AvroSchema.of(MyPOJO.class)).topic("my-topic").create();` ---- 2020-01-07 00:12:24 UTC - Mathieu Druart: when I do this command : `./pulsar-admin schemas get my-topic` ---- 2020-01-07 00:12:33 UTC - Mathieu Druart: I have this answer : ---- 2020-01-07 00:13:39 UTC - Mathieu Druart: ```{ "version": 0, "schemaInfo": { "name": "my-topic", "schema": { "type": "record", "name": "MyPOJO", "namespace": "package.of.my.pojo", "fields": [ { "name": "objet", "type": [ "null", "string" ] }, { "name": "action", "type": [ "null", "string" ] }, { "name": "nb", "type": "int" } ] }, "type": "AVRO", "properties": { "__alwaysAllowNull": "true" } } }``` ---- 2020-01-07 00:17:43 UTC - Mathieu Druart: I have another Consumer (in addition to the Function) for the same Topic, who has no problem to consume directly "MyPOJO" messages ---- 2020-01-07 00:52:23 UTC - juraj: since order is not guaranteed for a multi-topic consumer, is cumulative acknowledgement [with a msgid from just one of the topics] supposed to work [and ack all older-than-msgid messages from all topics of the consumer]? ---- 2020-01-07 02:46:23 UTC - Jacob O'Farrell: This is very cool! Thank you for sharing ---- 2020-01-07 03:37:28 UTC - Fei He: @Fei He has joined the channel ---- 2020-01-07 04:08:38 UTC - fuxiang: @fuxiang has joined the channel ---- 2020-01-07 04:52:58 UTC - Walter: Thank you ---- 2020-01-07 08:28:57 UTC - Vladimir Shchur: No, it will only take effect for the topic this message belongs to ----
