2019-01-16 09:17:48 UTC - Nguyễn Hữu Phong: @jia zhai segment is like ledger, Each Ledger consisting of one or more Fragments. So each segment consisting of one or more Fragments, true or false +1 : jia zhai ---- 2019-01-16 09:24:54 UTC - Nguyễn Hữu Phong: @jia zhai bookie save segment or save entry ---- 2019-01-16 09:26:45 UTC - Nguyễn Hữu Phong: Can you explain it ? Thanks ---- 2019-01-16 09:33:51 UTC - bossbaby: pulsar is kidding me, a day ago, it workded at local then i deploy it in my vps, sadly that it not successful until night, it worked while im not edit anything, im go home and now, it not work. joy : jia zhai ---- 2019-01-16 09:35:38 UTC - bossbaby: i think entry like segment slightly_smiling_face : jia zhai ---- 2019-01-16 10:04:53 UTC - Guangzhong Yao: Entry is an array of bytes, as the basic element in ledger/segment. +1 : jia zhai ---- 2019-01-16 11:12:36 UTC - naga: @naga has joined the channel ---- 2019-01-16 13:32:10 UTC - Harald Gustafsson: I use TLS as authentication, now I tried also sending a few curl commands in, to see if I could delete a topic and namespace. I managed to delete the topic but not the namespace. I use the same key/cert for topic as for namespace deletion, but the namespace deletetion gives a HTTP 401 Authentication required. I use my super user key/cert. any ideas? ---- 2019-01-16 13:32:45 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem <https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/> ["<persistent://test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/t1\\data>"] ---- 2019-01-16 13:33:01 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem -X DELETE <https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/t1%5cdata> ---- 2019-01-16 13:33:17 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem <https://136.225.152.185:30754/admin/v2/persistent/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/> [] ---- 2019-01-16 13:33:31 UTC - Harald Gustafsson: curl --cacert certs/cacert.pem --cert certs/superUser.bundle.pem -X DELETE <https://136.225.152.185:30754/admin/v2/namespaces/test-20632f25-e674-46c1-a25e-e7bcb74e6d85/testns1/> {"reason":"HTTP 401 Authentication required"} ---- 2019-01-16 13:33:41 UTC - Harald Gustafsson: Any ideas ---- 2019-01-16 13:57:18 UTC - jia zhai: in this pic, you could treat entry as each message produced by pulsar ---- 2019-01-16 13:58:05 UTC - jia zhai: true ---- 2019-01-16 13:59:53 UTC - jia zhai: all entries in this pic is belongs to one segment ---- 2019-01-16 14:32:38 UTC - Bogdan BUNECI: Can I use complex types (arrays of Objects) with Pulsar/Presto ? ---- 2019-01-16 14:33:13 UTC - Bogdan BUNECI: Basic types are ok. ---- 2019-01-16 14:49:08 UTC - jia zhai: in this pic, you could treat entry as each message produced by pulsar ---- 2019-01-16 14:49:21 UTC - jia zhai: all entries in this pic is belongs to one segment ---- 2019-01-16 15:05:26 UTC - Sijie Guo: currently broker hasn’t enabled tls settings in its usage of bookkeeper. so it might require some code changes. do you mind creating a github issue for us? ---- 2019-01-16 15:24:48 UTC - Brian: sure, will do ---- 2019-01-16 15:39:25 UTC - Bogdan BUNECI: I’m using Java producer, pulsar version 2.2.1. ---- 2019-01-16 15:50:29 UTC - Grant Wu: <https://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats> ---- 2019-01-16 15:50:38 UTC - Grant Wu: So not exactly, no ---- 2019-01-16 16:06:47 UTC - Bogdan BUNECI: So, if I attach a schema to a topic I can use only basic types (int, ... string) ? ---- 2019-01-16 16:07:05 UTC - Grant Wu: No, there’s a list of types there ---- 2019-01-16 16:07:26 UTC - Bogdan BUNECI: Sorry, I will read again ---- 2019-01-16 16:07:45 UTC - Grant Wu: There’s no support as far as I can see for arrays of Objects. You need to serialize the arrays of objects to a Protobuf/JSON object/Avro thing first. ---- 2019-01-16 16:08:31 UTC - Bogdan BUNECI: Objects are in Avro format ---- 2019-01-16 16:08:36 UTC - Bogdan BUNECI: Of course ---- 2019-01-16 16:08:53 UTC - Grant Wu: Ah, okay. ---- 2019-01-16 16:08:54 UTC - Bogdan BUNECI: With AvroSchemaOf class ---- 2019-01-16 16:10:00 UTC - Bogdan BUNECI: I’m producing avro records ---- 2019-01-16 16:10:30 UTC - Grant Wu: Okay, I’ve never actually worked with this :sweat_smile: ---- 2019-01-16 16:10:35 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Schema.html#AVRO-java.lang.Class-> seems to imply it takes a Class ---- 2019-01-16 16:10:38 UTC - Bogdan BUNECI: The fields with basic types are ok ... ---- 2019-01-16 16:11:00 UTC - Grant Wu: I would wait for @Matteo Merli or @Sijie Guo to respond ---- 2019-01-16 16:11:53 UTC - Bogdan BUNECI: Ok, Thanks !!!! ---- 2019-01-16 16:29:19 UTC - Matteo Merli: I think that array/map type should work in Presto. I can double check in a bit (and definitely this should be clarified in the docs) ---- 2019-01-16 16:33:48 UTC - Ivan Kelly: one pulsar "segment". I think the confusion is that we have different terminology in pulsar and bookkeeper, but sometimes we use them interchangeably +1 : jia zhai ---- 2019-01-16 17:20:28 UTC - Bogdan BUNECI: Please ! ---- 2019-01-16 17:20:46 UTC - Bogdan BUNECI: I can send you the code ---- 2019-01-16 17:21:01 UTC - Bogdan BUNECI: Thanks ;) ---- 2019-01-16 17:50:23 UTC - Matteo Merli: @Bogdan BUNECI I actually take my statement back. Array fields are not being supported right now. I think it shouldn’t be (hard) to add to support though ---- 2019-01-16 17:51:39 UTC - Bogdan BUNECI: ;) ---- 2019-01-16 17:51:47 UTC - Bogdan BUNECI: I saw ---- 2019-01-16 17:53:06 UTC - Bogdan BUNECI: But just in pulsar topic (with avro schema) should work ? ---- 2019-01-16 17:53:33 UTC - Matteo Merli: Yes, the avro ser/de and validation are ok ---- 2019-01-16 17:53:56 UTC - Matteo Merli: the missing part is bridging an “avro” array into Presto ---- 2019-01-16 17:54:17 UTC - Bogdan BUNECI: We tried, but the nested objects were empty ---- 2019-01-16 17:54:28 UTC - Bogdan BUNECI: In pulsar ---- 2019-01-16 17:54:54 UTC - Matteo Merli: What do you mean? When querying in Presto? ---- 2019-01-16 17:55:00 UTC - Bogdan BUNECI: Nope ---- 2019-01-16 17:55:49 UTC - Bogdan BUNECI: When logging from console consumer ---- 2019-01-16 17:56:14 UTC - Bogdan BUNECI: I can send you all the details to spare some time ---- 2019-01-16 17:56:35 UTC - Matteo Merli: Yes, please, can you open an issue in github with all the details? ---- 2019-01-16 17:56:52 UTC - Bogdan BUNECI: Ok ! ---- 2019-01-16 17:57:24 UTC - Matteo Merli: In general, we’re just passing the object to avro to get us back a byte[] and same (the reverse) in consumer to deserialize it back. ---- 2019-01-16 17:57:46 UTC - Matteo Merli: We shouldn’t be breaking up the internals there ---- 2019-01-16 18:01:05 UTC - Bogdan BUNECI: Than we will double check and if necessary we will open an issue ---- 2019-01-16 19:24:50 UTC - Emma Pollum: I'm having trouble getting my replication to connect. Where does pulsar get the IP for inboundConnection to replicate to ---- 2019-01-16 20:51:00 UTC - Ivan Kelly: @Emma Pollum it's part of the cluster configuration i think ---- 2019-01-16 20:51:56 UTC - Ivan Kelly: <http://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Clusters.html#updateCluster-java.lang.String-org.apache.pulsar.common.policies.data.ClusterData-> ---- 2019-01-16 20:52:24 UTC - Ivan Kelly: brokerServiceUrl in the ClusterData ---- 2019-01-16 20:53:13 UTC - Ivan Kelly: the host in the url should be a dns name backed by multiple ips or a VIP, for fault tolerance ---- 2019-01-16 22:56:38 UTC - Grant Wu: @Jerry Peng Hey, follow-up on this - is that the same for when I put it into a function config YAML? ---- 2019-01-16 23:30:06 UTC - Stepan: Is there any way to send delayed messages in pulsar? Either directly by specifying "I want this message to be delivered in 10 minutes" or with a workaround like specifying TTL on a namespace and then instead of deleting expired messages send them to a specified topic or something like that. ---- 2019-01-16 23:51:53 UTC - Grant Wu: Would it be possible to adjust the semantics of `pulsar-admin functions update` to more closely match that of the HTTP PUT verb that it uses? ---- 2019-01-16 23:52:35 UTC - Grant Wu: HTTP PUT is supposed to “creates a new resource *or* replaces a representation of the target resource with the request payload.” (<https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PUT>, emphasis mine) ---- 2019-01-16 23:53:24 UTC - Grant Wu: However right now if you try to use `pulsar-admin functions update` on a function that doesn’t exist it fails ---- 2019-01-17 00:38:32 UTC - Emma Pollum: I recently deployed a function to pulsar, then deleted it. The subscription remained, however. When I do 'pulsar-admin topics subscriptions TOPICNAME' the function's subscription shows up. When I do 'pulsar-admin topics unsubscribe' it says it can't find the subscription that it JUST listed. Any ideas on how I can get rid of this dangling subscription? ---- 2019-01-17 00:40:51 UTC - Sijie Guo: currently it is not supported yet. there are two outstanding efforts on supporting this feature:
- <https://github.com/apache/pulsar/pull/3155> - <https://github.com/apache/pulsar/issues/2375> ---- 2019-01-17 01:13:23 UTC - Ivan Kelly: @Emma Pollum sounds like a bug. maybe the broker is holding onto the subscription object even though the managed-ledger is deleted ---- 2019-01-17 01:50:36 UTC - Emma Pollum: Unfortunately, the subscription is still filling with messages, can you think of another way to remove the subscription? ---- 2019-01-17 01:51:29 UTC - Matteo Merli: I believe that should have been fixed in the 2.2.1 release (for the pulsar-admin command) ---- 2019-01-17 01:51:47 UTC - Matteo Merli: (issue was around the `/` in the subscription name ---- 2019-01-17 01:53:41 UTC - Emma Pollum: I'm in 2.2.1 ---- 2019-01-17 01:56:18 UTC - Matteo Merli: ok, sorry. that wasn’t actually merged: <https://github.com/apache/pulsar/pull/2910> ---- 2019-01-17 01:57:05 UTC - Matteo Merli: alternative is to make a REST call directly and do the URLencode of the subscription name ---- 2019-01-17 01:58:45 UTC - Matteo Merli: that would be something like: `curl -XDELETE '<http://localhost:8080/admin/v2/persistent/public/default/my-topic/subscription/TENANT%2FNAMESPACE%2FFUNCTION>'` ---- 2019-01-17 04:01:50 UTC - naga: can some one guide me to a doc or video on use case for getting data from pulsar. can pulsar call apis? ---- 2019-01-17 04:37:19 UTC - bossbaby: i found a bug and fixed it: i used tlsCiphers:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and important, if you have a many server broker, you must setup 1 cert in all server +1 : jia zhai ---- 2019-01-17 04:47:42 UTC - jia zhai: @bossbaby you are welcome to make a PR to improve the document. ---- 2019-01-17 04:49:23 UTC - jia zhai: @naga Do you mean to have a try for Pulsar to produce and consume data? <http://pulsar.apache.org/docs/en/standalone/> ---- 2019-01-17 04:51:26 UTC - jia zhai: Here above is a simple one. to consume data: `bin/pulsar-client consume -s "my-sub" my-topic -n 0` to produce data: `bin/pulsar-client produce my-topic --messages "hello-pulsar"` ---- 2019-01-17 05:02:08 UTC - naga: I have some APIs to send data to salesforce and I need call the APIs once the data is in Pulsar. is it possible? ---- 2019-01-17 05:02:53 UTC - naga: Produce part , I am clear, I need to understand subscribe part ---- 2019-01-17 05:03:06 UTC - naga: can we trigger some APis already availble? ---- 2019-01-17 05:07:37 UTC - Nguyễn Hữu Phong: fragment is belongs to ledger or segment ? @jia zhai ---- 2019-01-17 05:09:01 UTC - jia zhai: ---- 2019-01-17 05:09:17 UTC - jia zhai: at the bottom, it is entries. ---- 2019-01-17 05:13:03 UTC - Nguyễn Hữu Phong: So, segment is like ledger, right ? ---- 2019-01-17 05:13:50 UTC - jia zhai: yes, in bookkeeper it is called ledger. ---- 2019-01-17 05:22:45 UTC - Nguyễn Hữu Phong: segment is like ledger, so, in this pic, fragment is belongs to one segment ? ---- 2019-01-17 05:28:56 UTC - naga: or can Pulsar forward the data to subscribers? ---- 2019-01-17 05:33:54 UTC - naga: any suggestions to achive this? ---- 2019-01-17 05:37:42 UTC - Matteo Merli: @naga take a look at consumer in the API: <http://pulsar.apache.org/docs/en/client-libraries-java/#consumers> ---- 2019-01-17 05:39:29 UTC - naga: :+1: ---- 2019-01-17 05:43:03 UTC - naga: I Have gone through and I understand that it is pull from subscriber. which needs to listen and pull whenever message available. ---- 2019-01-17 05:43:45 UTC - naga: Is there any tool /service which I can utilise to call other services to insert and update data ---- 2019-01-17 05:48:12 UTC - Matteo Merli: You can set a message listener to be invoked whenever a message is available: <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#messageListener-org.apache.pulsar.client.api.MessageListener-> ---- 2019-01-17 05:58:39 UTC - naga: When a MessageListener is set, application will receive messages through it. Calls to Consumer.receive() will not be allowed. ---- 2019-01-17 05:58:46 UTC - naga: thanks it is helpful ---- 2019-01-17 06:01:54 UTC - jia zhai: yes ---- 2019-01-17 06:57:46 UTC - bossbaby: Pulsar not support set retention in different topic in the same namespace, its right? ---- 2019-01-17 06:58:40 UTC - Sijie Guo: currently most of the polices are available at namespace level. there is an ongoing work to add topic-level policy. +1 : bossbaby ---- 2019-01-17 07:15:44 UTC - bossbaby: thanks you bro ---- 2019-01-17 08:58:32 UTC - Yifan: Hi, all, I am having problem seeing my tenant in Pulsar Dashboard. When I use pulsar-admin to list the tenants, it is there in the list, but not in Dashboard ---- 2019-01-17 09:07:16 UTC - Yifan: Mmm... it appeared in dashboard now... ---- 2019-01-17 09:08:36 UTC - Sijie Guo: @Yifan it works based on stats. If it doesn’t have traffic, there is no stats and then the topics won’t show up ---- 2019-01-17 09:09:04 UTC - Yifan: Thanks @Sijie Guo Make sense. ----
