2019-06-24 09:49:18 UTC - Alexandre DUVAL: Hi, interesting point. When I produced message I applied String schema on producer creation. But now there is another service which is pushing into this topic using json formatted without precising schema, so is pulsar determines json schema automatically and there is two schema on my topic or something related? ---- 2019-06-24 13:16:38 UTC - Goclis: @Goclis has joined the channel ---- 2019-06-24 14:14:05 UTC - Darren Hobbs: Hi, I’m investigating using pulsar as the store for an event-source. It looks like a good fit with tiered storage and infinite retention. First question, is this a bad idea? Second question: one requirement I have is that event processors must be up-to-date - they should not publish an event if they have not seen all previous events. eg. a banking system should not process a withdrawal unless it can guarantee it knows the current balance. In a database I would solve this by incrementing a client-side counter and having a unique constraint on (aggregate_id, counter) so if another producer gets there first it will fail the insert and I can recalculate based on the latest event. Is there an equivalent way to achieve this in pulsar? ---- 2019-06-24 14:42:43 UTC - Kiran Krishnamurthy: @Kiran Krishnamurthy has joined the channel ---- 2019-06-24 14:46:14 UTC - Chris Bartholomew: @Darren Hobbs I am working with a colleague on a PoC that uses Pulsar as an event source. It seems like a good idea to us. We are in the early stages (planning), but I can let you know how it goes. ---- 2019-06-24 15:16:39 UTC - Jeremy Taylor: @Darren Hobbs Hi, this is a slightly off-topic answer, but it sounds like you probably should consider using a higher-level system like <https://juxt.pro/crux> (MIT license) to achieve the kinds of database-like semantics you are looking for without losing the benefits of event-sourcing (disclosure: I am the product manager for Crux). Crux currently works on top of Kafka but we are actively considering support for Pulsar as an alternative backend, specifically because it has a strong set of features for infinite retention use-cases (i.e. I believe the answer to your first question is that it's definitely _not_ a bad a idea).
Crux would solve your event processor requirement by providing serialised "compare-and-swap" operations and consistent reads across query nodes (specified using transaction times). We are aiming to announce a formal beta release ahead of our session @ Strange Loop in September (<https://www.thestrangeloop.com/2019/temporal-databases-for-streaming-architectures.html>). I would be very happy to discuss this further with anyone who may be interested :slightly_smiling_face: ---- 2019-06-24 15:32:38 UTC - David Kjerrumgaard: +1 on Pulsar support for Crux..... :smiley: crossed_fingers : Jeremy Taylor ---- 2019-06-24 17:22:30 UTC - Grant Wu: How do we get the MessageId after you produce a message in the Python client? ---- 2019-06-24 17:22:38 UTC - Grant Wu: I see you can get the last sequence ID but I don’t think that’s the same? ---- 2019-06-24 17:43:48 UTC - Grant Wu: @Matteo Merli thoughts? I know the Java client can do it: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/Producer.html#send-T-> ---- 2019-06-24 18:04:46 UTC - David Kjerrumgaard: Perhaps....or something similar causing schema conflicts ---- 2019-06-24 18:18:05 UTC - Adam Schwab: @Adam Schwab has joined the channel ---- 2019-06-24 19:31:16 UTC - Sree Vaddi: we have `pulsar` channel now, on `<http://the-asf.slack.com|the-asf.slack.com>` :slightly_smiling_face: ---- 2019-06-24 23:46:25 UTC - Sree Vaddi: <https://www.meetup.com/SF-Bay-Area-Apache-Pulsar-Meetup/events/zpsdcryzjbwb/> ---- 2019-06-24 23:48:36 UTC - Sree Vaddi: i am here at the location, if anyone planning to come early. ---- 2019-06-25 01:38:05 UTC - Sree Vaddi: Starting soon.... ---- 2019-06-25 01:51:00 UTC - Sijie Guo: @Sree Vaddi it might be worth sending this to a different channel. <#C5Z4T36F7|general> isn’t a good place for broadcasting this kind of messages. +1 : Sree Vaddi ---- 2019-06-25 04:00:11 UTC - Lfwen: @Lfwen has joined the channel ---- 2019-06-25 05:06:32 UTC - Venugopal Doddi: @Sijie Guo Need some help on the attached disk configuration(storage) for the bookkeepers for a persistent storage layer for the pulsar to manage the topics retention and so on.. ---- 2019-06-25 05:06:46 UTC - Venugopal Doddi: any help or reference can you share on this? ---- 2019-06-25 05:07:53 UTC - Venugopal Doddi: right now i have bookkeeper and bookie are on same node and currently holding the default storage that comes as part of provisioning.. they are on OST ---- 2019-06-25 05:29:02 UTC - Venugopal Doddi: in specific, when i use this command `<persistent://my-tenant/my-namespace/my-topic>` to create the topic.. where it will get created? how much space its allocation by default.. and how will i know once the data is produced on this topic is consumed and its deleted from the storage.. ---- 2019-06-25 05:29:16 UTC - Venugopal Doddi: seems like a basic.. but just trying to understand. ---- 2019-06-25 05:42:46 UTC - Sijie Guo: > create the topic.. where it will get created? when the topic is created, only metadata is created. there is no additional allocation for it. the storage is allocated when you start using it. if you don’t use it, there is no space is allocated for it. > how will i know once the data is produced on this topic is consumed pulsar exposed a lot of metrics. you can use `pulsar-admin stats` and `pulsar-admin stats-internal` to know the stats. ---- 2019-06-25 05:57:24 UTC - Venugopal Doddi: thanks @Sijie Guo also one more help on the DLQ, this is some setting i need to make on any conf file or , should be in the client application? ---- 2019-06-25 05:57:54 UTC - Sijie Guo: DLQ is on the client side. ---- 2019-06-25 05:59:05 UTC - Venugopal Doddi: and i have to pre-create the DLQ topic and the no of redelivery attemps? ---- 2019-06-25 06:00:25 UTC - Sijie Guo: by default the topic is created automatically. unless you have configured authentication/authorization and other settings that disable/prevent topic auto creation. ---- 2019-06-25 06:04:04 UTC - Venugopal Doddi: when you say default the topic created automatically.. DLQ will have a specific topic name? ---- 2019-06-25 06:04:06 UTC - Venugopal Doddi: what was that ---- 2019-06-25 06:13:02 UTC - Sijie Guo: no. I mean by default all the pulsar topics are created automatically when the producers and consumers attempt to produce or consume from it the first time. ---- 2019-06-25 06:16:02 UTC - Venugopal Doddi: ohk.. if you dont mind.. can you share some of the client consumer examples where the dead letter topic redeliveries are configured? ---- 2019-06-25 06:18:30 UTC - Sijie Guo: <http://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic> ---- 2019-06-25 06:18:36 UTC - Sijie Guo: check out this documentation ---- 2019-06-25 06:47:59 UTC - linpeng: @linpeng has joined the channel ----
