2019-07-21 17:55:38 UTC - mapa: @mapa has joined the channel ---- 2019-07-21 17:58:28 UTC - mapa: does pulsar have altrnative for kstream ---- 2019-07-21 17:58:29 UTC - mapa: ? ---- 2019-07-21 18:00:59 UTC - David Kjerrumgaard: @mapa I would recommend looking at Heron Streamlets for this. <https://streaml.io/blog/heron-streamlet-api> ---- 2019-07-21 18:02:01 UTC - mapa: thank you ---- 2019-07-21 18:02:19 UTC - mapa: @David Kjerrumgaard ---- 2019-07-21 18:03:59 UTC - David Kjerrumgaard: @mapa No problem....It also supports joins. <https://apache.github.io/incubator-heron/docs/concepts/streamlet-api/> ---- 2019-07-21 18:13:49 UTC - Jerry Peng: @mapa you can use Pulsar functions ---- 2019-07-21 18:14:58 UTC - Jerry Peng: @mapa <http://pulsar.apache.org/docs/en/functions-overview/> ---- 2019-07-21 19:14:59 UTC - Jon Bock: @mapa Are you asking because you already have a KStreams application and you’re wondering how to port it to Pulsar, or is there a specific capability in KStreams that you’re looking for? As Jerry says, most processing tasks can be handled by Pulsar Functions, or as David says there are a number of systems for stream processing that can be used with Pulsar if you have one of the scenarios that Pulsar functions can’t handle. ---- 2019-07-22 01:56:51 UTC - mapa: not really im looking for a higher abstraction function that acts like kstream and ktable for ETL ---- 2019-07-22 01:57:30 UTC - mapa: @Jon Bock we are in the process of testing out pulsar running side by side with our kafka ---- 2019-07-22 01:58:23 UTC - mapa: since we are a GO and JAVA team we looked at what are the functions needed for use to migrate existing consumers to pulsar ---- 2019-07-22 02:27:43 UTC - Jerry Peng: @mapa awesome! Pulsar Functions is great for etl workloads ---- 2019-07-22 03:49:15 UTC - yansuopeng: @yansuopeng has joined the channel ---- 2019-07-22 05:45:39 UTC - divyasree: Hi Sijieg, I am able to connect via proxy after giving different port number as suggested. Thanks :slightly_smiling_face: ---- 2019-07-22 06:46:12 UTC - pradeep: Hi Team and @Matteo Merli, How does the pulsar cluster (single logical cluster spread across multiple datacenter) behave in case of network breakage in between data-centers? Will there be split brain in pulsar cluster or pulsar handles this in different way? ---- 2019-07-22 08:11:48 UTC - Sijie Guo: cool ---- 2019-07-22 08:34:52 UTC - Sijie Guo: @pradeep if you add correct rack informations to bookies (you can use `bin/pulsar-admin bookies`), pulsar will place the data using rack-aware policy. It will then handle network breakage properly. If you have more than (include) 3 data centers, if one data center goes down, the pulsar cluster can still work as expect. ---- 2019-07-22 08:41:59 UTC - pradeep: In case of 2 data centers , and replication factor 4 (rack aware, 2 replica in each DC), if one data center is down during write, will the write go through? if yes, once the DC comes up, will it start replicating to maintain the replication factor. ---- 2019-07-22 08:47:52 UTC - Sijie Guo: > if one data center is down during write, will the write go through?
it is a complicated situation. - the write can go through, since it will change ensemble. - but if your broker crashes, it has to recover ledgers. recover require ack-quorum-size of bookies are alive. so if you set ack-quorum-size to be more than 2 (says 3), when a dc goes down, you can recover; but if you set ack-quorum-size to 2, it can lead to split-brain situation, which can cause data-loss during data recovery. ----
