2018-04-04 12:03:55 UTC - Karthik Palanivelu: Yes that is correct, connector from Druid to pulsar to ingest the data from topic ---- 2018-04-04 12:04:26 UTC - Karthik Palanivelu: And currently what is the file size it supports? ---- 2018-04-04 15:49:32 UTC - Matteo Merli: the max message size is 5MB ---- 2018-04-04 16:55:23 UTC - Igor Zubchenok: Hi guys! How can I create a subscription with Pulsar Java Client without actual message fetching until I ready to process them? ---- 2018-04-04 17:10:27 UTC - Matteo Merli: Not sure what’s your intention here, though there are different options for different cases: * You can use REST/Java/CLI admin API to create a subscription without attaching a consumer: `bin/pulsar-admin persistent create-subscription $MY_TOPIC --subscription $MY_SUBSCRIPTION` * You can use a consumer with `receiverQueueSize=0`. That would disable the prefetching so a consumers only fetches messages when it’s ready to process (eg: when it calls `receive()`). ---- 2018-04-04 17:11:00 UTC - Igor Zubchenok: Better option for me: when I create consumer I need to start with all unacknowledged messages? Currently I get only messages since subscription is created by consumer. ---- 2018-04-04 17:12:08 UTC - Matteo Merli: “unacknowledged messages” --> messages retained by subscription and not acknowledged by consumer ---- 2018-04-04 17:12:29 UTC - Igor Zubchenok: Thanks a lot, I found admin.persistentTopics().createSubscriptionAsync method. ---- 2018-04-04 17:12:30 UTC - Matteo Merli: the subscription needs to be created and pointing at some message ---- 2018-04-04 17:12:50 UTC - Igor Zubchenok: How can I point it to the earliest message when I create a subscription with consumer? ---- 2018-04-04 17:13:44 UTC - Matteo Merli: We have the option to create a subscription on “earliest” message in coming 2.0 release ---- 2018-04-04 17:15:02 UTC - Igor Zubchenok: So workaround is to create a subscription with MessageId.earliest via admin and then create a consumer, right? ---- 2018-04-04 17:15:07 UTC - Matteo Merli: <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#subscriptionInitialPosition-org.apache.pulsar.client.api.SubscriptionInitialPosition-> ---- 2018-04-04 17:15:40 UTC - Matteo Merli: > So workaround is to create a subscription with MessageId.earliest via admin and then create a consumer, right?
Yes, that would effectively the same. A bit less convenient, but the same ---- 2018-04-04 17:16:14 UTC - Igor Zubchenok: Looking forward for 2.0, it looks much better designed ---- 2018-04-04 17:16:25 UTC - Igor Zubchenok: Is it a long way to get it released? ---- 2018-04-04 17:16:53 UTC - Matteo Merli: few weeks only +1 : Igor Zubchenok, Ali Ahmed ---- 2018-04-04 17:52:41 UTC - Piotr: Can´t you use a Reader as well? Or is that also in 2.0? ---- 2018-04-04 17:52:58 UTC - Piotr: won´t be a subscription, but might work with your use case ---- 2018-04-04 18:24:45 UTC - Matteo Merli: reader is available since 1.20 ---- 2018-04-04 18:25:34 UTC - Matteo Merli: though you, the difference is that with subscription you just need to set where to initialize it and then it will keep track of poisiton, while reader you have to do it each time ---- 2018-04-04 18:47:38 UTC - Igor Zubchenok: Subscriptions works better for me. ---- 2018-04-04 19:38:39 UTC - Niraj Punjabi: @Niraj Punjabi has joined the channel ---- 2018-04-04 20:24:43 UTC - Piotr: @Igor Zubchenok slack tip: You can delete messages, like in this case when you have asked about something and manage to solve it yourself before anyone else has seen/answered it :wink: ---- 2018-04-05 01:52:42 UTC - Igor Zubchenok: thanks, will do next time. ---- 2018-04-05 02:44:39 UTC - Eric Asuncion: @Eric Asuncion has joined the channel ---- 2018-04-05 05:00:48 UTC - Vicky Kak: @Vicky Kak has joined the channel ---- 2018-04-05 05:17:54 UTC - Vicky Kak: Hey Guys, I have been using the Kafka for some time. I am wondering if there is a Kafka Source Connect feature plan for the Apache Pulsar, I can see the funtions which are similar to the Kafka Stream. Here is the kafka source e.g that I have been expected to port to Apache Pulsar <https://github.com/jcustenborder/kafka-connect-twitter> ---- 2018-04-05 05:19:09 UTC - Vicky Kak: I have been watching <https://streaml.io/> for some time, they have got useful resources, however did not find what I wanted. ---- 2018-04-05 06:54:38 UTC - Ali Ahmed: @Vicky Kak there are plans to introduce connectors on pulsar based on pulsar functions and to be releases as part 2.0 ---- 2018-04-05 06:55:03 UTC - Ali Ahmed: porting or writing new connectors will be relatively simple ---- 2018-04-05 06:56:03 UTC - Ali Ahmed: also since pulsar functions will support python natively sources and sinks can be written or ported in python also ---- 2018-04-05 07:36:08 UTC - Vicky Kak: @Ali Ahmed I am not able to see the JIRA for the pulsar, I would be interested to know the initiative from contributers perspective. It would be great if you can point out to the discussion thread about the same. ---- 2018-04-05 07:38:07 UTC - Vicky Kak: I am also thinking of working in the pulsar based sink for the Apache Gobblin i.e you can see the Kafka sink here <https://gobblin.readthedocs.io/en/latest/sinks/Kafka/> ---- 2018-04-05 07:39:01 UTC - Vicky Kak: I am currently using the Kafka/Apache Gobblin. I would be interested to take a look at the Pulsar, looks interesting option. ---- 2018-04-05 07:41:55 UTC - Ali Ahmed: We are still in planning stages , but in general they are two parts to a source a sink system the general logic and the executor. ---- 2018-04-05 07:42:49 UTC - Ali Ahmed: gobblin , nifi , kafka emulation are all valid approaches but the general consensus is to build on the foundations here <https://github.com/openconnectors/connectors> ---- 2018-04-05 07:43:40 UTC - Ali Ahmed: the long term goal is to incubate these connectors under the open messaging group <http://openmessaging.cloud/> ---- 2018-04-05 07:45:17 UTC - Ali Ahmed: and provide an execution an layer on top of pulsar functions and other execution engines like local jvm process etc ---- 2018-04-05 07:54:49 UTC - Vicky Kak: Thanks. ----
