2019-04-02 12:55:36 UTC - Daniel Ferreira Jorge: Hello again! I have a scenario where I need to consume from a X partitions topic A and produce to a Y partitions topic B, using a failover subscription. This mean I will have more than 1 processes (up to X, but not necessarily equal to X) consuming from A and producing to B. How can I produce to B in an idempotent manner? Is it even possible? I do not see how, using sequence IDs derived from the message IDs from topic A, I can guarantee no duplication on B... Are there any best practices for this kind of scenarios? ---- 2019-04-02 15:05:50 UTC - Ryan Samo: @Matteo Merli is there a way to get the MessageIdImpl() by a given time stamp? I’m dealing with a situation where I have 18 partitions but need to read data that has already been acknowledged in the past. Like if you didn’t process a few messages on the consumer side, we need a way to make up the data gap as easy as possible. Thoughts? ---- 2019-04-02 16:15:08 UTC - Sanjeev Kulkarni: You need to specify the name. Just don’t specify namespace and tenant ---- 2019-04-02 16:17:25 UTC - Ryan Samo: Just trying to come up with a way to position the cursor for a reader or consumer more easily. I do see that there is a consumer.seek being enhanced to use time, that might be exactly what I need ---- 2019-04-02 16:49:51 UTC - ever: @ever has joined the channel ---- 2019-04-02 21:43:49 UTC - Thor Sigurjonsson: Hey, I have a question. Is there some way to get to Message Properties in Pulsar Functions? I'm looking at processing metadata about messages in functions without touching the value/data payload. It looks like we have that in the client API but I'm not finding it in the Functions API. ---- 2019-04-02 21:46:17 UTC - Sanjeev Kulkarni: @Thor Sigurjonsson Context.getCurrentRecord().getProperties() ---- 2019-04-02 21:46:23 UTC - Sijie Guo: In java function, you can use `context.getCurrentRecord().getProperties()` to retrieve all the properties associated with the current message. ---- 2019-04-02 21:46:59 UTC - Thor Sigurjonsson: @Sanjeev Kulkarni @Sijie Guo Thanks! Is there also a way to publish messages with Properties? ---- 2019-04-02 21:49:20 UTC - Sanjeev Kulkarni: Currently not, but something that can be added fairly easily. ---- 2019-04-02 21:49:41 UTC - Sanjeev Kulkarni: Let me know if you want to take a crack at it ---- 2019-04-02 22:09:43 UTC - Devin G. Bost: Does anyone have any suggestions for debugging pulsar functions? Ideally, I'd like to be able to set a breakpoint and step through my function when it gets data. ---- 2019-04-02 22:11:49 UTC - Ali Ahmed: @Devin G. Bost we don’t have a local runner yet it will take some work to implement it , options are to attach a agent based debugger or set up jvm flags in standalone mode and run the function in threaded mode ---- 2019-04-02 22:14:27 UTC - Grant Wu: printf debugging? ---- 2019-04-02 22:14:47 UTC - Grant Wu: Also can’t you run the function locally ---- 2019-04-02 22:14:52 UTC - Grant Wu: `localrun` etc. ---- 2019-04-02 23:22:03 UTC - Devin G. Bost: What would be involved in creating a local runner? ---- 2019-04-02 23:29:26 UTC - Ali Ahmed: depends on the level of capability a simple one man be built with mocks ---- 2019-04-02 23:30:21 UTC - Ali Ahmed: here is a sample of what needs to be done ``` <https://github.com/apache/pulsar/blob/6142a4c5b03f5a4e26a774fd57ab458844255786/pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java> ``` +1 : Devin G. Bost ---- 2019-04-03 00:07:00 UTC - Grant Wu: Sorry, I meant literally this: @Devin G. Bost <https://pulsar.apache.org/docs/en/functions-overview/#local-run-mode> +1 : Devin G. Bost ---- 2019-04-03 00:07:17 UTC - Grant Wu: Like, if it’s running locally, the process is local, you can attach a debugger just as you would to any other local process, etc. +1 : Devin G. Bost ---- 2019-04-03 01:17:42 UTC - Sree Vaddi: <https://thenewstack.io/10-min-survey-data-streamings-role-in-application-architecture/> -1 : Grant Wu, Khoa Tran, Sijie Guo, Guangzhong Yao, Kendall Magesh-Davis ---- 2019-04-03 02:01:12 UTC - Thor Sigurjonsson: @Sanjeev Kulkarni Yes, I'd like that. It would be my first on this, but I'm up for it. I have started looking at what's involved and have begun getting things all wired and set up. I may have some questions along the way if that is OK. I'd love to contribute more once I've gotten this one thing through too. ---- 2019-04-03 02:03:02 UTC - Sanjeev Kulkarni: Any place you get stuck please feel free to either ping general salck or directly to me ---- 2019-04-03 02:05:38 UTC - Thor Sigurjonsson: Thank you! I forked it and have just been starting working through getting the project compiling and seeing tests run... I have a first stab at the changes and will want to add tests and play with that a bit more. Right now I'm also just getting familiar with the structure of the project and what needs to be compiled and working before the functions module will be working that way... ---- 2019-04-03 02:06:23 UTC - Thor Sigurjonsson: Any tips on the project structure may be helpful as I orient myself. ---- 2019-04-03 02:06:47 UTC - Thor Sigurjonsson: I started with: <https://pulsar.apache.org/en/contributing/> ---- 2019-04-03 02:09:05 UTC - Thor Sigurjonsson: I'm guessing I need to work with `pulsar-functions-api` and `pulsar-functions-instance` to get these changes in. ---- 2019-04-03 02:10:30 UTC - Sanjeev Kulkarni: Yes for this work the main classes that you will touch are the context interface in functions api and the context impl in the instance ---- 2019-04-03 02:11:23 UTC - Sanjeev Kulkarni: That’s for java. Though it will be nice to also add this to python as well. Those will be in context.py and contextimol.oy ---- 2019-04-03 02:13:02 UTC - Thor Sigurjonsson: Nice, yes that's a good idea -- have that api symmetry. Good, yes those are the files I have been touching. ---- 2019-04-03 02:13:35 UTC - Kendall Magesh-Davis: Please don’t (at) here for random things +1 : Sijie Guo, Khoa Tran, Guangzhong Yao, Sree Vaddi ---- 2019-04-03 06:34:14 UTC - Gaurav Ashok: @Gaurav Ashok has joined the channel ---- 2019-04-03 08:01:47 UTC - Michael Bongartz: @Michael Bongartz has joined the channel ---- 2019-04-03 08:23:57 UTC - Michael Bongartz: Hi there ! I’m not sure this is the right place to ask my question but I’ll try anyway. I’ve set up a Pulsar cluster with separate VMs for proxies/brokers/bookies/Znodes and have been testing it for a few days perfectly. I implemented TLS encryption, tested it, everything still working flawlessly.
Yesterday, I tried to configure client authentication based on token (exclusively on proxies for a start) by following this documentation: <https://pulsar.apache.org/docs/en/security-token-admin/> The error I get when starting up pulsar proxy is : [sun.misc.Launcher$AppClassLoader@6996db8] error Uncaught exception in thread main: Failed to load an authentication provider. From what I understand (after having looked at the code), it couldn’t find/load the class org.apache.pulsar.broker.authentication.AuthenticationProviderToken for some reason. I am using pulsar release 2.3.0, openjdk version 1.8.0_201 on CentOS 7, I already checked for jars integrity, when I list classes that are packaged in org.apache.pulsar-pulsar-broker-common-2.3.0.jar I can see the AuthenticationProviderToken.class and when using AuthenticationProviderTls as authentication provider, pulsar proxy starts without such error. Is anyone successfuly using client authentication based on token ? Could someone maybe help ? ---- 2019-04-03 09:01:57 UTC - jia zhai: @Michael Bongartz what is the auth conf in your proxy.conf file? ----
