2020-01-02 13:48:04 UTC - Naby: Yes. ---- 2020-01-02 16:12:07 UTC - hugues DESLANDES: hi everybody. I am testing the queryable state with the python client using the the standalone pulsar with docker (release 2.4.2) I tried to delete the state for one row and got an error like : context.del_counter(word)\n File \"/pulsar/instances/python-instance/contextimpl.py\", line 234, in del_counter\n return self.state_context.delete(key)\nAttributeError: 'BKManagedStateContext' object has no attribute 'delete'\n" After investigation, it seems that the BKManagedStateContext has a delete_key attribute : is ther a misspelling or my misunderstanding ? ---- 2020-01-02 16:13:15 UTC - Sree Vaddi: :bouquet: I Wish You And Your Family A Happy And Prosperous New Year 2020 :bouquet: ---- 2020-01-02 16:25:34 UTC - Naby: Hi @Sijie Guo. I don’t know much about terraform and I was wondering if you can clarify to me where some of the variables were defined in the playbook such as `hostvars` and `inventory_hostname` since there’s no vars.yml file. Thanks. ---- 2020-01-02 16:39:06 UTC - Karthik Ramasamy: Apache pulsar discussion- no 1 in hacker news. ---- 2020-01-02 16:39:15 UTC - Karthik Ramasamy: +1 : Chris Bartholomew, LazyDevLeo, Sree Vaddi, Vladimir Shchur, Ali Ahmed, Yuvaraj Loganathan, Adrian Plani, Sijie Guo, jia zhai heart_eyes : Yuvaraj Loganathan, Sijie Guo, jia zhai bananadance : Yuvaraj Loganathan, Sijie Guo, jia zhai 100 : Yuvaraj Loganathan, Sijie Guo, jia zhai ---- 2020-01-02 17:15:14 UTC - Olav Groenaas Gjerde: @Olav Groenaas Gjerde has joined the channel ---- 2020-01-02 17:19:44 UTC - David Kjerrumgaard: @vikash You can use any of the connectors from the open source code as a reference for an Apache Ignite Sink connector. <https://github.com/apache/pulsar/tree/master/pulsar-io> ---- 2020-01-02 18:45:09 UTC - LazyDevLeo: @LazyDevLeo has joined the channel ---- 2020-01-02 22:08:47 UTC - Yuvaraj Loganathan: And it is still going strong.. +1 : Shivji Kumar Jha ---- 2020-01-02 22:28:06 UTC - Arun Kumar: @Arun Kumar has joined the channel ---- 2020-01-02 22:34:56 UTC - Arun Kumar: I am the founder of a small startup called BayMap Systems. We are exploring the possibilities of using pulsar for our IOT ( for stream processing (pulsar functions) and as a broker. Had a few newbie questions
1. Are pulsar functions "light weight threads" like fibres or are based on event loops like node.js or spring webflux ? Or are they general threads and there is a vertical limit on spawning threads like in Java(due to the stack size limitation? 2. What are good sources for pulsar ( Can we use spring webflux with pulsar or dropwizard for that matter ) for routing click requests from the browser 3. What are the best suitable sinks for pulsar ( DBs --- SQL , NoSQL ? ) Both timeseries and regular RDBMS ? 4. One can connect from the browser to a kafka topic(this greatly helps in real time graphs ). Is this possible in pulsar 5. How good a platform pulsar is for data processing ? Is Flink needed with Pulsar or we can do everything on pulsar itself Best, Arun ---- 2020-01-02 22:36:23 UTC - Arun Kumar: Anybody in this forum from the company ActorCloud ? ---- 2020-01-02 22:49:17 UTC - Bendik: @Bendik has joined the channel ---- 2020-01-02 23:44:58 UTC - Arun Kumar: Hello, After some reading, I cleared my first question --- hence sharing what I learnt There are "function worker" threads if using java and leadership as well. The function threads are given function assignments. This means that this is not a thread per function model and is similar to the event loop model of most reactive systems. But the limitation is that this type of thread model is only supported in Java ---- 2020-01-02 23:47:17 UTC - Arun Kumar: And the assignment of functions again happens through a topic called "Assignment Topic" ---- 2020-01-02 23:47:31 UTC - Arun Kumar: to the function worker thread pool threads ---- 2020-01-02 23:52:02 UTC - Arun Kumar: <https://github.com/apache/pulsar/wiki/PIP-15:-Pulsar-Functions> ---- 2020-01-03 00:09:28 UTC - David Kjerrumgaard: @Arun Kumar FWIW, Pulsar supports websockets, so that might allow you to connect to a pulsar topic via a browser-based javascript function to feed real-time graphs. ---- 2020-01-03 00:10:18 UTC - David Kjerrumgaard: @Arun Kumar There are several sources and sinks currently available, and the framework is extensible to support custom developed IO connectors as well. Are there specific ones you are looking for? ---- 2020-01-03 00:11:50 UTC - David Kjerrumgaard: @Arun Kumar "How good a platform pulsar is for data processing ?" Well, it depends on what you are trying to accomplish. For simple ETL and data manipulation tasks, Pulsar functions are sufficient. However, it also supports integration with Flink, Storm, Spark, etc for more complex event processing if you need it. ---- 2020-01-03 00:19:58 UTC - Jason Liu: hey guys, Is there an easy to toggle my log4j level in a runner cluster? thanks ---- 2020-01-03 00:20:44 UTC - Matteo Merli: Yes, log4j is monitoring changes to the config file ---- 2020-01-03 00:21:21 UTC - Jason Liu: great. thanks ---- 2020-01-03 00:21:57 UTC - Arun Kumar: Thanks a ton David. This helps. The big question whether to use flink or not depends on how exactly a function is executed. If each function is executed as a separate thread, spawned each time.... then it is very heavy. If it is like the reactive event loops with a set of worker threads executes function assignments, then it is awesome. Flink uses actors beneath the surface ( few threads taking up 10s of thousands of actors ) ---- 2020-01-03 00:22:33 UTC - Arun Kumar: My use case is for real time analytics for Internet Of Things ---- 2020-01-03 00:23:34 UTC - Arun Kumar: My concern is --- vertical scaling. I am confident of the horizontal scaling part. But the eventual costs depend on how well we scale vertically as well ---- 2020-01-03 00:24:33 UTC - Adrian Plani: Hey Arun, it really depends on your throughput, complexity of computation and whether you have the dev know how to run flink :grinning: ---- 2020-01-03 00:24:54 UTC - Adrian Plani: What sort of computations are being performed? ---- 2020-01-03 00:25:35 UTC - Arun Kumar: JSON streams received need to be evaluated against rules and based on the pattern matches and filtering, persisted into databases ---- 2020-01-03 00:26:15 UTC - Sanjeev Kulkarni: @Arun Kumar The thread is not spawned for every message. The spawning of the thread happens when you create the function and the thread gets deleted when you delete the function +1 : David Kjerrumgaard ---- 2020-01-03 00:27:07 UTC - Adrian Plani: Sounds like pulsar functions would work (to start with) ---- 2020-01-03 00:28:32 UTC - Arun Kumar: Thanks Adrian and Sanjeev. +1 : Adrian Plani ---- 2020-01-03 00:57:35 UTC - Arun Kumar: 1. React/Angular(Browser) ---> 2. Spring Boot(Webflux-reactive) or DropWizard or node.js ----> 3. pub/sub to topics in pulsar. The question is --> For point#2 of the above flow ---> which software stack in Java works well with pulsar. For example Spring Boot(webflux) has awesome integration with Kafka. Similarly for pulsar, which one ? ---- 2020-01-03 01:05:51 UTC - Arun Kumar: <https://objectpartners.com/2018/08/07/apache-kafka-and-reactive-spring-boot/> ---- 2020-01-03 01:05:57 UTC - Arun Kumar: Similar one for pulsar ---- 2020-01-03 01:58:07 UTC - Rahul Govind: @Rahul Govind has joined the channel ---- 2020-01-03 02:08:38 UTC - jia zhai: @Addison Higham :+1::+1: ---- 2020-01-03 02:17:38 UTC - Addison Higham: Thanks! Now I just need to blog post it but it has gone really well! +1 : jia zhai ---- 2020-01-03 03:17:28 UTC - colinyin: @colinyin has joined the channel ---- 2020-01-03 04:19:36 UTC - Dan: @Dan has joined the channel ---- 2020-01-03 04:48:48 UTC - Rajitha: Hi Everyone, I am a beginner in using pulsar..I am currently working on a microservices based architecture where all the services are connected through pulsar queues ..with new feature flows adding to this system (different events each requiring a different processing logic and priority), I wanted to understand if having multiple pulsar topics being handled by different thread pools within the services is a good approach to have given that pulsar topics are cheap.. Is there a recommended way/standard anyone of you have that can help me to proceed? ---- 2020-01-03 05:54:41 UTC - jia zhai: looking forward to it ---- 2020-01-03 07:06:02 UTC - richi.wang: @richi.wang has joined the channel ---- 2020-01-03 07:30:54 UTC - Karthik Ramasamy: @Addison Higham - you need to write a blog and talk in conferences ----
