Sure thing. https://debezium.io is an open-source project largely developed by Red Hat that standardizes change data capture across a variety of data sources. It ships with connectors for Oracle, SQL Server, MySQL, PostgreSQL, and MongoDB, and typically records changes into topics in Apache Kafka.
CouchDB, of course, has a change capture feed as a public API endpoint, so a connector should be a smaller effort than consuming the WAL from a traditional RDMS. Integrating with Debezium would provide an easy way to translate the _changes feed into a Kafka topic which plugs us into a much larger ecosystem of tools and alleviates the need for every consumer of data in CouchDB to build a bespoke “follower” of the _changes feed. Happy to dig up answers to other questions and to make connections with Kafka / Debezium experts where it makes sense. Cheers, Adam > On Apr 2, 2021, at 5:44 PM, Bessenyei Balázs Donát <bes...@apache.org> wrote: > > Thank you for sharing the idea, Adam. I like it! > Can you please share some details? > > > Donat > > On Wed, Mar 31, 2021 at 2:14 AM Adam Kocoloski <kocol...@apache.org> wrote: >> >> I would bias towards ecosystem integrations, e.g. a metrics exporter for >> Prometheus (I think one is in the works), or a Debezium connector for the >> _changes feed (I haven’t seen any movement here). I suppose the Debezium >> connector would ideally be part of the Debezium project itself, but that >> feels like one that is reasonably self-contained and would provide a good >> return on investment. Happy to help expand on that one a bit if you think >> it’s worthwhile. >> >> Adam >> >>> On Mar 30, 2021, at 4:42 AM, Bessenyei Balázs Donát <bes...@apache.org> >>> wrote: >>> >>> Wow, I didn't realize we are that far into the timeline. >>> >>> I actually don't have a well-scoped idea for GSoC this year. >>> Does anyone have good candidates for a project? >>> >>> >>> Donat >>> >>> On Mon, Mar 29, 2021 at 6:23 PM Joan Touzet <woh...@apache.org> wrote: >>>> >>>> https://community.apache.org/gsoc.html might be of help. >>>> >>>> If you are looking to talk to other Apache projects that have done this >>>> before, you could reach out on the dev@community.a.o list. >>>> >>>> >>>> https://lists.apache.org/thread.html/r189a563fe003ad8f0e4c298e18fad4da8d4b2854bd2a5d741ae3ac45%40%3Cdev.community.apache.org%3E >>>> >>>> Note the dependency on JIRA: >>>> >>>>> All ASF projects are invited to submit their ideas to their issue >>>>> tracker, please be sure to add the labels “gsoc2021” and “mentor” so that >>>>> we can automatically include them in our list of subjects. If your >>>>> project does not use JIRA please contact d...@community.apache.org. >>>> >>>> so you'll need to post at dev@community to get included in the master list. >>>> >>>> Student applications start tomorrow for 2 weeks, so you'll need to get a >>>> move on... If I'm around on chat I can try and help a bit. >>>> >>>> -Joan >>>> >>>> On 29/03/2021 01:20, Bessenyei Balázs Donát wrote: >>>>> If there are any projects that don't exceed my CouchDB / erlang / JS >>>>> knowledge, I'd make sure I'm available enough to support someone doing >>>>> a GSoC with us. >>>>> What's the workflow here? Do we have to apply as a project? Do we have >>>>> to propose projects? >>>>> I did look at "Prospective ASF mentors: read this" of [1], but I don't >>>>> see what it looks like for a project. Do we need a vote here? >>>>> >>>>> >>>>> Donat >>>>> >>>>> [1]: https://community.apache.org/gsoc.html >>>>> >>>>> On Sun, Mar 28, 2021 at 11:52 PM Joan Touzet <woh...@apache.org> wrote: >>>>>> >>>>>> The ASF often ends up doing GSoC. I don't think we've ever had the >>>>>> sponsor within the project for it (or for Outreachy, for that matter). >>>>>> >>>>>> The most critical part is being available on a regular basis for proper >>>>>> mentoring. If you don't think you can get that into your schedule, don't >>>>>> volunteer. Assume you will get zero support from any other developer >>>>>> (not true, but best to plan for the worst case situation...) >>>>>> >>>>>> The second most critical part is to come up with a self-contained >>>>>> project that makes sense for CouchDB. The most obvious thing to me would >>>>>> be Fauxton work, esp. as it falls into the "sweet spot" of JS >>>>>> development. I dunno how good of a target main is, given how in flux it >>>>>> is; others might have a better take on that. There's also this PR that >>>>>> never finished up: >>>>>> >>>>>> https://github.com/apache/couchdb/issues/1254 >>>>>> >>>>>> These topics are all probably too big, but maybe one of them could be >>>>>> cut down to something summer-sized: >>>>>> >>>>>> https://github.com/apache/couchdb/projects/1 >>>>>> >>>>>> Thanks for taking on this initiative! I know for a fact I won't have >>>>>> time this summer, or I'd agree to join you. >>>>>> >>>>>> -Joan >>>>>> >>>>>> On 28/03/2021 15:59, Bessenyei Balázs Donát wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> I've just seen that the ASF is accepted as a mentoring organisation >>>>>>> for GSoC 2021. >>>>>>> Is CouchDB interested in participating? >>>>>>> I've never done a GSoC before, but I'd certainly be interested. I'd be >>>>>>> happy to help a student contribute to CouchDB. >>>>>>> >>>>>>> What do you all think? >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Donat >>>>>>> >>
