Hey Julian, Your pessimism in this matter is understandable but regrettable!
It would be great to see this effort become part of mainline Druid. It is a more maintainable approach than a separate repo, because it gets rid of the risk of interface drift, and it makes sure that all the tests are run whenever we do a Druid release. It's more upfront work for you (and for us), but Spark and Druid are both important OSS projects and I think it is good to encourage better integration between them. I have also written in the past about the importance of us getting better at accepting contributions (at https://s.apache.org/aqicd). It is not always easy, since reviewing contributions takes time, and it is mostly done on a volunteer basis. But I think if you are game to work with us on this one, let's try to get it in. I say that out of pure idealism, not having looked at the design or code at all 🙂 In the mail I linked, I had written: > For contributors, focusing on UX and tests means writing out (in natural > language) how your patch changes user experience, and why you think this > change is a good idea. It also means having good testing of the new stuff > you're adding, and writing out (in natural language) why you think your > tests cover all the important cases. Speaking as a person that has reviewed > a lot of code: these natural language descriptions are *very helpful*, > especially when they add context to the patch. Don't make reviewers > reverse-engineer your code to guess what you were thinking. As I said, I haven't looked at your design doc or PR yet. But if they cover the above stuff, could you please point me to the right places that have the most up-to-date info, and I will put my money where my mouth is and review them in the way that I suggested in that thread. (i.e., focusing on user experience and test coverage.) By the way, I think the mailing list chomped your links. I'll reproduce them here. 1) Mailing list: https://lists.apache.org/thread.html/r8219a7be0583ae3d9a2303fa7f21872782cf0703812a410bb62acfef%40%3Cdev.druid.apache.org%3E 2) Slack: https://the-asf.slack.com/archives/CJ8D1JTB8/p1581452302483600 3) GitHub: https://github.com/apache/druid/issues/9780 4) Pull request: https://github.com/apache/druid/pull/10920 On Tue, Feb 23, 2021 at 10:37 PM Julian Jaffe <julianfja...@gmail.com> wrote: > > Hey Druids, > > Last April, there was some discussion on this mailing list, Slack, and > GitHub around building Spark-Druid connectors. After working up a rough > cut, the effort was dormant until a few weeks ago when I returned to it. > I’ve opened a pull request for the connectors, but I don’t realistically > expect it to be accepted. Am I too pessimistic in my assumptions here? > Otherwise, what’s the best course of action - create a standalone repo and > add a link in the Druid docs? > > Julian >
