Re: Spark Druid connectors, take 2

i >> wrote: >> >>> Will, Julian, >>>See responses below tagged with [Rajiv] in blue: >>> >>> From: Will Xu >>> Date: Tuesday, August 8, 2023 at 9:27 AM >>> To: dev@druid.apache.org >>> Subject: Re: Spark Dr

Re: Spark Druid connectors, take 2

Maytas > > > > On Wed, Aug 9, 2023 at 2:14 PM Rajiv Mordani > wrote: > > > Will, Julian, > > See responses below tagged with [Rajiv] in blue: > > > > From: Will Xu > > Date: Tuesday, August 8, 2023 at 9:27 AM > > To: dev@druid.apache

Re: Spark Druid connectors, take 2

wrote: > Will, Julian, > See responses below tagged with [Rajiv] in blue: > > From: Will Xu > Date: Tuesday, August 8, 2023 at 9:27 AM > To: dev@druid.apache.org > Subject: Re: Spark Druid connectors, take 2 > !! External Email > > For which versio

Re: Spark Druid connectors, take 2

Looks like the “blue” fonts didn’t go through to the mailing list . However it is still tagged with [Rajiv]. * Rajiv From: Rajiv Mordani Date: Wednesday, August 9, 2023 at 2:14 PM To: dev@druid.apache.org Subject: Re: Spark Druid connectors, take 2 !! External Email Will, Julian

Re: Spark Druid connectors, take 2

Will, Julian, See responses below tagged with [Rajiv] in blue: From: Will Xu Date: Tuesday, August 8, 2023 at 9:27 AM To: dev@druid.apache.org Subject: Re: Spark Druid connectors, take 2 !! External Email For which version to target, I think we should survey the Druid community

Re: Spark Druid connectors, take 2

Yes, it does make sense. For #2 (Spark reads Druid), I think Spark also needs to be able to get schema from Druid. This is probably a query to the broker. I wonder what's the UX look like for Spark SQL users on how they specify the schema. Would they create an EXTERNAL TABLE in Spark that maps to

Re: Spark Druid connectors, take 2

For proper disclosure, it's been a while since I used Druid, but here's my 2 cents w.r.t Will's question (based on what I originally wrote in this design doc ): 1. *Spark writes to Druid*: 1. Based on

Re: Spark Druid connectors, take 2

For which version to target, I think we should survey the Druid community and get input. In your case, which version are you currently deploying? Historical experience tells me we should target current and current-1 (3.4.x and 3.3.x) In terms of the writer (Spark writes to Druid), what's the user

Spark Druid connectors, take 2

Hey all, There was talk earlier this year about resurrecting the effort to add direct Spark readers and writers to Druid. Rather than repeat the previous attempt and parachute in with updated connectors, I’d like to start by building a little more consensus around what the Druid dev community

Re: Spark-Druid Connectors

Bimonthly ping for reviews :) I’m perfectly willing to hop on Slack or a video call to walk through the code and design as well if potential reviewers would find that helpful. > On Apr 14, 2021, at 10:06 AM, Julian Jaffe wrote: > >  > Hey Samarth, > > I’m overjoyed to hear that! The PR is

Re: Spark-Druid Connectors

Hey Samarth, I’m overjoyed to hear that! The PR is here: https://github.com/apache/druid/pull/10920. I’ll add you as a reviewer as well when I have a moment. Thanks, Julian > On Apr 14, 2021, at 12:09 AM, Samarth Jain wrote: > > Hi Julian, > > I would be happy to review your Spark-Druid

Re: Spark-Druid Connectors

Hi Julian, I would be happy to review your Spark-Druid connector PRs. Ingesting data into Druid using Spark SQL and Dataframe API is something we are very keen to onboard. Could you point me to them or alternatively add me as a reviewer? - Samarth On Tue, Apr 13, 2021 at 11:51 PM Julian Jaffe

Re: Spark-Druid Connectors

Hey Gian and other Druids, Is there anything I can do to encourage reviews of this code? Would a dev guide or design doc be helpful to reviewers? Can I bribe someone with chocolate :)? Thanks, Julian > On Mar 2, 2021, at 9:53 AM, Gian Merlino wrote: > > Thank you! > >> On Thu, Feb 25, 2021

Re: Spark-Druid Connectors

Thank you! On Thu, Feb 25, 2021 at 12:03 AM Julian Jaffe wrote: > Hey Gian, > > I’d be overjoyed to be proven wrong! For what it’s worth, my pessimism was > not driven by a lack of faith in the Druid community or the Druid > committers but by the fact that these connectors may be an awkward fit

Re: Spark-Druid Connectors

Hey Gian, I’d be overjoyed to be proven wrong! For what it’s worth, my pessimism was not driven by a lack of faith in the Druid community or the Druid committers but by the fact that these connectors may be an awkward fit in the Druid code base without more buy-in from the community writ

Re: Spark-Druid Connectors

ast April, there was some discussion on this mailing list, Slack, and > GitHub around building Spark-Druid connectors. After working up a rough > cut, the effort was dormant until a few weeks ago when I returned to it. > I’ve opened a pull request for the connectors, but I don’t realistically >

Spark-Druid Connectors

Hey Druids, Last April, there was some discussion on this mailing list, Slack, and GitHub around building Spark-Druid connectors. After working up a rough cut, the effort was dormant until a few weeks ago when I returned to it. I’ve opened a pull request for the connectors, but I don’t

Spark-Druid Connectors Proposal

Hey all, There have been ongoing discussions on this list and in Slack about improving interoperability between Spark and Druid by creating Spark connectors that can read from and write to Druid clusters. As these discussions have begun to converge on a potential solution, I've opened a proposal