Re: Spark Druid connectors, take 2

jiv Mordani >> wrote: >> >>> Will, Julian, >>>See responses below tagged with [Rajiv] in blue: >>> >>> From: Will Xu >>> Date: Tuesday, August 8, 2023 at 9:27 AM >>> To: dev@druid.apache.org >>> Subject: Re

Re: Spark Druid connectors, take 2

aytas > > > > On Wed, Aug 9, 2023 at 2:14 PM Rajiv Mordani > wrote: > > > Will, Julian, > > See responses below tagged with [Rajiv] in blue: > > > > From: Will Xu > > Date: Tuesday, August 8, 2023 at 9:27 AM > > To: dev@dr

Re: Spark Druid connectors, take 2

wrote: > Will, Julian, > See responses below tagged with [Rajiv] in blue: > > From: Will Xu > Date: Tuesday, August 8, 2023 at 9:27 AM > To: dev@druid.apache.org > Subject: Re: Spark Druid connectors, take 2 > !! External Email > > For which versio

Re: Spark Druid connectors, take 2

Looks like the “blue” fonts didn’t go through to the mailing list 😊. However it is still tagged with [Rajiv]. * Rajiv From: Rajiv Mordani Date: Wednesday, August 9, 2023 at 2:14 PM To: dev@druid.apache.org Subject: Re: Spark Druid connectors, take 2 !! External Email Will, Julian

Re: Spark Druid connectors, take 2

Will, Julian, See responses below tagged with [Rajiv] in blue: From: Will Xu Date: Tuesday, August 8, 2023 at 9:27 AM To: dev@druid.apache.org Subject: Re: Spark Druid connectors, take 2 !! External Email For which version to target, I think we should survey the Druid community

Re: Spark Druid connectors, take 2

Yes, it does make sense. For #2 (Spark reads Druid), I think Spark also needs to be able to get schema from Druid. This is probably a query to the broker. I wonder what's the UX look like for Spark SQL users on how they specify the schema. Would they create an EXTERNAL TABLE in Spark that maps to a

Re: Spark Druid connectors, take 2

For proper disclosure, it's been a while since I used Druid, but here's my 2 cents w.r.t Will's question (based on what I originally wrote in this design doc ): 1. *Spark writes to Druid*: 1. Based on

Re: Spark Druid connectors, take 2

For which version to target, I think we should survey the Druid community and get input. In your case, which version are you currently deploying? Historical experience tells me we should target current and current-1 (3.4.x and 3.3.x) In terms of the writer (Spark writes to Druid), what's the user

Spark Druid connectors, take 2

Hey all, There was talk earlier this year about resurrecting the effort to add direct Spark readers and writers to Druid. Rather than repeat the previous attempt and parachute in with updated connectors, I’d like to start by building a little more consensus around what the Druid dev community w