It would be interesting to add ai support to query data using natural
language. I have experience with other similar query platforms . Happy to
works and collaborate to make this happen.

On Fri, Nov 13, 2026 at 9:26 PM Holden Karau via dev <[email protected]>
wrote:

> I think it's interesting but I'm hesistant to add only PySpark functions,
> there's no reason we can't build a proper cross-language implementation
> like the rest of our Spark functionality. I've got a bit of a larger WIP
> doc for looking at how we could add a common set (and how we might do it
> differently than "regular" UDFs), would love to collaborate on that
> together if your open to it :)
>
> https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing
>
>
> On 2026-11-12 00:37 UTC Alejandro Morgante wrote:
> > (Resending now that I’m subscribed, as my earlier message doesn’t appear
> in
> > the public archive. Apologies if you receive this twice)
> >
> > Hi Spark devs,
> >
> > I’d like to propose *ai_query* for *PySpark*. It takes a prompt column
> and
> > returns the model’s response as a column. *Databricks SQL* and *Google
> > BigQuery *offer similar capabilities through *ai_query
> > <
> https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query>
> > *and
> > *AI*.*GENERATE*
> > <
> https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate
> >,
> > respectively. Here’s what this could look like in a *PySpark DataFrame.*
> >
> > from pyspark.sql.ai import ai_query, register_modelfrom
> > pyspark.sql.functions import col, concat, lit
> > register_model(
> >     "summarizer",
> >     "databricks-gpt-oss-20b",
> >     provider="databricks",
> >     profile="my-workspace",
> >     auth_type="databricks-cli",
> > )
> > df.withColumn(
> >     "summary",
> >     ai_query(
> >         endpoint="summarizer",
> >         request=concat(lit("Summarize in one sentence: "),
> col("description")),
> >     ),
> > )
> >
> > *register_model* maps *summarizer* to a model endpoint. You can switch
> > between *Databricks Model Serving* and *Amazon Bedrock* by changing the
> > registration, while keeping the same DataFrame code.
> > *The prototype passed end-to-end tests with both providers using Spark
> and
> > Arrow.*
> > What do you think of this *API* for *PySpark*? *I’d love to hear how
> you’d
> > use it and what you’d change.*
> >
> > Draft PR #60311 <https://github.com/apache/spark/pull/60311>
> > JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404>
> >
> > Thanks,
> > Morgan
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [email protected]
>
>

Reply via email to