It would be interesting to add ai support to query data using natural language. I have experience with other similar query platforms . Happy to works and collaborate to make this happen.
On Fri, Nov 13, 2026 at 9:26 PM Holden Karau via dev <[email protected]> wrote: > I think it's interesting but I'm hesistant to add only PySpark functions, > there's no reason we can't build a proper cross-language implementation > like the rest of our Spark functionality. I've got a bit of a larger WIP > doc for looking at how we could add a common set (and how we might do it > differently than "regular" UDFs), would love to collaborate on that > together if your open to it :) > > https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing > > > On 2026-11-12 00:37 UTC Alejandro Morgante wrote: > > (Resending now that I’m subscribed, as my earlier message doesn’t appear > in > > the public archive. Apologies if you receive this twice) > > > > Hi Spark devs, > > > > I’d like to propose *ai_query* for *PySpark*. It takes a prompt column > and > > returns the model’s response as a column. *Databricks SQL* and *Google > > BigQuery *offer similar capabilities through *ai_query > > < > https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query> > > *and > > *AI*.*GENERATE* > > < > https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate > >, > > respectively. Here’s what this could look like in a *PySpark DataFrame.* > > > > from pyspark.sql.ai import ai_query, register_modelfrom > > pyspark.sql.functions import col, concat, lit > > register_model( > > "summarizer", > > "databricks-gpt-oss-20b", > > provider="databricks", > > profile="my-workspace", > > auth_type="databricks-cli", > > ) > > df.withColumn( > > "summary", > > ai_query( > > endpoint="summarizer", > > request=concat(lit("Summarize in one sentence: "), > col("description")), > > ), > > ) > > > > *register_model* maps *summarizer* to a model endpoint. You can switch > > between *Databricks Model Serving* and *Amazon Bedrock* by changing the > > registration, while keeping the same DataFrame code. > > *The prototype passed end-to-end tests with both providers using Spark > and > > Arrow.* > > What do you think of this *API* for *PySpark*? *I’d love to hear how > you’d > > use it and what you’d change.* > > > > Draft PR #60311 <https://github.com/apache/spark/pull/60311> > > JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404> > > > > Thanks, > > Morgan > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
