Hi Morgan, Thanks for sharing the draft document and your prompt acknowledgement. I see this as a step towards bringing in AI functions within spark. Thanks again for sharing the same & happy to work together.
Regards, Hemil On Sat, Nov 14, 2026 at 1:15 AM Alejandro Morgante < [email protected]> wrote: > Hi everyone, > > Thanks for the interest! We're working on an SPIP for `ai_query` with > Daniel Tenedorio and Norio Akagi. > > The PySpark PR is a reference prototype. The direction we're exploring is > a cross-language implementation: a native SQL function backed by Catalyst, > with DataFrame wrappers and a shared execution backend. > > Holden, we're using your broader proposal as a guide and reference, and > we'd like to coordinate the designs. > > Our shared draft is here: > > https://docs.google.com/document/d/1mAivaYyJaB_Hi676TvrXJA8CXsC1JS6fxJVGeo1yi5A/edit > > You're both very welcome to contribute to the SPIP and implementation. > We'd be happy to work on this together! > > Morgan > > El vie, 13 nov 2026 a las 23:32, Hemil Shah (<[email protected]>) > escribió: > >> It would be interesting to add ai support to query data using natural >> language. I have experience with other similar query platforms . Happy to >> works and collaborate to make this happen. >> >> On Fri, Nov 13, 2026 at 9:26 PM Holden Karau via dev < >> [email protected]> wrote: >> >>> I think it's interesting but I'm hesistant to add only PySpark >>> functions, there's no reason we can't build a proper cross-language >>> implementation like the rest of our Spark functionality. I've got a bit of >>> a larger WIP doc for looking at how we could add a common set (and how we >>> might do it differently than "regular" UDFs), would love to collaborate on >>> that together if your open to it :) >>> >>> https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing >>> >>> >>> On 2026-11-12 00:37 UTC Alejandro Morgante wrote: >>> > (Resending now that I’m subscribed, as my earlier message doesn’t >>> appear in >>> > the public archive. Apologies if you receive this twice) >>> > >>> > Hi Spark devs, >>> > >>> > I’d like to propose *ai_query* for *PySpark*. It takes a prompt column >>> and >>> > returns the model’s response as a column. *Databricks SQL* and *Google >>> > BigQuery *offer similar capabilities through *ai_query >>> > < >>> https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query >>> > >>> > *and >>> > *AI*.*GENERATE* >>> > < >>> https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate >>> >, >>> > respectively. Here’s what this could look like in a *PySpark >>> DataFrame.* >>> > >>> > from pyspark.sql.ai import ai_query, register_modelfrom >>> > pyspark.sql.functions import col, concat, lit >>> > register_model( >>> > "summarizer", >>> > "databricks-gpt-oss-20b", >>> > provider="databricks", >>> > profile="my-workspace", >>> > auth_type="databricks-cli", >>> > ) >>> > df.withColumn( >>> > "summary", >>> > ai_query( >>> > endpoint="summarizer", >>> > request=concat(lit("Summarize in one sentence: "), >>> col("description")), >>> > ), >>> > ) >>> > >>> > *register_model* maps *summarizer* to a model endpoint. You can switch >>> > between *Databricks Model Serving* and *Amazon Bedrock* by changing the >>> > registration, while keeping the same DataFrame code. >>> > *The prototype passed end-to-end tests with both providers using Spark >>> and >>> > Arrow.* >>> > What do you think of this *API* for *PySpark*? *I’d love to hear how >>> you’d >>> > use it and what you’d change.* >>> > >>> > Draft PR #60311 <https://github.com/apache/spark/pull/60311> >>> > JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404> >>> > >>> > Thanks, >>> > Morgan >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>>
