Hi Morgan,

Thanks for sharing the draft document and your prompt acknowledgement. I
see this as a step towards bringing in AI functions within spark. Thanks
again for sharing the same & happy to work together.

Regards,
Hemil

On Sat, Nov 14, 2026 at 1:15 AM Alejandro Morgante <
[email protected]> wrote:

> Hi everyone,
>
> Thanks for the interest! We're working on an SPIP for `ai_query` with
> Daniel Tenedorio and Norio Akagi.
>
> The PySpark PR is a reference prototype. The direction we're exploring is
> a cross-language implementation: a native SQL function backed by Catalyst,
> with DataFrame wrappers and a shared execution backend.
>
> Holden, we're using your broader proposal as a guide and reference, and
> we'd like to coordinate the designs.
>
> Our shared draft is here:
>
> https://docs.google.com/document/d/1mAivaYyJaB_Hi676TvrXJA8CXsC1JS6fxJVGeo1yi5A/edit
>
> You're both very welcome to contribute to the SPIP and implementation.
> We'd be happy to work on this together!
>
> Morgan
>
> El vie, 13 nov 2026 a las 23:32, Hemil Shah (<[email protected]>)
> escribió:
>
>> It would be interesting to add ai support to query data using natural
>> language. I have experience with other similar query platforms . Happy to
>> works and collaborate to make this happen.
>>
>> On Fri, Nov 13, 2026 at 9:26 PM Holden Karau via dev <
>> [email protected]> wrote:
>>
>>> I think it's interesting but I'm hesistant to add only PySpark
>>> functions, there's no reason we can't build a proper cross-language
>>> implementation like the rest of our Spark functionality. I've got a bit of
>>> a larger WIP doc for looking at how we could add a common set (and how we
>>> might do it differently than "regular" UDFs), would love to collaborate on
>>> that together if your open to it :)
>>>
>>> https://docs.google.com/document/d/16WwnJjON3mT0ls1aF20nlHMS72-wU1LQfR9BTH5x_vM/edit?usp=sharing
>>>
>>>
>>> On 2026-11-12 00:37 UTC Alejandro Morgante wrote:
>>> > (Resending now that I’m subscribed, as my earlier message doesn’t
>>> appear in
>>> > the public archive. Apologies if you receive this twice)
>>> >
>>> > Hi Spark devs,
>>> >
>>> > I’d like to propose *ai_query* for *PySpark*. It takes a prompt column
>>> and
>>> > returns the model’s response as a column. *Databricks SQL* and *Google
>>> > BigQuery *offer similar capabilities through *ai_query
>>> > <
>>> https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query
>>> >
>>> > *and
>>> > *AI*.*GENERATE*
>>> > <
>>> https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate
>>> >,
>>> > respectively. Here’s what this could look like in a *PySpark
>>> DataFrame.*
>>> >
>>> > from pyspark.sql.ai import ai_query, register_modelfrom
>>> > pyspark.sql.functions import col, concat, lit
>>> > register_model(
>>> >     "summarizer",
>>> >     "databricks-gpt-oss-20b",
>>> >     provider="databricks",
>>> >     profile="my-workspace",
>>> >     auth_type="databricks-cli",
>>> > )
>>> > df.withColumn(
>>> >     "summary",
>>> >     ai_query(
>>> >         endpoint="summarizer",
>>> >         request=concat(lit("Summarize in one sentence: "),
>>> col("description")),
>>> >     ),
>>> > )
>>> >
>>> > *register_model* maps *summarizer* to a model endpoint. You can switch
>>> > between *Databricks Model Serving* and *Amazon Bedrock* by changing the
>>> > registration, while keeping the same DataFrame code.
>>> > *The prototype passed end-to-end tests with both providers using Spark
>>> and
>>> > Arrow.*
>>> > What do you think of this *API* for *PySpark*? *I’d love to hear how
>>> you’d
>>> > use it and what you’d change.*
>>> >
>>> > Draft PR #60311 <https://github.com/apache/spark/pull/60311>
>>> > JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404>
>>> >
>>> > Thanks,
>>> > Morgan
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [email protected]
>>>
>>>

Reply via email to