(Resending now that I’m subscribed, as my earlier message doesn’t appear in
the public archive. Apologies if you receive this twice)

Hi Spark devs,

I’d like to propose *ai_query* for *PySpark*. It takes a prompt column and
returns the model’s response as a column. *Databricks SQL* and *Google
BigQuery *offer similar capabilities through *ai_query
<https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query>
*and
*AI*.*GENERATE*
<https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate>,
respectively. Here’s what this could look like in a *PySpark DataFrame.*

from pyspark.sql.ai import ai_query, register_modelfrom
pyspark.sql.functions import col, concat, lit
register_model(
    "summarizer",
    "databricks-gpt-oss-20b",
    provider="databricks",
    profile="my-workspace",
    auth_type="databricks-cli",
)
df.withColumn(
    "summary",
    ai_query(
        endpoint="summarizer",
        request=concat(lit("Summarize in one sentence: "), col("description")),
    ),
)

*register_model* maps *summarizer* to a model endpoint. You can switch
between *Databricks Model Serving* and *Amazon Bedrock* by changing the
registration, while keeping the same DataFrame code.
*The prototype passed end-to-end tests with both providers using Spark and
Arrow.*
What do you think of this *API* for *PySpark*? *I’d love to hear how you’d
use it and what you’d change.*

Draft PR #60311 <https://github.com/apache/spark/pull/60311>
JIRA SPARK-61404 <https://issues.apache.org/jira/browse/SPARK-61404>

Thanks,
Morgan

Reply via email to