Alejandro Morgante created SPARK-61404:
------------------------------------------
Summary: Add ai_query for remote model inference in PySpark
Key: SPARK-61404
URL: https://issues.apache.org/jira/browse/SPARK-61404
Project: Spark
Issue Type: New Feature
Components: PySpark
Affects Versions: 5.0.0
Reporter: Alejandro Morgante
Add a PySpark column function that sends a prompt to a registered model and
returns the response as text. This lets DataFrame transformations summarize or
classify text using the same expression with Databricks Model Serving or Amazon
Bedrock.
Register a Databricks endpoint and use its alias to summarize each document:
{code:python}
from pyspark.sql.ai import ai_query, register_model
from pyspark.sql.functions import col, concat, lit
register_model(
"summarizer",
"databricks-gpt-oss-20b",
provider="databricks",
profile="my-workspace",
auth_type="databricks-cli",
)
df.withColumn(
"summary",
ai_query(
endpoint="summarizer",
request=concat(lit("Summarize in one sentence: "), col("description")),
),
)
{code}
The draft uses an iterator Pandas UDF and resolves credentials on the workers.
A null prompt yields null. Spark task retries and repeated actions can make
additional billable model calls.
The prototype passed end-to-end tests on local Spark 4.3 with Arrow against
Databricks Model Serving (GPT OSS 20B) and Amazon Bedrock Converse (Nova Micro).
[Draft PR #60311|https://github.com/apache/spark/pull/60311]. Feedback is
welcome on the API and the scope of the first version.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]