Alejandro Morgante created SPARK-61404:
------------------------------------------

             Summary: Add ai_query for remote model inference in PySpark
                 Key: SPARK-61404
                 URL: https://issues.apache.org/jira/browse/SPARK-61404
             Project: Spark
          Issue Type: New Feature
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: Alejandro Morgante


Add a PySpark column function that sends a prompt to a registered model and 
returns the response as text. This lets DataFrame transformations summarize or 
classify text using the same expression with Databricks Model Serving or Amazon 
Bedrock.

Register a Databricks endpoint and use its alias to summarize each document:

{code:python}
from pyspark.sql.ai import ai_query, register_model
from pyspark.sql.functions import col, concat, lit

register_model(
    "summarizer",
    "databricks-gpt-oss-20b",
    provider="databricks",
    profile="my-workspace",
    auth_type="databricks-cli",
)
df.withColumn(
    "summary",
    ai_query(
        endpoint="summarizer",
        request=concat(lit("Summarize in one sentence: "), col("description")),
    ),
)
{code}

The draft uses an iterator Pandas UDF and resolves credentials on the workers. 
A null prompt yields null. Spark task retries and repeated actions can make 
additional billable model calls.

The prototype passed end-to-end tests on local Spark 4.3 with Arrow against 
Databricks Model Serving (GPT OSS 20B) and Amazon Bedrock Converse (Nova Micro).

[Draft PR #60311|https://github.com/apache/spark/pull/60311]. Feedback is 
welcome on the API and the scope of the first version.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to