[
https://issues.apache.org/jira/browse/SPARK-61404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61404:
-----------------------------------
Labels: pull-request-available (was: )
> Add ai_query for remote model inference in PySpark
> --------------------------------------------------
>
> Key: SPARK-61404
> URL: https://issues.apache.org/jira/browse/SPARK-61404
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Alejandro Morgante
> Priority: Major
> Labels: pull-request-available
>
> Add a PySpark column function that sends a prompt to a registered model and
> returns the response as text. This lets DataFrame transformations summarize
> or classify text using the same expression with Databricks Model Serving or
> Amazon Bedrock.
> Register a Databricks endpoint and use its alias to summarize each document:
> {code:python}
> from pyspark.sql.ai import ai_query, register_model
> from pyspark.sql.functions import col, concat, lit
> register_model(
> "summarizer",
> "databricks-gpt-oss-20b",
> provider="databricks",
> profile="my-workspace",
> auth_type="databricks-cli",
> )
> df.withColumn(
> "summary",
> ai_query(
> endpoint="summarizer",
> request=concat(lit("Summarize in one sentence: "),
> col("description")),
> ),
> )
> {code}
> The draft uses an iterator Pandas UDF and resolves credentials on the
> workers. A null prompt yields null. Spark task retries and repeated actions
> can make additional billable model calls.
> The prototype passed end-to-end tests on local Spark 4.3 with Arrow against
> Databricks Model Serving (GPT OSS 20B) and Amazon Bedrock Converse (Nova
> Micro).
> [Draft PR #60311|https://github.com/apache/spark/pull/60311]. Feedback is
> welcome on the API and the scope of the first version.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]