This is an automated email from the ASF dual-hosted git repository.
HappenLee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 07b7eeb50b3 [doc](embed) Update multimodal embedding instructions
(#4036)
07b7eeb50b3 is described below
commit 07b7eeb50b3d617b5424ee64564abed429a8a116
Author: linrrarity <[email protected]>
AuthorDate: Thu Aug 6 09:42:58 2026 +0800
[doc](embed) Update multimodal embedding instructions (#4036)
## Versions
- [x] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [ ] Chinese
- [ ] English
- [ ] Japanese candidate translation needed
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
- [ ] Updated required version and language counterparts, or explained
why not
- [ ] If only one language changed, confirmed whether source/translation
counterparts need sync
---
docs/key-features/embedding.mdx | 36 +++++-
.../ai-functions/distance-functions/embed.md | 117 ++++++++++++++++++--
.../ai-functions/distance-functions/embed.md | 119 ++++++++++++++++++--
.../ai-functions/distance-functions/embed.md | 123 +++++++++++++++++++--
.../version-4.x/key-features/embedding.mdx | 36 +++++-
.../ai-functions/distance-functions/embed.md | 121 ++++++++++++++++++--
6 files changed, 504 insertions(+), 48 deletions(-)
diff --git a/docs/key-features/embedding.mdx b/docs/key-features/embedding.mdx
index 6c28c7404ad..7852b959def 100644
--- a/docs/key-features/embedding.mdx
+++ b/docs/key-features/embedding.mdx
@@ -12,7 +12,7 @@ keywords:
slug: /key-features/embedding
image: /images/next/key-features/embedding.jpg
last_update:
- date: 2026-05-10
+ date: 2026-08-04
author: Apache Doris
featureCard:
tags:
@@ -91,6 +91,40 @@ FROM notes ORDER BY d ASC LIMIT 1;
The `INSERT` precomputes one vector per row. The `SELECT` embeds the query
string once at runtime, then ranks rows by cosine distance. Two API calls in
total, both hidden inside SQL.
+## Multimodal quick start {#multimodal-quick-start}
+
+For a multimodal embedding, pass a JSON object instead of a text string. The
object must contain the file `uri` and its MIME `content_type`. The URI can be
an HTTP or HTTPS URL, or an S3-compatible URI. Confirm that your configured
provider and model support the input media type.
+
+This example embeds a video stored in S3-compatible object storage:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+**Example result**
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+The vector dimension depends on the configured provider and model, so the
result above is only an example. For an HTTP or HTTPS URL, only `uri` and
`content_type` are required. For an S3-compatible URI, Doris generates a
presigned URL and supports either AK/SK credentials or IAM role authentication
with `role_arn` and `external_id`.
+
## When should you use the Apache Doris EMBED function? {#when}
The Apache Doris `EMBED()` function fits RAG and semantic search over corpora
that already live in Apache Doris, backfills that keep the work inside SQL,
hybrid pipelines that pair vectors with keyword filters, and multimodal
embeddings whose source files live in S3. It is not a fit for per-row embedding
over millions of rows on the hot path, models outside the supported provider
list, or air-gapped clusters with no path to the provider.
diff --git
a/docs/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
b/docs/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
index 9a15b470aaf..340eefcee9c 100644
--- a/docs/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
+++ b/docs/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
@@ -2,7 +2,7 @@
{
"title": "EMBED",
"language": "en",
- "description": "Generates a semantic embedding vector based on the input
text, representing the semantic information of the text."
+ "description": "Generates an embedding vector for text, image, video, or
audio input."
}
---
@@ -27,30 +27,125 @@ under the License.
## Description
-Generates a semantic embedding vector based on the input text, representing
the semantic information of the text. It can be used for similarity
calculation, retrieval, and other scenarios.
+Generates an embedding vector for text or multimodal input. The vector can be
used for similarity calculation, retrieval, and other scenarios.
+
+For multimodal embedding, pass a JSON object that describes an image, video,
or audio file. Before using a media type, confirm that the configured AI
provider and model support it.
## Syntax
```sql
-EMBED([<resource_name>], <text>)
+EMBED([<resource_name>], <input>)
```
## Parameters
-| Parameter | Description |
-| ----------------- | ------------------------- |
-| `<resource_name>` | The specified resource name|
-| `<text>` | The text to generate the embedding vector for |
+| Parameter | Description |
+| ----------------- | ----------- |
+| `<resource_name>` | Optional. The AI resource used to generate the
embedding. If omitted, Doris uses the session variable `default_ai_resource`. |
+| `<input>` | The content to embed. It can be a string for text
embedding, or a JSON object for multimodal embedding. |
+
+The multimodal JSON object contains the following fields:
+
+| Field | Required | Description |
+| ----- | -------- | ----------- |
+| `uri` | Yes | An `http://` or `https://` URL, or an S3-compatible URI in the
form `s3://<bucket>/<key>`. |
+| `content_type` | Yes | The media MIME type. Values beginning with `image/`,
`video/`, or `audio/` are recognized. The AI provider and model must support
that media type. |
+| `endpoint` | For S3-compatible URIs | The S3-compatible storage endpoint. |
+| `region` | For S3-compatible URIs | The storage region. |
+| `ak` | No | Access key for AK/SK authentication. |
+| `sk` | No | Secret key for AK/SK authentication. |
+| `role_arn` | No | IAM role ARN for role-based authentication. |
+| `external_id` | No | External ID used when assuming the IAM role. |
+
+For an HTTP or HTTPS URL, only `uri` and `content_type` are required. For an
S3-compatible URI, Doris uses the storage credentials to generate a presigned
URL. Both AK/SK and IAM role authentication are supported.
## Return Value
-The return type is ARRAY<FLOAT>, representing the generated vector.
+The return type is `ARRAY<FLOAT>`, representing the generated vector.
Returns NULL if the input value is NULL.
-The result is generated by a large language model, so the returned content is
not fixed.
+The vector dimension and values depend on the AI provider and model.
+
+## Multimodal Examples
+
+The following example generates an embedding for an image available through an
HTTPS URL:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "https://example.com/images/product.png";,
+ "content_type": "image/png"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+The following example reads a video from S3-compatible storage by using AK/SK
authentication:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+IAM role authentication can be used instead of AK/SK. The following example
generates an image embedding:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/images/product.png",
+ "content_type": "image/png",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "role_arn": "arn:aws:iam::<account_id>:role/<role_name>",
+ "external_id": "<external_id>"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
-## Example
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 3072 |
++---------------------------+
+```
+
+The preceding results are examples. The actual embedding dimension depends on
the configured provider and model.
+
+## Text Example
The following table simulates a company's code of conduct.
@@ -166,4 +261,4 @@ LIMIT 3;
| 4 | Meeting Room Reservation | Meeting rooms can be reserved in
advance through the OA system, with time and number of participants specified.
| 30.90449328294426 |
| 2 | Leave Policy | Employees must apply for leave in the
system in advance. If the leave is longer than three days, approval from the
direct manager is required. | 31.060405636536416 |
+------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+
-```
\ No newline at end of file
+```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
index bc4d494b563..ea85acb33a9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
@@ -2,7 +2,7 @@
{
"title": "EMBED",
"language": "zh-CN",
- "description": "根据输入文本生成语义嵌入向量，用于表示文本的语义信息，可用于相似度计算、检索等场景。"
+ "description": "根据文本、图像、视频或音频输入生成嵌入向量。"
}
---
@@ -27,31 +27,126 @@ under the License.
## 描述
-根据输入文本生成语义嵌入向量，用于表示文本的语义信息，可用于相似度计算、检索等场景。
+根据文本或多模态输入生成嵌入向量，可用于相似度计算、检索等场景。
+
+生成多模态嵌入向量时，需要传入描述图像、视频或音频文件的 JSON 对象。使用某种媒体类型前，请确认配置的 AI Provider 和模型支持该类型。
## 语法
```sql
-EMBED([<resource_name>], <text>)
+EMBED([<resource_name>], <input>)
```
## 参数
-| 参数 | 说明 |
-| ---------- | -------- |
-| `<resource_name>`| 指定的资源名称|
-| `<text>` | 生成嵌入向量的文本 |
+| 参数 | 说明 |
+| ---- | ---- |
+| `<resource_name>` | 可选。用于生成嵌入向量的 AI Resource 名称。省略时，Doris 使用会话变量
`default_ai_resource` 指定的资源。 |
+| `<input>` | 需要生成嵌入向量的内容。文本嵌入传入字符串，多模态嵌入传入 JSON 对象。 |
+
+多模态 JSON 对象包含以下字段：
+
+| 字段 | 是否必需 | 说明 |
+| ---- | -------- | ---- |
+| `uri` | 是 | `http://` 或 `https://` URL，或者格式为 `s3://<bucket>/<key>` 的 S3 兼容存储
URI。 |
+| `content_type` | 是 | 媒体的 MIME 类型。以 `image/`、`video/` 或 `audio/` 开头的值会被识别，且
AI Provider 和模型必须支持对应的媒体类型。 |
+| `endpoint` | 使用 S3 兼容存储 URI 时必需 | S3 兼容存储的 Endpoint。 |
+| `region` | 使用 S3 兼容存储 URI 时必需 | 对象存储所在的 Region。 |
+| `ak` | 否 | 使用 AK/SK 鉴权时的 Access Key。 |
+| `sk` | 否 | 使用 AK/SK 鉴权时的 Secret Key。 |
+| `role_arn` | 否 | 使用 IAM Role 鉴权时的 Role ARN。 |
+| `external_id` | 否 | Assume Role 时使用的 External ID。 |
+
+使用 HTTP 或 HTTPS URL 时，只需提供 `uri` 和 `content_type`。使用 S3 兼容存储 URI 时，Doris
会根据对象存储凭证生成预签名 URL，支持 AK/SK 和 IAM Role 两种鉴权方式。
## 返回值
-返回类型为 ARRAY<FLOAT> 代表所生成的向量
+返回类型为 `ARRAY<FLOAT>`，表示生成的嵌入向量。
+
+当输入值为 NULL 时返回 NULL。
+
+向量的维度和值由 AI Provider 和模型决定。
+
+## 多模态示例
+
+以下示例为可通过 HTTPS URL 访问的图像生成嵌入向量：
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "https://example.com/images/product.png";,
+ "content_type": "image/png"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+以下示例使用 AK/SK 鉴权，从 S3 兼容对象存储读取视频并生成嵌入向量：
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+也可以使用 IAM Role 代替 AK/SK 进行鉴权。以下示例生成图像嵌入向量：
-当输入值为 NULL 时返回 NULL
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/images/product.png",
+ "content_type": "image/png",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "role_arn": "arn:aws:iam::<account_id>:role/<role_name>",
+ "external_id": "<external_id>"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
-结果为大模型生成，所以返回内容并不固定
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 3072 |
++---------------------------+
+```
+
+以上结果仅供参考，实际的嵌入向量维度取决于配置的 Provider 和模型。
-## 示例
+## 文本示例
下表模拟某公司的行为手册
@@ -168,4 +263,4 @@ LIMIT 3;
| 4 | Meeting Room Reservation | Meeting rooms can be reserved in
advance through the OA system, with time and number of participants specified.
| 30.90449328294426 |
| 2 | Leave Policy | Employees must apply for leave in the
system in advance. If the leave is longer than three days, approval from the
direct manager is required. | 31.060405636536416 |
+------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+
-```
\ No newline at end of file
+```
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
index bc4d494b563..f08de628dfe 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
@@ -2,7 +2,7 @@
{
"title": "EMBED",
"language": "zh-CN",
- "description": "根据输入文本生成语义嵌入向量，用于表示文本的语义信息，可用于相似度计算、检索等场景。"
+ "description": "根据文本、图像、视频或音频输入生成嵌入向量。"
}
---
@@ -27,31 +27,130 @@ under the License.
## 描述
-根据输入文本生成语义嵌入向量，用于表示文本的语义信息，可用于相似度计算、检索等场景。
+根据文本或多模态输入生成嵌入向量，可用于相似度计算、检索等场景。
+
+:::note
+多模态 Embedding 自 Apache Doris 4.1.5 版本起支持。
+:::
+
+生成多模态嵌入向量时，需要传入描述图像、视频或音频文件的 JSON 对象。使用某种媒体类型前，请确认配置的 AI Provider 和模型支持该类型。
## 语法
```sql
-EMBED([<resource_name>], <text>)
+EMBED([<resource_name>], <input>)
```
## 参数
-| 参数 | 说明 |
-| ---------- | -------- |
-| `<resource_name>`| 指定的资源名称|
-| `<text>` | 生成嵌入向量的文本 |
+| 参数 | 说明 |
+| ---- | ---- |
+| `<resource_name>` | 可选。用于生成嵌入向量的 AI Resource 名称。省略时，Doris 使用会话变量
`default_ai_resource` 指定的资源。 |
+| `<input>` | 需要生成嵌入向量的内容。文本嵌入传入字符串，多模态嵌入传入 JSON 对象。 |
+
+多模态 JSON 对象包含以下字段：
+
+| 字段 | 是否必需 | 说明 |
+| ---- | -------- | ---- |
+| `uri` | 是 | `http://` 或 `https://` URL，或者格式为 `s3://<bucket>/<key>` 的 S3 兼容存储
URI。 |
+| `content_type` | 是 | 媒体的 MIME 类型。以 `image/`、`video/` 或 `audio/` 开头的值会被识别，且
AI Provider 和模型必须支持对应的媒体类型。 |
+| `endpoint` | 使用 S3 兼容存储 URI 时必需 | S3 兼容存储的 Endpoint。 |
+| `region` | 使用 S3 兼容存储 URI 时必需 | 对象存储所在的 Region。 |
+| `ak` | 否 | 使用 AK/SK 鉴权时的 Access Key。 |
+| `sk` | 否 | 使用 AK/SK 鉴权时的 Secret Key。 |
+| `role_arn` | 否 | 使用 IAM Role 鉴权时的 Role ARN。 |
+| `external_id` | 否 | Assume Role 时使用的 External ID。 |
+
+使用 HTTP 或 HTTPS URL 时，只需提供 `uri` 和 `content_type`。使用 S3 兼容存储 URI 时，Doris
会根据对象存储凭证生成预签名 URL，支持 AK/SK 和 IAM Role 两种鉴权方式。
## 返回值
-返回类型为 ARRAY<FLOAT> 代表所生成的向量
+返回类型为 `ARRAY<FLOAT>`，表示生成的嵌入向量。
+
+当输入值为 NULL 时返回 NULL。
-当输入值为 NULL 时返回 NULL
+向量的维度和值由 AI Provider 和模型决定。
-结果为大模型生成，所以返回内容并不固定
+## 多模态示例
-## 示例
+以下示例为可通过 HTTPS URL 访问的图像生成嵌入向量：
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "https://example.com/images/product.png";,
+ "content_type": "image/png"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+以下示例使用 AK/SK 鉴权，从 S3 兼容对象存储读取视频并生成嵌入向量：
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+也可以使用 IAM Role 代替 AK/SK 进行鉴权。以下示例生成图像嵌入向量：
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/images/product.png",
+ "content_type": "image/png",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "role_arn": "arn:aws:iam::<account_id>:role/<role_name>",
+ "external_id": "<external_id>"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 3072 |
++---------------------------+
+```
+
+以上结果仅供参考，实际的嵌入向量维度取决于配置的 Provider 和模型。
+
+## 文本示例
下表模拟某公司的行为手册
@@ -168,4 +267,4 @@ LIMIT 3;
| 4 | Meeting Room Reservation | Meeting rooms can be reserved in
advance through the OA system, with time and number of participants specified.
| 30.90449328294426 |
| 2 | Leave Policy | Employees must apply for leave in the
system in advance. If the leave is longer than three days, approval from the
direct manager is required. | 31.060405636536416 |
+------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+
-```
\ No newline at end of file
+```
diff --git a/versioned_docs/version-4.x/key-features/embedding.mdx
b/versioned_docs/version-4.x/key-features/embedding.mdx
index 6c28c7404ad..7852b959def 100644
--- a/versioned_docs/version-4.x/key-features/embedding.mdx
+++ b/versioned_docs/version-4.x/key-features/embedding.mdx
@@ -12,7 +12,7 @@ keywords:
slug: /key-features/embedding
image: /images/next/key-features/embedding.jpg
last_update:
- date: 2026-05-10
+ date: 2026-08-04
author: Apache Doris
featureCard:
tags:
@@ -91,6 +91,40 @@ FROM notes ORDER BY d ASC LIMIT 1;
The `INSERT` precomputes one vector per row. The `SELECT` embeds the query
string once at runtime, then ranks rows by cosine distance. Two API calls in
total, both hidden inside SQL.
+## Multimodal quick start {#multimodal-quick-start}
+
+For a multimodal embedding, pass a JSON object instead of a text string. The
object must contain the file `uri` and its MIME `content_type`. The URI can be
an HTTP or HTTPS URL, or an S3-compatible URI. Confirm that your configured
provider and model support the input media type.
+
+This example embeds a video stored in S3-compatible object storage:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+**Example result**
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+The vector dimension depends on the configured provider and model, so the
result above is only an example. For an HTTP or HTTPS URL, only `uri` and
`content_type` are required. For an S3-compatible URI, Doris generates a
presigned URL and supports either AK/SK credentials or IAM role authentication
with `role_arn` and `external_id`.
+
## When should you use the Apache Doris EMBED function? {#when}
The Apache Doris `EMBED()` function fits RAG and semantic search over corpora
that already live in Apache Doris, backfills that keep the work inside SQL,
hybrid pipelines that pair vectors with keyword filters, and multimodal
embeddings whose source files live in S3. It is not a fit for per-row embedding
over millions of rows on the hot path, models outside the supported provider
list, or air-gapped clusters with no path to the provider.
diff --git
a/versioned_docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
b/versioned_docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
index 9a15b470aaf..373cc17b6ee 100644
---
a/versioned_docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
+++
b/versioned_docs/version-4.x/sql-manual/sql-functions/ai-functions/distance-functions/embed.md
@@ -2,7 +2,7 @@
{
"title": "EMBED",
"language": "en",
- "description": "Generates a semantic embedding vector based on the input
text, representing the semantic information of the text."
+ "description": "Generates an embedding vector for text, image, video, or
audio input."
}
---
@@ -27,30 +27,129 @@ under the License.
## Description
-Generates a semantic embedding vector based on the input text, representing
the semantic information of the text. It can be used for similarity
calculation, retrieval, and other scenarios.
+Generates an embedding vector for text or multimodal input. The vector can be
used for similarity calculation, retrieval, and other scenarios.
+
+:::note
+Multimodal embedding is supported since Apache Doris 4.1.5.
+:::
+
+For multimodal embedding, pass a JSON object that describes an image, video,
or audio file. Before using a media type, confirm that the configured AI
provider and model support it.
## Syntax
```sql
-EMBED([<resource_name>], <text>)
+EMBED([<resource_name>], <input>)
```
## Parameters
-| Parameter | Description |
-| ----------------- | ------------------------- |
-| `<resource_name>` | The specified resource name|
-| `<text>` | The text to generate the embedding vector for |
+| Parameter | Description |
+| ----------------- | ----------- |
+| `<resource_name>` | Optional. The AI resource used to generate the
embedding. If omitted, Doris uses the session variable `default_ai_resource`. |
+| `<input>` | The content to embed. It can be a string for text
embedding, or a JSON object for multimodal embedding. |
+
+The multimodal JSON object contains the following fields:
+
+| Field | Required | Description |
+| ----- | -------- | ----------- |
+| `uri` | Yes | An `http://` or `https://` URL, or an S3-compatible URI in the
form `s3://<bucket>/<key>`. |
+| `content_type` | Yes | The media MIME type. Values beginning with `image/`,
`video/`, or `audio/` are recognized. The AI provider and model must support
that media type. |
+| `endpoint` | For S3-compatible URIs | The S3-compatible storage endpoint. |
+| `region` | For S3-compatible URIs | The storage region. |
+| `ak` | No | Access key for AK/SK authentication. |
+| `sk` | No | Secret key for AK/SK authentication. |
+| `role_arn` | No | IAM role ARN for role-based authentication. |
+| `external_id` | No | External ID used when assuming the IAM role. |
+
+For an HTTP or HTTPS URL, only `uri` and `content_type` are required. For an
S3-compatible URI, Doris uses the storage credentials to generate a presigned
URL. Both AK/SK and IAM role authentication are supported.
## Return Value
-The return type is ARRAY<FLOAT>, representing the generated vector.
+The return type is `ARRAY<FLOAT>`, representing the generated vector.
Returns NULL if the input value is NULL.
-The result is generated by a large language model, so the returned content is
not fixed.
+The vector dimension and values depend on the AI provider and model.
+
+## Multimodal Examples
-## Example
+The following example generates an embedding for an image available through an
HTTPS URL:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "https://example.com/images/product.png";,
+ "content_type": "image/png"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+The following example reads a video from S3-compatible storage by using AK/SK
authentication:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/videos/demo.mp4",
+ "content_type": "video/mp4",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "ak": "<access_key>",
+ "sk": "<secret_key>"
+ }' AS JSON)
+ )
+) AS video_embedding_dimension;
+```
+
+```text
++---------------------------+
+| video_embedding_dimension |
++---------------------------+
+| 2560 |
++---------------------------+
+```
+
+IAM role authentication can be used instead of AK/SK. The following example
generates an image embedding:
+
+```sql
+SELECT ARRAY_SIZE(
+ EMBED(
+ 'multimodal_embed_resource',
+ CAST('{
+ "uri": "s3://example-bucket/images/product.png",
+ "content_type": "image/png",
+ "endpoint": "s3.us-east-1.amazonaws.com",
+ "region": "us-east-1",
+ "role_arn": "arn:aws:iam::<account_id>:role/<role_name>",
+ "external_id": "<external_id>"
+ }' AS JSON)
+ )
+) AS image_embedding_dimension;
+```
+
+```text
++---------------------------+
+| image_embedding_dimension |
++---------------------------+
+| 3072 |
++---------------------------+
+```
+
+The preceding results are examples. The actual embedding dimension depends on
the configured provider and model.
+
+## Text Example
The following table simulates a company's code of conduct.
@@ -166,4 +265,4 @@ LIMIT 3;
| 4 | Meeting Room Reservation | Meeting rooms can be reserved in
advance through the OA system, with time and number of participants specified.
| 30.90449328294426 |
| 2 | Leave Policy | Employees must apply for leave in the
system in advance. If the leave is longer than three days, approval from the
direct manager is required. | 31.060405636536416 |
+------+-----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+
-```
\ No newline at end of file
+```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://example.com/images/product.png"](https://example.com/images/product.png"){kind=link}