This is an automated email from the ASF dual-hosted git repository.
ruinova pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/dubbo-go-samples.git
The following commit(s) were added to refs/heads/main by this push:
new 64e9409d feat: Add multi-LLM provider support (#932)
64e9409d is described below
commit 64e9409d11644e2e3ec5d6a4ddbbd88e1682eead
Author: CAICAII <[email protected]>
AuthorDate: Fri Oct 17 06:56:49 2025 -0700
feat: Add multi-LLM provider support (#932)
* feat: Add multi-LLM provider support
---
llm/.env.example | 39 +++++++--
llm/README.md | 67 ++++++++++++---
llm/README_zh.md | 65 ++++++++++++---
llm/config/config.go | 104 +++++++++++++++++++++--
llm/go-server/cmd/server.go | 30 ++++---
llm/go-server/llm/llm_service.go | 176 +++++++++++++++++++++++++++++++++++++++
llm/start_servers.sh | 28 +++++--
7 files changed, 450 insertions(+), 59 deletions(-)
diff --git a/llm/.env.example b/llm/.env.example
index f3686ffa..0bbced35 100644
--- a/llm/.env.example
+++ b/llm/.env.example
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -13,14 +12,40 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
+# LLM Models Configuration
+LLM_MODELS = llava:7b, qwen2.5:7b
-OLLAMA_MODELS= qwen2.5:7b, llava:7b
+# Provider Configuration
+LLM_PROVIDER = ollama
+LLM_BASE_URL = http://localhost:11434
+# LLM_API_KEY = your-api-key-here
+
+# Legacy Configuration (for backward compatibility)
OLLAMA_URL = http://localhost:11434
-TIME_OUT_SECOND = 300
-NACOS_URL = localhost:8848
+# Server Configuration
+MODEL_NAME = llava:7b
+SERVER_PORT = 20020
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
MAX_CONTEXT_COUNT = 3
-MODEL_NAME = qwen2.5:7b
-SERVER_PORT = 20000
+
+# Example configurations:
+#
+# Ollama (Local):
+# LLM_MODELS = llava:7b, qwen2.5:7b
+# LLM_PROVIDER = ollama
+# LLM_BASE_URL = http://localhost:11434
+#
+# OpenAI:
+# LLM_MODELS = gpt-4, gpt-3.5-turbo
+# LLM_PROVIDER = openai
+# LLM_BASE_URL = https://api.openai.com/v1
+# LLM_API_KEY = your-openai-api-key
+#
+# Anthropic:
+# LLM_MODELS = claude-3-sonnet, claude-3-haiku
+# LLM_PROVIDER = anthropic
+# LLM_BASE_URL = https://api.anthropic.com/v1
+# LLM_API_KEY = your-anthropic-api-key
diff --git a/llm/README.md b/llm/README.md
index 27556f35..29f931a4 100644
--- a/llm/README.md
+++ b/llm/README.md
@@ -2,22 +2,23 @@
## 1. **Introduction**
-This sample demonstrates how to integrate **large language models (LLM)** in
**Dubbo-go**, allowing the server to invoke the Ollama model for inference and
return the results to the client via Dubbo RPC. It supports multiple model
deployment with multiple instances per model.
+This sample demonstrates how to integrate **large language models (LLM)** in
**Dubbo-go**, allowing the server to invoke various LLM providers for inference
and return the results to the client via Dubbo RPC. It supports multiple LLM
providers including Ollama, OpenAI, Anthropic Claude, and Azure OpenAI, with
multiple model deployment and multiple instances per model.
## 2. **Preparation**
-### **Install Ollama**
+### **Choose Your LLM Provider**
+This sample supports multiple LLM providers. Choose one based on your needs:
+
+#### **Option 1: Ollama (Local, Free)**
Ollama is a local language model platform that supports fast inference.
**Quick Installation**:
-
```shell
$ curl -fsSL https://ollama.com/install.sh | sh
```
**Manual Installation**:
-
```shell
$ mkdir -p ~/ollama
$ cd ~/ollama
@@ -28,14 +29,20 @@ $ source ~/.bashrc
$ ollama serve
```
-### **Download Models**
-
+**Download Models**:
```shell
$ ollama pull llava:7b
$ ollama pull qwen2.5:7b # Optional: download additional models
```
-You can pull your preferred models and configure them in the `.env` file.
+#### **Option 2: OpenAI API (Cloud, Paid)**
+Requires an OpenAI API key. Get one from [OpenAI
Platform](https://platform.openai.com/).
+
+#### **Option 3: Anthropic Claude API (Cloud, Paid)**
+Requires an Anthropic API key. Get one from [Anthropic
Console](https://console.anthropic.com/).
+
+#### **Option 4: Azure OpenAI (Cloud, Paid)**
+Requires an Azure OpenAI resource. Set up at [Azure
Portal](https://portal.azure.com/).
### **Install Nacos**
@@ -82,12 +89,50 @@ $ cp .env.example .env
### **Configuration**
-The `.env` file supports multiple model configurations, example:
+The `.env` file supports multiple LLM providers. Here are examples for each
provider:
+
+#### **Ollama Configuration**:
+```text
+LLM_PROVIDER = ollama
+LLM_MODELS = llava:7b, qwen2.5:7b
+LLM_BASE_URL = http://localhost:11434
+MODEL_NAME = llava:7b
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+
+#### **OpenAI Configuration**:
+```text
+LLM_PROVIDER = openai
+LLM_MODELS = gpt-4, gpt-3.5-turbo
+LLM_BASE_URL = https://api.openai.com/v1
+LLM_API_KEY = your-openai-api-key
+MODEL_NAME = gpt-4
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+
+#### **Anthropic Configuration**:
+```text
+LLM_PROVIDER = anthropic
+LLM_MODELS = claude-3-sonnet-20240229, claude-3-haiku-20240307
+LLM_BASE_URL = https://api.anthropic.com/v1
+LLM_API_KEY = your-anthropic-api-key
+MODEL_NAME = claude-3-sonnet-20240229
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+#### **Azure OpenAI Configuration**:
```text
-# Configure multiple models, comma-separated, spaces allowed
-OLLAMA_MODELS = llava:7b, qwen2.5:7b
-OLLAMA_URL = http://localhost:11434
+LLM_PROVIDER = azure-openai
+LLM_MODELS = gpt-4, gpt-35-turbo
+LLM_BASE_URL =
https://your-resource.openai.azure.com/openai/deployments/your-deployment
+LLM_API_KEY = your-azure-openai-api-key
+MODEL_NAME = gpt-4
NACOS_URL = nacos://localhost:8848
TIME_OUT_SECOND = 300
MAX_CONTEXT_COUNT = 3
diff --git a/llm/README_zh.md b/llm/README_zh.md
index b7329e28..97513533 100644
--- a/llm/README_zh.md
+++ b/llm/README_zh.md
@@ -6,18 +6,19 @@
## 2. **准备工作**
-### **安装 Ollama**
+### **选择您的 LLM 提供商**
+本案例支持多种 LLM 提供商。根据您的需求选择：
+
+#### **选项 1: Ollama (本地，免费)**
Ollama 是一个本地运行的大语言模型平台，支持快速推理。
**快速安装**：
-
```shell
$ curl -fsSL https://ollama.com/install.sh | sh
```
**手动安装**：
-
```shell
$ mkdir -p ~/ollama
$ cd ~/ollama
@@ -28,14 +29,20 @@ $ source ~/.bashrc
$ ollama serve
```
-### 下载模型
-
+**下载模型**：
```shell
$ ollama pull llava:7b
$ ollama pull qwen2.5:7b # 可选：下载其他模型
```
-您可以自行 pull 需要的模型，并在 `.env` 文件中配置要使用的模型列表。
+#### **选项 2: OpenAI API (云端，付费)**
+需要 OpenAI API 密钥。从 [OpenAI Platform](https://platform.openai.com/) 获取。
+
+#### **选项 3: Anthropic Claude API (云端，付费)**
+需要 Anthropic API 密钥。从 [Anthropic Console](https://console.anthropic.com/) 获取。
+
+#### **选项 4: Azure OpenAI (云端，付费)**
+需要 Azure OpenAI 资源。在 [Azure Portal](https://portal.azure.com/) 设置。
### **安装 Nacos**
@@ -81,12 +88,50 @@ $ cp .env.example .env
### **配置说明**
-`.env` 文件支持配置多个模型，示例：
+`.env` 文件支持多种 LLM 提供商。以下是各提供商的配置示例：
+
+#### **Ollama 配置**：
+```text
+LLM_PROVIDER = ollama
+LLM_MODELS = llava:7b, qwen2.5:7b
+LLM_BASE_URL = http://localhost:11434
+MODEL_NAME = llava:7b
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+
+#### **OpenAI 配置**：
+```text
+LLM_PROVIDER = openai
+LLM_MODELS = gpt-4, gpt-3.5-turbo
+LLM_BASE_URL = https://api.openai.com/v1
+LLM_API_KEY = your-openai-api-key
+MODEL_NAME = gpt-4
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+
+#### **Anthropic 配置**：
+```text
+LLM_PROVIDER = anthropic
+LLM_MODELS = claude-3-sonnet-20240229, claude-3-haiku-20240307
+LLM_BASE_URL = https://api.anthropic.com/v1
+LLM_API_KEY = your-anthropic-api-key
+MODEL_NAME = claude-3-sonnet-20240229
+NACOS_URL = nacos://localhost:8848
+TIME_OUT_SECOND = 300
+MAX_CONTEXT_COUNT = 3
+```
+#### **Azure OpenAI 配置**：
```text
-# 支持配置多个模型，使用逗号分隔，支持带空格
-OLLAMA_MODELS = llava:7b, qwen2.5:7b
-OLLAMA_URL = http://localhost:11434
+LLM_PROVIDER = azure-openai
+LLM_MODELS = gpt-4, gpt-35-turbo
+LLM_BASE_URL =
https://your-resource.openai.azure.com/openai/deployments/your-deployment
+LLM_API_KEY = your-azure-openai-api-key
+MODEL_NAME = gpt-4
NACOS_URL = nacos://localhost:8848
TIME_OUT_SECOND = 300
MAX_CONTEXT_COUNT = 3
diff --git a/llm/config/config.go b/llm/config/config.go
index 5a0b1e22..2943b1d0 100644
--- a/llm/config/config.go
+++ b/llm/config/config.go
@@ -29,9 +29,33 @@ import (
"github.com/joho/godotenv"
)
+// Default URLs for different LLM providers
+const (
+ DefaultOllamaURL = "http://localhost:11434";
+ DefaultOpenAIURL = "https://api.openai.com/v1";
+)
+
+// LLMConfig represents configuration for a specific LLM model
+type LLMConfig struct {
+ Model string // Model name (e.g., "gpt-4", "claude-3-sonnet",
"llava:7b")
+ Provider string // Provider type (e.g., "openai", "anthropic", "ollama")
+ BaseURL string // Base URL for the model service
+ APIKey string // API key for the model service
+}
+
type Config struct {
- OllamaModels []string
- OllamaURL string
+ // LLM Models Configuration - Each model inherits global provider
settings
+ LLMModels []LLMConfig // Array of LLM models, all sharing the same
provider, URL, and key
+
+ // Legacy fields (for backward compatibility)
+ LLMProvider string // Single provider for all models
+ LLMModelsList []string // List of model names only
+ LLMBaseURL string // Base URL for LLM service
+ LLMAPIKey string // API key for LLM service
+ OllamaModels []string
+ OllamaURL string
+
+ // Common Configuration
TimeoutSeconds int
NacosURL string
MaxContextCount int
@@ -57,10 +81,23 @@ func Load(envFile string) (*Config, error) {
return
}
- modelsEnv := os.Getenv("OLLAMA_MODELS")
+ // Load default LLM provider (for backward compatibility)
+ llmProvider := os.Getenv("LLM_PROVIDER")
+ if llmProvider == "" {
+ // Fallback to legacy Ollama configuration
+ llmProvider = "ollama"
+ }
+ config.LLMProvider =
strings.ToLower(strings.TrimSpace(llmProvider))
+
+ // Load models - try LLM_MODELS first, then fallback to
OLLAMA_MODELS for backward compatibility
+ modelsEnv := os.Getenv("LLM_MODELS")
if modelsEnv == "" {
- configErr = fmt.Errorf("error: OLLAMA_MODELS
environment variable is not set")
- return
+ // Backward compatibility: try OLLAMA_MODELS
+ modelsEnv = os.Getenv("OLLAMA_MODELS")
+ if modelsEnv == "" {
+ configErr = fmt.Errorf("error: LLM_MODELS or
OLLAMA_MODELS environment variable is not set")
+ return
+ }
}
modelsList := strings.Split(modelsEnv, ",")
@@ -72,7 +109,13 @@ func Load(envFile string) (*Config, error) {
return
}
- config.OllamaModels = modelsList
+ // Keep legacy fields for backward compatibility
+ config.LLMModelsList = modelsList
+
+ // For backward compatibility, also set OllamaModels
+ if config.LLMProvider == "ollama" {
+ config.OllamaModels = modelsList
+ }
modelName := os.Getenv("MODEL_NAME")
if modelName == "" {
@@ -104,12 +147,51 @@ func Load(envFile string) (*Config, error) {
return
}
+ // Load LLM base URL and API key
+ llmBaseURL := os.Getenv("LLM_BASE_URL")
+ llmAPIKey := os.Getenv("LLM_API_KEY")
+
+ // For backward compatibility with Ollama
ollamaURL := os.Getenv("OLLAMA_URL")
- if ollamaURL == "" {
- configErr = fmt.Errorf("OLLAMA_URL is not set")
+ if llmBaseURL == "" && ollamaURL != "" {
+ // Use OLLAMA_URL as fallback for LLM_BASE_URL
+ llmBaseURL = ollamaURL
+ }
+
+ // Set default URL for providers if not configured
+ if llmBaseURL == "" && config.LLMProvider == "ollama" {
+ llmBaseURL = DefaultOllamaURL
+ }
+ if llmBaseURL == "" && config.LLMProvider == "openai" {
+ llmBaseURL = DefaultOpenAIURL
+ }
+
+ // Validate configuration based on provider
+ if config.LLMProvider != "ollama" && config.LLMProvider !=
"openai" && llmBaseURL == "" {
+ configErr = fmt.Errorf("LLM_BASE_URL is required for %s
provider", config.LLMProvider)
return
}
- config.OllamaURL = ollamaURL
+
+ // Set URLs and API key
+ config.LLMBaseURL = llmBaseURL
+ config.LLMAPIKey = llmAPIKey
+
+ // For backward compatibility, also set OllamaURL
+ if config.LLMProvider == "ollama" {
+ config.OllamaURL = llmBaseURL
+ }
+
+ // Create LLMConfig array - simple and practical
+ config.LLMModels = make([]LLMConfig, len(modelsList))
+ for i, model := range modelsList {
+ // Each model inherits the global provider settings
+ config.LLMModels[i] = LLMConfig{
+ Model: model,
+ Provider: config.LLMProvider,
+ BaseURL: config.LLMBaseURL,
+ APIKey: config.LLMAPIKey,
+ }
+ }
timeoutStr := os.Getenv("TIME_OUT_SECOND")
if timeoutStr == "" {
@@ -151,6 +233,10 @@ func GetConfig() (*Config, error) {
}
func (c *Config) DefaultModel() string {
+ if len(c.LLMModels) > 0 {
+ return c.LLMModels[0].Model
+ }
+ // Fallback to legacy OllamaModels for backward compatibility
if len(c.OllamaModels) > 0 {
return c.OllamaModels[0]
}
diff --git a/llm/go-server/cmd/server.go b/llm/go-server/cmd/server.go
index ca2c6cff..49497026 100644
--- a/llm/go-server/cmd/server.go
+++ b/llm/go-server/cmd/server.go
@@ -34,18 +34,18 @@ import (
"github.com/dubbogo/gost/log/logger"
"github.com/tmc/langchaingo/llms"
- "github.com/tmc/langchaingo/llms/ollama"
)
import (
"github.com/apache/dubbo-go-samples/llm/config"
+ "github.com/apache/dubbo-go-samples/llm/go-server/llm"
chat "github.com/apache/dubbo-go-samples/llm/proto"
)
var cfg *config.Config
type ChatServer struct {
- llm *ollama.LLM
+ llmService *llm.LLMService
}
func NewChatServer() (*ChatServer, error) {
@@ -53,16 +53,20 @@ func NewChatServer() (*ChatServer, error) {
return nil, fmt.Errorf("MODEL_NAME environment variable is not
set")
}
- llm, err := ollama.New(
- ollama.WithModel(cfg.ModelName),
- ollama.WithServerURL(cfg.OllamaURL),
+ // Create LLM service based on provider
+ llmService, err := llm.NewLLMService(
+ llm.LLMProvider(cfg.LLMProvider),
+ cfg.ModelName,
+ cfg.LLMBaseURL,
+ cfg.LLMAPIKey,
)
if err != nil {
- return nil, fmt.Errorf("failed to initialize model %s: %v",
cfg.ModelName, err)
+ return nil, fmt.Errorf("failed to initialize LLM service for
model %s: %v", cfg.ModelName, err)
}
- logger.Infof("Initialized model: %s", cfg.ModelName)
- return &ChatServer{llm: llm}, nil
+ logger.Infof("Initialized %s model: %s", cfg.LLMProvider, cfg.ModelName)
+
+ return &ChatServer{llmService: llmService}, nil
}
func (s *ChatServer) Chat(ctx context.Context, req *chat.ChatRequest, stream
chat.ChatService_ChatServer) (err error) {
@@ -73,8 +77,8 @@ func (s *ChatServer) Chat(ctx context.Context, req
*chat.ChatRequest, stream cha
}
}()
- if s.llm == nil {
- return fmt.Errorf("LLM model is not initialized")
+ if s.llmService == nil {
+ return fmt.Errorf("LLM service is not initialized")
}
if len(req.Messages) == 0 {
@@ -114,10 +118,10 @@ func (s *ChatServer) Chat(ctx context.Context, req
*chat.ChatRequest, stream cha
messages = append(messages, messageContent)
}
- _, err = s.llm.GenerateContent(
+ err = s.llmService.GenerateContent(
ctx,
messages,
- llms.WithStreamingFunc(func(ctx context.Context, chunk []byte)
error {
+ func(ctx context.Context, chunk []byte) error {
if chunk == nil {
return nil
}
@@ -125,7 +129,7 @@ func (s *ChatServer) Chat(ctx context.Context, req
*chat.ChatRequest, stream cha
Content: string(chunk),
Model: cfg.ModelName,
})
- }),
+ },
)
if err != nil {
logger.Errorf("GenerateContent failed with model %s: %v\n",
cfg.ModelName, err)
diff --git a/llm/go-server/llm/llm_service.go b/llm/go-server/llm/llm_service.go
new file mode 100644
index 00000000..68732919
--- /dev/null
+++ b/llm/go-server/llm/llm_service.go
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package llm
+
+import (
+ "context"
+ "fmt"
+ "os"
+)
+
+import (
+ "github.com/tmc/langchaingo/llms"
+ "github.com/tmc/langchaingo/llms/anthropic"
+ "github.com/tmc/langchaingo/llms/ollama"
+ "github.com/tmc/langchaingo/llms/openai"
+)
+
+import (
+ "github.com/apache/dubbo-go-samples/llm/config"
+)
+
+// LLMProvider represents different LLM service providers
+type LLMProvider string
+
+const (
+ ProviderOllama LLMProvider = "ollama"
+ ProviderOpenAI LLMProvider = "openai"
+ ProviderAnthropic LLMProvider = "anthropic"
+ ProviderAzureOpenAI LLMProvider = "azure-openai"
+)
+
+// LLMService wraps langchaingo LLM with provider information
+type LLMService struct {
+ llm llms.Model
+ provider LLMProvider
+ model string
+}
+
+// NewLLMService creates a new LLM service based on provider configuration
+func NewLLMService(provider LLMProvider, model string, baseURL string, apiKey
string) (*LLMService, error) {
+ var llm llms.Model
+ var err error
+
+ switch provider {
+ case ProviderOllama:
+ if baseURL == "" {
+ baseURL = config.DefaultOllamaURL
+ }
+ llm, err = ollama.New(
+ ollama.WithModel(model),
+ ollama.WithServerURL(baseURL),
+ )
+ if err != nil {
+ return nil, fmt.Errorf("failed to initialize Ollama
model %s: %v", model, err)
+ }
+
+ case ProviderOpenAI:
+ if apiKey == "" {
+ apiKey = os.Getenv("OPENAI_API_KEY")
+ }
+ if apiKey == "" {
+ return nil, fmt.Errorf("OpenAI API key is required")
+ }
+
+ opts := []openai.Option{
+ openai.WithModel(model),
+ openai.WithToken(apiKey),
+ }
+ if baseURL != "" {
+ opts = append(opts, openai.WithBaseURL(baseURL))
+ }
+
+ llm, err = openai.New(opts...)
+ if err != nil {
+ return nil, fmt.Errorf("failed to initialize OpenAI
model %s: %v", model, err)
+ }
+
+ case ProviderAnthropic:
+ if apiKey == "" {
+ apiKey = os.Getenv("ANTHROPIC_API_KEY")
+ }
+ if apiKey == "" {
+ return nil, fmt.Errorf("Anthropic API key is required")
+ }
+
+ opts := []anthropic.Option{
+ anthropic.WithModel(model),
+ anthropic.WithToken(apiKey),
+ }
+ if baseURL != "" {
+ opts = append(opts, anthropic.WithBaseURL(baseURL))
+ }
+
+ llm, err = anthropic.New(opts...)
+ if err != nil {
+ return nil, fmt.Errorf("failed to initialize Anthropic
model %s: %v", model, err)
+ }
+
+ case ProviderAzureOpenAI:
+ if apiKey == "" {
+ apiKey = os.Getenv("AZURE_OPENAI_API_KEY")
+ }
+ if apiKey == "" {
+ return nil, fmt.Errorf("Azure OpenAI API key is
required")
+ }
+
+ // Azure OpenAI uses OpenAI client with specific configuration
+ opts := []openai.Option{
+ openai.WithModel(model),
+ openai.WithToken(apiKey),
+ openai.WithAPIType(openai.APITypeAzure),
+ }
+ if baseURL != "" {
+ opts = append(opts, openai.WithBaseURL(baseURL))
+ }
+
+ llm, err = openai.New(opts...)
+ if err != nil {
+ return nil, fmt.Errorf("failed to initialize Azure
OpenAI model %s: %v", model, err)
+ }
+
+ default:
+ return nil, fmt.Errorf("unsupported LLM provider: %s", provider)
+ }
+
+ return &LLMService{
+ llm: llm,
+ provider: provider,
+ model: model,
+ }, nil
+}
+
+// GenerateContent generates content using the LLM with streaming support
+func (s *LLMService) GenerateContent(ctx context.Context, messages
[]llms.MessageContent, callback func(ctx context.Context, chunk []byte) error)
error {
+ _, err := s.llm.GenerateContent(
+ ctx,
+ messages,
+ llms.WithStreamingFunc(callback),
+ )
+ return err
+}
+
+// GetProvider returns the provider type
+func (s *LLMService) GetProvider() LLMProvider {
+ return s.provider
+}
+
+// GetModel returns the model name
+func (s *LLMService) GetModel() string {
+ return s.model
+}
+
+// GetSupportedProviders returns a list of supported providers
+func GetSupportedProviders() []LLMProvider {
+ return []LLMProvider{
+ ProviderOllama,
+ ProviderOpenAI,
+ ProviderAnthropic,
+ ProviderAzureOpenAI,
+ }
+}
diff --git a/llm/start_servers.sh b/llm/start_servers.sh
index 882aa2f1..dd34365b 100755
--- a/llm/start_servers.sh
+++ b/llm/start_servers.sh
@@ -38,20 +38,30 @@ get_env_value() {
echo "$value"
}
-# Get models from .env file
-OLLAMA_MODELS=$(get_env_value "OLLAMA_MODELS")
+# Get LLM provider and models from .env file
+LLM_PROVIDER=$(get_env_value "LLM_PROVIDER")
+if [ -z "$LLM_PROVIDER" ]; then
+ LLM_PROVIDER="ollama" # Default provider for backward compatibility
+fi
+
+# Get models - try LLM_MODELS first, then fallback to OLLAMA_MODELS for
backward compatibility
+MODELS=$(get_env_value "LLM_MODELS")
+if [ -z "$MODELS" ]; then
+ # Backward compatibility: try OLLAMA_MODELS
+ MODELS=$(get_env_value "OLLAMA_MODELS")
+fi
-# Check if OLLAMA_MODELS is empty
-if [ -z "$OLLAMA_MODELS" ]; then
- echo "Error: OLLAMA_MODELS not found in .env file"
- echo "Please make sure .env file contains a line like: OLLAMA_MODELS =
llava:7b, qwen2.5:7b"
+if [ -z "$MODELS" ]; then
+ echo "Error: LLM_MODELS or OLLAMA_MODELS not found in .env file"
+ echo "Please make sure .env file contains a line like: LLM_MODELS =
llava:7b, qwen2.5:7b"
exit 1
fi
-echo "Found models: $OLLAMA_MODELS"
+echo "Found LLM provider: $LLM_PROVIDER"
+echo "Found models: $MODELS"
# Convert comma-separated string to array, handling spaces
-IFS=',' read -ra MODELS <<< "$OLLAMA_MODELS"
+IFS=',' read -ra MODELS <<< "$MODELS"
current_port=$START_PORT
@@ -84,4 +94,4 @@ echo "All servers started. Total instances: $((${#MODELS[@]}
* INSTANCES_PER_MOD
echo "Use Ctrl+C to stop all servers."
# Wait for all background processes
-wait
\ No newline at end of file
+wait
\ No newline at end of file
