utkarsharma2 opened a new pull request, #34921:
URL: https://github.com/apache/airflow/pull/34921
This PR is part of our larger effort to add first-class integrations to
support LLMOps that was presented at Airflow Summit. This PR specifically adds
the Cohere Provider. Cohere is a renowned platform offering a range of AI
Models tailored for various NLP tasks. In this iteration, we are integrating
with their Embeddings Model.
The primary objective of this Provider is to present users with an
alternative embedding model. This allows them to generate vectors for their
proprietary data, a pivotal step towards establishing integrations with LLM
models like ChatGPT.
Example DAG:
The `CohereEmbeddingOperator` can accept either a list of strings or a
callable returning a list of strings.
```
from datetime import datetime
from airflow import DAG
from airflow.providers.cohere.operators.embedding import
CohereEmbeddingOperator
with DAG("example_cohere_embedding", schedule=None,
start_date=datetime(2023, 1, 1), catchup=False) as dag:
texts = [
"On Kernel-Target Alignment. We describe a family of global
optimization procedures",
" that automatically decompose optimization problems into smaller
loosely coupled",
" problems, then combine the solutions of these with message passing
algorithms.",
]
def get_text():
return texts
CohereEmbeddingOperator(input_text=texts, task_id="embedding_via_text")
CohereEmbeddingOperator(input_callable=get_text,
task_id="embedding_via_callable")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
