Great thanks Radek! works like a charm!
kind regards
Marco
On Tue, Apr 1, 2025 at 9:34 AM Sofia’s World <mmistr...@gmail.com> wrote:
> I see .I should have read code better..I did not see run inference is part
> of beam rather than the specific example.thanks
>
> On Tue, 1 Apr 2025, 09:02 Radek Stankiewicz, <radosl...@google.com> wrote:
>
>> hey Marco,
>> In your case, as your model inference is remote and has a custom handler,
>> the only difference is that RunInference transform is adding batching
>> before invoking the handler.
>> Today this handler is pretty simple but I would imagine that this
>> RunInference in the future may introduce certain optimizations, like async
>> calls which today you would have to develop yourself in parDo.
>>
>> Radek
>>
>>
>> On Tue, Apr 1, 2025 at 6:51 AM Sofia’s World <mmistr...@gmail.com> wrote:
>>
>>> Hi Radek
>>> uhm, how does that differ from creating something like this?
>>> am i missing something?
>>>
>>> Kind regards
>>> Marco
>>>
>>>
>>> class LLMProcessor(beam.DoFn):
>>>
>>> def __init__(self, key):
>>> """Initiate the OAI API client."""
>>> self.client = OpenAI(
>>> api_key=key
>>> )
>>>
>>> def process(self, element: str):
>>> logging.info(f'Input elements:{element}')
>>>
>>> return self.client.do_some_processing(..)
>>>
>>>
>>> On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user <
>>> user@beam.apache.org> wrote:
>>>
>>>> hi Sofia,
>>>>
>>>> here you have nice example
>>>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb
>>>>
>>>> where CloudVisionModelHandler is custom code that can invoke any client
>>>> library.
>>>> you can pass the key as one of the constructors
>>>> to CloudVisionModelHandler or you can load it from any preferred secret
>>>> managers e.g.
>>>>
>>>> def load_model(self):
>>>> """Initiate the OAI API client."""
>>>> client = OpenAI(
>>>>
>>>>
>>>> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8')
>>>> )
>>>> return client
>>>>
>>>> def run_inference(self, batch, oai_client, inference):
>>>> response = oai_client.responses.create( # your LLM magic goes here.
>>>> [..]
>>>>
>>>> using env variables won't work here as like you've noticed, beam is
>>>> running on multiple machines and you can't set env variables there.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <mmistr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello
>>>>> presumably is possible to kick off a beam process that invokes an LLM>?
>>>>> but the only issue i have is how/where do i store the OpenAI key for
>>>>> example
>>>>>
>>>>> in my c current colab/pc setup i have the key configured in my
>>>>> environment.. but Beam will run
>>>>> on multiple machines... how do i configure the OPENAI_KEY for example?
>>>>>
>>>>> thanks and regards
>>>>> Marco
>>>>>
>>>>
