hey Marco, In your case, as your model inference is remote and has a custom handler, the only difference is that RunInference transform is adding batching before invoking the handler. Today this handler is pretty simple but I would imagine that this RunInference in the future may introduce certain optimizations, like async calls which today you would have to develop yourself in parDo.
Radek On Tue, Apr 1, 2025 at 6:51 AM Sofia’s World <[email protected]> wrote: > Hi Radek > uhm, how does that differ from creating something like this? > am i missing something? > > Kind regards > Marco > > > class LLMProcessor(beam.DoFn): > > def __init__(self, key): > """Initiate the OAI API client.""" > self.client = OpenAI( > api_key=key > ) > > def process(self, element: str): > logging.info(f'Input elements:{element}') > > return self.client.do_some_processing(..) > > > On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user < > [email protected]> wrote: > >> hi Sofia, >> >> here you have nice example >> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb >> >> where CloudVisionModelHandler is custom code that can invoke any client >> library. >> you can pass the key as one of the constructors >> to CloudVisionModelHandler or you can load it from any preferred secret >> managers e.g. >> >> def load_model(self): >> """Initiate the OAI API client.""" >> client = OpenAI( >> >> >> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8') >> ) >> return client >> >> def run_inference(self, batch, oai_client, inference): >> response = oai_client.responses.create( # your LLM magic goes here. >> [..] >> >> using env variables won't work here as like you've noticed, beam is >> running on multiple machines and you can't set env variables there. >> >> >> >> >> >> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <[email protected]> >> wrote: >> >>> Hello >>> presumably is possible to kick off a beam process that invokes an LLM>? >>> but the only issue i have is how/where do i store the OpenAI key for >>> example >>> >>> in my c current colab/pc setup i have the key configured in my >>> environment.. but Beam will run >>> on multiple machines... how do i configure the OPENAI_KEY for example? >>> >>> thanks and regards >>> Marco >>> >>
