Thanks for your response Carlos I have a follow up question. Currently the steps followed by us are as follows :
i. We have datasets of images present in our GCS bucket ii. From there we download a number of selected images ( might be from the same or different datasets ) iii. After that we need to upload all those downloaded raw images into the client's GCS bucket iv. Then we use the GCS_URI of each of those individual images uploaded in the client's bucket to perform some operation and create a jsonl v. Finally we upload that jsonl file ( containing GCS_URI of each of those individual images uploaded in the client's bucket ) into the client's same GCS Bucket Are we doing this right? Or is there any other better / simpler way to achieve the same objective? On Thu, Sep 21, 2023 at 5:58 PM Carlos Magalhães < carlos.magalh...@dp6.com.br> wrote: > Hey Puneet, > You can use the GCS to GCS operator, you'll only need a service account > with access in both buckets. > > https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/transfers/gcs_to_gcs/index.html > > Carlos Magalhães (Ele/He/Él) > Data Engineer > carlos.magalh...@dp6.com.br > mobile: +55 (11) 9 3229-1706 > phone: +55 (11) 3500-4570 <carlos.magalh...@dp6.com.br> > > DP6 > You imagine, our data science makes it possible > > > On Thu, Sep 21, 2023 at 4:17 AM Puneet Jindal < > puneet.jin...@tensormatics.com> wrote: > >> Hi Airflow community, >> >> I have a question for you. >> >> I need to transfer a large number of files(images/videos) from my >> organisation's GCS bucket to my client's GCS bucket. >> >> How can I do that? >> >> Thanks, >> Puneet >> > > > AVISO LEGAL: Esta mensagem e anexo(s) contém informações confidenciais > e/ou legalmente protegidas. Caso tenha recebido por engano, favor > devolvê-la ao remetente e eliminá-la do seu sistema, não divulgando ou > utilizando a totalidade ou parte destas informações > LEGAL NOTICE: This message and attached document(s) contain information of > a confidential nature that is legally protected. If you have received this > message by mistake, please reply to the sender, delete it from your system > and do not disclose or use the information, in whole or in part.
