For gcpTempLocation, you can definitely change this: https://cloud.google.com/dataflow/docs/guides/setting-pipeline-options.
On Thu, Feb 27, 2025 at 5:46 PM Lam Nguyen <lamnguyen110...@gmail.com> wrote: > Hi team, > > We have a very simple use case where a streaming Dataflow job reads KMS > encrypted messages (with lots of PII) from a PubSub topic. Our security > team raised a concern that some of the decrypted data might be stored at > runtime in an internal GCS bucket location. This internal bucket seems to > be encrypted with Google-managed key and cannot work with user-defined key. > The concern here is that anyone having access to the bucket can see the > content and thus the PII data. > > 1. Is the location of the temp data parameterizable by the user? (ie the > gcpTempLocation option) > 2. If not, what is the TTL for the temp data and how can anyone access it? > > Appreciate any pointer/confirmation! Thank you! >
