Re: On Auto-creating GCS buckets on behalf of users

IMHO, we're erroring a bit to far on making it hard to get started. I would lean towards automatically creating (and using) a bucket, provided it had a name that was unlikely to conflict with others and very obvious when one saw it. (Logging is important too, but also very often ignored and not pre

Re: On Auto-creating GCS buckets on behalf of users

I like the idea of simplifying the user experience by automating part of the initial setup. On the other hand, I see why silently creating billed resources like a GCS bucket could be an issue. I don't think creating an empty bucket is an issue since it doesn't incur any charges yet, but at least lo

Re: On Auto-creating GCS buckets on behalf of users

I do not have a strong opinion about this either way. I think this is fundamentally a UX tradeoff between making it easier to get started and potentially creating unwanted/misconfigured items. I do not have data about what would be more preferable for most users. I believe either option would be fi

Re: On Auto-creating GCS buckets on behalf of users

I think creating the bucket makes sense since it is an improvement in the users experience and simplifies first time users setup needs. We should be clear to tell users that we are doing this on their behalf. On Mon, Jun 22, 2020 at 1:26 PM Pablo Estrada wrote: > Hi everyone, > I've gotten aroun

Re: On Auto-creating GCS buckets on behalf of users

Hi everyone, I've gotten around to making this change, and Udi has been gracious to review it[1]. I figured we have not fully answered the larger question of whether we would truly like to make this change. Here are some thoughts giving me pause: 1. Appropriate defaults - We are not sure we can s

Re: On Auto-creating GCS buckets on behalf of users

I agree with the benefits of auto-creating buckets from an ease of use perspective. My counter argument is that the auto created buckets may not have the right settings for the users. A bucket has multiple settings, some required as (name, storage class) and some optional (acl policy, encryption, r

Re: On Auto-creating GCS buckets on behalf of users

Another idea would be to put default bucket preferences in a .beamrc file so you don't have to remember to pass it every time (this could also contain other default flag values). On Tue, Jul 23, 2019 at 1:43 PM Robert Bradshaw wrote: > On Tue, Jul 23, 2019 at 10:26 PM Chamikara Jayalath > wro

Re: On Auto-creating GCS buckets on behalf of users

On Tue, Jul 23, 2019 at 10:26 PM Chamikara Jayalath wrote: > > On Tue, Jul 23, 2019 at 1:10 PM Kyle Weaver wrote: >> >> I agree with David that at least clearer log statements should be added. >> >> Udi, that's an interesting idea, but I imagine the sheer number of existing >> flags (including m

Re: On Auto-creating GCS buckets on behalf of users

On Tue, Jul 23, 2019 at 1:10 PM Kyle Weaver wrote: > I agree with David that at least clearer log statements should be added. > > Udi, that's an interesting idea, but I imagine the sheer number of > existing flags (including many SDK-specific flags) would make it difficult > to implement. In addi

Re: On Auto-creating GCS buckets on behalf of users

I agree with David that at least clearer log statements should be added. Udi, that's an interesting idea, but I imagine the sheer number of existing flags (including many SDK-specific flags) would make it difficult to implement. In addition, uniform argument names wouldn't necessarily ensure unifo

Re: On Auto-creating GCS buckets on behalf of users

+1 to have a consistent experience across SDKs, and do bucket creation by default, specifically: - Temp locations should be optional. - Autocreation behavior should be documented. - The messages ("using bucket X", or "creating bucket X since temp_location is not specified") should be visible in con

Re: On Auto-creating GCS buckets on behalf of users

Java SDK creates one regional bucket per project and region combination . So it's not a l

Re: On Auto-creating GCS buckets on behalf of users

I would go for #1 since it's a better user experience. Especially for new users who don't understand every step involved on staging/deploying. It's just another (unnecessary) mental concept they don't have to be aware of. Anything that makes it closer to only providing the `--runner` flag without a

Re: On Auto-creating GCS buckets on behalf of users

Do we clean up auto created GCS buckets ? If there's no good way to cleanup, I think it might be better to make this opt-in. Thanks, Cham On Tue, Jul 23, 2019 at 3:25 AM Robert Bradshaw wrote: > I think having a single, default, auto-created temporary bucket per > project for use in GCP (when

Re: On Auto-creating GCS buckets on behalf of users

I think having a single, default, auto-created temporary bucket per project for use in GCP (when running on Dataflow, or running elsewhere but using GCS such as for this BQ load files example), though not ideal, is the best user experience. If we don't want to be automatically creating such things