Sometimes in software, we try to be helpful by doing a bunch of work for
the user, but it ends up causing a bunch of confusion and maintenance
headaches.  I think these decorators are an example of that.

What `unify_bucket_name_and_key` does is, it tries to make it so you could
pass the full S3 URI to `key`, or you can provide them separately as bucket
and key, and the decorator will "figure it out".

Then this decorator can also be combined with another one,
@provide_bucket_name, which, if you don't provide the bucket name, it will
grab it from an airflow connection.

Then, sometimes the params are again passed through another function
`get_s3_bucket_key`.

This requires us to add tests to ensure that the decorators are applied in
the correct order, and now, to handle the various async cases (the
decorated fn may be either coroutine or async iterator).

It's a bunch of code and complexity that, really does not need to exist.
I.e. what if bucket were simply bucket, and key were simply key, always?

If it were up to me, I'd just remove all of it.  Just force the user to
supply a bucket and key separately.

We can of course do this since semver allows it.  But some portion of users
will be depending on this change and will have to change their code.

Curious what others think.

Reply via email to