Or (and here things get interesting and I am looking forward to discussions it can bring) alternatively - maybe we could make the pattern **work** actually .... If we could dynamically instrument such `execute` methods and do all the pre/post processing on the operator, maybe it could be a useful reuse of the operators we have.
As the good "product and management" advice is - when you lead a pack of runners you should often look back and see if they are maybe not turning so that you can course correct when they do... It might be the case that maybe we should let the natural instincts of our users guide us :) Would love to hear what others think J On Tue, Feb 27, 2024 at 8:09 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello here, > > I have seen recently at least a few times that our users started to use a > strange pattern > > @task(task_id='some_id', provide_context=True) > def some_dummy_task(**context): > ti = context['ti'] > cmd2 = 'echo "pushing 2"' > dbt_task = BashOperator(task_id='some_dummy_task_id', > bash_command=cmd2, > do_xcom_push=True) > dbt_task.execute(context)` > > This **works** for them but then - for example - they complain that xcom > is not pushed (well of course it's not if you use operator and manually run > execute). > > Now - while this is a bad pattern - obviously - it seems that our users > **think** they can do itl. And maybe we can do something to prevent them > shooting themselves in their foot? > > I do not know that well the @task decoration magic, but maybe we could > somehow detect the case where someone instantiates the operator (and runs > execute) inside a decorated task and give a helpful error in this case? I > am afraid people will start using it more and more and the sooner we add > protection against it, the better chance we have to contain it. > > J. > > >
