The operator is a thin wrapper around their SDK. The qubole SDK makes heavy use of **kwargs, and the operator just passes them through.
Short of writing our own operator with named Params to keep the base operator happy and then delegate to the qubole sdk, there’s no other way to silence the warnings to my knowledge. B Sent from a device with less than stellar autocorrect > On Sep 28, 2018, at 7:56 PM, Chris Palmer <ch...@crpalmer.com> wrote: > > Doesn't this warning come from the BaseOperator class - > https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2511 > ? > > Are you passing unused arguments to the QuboleOperator, or do you not > control the instantiation of those operators? > > Chris > > On Fri, Sep 28, 2018 at 7:39 PM Sean Carey <sean.ca...@aurishealth.com> > wrote: > >> Haha yes, good point. Thanks! >> >> Sean >> >> >> On 9/28/18, 6:13 PM, "Ash Berlin-Taylor" <a...@apache.org> wrote: >> >> Sounds good for your use, certainly. >> >> I mainly wanted to make sure other people knew before blindly >> equipping a foot-canon :) >> >> -ash >> >>> On 29 Sep 2018, at 00:09, Sean Carey <sean.ca...@aurishealth.com> >> wrote: >>> >>> Thanks, Ash. I understand what you're saying. The warnings are >> coming from the Qubole operator. We get a lot of this: >>> >>> PendingDeprecationWarning: Invalid arguments were passed to >> QuboleOperator. Support for passing such arguments will be dropped in >> Airflow 2.0. Invalid arguments were: >>> *args: () >>> **kwargs: {...} >>> category=PendingDeprecationWarning >>> >>> We've spoken to Qubole about this and they plan to address it. In >> the meantime, it generates a ton of noise in our logs which we'd like to >> suppress -- we are aware of the issue, and don't need to be told about it >> every 5 seconds. >>> >>> As for it suddenly breaking, being that this is pending Airflow 2.0 >> I feel the risk is low and when we do upgrade it will be thoroughly tested. >>> >>> Thanks! >>> >>> Sean >>> >>> >>> On 9/28/18, 5:01 PM, "Ash Berlin-Taylor" <a...@apache.org> wrote: >>> >>> What deprecation warnings are you getting? Are they from Airflow >> itself (i.e. things Airflow is calling like flask_wtf, etc) or of your use >> of Airflow? >>> >>> If it is the form could you check and see if someone has already >> reported a Jira issue so we can fix them? >> https://issues.apache.org/jira/issues/?jql=project%3DAIRFLOW >>> >>> If it is the latter PLEASE DO NOT IGNORE THESE. >>> >>> Deprecation warnings are how we, the Airflow community tell users >> that you need to make a change to your DAG/code/config to upgrade things. >> If you silence these warnings you will have a much harder time upgrading to >> new versions of Airflow (read: you might suddenly find that things stop >> working because you turned of the warnings.) >>> >>> -ash >>> >>>> On 28 Sep 2018, at 22:52, Sean Carey <sean.ca...@aurishealth.com> >> wrote: >>>> >>>> Hello, >>>> >>>> I’ve been looking for a way to suppress the >> PendingDeprecationWarning messages cluttering our airflow logs and I have a >> working solution which I thought I would share. >>>> >>>> In order to do this, you first need to configure airflow for custom >> logging using steps 1-4 here: >>>> >>>> >> https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage >>>> >>>> (note that although the document is for Azure remote logging you >> don’t actually need azure for this) >>>> >>>> Next, modify the log_config.py script created in the step above as >> follows: >>>> >>>> >>>> 1. Import logging >>>> 2. Define the filter class: >>>> >>>> >>>> >>>> class DeprecationWarningFilter(logging.Filter): >>>> >>>> def filter(self, record): >>>> >>>> allow = 'DeprecationWarning' not in record.msg >>>> >>>> return allow >>>> >>>> >>>> 1. Add a “filters” section to the LOGGING_CONFIG beneath >> “formatters: >>>> >>>> >>>> >>>> 'filters': { >>>> >>>> 'noDepWarn': { >>>> >>>> '()': DeprecationWarningFilter, >>>> >>>> } >>>> >>>> }, >>>> >>>> >>>> 1. For each of the handlers where you want to suppress the >> warnings (console, task, processor, or any of the remote log handlers you >> may be using) add the following line to its configuration: >>>> >>>> >>>> >>>> 'filters': ['noDepWarn'], >>>> >>>> Restart airflow and your logs should be clean. >>>> >>>> >>>> Sean Carey >>>> >>> >>> >>> >> >> >> >>
