turbaszek commented on issue #8765: URL: https://github.com/apache/airflow/issues/8765#issuecomment-691977374
Base for `airflow upgrade-check` command is merged (#9276 ) so we can decide what rules we would like to implement. Once we have an initial list I will create issues so multiple people can work on this. Based on this gdoc: > https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit?usp=sharing We may consider the following rules as a good start (those are major breaking changes): - `ConnTypeIsNotNullableRule` - Not-nullable conn_type column in connection table (done) - `UniqueConnIdRule` - Unique conn_id in connection table - `CutomOperatorUsesMetaclassRule` - BaseOperator uses metaclass - `UsingSQLFromBaseHookRule` - Remove SQL support in base_hook - `ChainBetwenDAGAndOperatorNotAllowedRule` - Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported - `AirflowMacroPluginRemovedRule` - Removal of airflow.AirflowMacroPlugin class - `NoAdditionalArgsInOperatorsRule` - Additional arguments passed to BaseOperator cause an exception - `MesosExecutorRemovedRule` - Removal of Mesos Executor and config related changes: - `HostnameCallableRule` - Unify `hostname_callable` option in `core` section - `StatNameHandlerNotSupportedRule` - Drop plugin support for stat_name_handler - `LoggingConfigurationRule` - Logging configuration has been moved to new section - `NoGCPServiceAccountKeyInConfigRule` - Remove gcp_service_account_keys option in airflow.cfg file - `FernetEnabledRule` - Fernet is enabled by default - `KubernetesWorkerAnnotationsRule` - Changes to propagating Kubernetes worker annotations - `LegacyUIDeprecatedRule` - Deprecate legacy UI in favor of FAB RBAC UI - `TaskHandlersMovedRule` - GCSTaskHandler has been moved, WasbTaskHandler has been moved, StackdriverTaskHandler has been moved , S3TaskHandler has been moved, ElasticsearchTaskHandler has been moved, CloudwatchTaskHandler has been moved - `SendGridMovedRule` - SendGrid emailer has been moved - `CustomExecutorsRequireFullPathRule` - Custom executors is loaded using full import path In case of operator related changes I would like to suggest a single rule `OperatorChangesRule`. It will use a map `old_operator_name -> list of possible problems` so we can create a single DagBag and scan all used operators and raise information about changes. It should also suggest what providers packages users should use. I think this is much better approach than having a rule for each operator... I think we should first **focus on the "check and warn" approach so we have any tool**. Once we have it we may consider **"check and apply changes"** - even, in this case, we should probably first focus on major problems (config, new imports, etc). The second step can use "report" generated by `airflow upgrade-check` to apply changes (still can be single command but run wit additional flag). Summoning all elders: @mik-laj @vikramkoka @ashb @potiuk @kaxil @feluelle @dimberman @houqp @ryw @Fokko ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
