cetingokhan commented on PR #62963: URL: https://github.com/apache/airflow/pull/62963#issuecomment-4440929715
Hello @gopidesupavan and @kaxil A bit of context on the design evolution and one open concern I'd appreciate feedback on. ### Architecture shift from the previous version The earlier version of this operator handled everything internally — schema discovery, SQL execution, and validation were all wired directly into the operator class. This PR aligns it with the pattern used by other common.ai operators: the operator is intentionally thin and orchestrates work through toolsets. This meant dropping some features I had explored (collecting multiple metrics from a single SQL query, grouping checks by category, etc.), but I think the result is a much cleaner and more approachable architecture. A big part of that is possible because of the toolset abstraction that was already in place — hat tip to everyone who built that foundation. ### Open concern: durable and missing Airflow config Currently, if durable=True is set on the operator but no durable backend is configured in airflow.cfg, the operator raises an error at runtime. The default is effectively "no config = no durable support", which makes durable=True an opt-in that silently breaks without clear guidance. One option worth considering: default the durable backend to file-based storage when no explicit config is present, so durable=True works out of the box without requiring any Airflow configuration. This would make the opt-in path much more forgiving for new users. ### General note This PR went through quite a few iterations — each one a learning exercise. The current state is as lean and extensible as I could make it. I know you're busy, but whenever you get a chance, I'd genuinely welcome any feedback on missing features or patterns that don't sit right with you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
