sfc-gh-madkins commented on issue #24456: URL: https://github.com/apache/airflow/issues/24456#issuecomment-1175727555
Tomek, Kamil -- Are we ready to merge code to main? ISV doesnt have any best practices when it comes to using "query_tag". You should see in the airflow snowflake connection instantiation code, an "application" parameter gets set for us to track consumption. I am working with the docs team to get an ecosystem section added for Snowpark. I can also coordinate other GTM activities that we might want to do with one or more airflow managed providers. On Sun, Jun 26, 2022 at 7:35 AM Kamil Breguła ***@***.***> wrote: > @sfc-gh-madkins <https://github.com/sfc-gh-madkins> I wonder if I should > set QUERY_TAG > <https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag> > and pass the following values: > > - Instance name > <https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#instance-name> > - DAG ID > - Task ID > - Run ID > > Does Snowflake have any recommendations for ISV on how to use the > query_tag that we can use here? Unfortunately, the public documentation > is quite poor on the description of this parameter and I do not know the > best practices on how it should be used. > > I also wonder what to do to make Snowflake customers know about this > integration. Currently, support for Snowflake in Airflow is not mentioned > in the end-customer Snowflake documentation. For example, we can add Apache > Airflow to the ecosystem section > <https://docs.snowflake.com/en/user-guide/ecosystem.html>. > There is also integration for Apache Beam added, but under the commercial > name - Google Dataflow, so adding Apache Airflow under the commercial name > - Cloud Composer <https://cloud.google.com/composer> is one solution. > [image: Screenshot 2022-06-26 at 14 22 46] > <https://user-images.githubusercontent.com/12058428/175813909-ca28f942-1832-4bb2-8c42-b281bc88c3c0.png> > But Apache Airflow has more service providers, e.g. Amazon Managed > Workflows for Apache Airflow (MWAA) > <https://docs.aws.amazon.com/mwaa/latest/userguide/what-is-mwaa.html> or > Astronomer <https://www.astronomer.io/>. In addition, Apache Beam can > also be run on-premises when using Apache Spark runner instead of Google > Dataflow Runner, which also allows us to use SnowflakeIO for Apache Beam > without Google Dataflow. > > For comparison, dbt has explicitly described Apache Airflow, and they even > published a guide on how to use Apache Airflow with dbt cloud. > > https://docs.getdbt.com/guides/orchestration/airflow-and-dbt-cloud/1-airflow-and-dbt-cloud > > https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production > > CC: @sfc-gh-pgancz <https://github.com/sfc-gh-pgancz>, @sfc-gh-sleslie > <https://github.com/sfc-gh-sleslie> > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/airflow/issues/24456#issuecomment-1166519231>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ATSRCU6T4ABIDOCNAZDWJ4LVRBFA5ANCNFSM5YZNCYZA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
