[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163381#comment-17163381 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Hi [~stevepak] I addressed your questions in this PR: [https://github.com/apache/airflow/pull/9951] Btw. we migrated to Github issues, and in case of troubleshooting questions I recommend using our slack: [https://apache-airflow-slack.herokuapp.com|https://apache-airflow-slack.herokuapp.com/] > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163060#comment-17163060 ] Steve Pak commented on AIRFLOW-6732: Hi Todd, I am trying to use you GoogleAdsHook and I am confused about the authentications required. Why do I need two? Also, instead of a yaml config could I pass in the service account keys? Apologies, if my questions are dumb, I am super new to this and am very confused > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063239#comment-17063239 ] ASF subversion and git services commented on AIRFLOW-6732: -- Commit 6b9b214e4c3b3afa8ea2e1a5c1e24993013d60ac in airflow's branch refs/heads/master from Todd de Quincey [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6b9b214 ] [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator (#7692) * [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator Provide ability to connect to Google Ads API using service account and execute arbitrary search (API call) using Google Ads Query language. Convert the API results to CSV and store in designated GCS bucket. Add google-ads to setup.py as a gcp package. * Update operators-and-hooks-ref for Google Ads hook and operator * Make updates for minor PR comments - Add api_version as an attribute of GoogleAdsHook - Update CLIENT_IDS in example_google_ads DAG to make it clear that they are dummy values - Add Google Ads API link in hook and operator docstrings * Add tests * Add flush to tempfile Required to ensure the csvfile within the context manager is correctly flushed before being uploaded to GCS. Without this flush, the written data is not correctly saved before upload * Fix docs * Move code to google.ads directory * fixup! Move code to google.ads directory * fixup! fixup! Move code to google.ads directory * fixup! fixup! fixup! Move code to google.ads directory * fixup! fixup! fixup! fixup! Move code to google.ads directory Co-authored-by: Todd de Quincey Co-authored-by: Todd de Quincey Co-authored-by: Tomek Urbaszek > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063238#comment-17063238 ] ASF subversion and git services commented on AIRFLOW-6732: -- Commit 6b9b214e4c3b3afa8ea2e1a5c1e24993013d60ac in airflow's branch refs/heads/master from Todd de Quincey [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6b9b214 ] [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator (#7692) * [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator Provide ability to connect to Google Ads API using service account and execute arbitrary search (API call) using Google Ads Query language. Convert the API results to CSV and store in designated GCS bucket. Add google-ads to setup.py as a gcp package. * Update operators-and-hooks-ref for Google Ads hook and operator * Make updates for minor PR comments - Add api_version as an attribute of GoogleAdsHook - Update CLIENT_IDS in example_google_ads DAG to make it clear that they are dummy values - Add Google Ads API link in hook and operator docstrings * Add tests * Add flush to tempfile Required to ensure the csvfile within the context manager is correctly flushed before being uploaded to GCS. Without this flush, the written data is not correctly saved before upload * Fix docs * Move code to google.ads directory * fixup! Move code to google.ads directory * fixup! fixup! Move code to google.ads directory * fixup! fixup! fixup! Move code to google.ads directory * fixup! fixup! fixup! fixup! Move code to google.ads directory Co-authored-by: Todd de Quincey Co-authored-by: Todd de Quincey Co-authored-by: Tomek Urbaszek > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063237#comment-17063237 ] ASF GitHub Bot commented on AIRFLOW-6732: - turbaszek commented on pull request #7692: [AIRFLOW-6732] Add GoogleAdsHook and GoogleAdsToGcsOperator URL: https://github.com/apache/airflow/pull/7692 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057017#comment-17057017 ] ASF GitHub Bot commented on AIRFLOW-6732: - toddy86 commented on pull request #7692: AIRFLOW-6732 Add GoogleAdsHook and GoogleAdsToGcsOperator URL: https://github.com/apache/airflow/pull/7692 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052321#comment-17052321 ] Todd de Quincey commented on AIRFLOW-6732: -- Hi [~turbaszek], Apologies for the radio silence. It has been an absolutely manic couple of weeks! I still have a few minor things to do (update the docstrings to sphinx format, add unit tests), but I will push to my forked repo tomorrow (early next week at the latest) and will let you know. Would welcome the early comments etc before putting in a PR. Would that work for you? Todd > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051948#comment-17051948 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Hi [~toddy86], any update? > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041101#comment-17041101 ] Michał Słowikowski commented on AIRFLOW-6732: - Thanks a lot [~toddy86], appreciated it. Have a great day! > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041080#comment-17041080 ] Todd de Quincey commented on AIRFLOW-6732: -- [~turbaszek], I will be taking you up on your kind offer of help! > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041078#comment-17041078 ] Todd de Quincey commented on AIRFLOW-6732: -- Hi [~mislow], I have created the following: * GoogleAdsHook - supports connecting to google ads via a service account using the yaml config and access to the search method from an authenticated service. The search method accepts an arbitrary Google Ads Query Language query and returns a list of Google Ads Row objects. See the following link for reference (not exactly the same, but similar use of the service and search methods) [https://developers.google.com/google-ads/api/docs/samples/get-campaigns] * GoogleAdsToGcsOperator - Takes a list of client_ids and a Google Ads Query Language query, gets the results from the API, converts them to a CSV and uploads them to a designated GCS bucket. FYI, the links you have provided above are for the Google Ad Words API. As I understand it, the Google Ads API (still in beta) is being developed to replace this API. > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041010#comment-17041010 ] Michał Słowikowski commented on AIRFLOW-6732: - Hi [~toddy86]. Could list hooks and operators are you going to implement? I am asking because I want to create operator for GA360AdsLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/list]) and GA360ReportingLinksRetrieveOperator ([https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/webPropertyAdWordsLinks/get]). > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040839#comment-17040839 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- No rush indeed, let me know if you need any help or review :) > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033431#comment-17033431 ] Todd de Quincey commented on AIRFLOW-6732: -- We didn't implement a GetAccounts operator. For our use case, we simply created a GoogleAdsHook and a GoogleAdsToGcsOperator which accepts the Google Ads Query Language (SQL like syntax) and stores the results in a CSV in the specified GCS bucket. But based on the hook, this should be easy to implement. Yep, we provided authentication via the yaml config file. This requires two connections to be setup in Airflow. 1) `google_ads_default` and 2) `google_cloud_default`. I should be finished with the tidy up of the code that I need to do in a few weeks. Presuming you aren't in too much of a rush at this stage? > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032229#comment-17032229 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- Great! I wanted to add GoogleAdsGetAccounts ([https://developers.google.com/google-ads/api/docs/account-management/listing-accounts)] and GoogleAdsGetReportOperator. Do you provide authentication using yaml config file? ** > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032225#comment-17032225 ] Todd de Quincey commented on AIRFLOW-6732: -- [~turbaszek], yeah I am happy to do this one. I have already created the custom hook and operator and we have used this in our workplace for the past 4-5 months. So I just need to do some tidy up, unit tests etc and it should be ready for submission. More than happy to get your input and help though. > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032221#comment-17032221 ] Tomasz Urbaszek commented on AIRFLOW-6732: -- [~toddy86] are you going to work on this? I am happy to help :) > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
