Hi,
I am trying to download reports using adword api. I am able to do that on
my local machine but when I try to establish a session on my remote unix
box, I get the following exception,
2018-06-10 13:24:02,326 ERROR [main] com.lendingclub.edh.restful.adaptor.
AdwordsRestClient: Failed to create OAuth credentials. Check OAuth settings
in the file. Exception: {}
shaded.com.lendingclub.edh.google.api.ads.common.lib.exception.
OAuthException: Credential could not be refreshed.
at shaded.com.lendingclub.edh.google.api.ads.common.lib.auth.
OfflineCredentials.generateCredential(OfflineCredentials.java:240)
at com.lendingclub.edh.restful.adaptor.AdwordsRestClient.connect(
AdwordsRestClient.java:98)
at com.lendingclub.edh.restful.adaptor.AdwordsLoadAction.run(
AdwordsLoadAction.java:91)
at com.lendingclub.edh.restful.adaptor.AdwordsLoadAction.main(
AdwordsLoadAction.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:
241)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
350)
at java.net.AbstractPlainSocketImpl.connectToAddress(
AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.
getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(
HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java
:1022)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java
:1020)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessController.doPrivilegedWithCombiner(AccessController
.java:782)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(
HttpURLConnection.java:1019)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(
AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(
HttpURLConnection.java:1316)
at sun.net.www.protocol.http.HttpURLConnection.access$100(HttpURLConnection
.java:91)
at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java
:1283)
at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java
:1281)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessController.doPrivilegedWithCombiner(AccessController
.java:782)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(
HttpURLConnection.java:1280)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(
HttpsURLConnectionImpl.java:250)
at shaded.com.lendingclub.edh.google.api.client.http.javanet.NetHttpRequest
.execute(NetHttpRequest.java:77)
at shaded.com.lendingclub.edh.google.api.client.http.HttpRequest.execute(
HttpRequest.java:981)
at shaded.com.lendingclub.edh.google.api.client.auth.oauth2.TokenRequest.
executeUnparsed(TokenRequest.java:283)
at shaded.com.lendingclub.edh.google.api.client.auth.oauth2.TokenRequest.
execute(TokenRequest.java:307)
at shaded.com.lendingclub.edh.google.api.client.auth.oauth2.Credential.
executeRefreshToken(Credential.java:570)
at shaded.com.lendingclub.edh.google.api.client.googleapis.auth.oauth2.
GoogleCredential.executeRefreshToken(GoogleCredential.java:248)
at shaded.com.lendingclub.edh.google.api.client.auth.oauth2.Credential.
refreshToken(Credential.java:489)
at shaded.com.lendingclub.edh.google.api.ads.common.lib.auth.OAuth2Helper.
callRefreshToken(OAuth2Helper.java:69)
at shaded.com.lendingclub.edh.google.api.ads.common.lib.auth.
OfflineCredentials.generateCredential(OfflineCredentials.java:234)
Please find my implementation attached with the post. The credentials work
on my local machine and I am able to retrieve reports. However the code to
obtain session fails at new
OfflineCredentials.Builder().generateCredential();
Please let me know if I am doing anything wrong.
@Override
public void connect() {
try {
this.logger.info("In connect method");
Credential credential = new OfflineCredentials.Builder()
.forApi(Api.ADWORDS)
.withClientSecrets(this.clientId, this.clientSecret)
.withRefreshToken(this.refreshToken)
.build()
.generateCredential();
this.logger.info("Connection object created");
this.setSession(new AdWordsSession.Builder()
.withDeveloperToken(this.developerToken)
.withOAuth2Credential(credential)
.build());
this.logger.info("Successuly established a session");
} catch (ValidationException ve) {
this.logger.error("Invalid configuration Exception: {}", ve);
} catch (OAuthException oe) {
this.logger.error("Failed to create OAuth credentials. Check
OAuth settings in the file. Exception: {}", oe);
}
}
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/c9bdba79-a6db-4bc2-95d4-5e19eb685055%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
package com.lendingclub.edh.restful.adaptor;
import java.io.IOException;
import java.io.InputStream;
import java.rmi.RemoteException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.joda.time.LocalDate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.api.ads.adwords.axis.factory.AdWordsServices;
import com.google.api.ads.adwords.axis.v201710.cm.ReportDefinitionField;
import com.google.api.ads.adwords.axis.v201710.cm.ReportDefinitionServiceInterface;
import com.google.api.ads.adwords.lib.client.AdWordsSession;
import com.google.api.ads.adwords.lib.client.reporting.ReportingConfiguration;
import com.google.api.ads.adwords.lib.factory.AdWordsServicesInterface;
import com.google.api.ads.adwords.lib.jaxb.v201710.DownloadFormat;
import com.google.api.ads.adwords.lib.jaxb.v201710.ReportDefinitionDateRangeType;
import com.google.api.ads.adwords.lib.utils.DetailedReportDownloadResponseException;
import com.google.api.ads.adwords.lib.utils.ReportDownloadResponse;
import com.google.api.ads.adwords.lib.utils.ReportDownloadResponseException;
import com.google.api.ads.adwords.lib.utils.ReportException;
import com.google.api.ads.adwords.lib.utils.v201710.ReportDownloaderInterface;
import com.google.api.ads.adwords.lib.utils.v201710.ReportQuery;
import com.google.api.ads.common.lib.auth.OfflineCredentials;
import com.google.api.ads.common.lib.auth.OfflineCredentials.Api;
import com.google.api.ads.common.lib.exception.OAuthException;
import com.google.api.ads.common.lib.exception.ValidationException;
import com.google.api.client.auth.oauth2.Credential;
/**
* Implementation of operations for adwords report download.
*
*/
public class AdwordsRestClient implements Restful {
private final Logger logger = LoggerFactory.getLogger(AdwordsRestClient.class);
private static final String EXTENSION = ".csv";
private Set<String> reportsWithoutDateColumn;
private String refreshToken;
private String clientId;
private String clientSecret;
private Map<String, String> customers;
private String developerToken;
private AdWordsSession session;
private String downloadPath;
private AdWordsServicesInterface adWordsServices;
private HdfsUtil hdfsUtil;
/**
* Class to make call to adword api and download reports.
* @param clientId : id of client for oAuth connection
* @param customers : map with clientCustomerName as key and clientCustomerId as value
* @param util : util for performing hadoop operations
* @param clientSecret : for oAuth connection
* @param refreshToken : for oAuth connection
* @param developerToken : for downloading reports
* @param downloadPath : location for writing files in hdfs after downloading.
*/
public AdwordsRestClient(String clientId,
Map<String, String> customers,
String clientSecret,
String refreshToken,
String developerToken,
HdfsUtil util,
String downloadPath) {
this.logger.info("clientId : {}, customers : {}, clientSecret : {}, refreshToken : {}, developerToken : {}, "
+ "downloadPath : {}", clientId, customers, clientSecret, refreshToken, developerToken, downloadPath);
this.clientId = clientId;
this.customers = customers;
this.clientSecret = clientSecret;
this.refreshToken = refreshToken;
this.developerToken = developerToken;
this.downloadPath = downloadPath;
this.setHdfsUtil(util);
this.setAdWordsServices(AdWordsServices.getInstance());
this.reportsWithoutDateColumn = AdwordsUtil.getReportsWithoutDateColumn();
this.logger.info("Done with constructor");
}
@Override
public void connect() {
try {
this.logger.info("In connect method");
Credential credential = new OfflineCredentials.Builder()
.forApi(Api.ADWORDS)
.withClientSecrets(this.clientId, this.clientSecret)
.withRefreshToken(this.refreshToken)
.build()
.generateCredential();
this.logger.info("Connection object created");
this.setSession(new AdWordsSession.Builder()
.withDeveloperToken(this.developerToken)
.withOAuth2Credential(credential)
.build());
this.logger.info("Successuly established a session");
} catch (ValidationException ve) {
this.logger.error("Invalid configuration Exception: {}", ve);
} catch (OAuthException oe) {
this.logger.error("Failed to create OAuth credentials. Check OAuth settings in the file. Exception: {}", oe);
}
}
@Override
public void call() {
if (this.getSession() == null) {
this.logger.error("Adword session is null");
return;
}
this.logger.info("In call method");
for (String clientCustomerId : this.customers.keySet()) {
this.session.setClientCustomerId(clientCustomerId);
for (String reportName : AdwordsUtil.getReportNames()) {
try {
downloadAndSaveToHdfs(reportName, LocalDate.now().minusDays(1), ReportDefinitionDateRangeType.YESTERDAY);
downloadAndSaveToHdfs(reportName, LocalDate.now(), ReportDefinitionDateRangeType.TODAY);
} catch (DetailedReportDownloadResponseException dre) {
// A DetailedReportDownloadResponseException will be thrown if the HTTP status code in the
// response indicates an error occurred and the response body contains XML with further
// information, such as the fieldPath and trigger.
String error = String.format("Report was not downloaded due to a %s with errorText '%s', "
+ "trigger '%s' and field path '%s'",
dre.getClass().getSimpleName(),
dre.getErrorText(),
dre.getTrigger(),
dre.getFieldPath());
this.logger.error(error);
} catch (ReportDownloadResponseException rde) {
// A ReportDownloadResponseException will be thrown if the HTTP status code in the response
// indicates an error occurred, but the response did not contain further details.
this.logger.error(String.format("Report was not downloaded due to: %s", rde));
} catch (ReportException re) {
// A ReportException will be thrown if the download failed due to a transport layer exception.
this.logger.error(String.format("Report was not downloaded due to transport layer exception: %s", re));
} catch (IOException ioe) {
// An IOException in this example indicates that the report's contents could not be written
// to the output file.
this.logger.error(String.format("Report was not written due to an IOException: %s", ioe));
}
}
}
}
/**
* Method to download reports from adword server.
* @param reportName : name of the report to download
* @param date : day for which reports need to be generated.
* @return reports as input stream
* @throws ReportDownloadResponseException : Exception while downloading report
* @throws ReportException : Exception while downloading report
* @throws IOException : Network exception
*/
public InputStream download(String reportName,
ReportDefinitionDateRangeType date)
throws ReportDownloadResponseException, ReportException, IOException {
List<String> fields = getFields(reportName);
this.logger.info("Sucessfully retrieved fields for {}. Fields size : {}", reportName, fields);
ReportQuery.Builder builder = new ReportQuery.Builder()
.fields(fields)
.from(AdwordsUtil.getReportNameEnum(reportName));
if (!this.reportsWithoutDateColumn.contains(reportName)) {
builder.during(date);
}
ReportQuery query = builder.build();
// Optional: Set the reporting configuration of the session to suppress header, column name, or
// summary rows in the report output. You can also configure this via your ads.properties
// configuration file. See AdWordsSession.Builder.from(Configuration) for details.
// In addition, you can set whether you want to explicitly include or exclude zero impression
// rows.
ReportingConfiguration reportingConfiguration =
new ReportingConfiguration.Builder()
.skipReportHeader(true)
.skipColumnHeader(false)
.skipReportSummary(false)
.includeZeroImpressions(false)
.build();
getSession().setReportingConfiguration(reportingConfiguration);
ReportDownloaderInterface reportDownloader =
getAdWordsServices().getUtility(getSession(), ReportDownloaderInterface.class);
// Set the property api.adwords.reportDownloadTimeout or call
// ReportDownloader.setReportDownloadTimeout to set a timeout (in milliseconds)
// for CONNECT and READ in report downloads.
ReportDownloadResponse response =
reportDownloader.downloadReport(query.toString(), DownloadFormat.CSV);
return response.getInputStream();
}
/**
* Method to get fields in a report so that report can be downloaded later with those fields.
* @param reportName : name of the report
* @return list of fields in a report
* @throws RemoteException rmi exception while downloading the file.
*/
public List<String> getFields(String reportName)
throws RemoteException {
List<String> fields = new ArrayList<String>();
ReportDefinitionServiceInterface reportDefinitionService = this.getAdWordsServices().get(this.getSession(),
ReportDefinitionServiceInterface.class);
ReportDefinitionField[] reportDefinitionFields = reportDefinitionService.getReportFields(
AdwordsUtil.getReportNameEnumForFields(reportName));
for (ReportDefinitionField reportDefinitionField : reportDefinitionFields) {
if (reportDefinitionField.getExclusiveFields() == null
|| reportDefinitionField.getExclusiveFields().length == 0) {
fields.add(reportDefinitionField.getFieldName());
}
}
return fields;
}
/**
* Method that downloads a report from adword api and writes it to hdfs.
* @param reportName : name of the report for download.
* @param localDate : time of download (today/yesterday), used to creating file name.
* @param dateRange : day of download (today/yesterday), used to fetch report for that day.
* @throws ReportDownloadResponseException caused while downloading report
* @throws ReportException caused while downloading report
* @throws IOException io exception
*/
public void downloadAndSaveToHdfs(String reportName,
LocalDate localDate,
ReportDefinitionDateRangeType dateRange)
throws ReportDownloadResponseException, ReportException, IOException {
this.logger.info("In downloadAndSaveToHdfs method");
String fileName = getFileName(localDate, reportName);
this.logger.info("fileName : {}", fileName);
InputStream reportStream = download(reportName, dateRange);
this.hdfsUtil.write(reportStream, fileName, this.downloadPath);
}
/**
* gets filename in the form - customer_name.report_name.mm_dd_yyyy.
* @param date : date for which report needs to be downloaded.
* @param reportName : report that needs to be downloaded.
* @return filename
*/
public String getFileName(LocalDate date,
String reportName) {
int day = date.getDayOfMonth();
int month = date.getMonthOfYear();
int year = date.getYear();
String clientCustomerName = this.customers.get(this.session.getClientCustomerId());
String fileName = String.format("%s#%s#%s_%s_%s%s",
clientCustomerName, reportName, month, day, year, EXTENSION);
this.logger.info("fileName : {}", fileName);
return fileName;
}
public AdWordsSession getSession() {
return this.session;
}
public void setSession(AdWordsSession session) {
this.session = session;
}
public AdWordsServicesInterface getAdWordsServices() {
return this.adWordsServices;
}
public void setAdWordsServices(AdWordsServicesInterface adWordsServices) {
this.adWordsServices = adWordsServices;
}
public HdfsUtil getHdfsUtil() {
return this.hdfsUtil;
}
public void setHdfsUtil(HdfsUtil hdfsUtil) {
this.hdfsUtil = hdfsUtil;
}
}
