[
https://issues.apache.org/jira/browse/BEAM-11980?focusedWorklogId=643048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-643048
]
ASF GitHub Bot logged work on BEAM-11980:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Aug/21 00:31
Start Date: 28/Aug/21 00:31
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #15394:
URL: https://github.com/apache/beam/pull/15394#discussion_r697775806
##########
File path:
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
##########
@@ -454,8 +458,31 @@ public SeekableByteChannel open(GcsPath path) throws
IOException {
@VisibleForTesting
SeekableByteChannel open(GcsPath path, GoogleCloudStorageReadOptions
readOptions)
throws IOException {
- return googleCloudStorage.open(
- new StorageResourceId(path.getBucket(), path.getObject()),
readOptions);
+ HashMap<String, String> baseLabels = new HashMap<>();
+ baseLabels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "");
+ baseLabels.put(MonitoringInfoConstants.Labels.SERVICE, "Storage");
+ baseLabels.put(MonitoringInfoConstants.Labels.METHOD, "Objects.get");
+ baseLabels.put(
+ MonitoringInfoConstants.Labels.RESOURCE,
+ GcpResourceIdentifiers.cloudStorageBucket(path.getBucket()));
+ baseLabels.put(
+ MonitoringInfoConstants.Labels.GCS_PROJECT_ID,
googleCloudStorageOptions.getProjectId());
+ baseLabels.put(MonitoringInfoConstants.Labels.GCS_BUCKET,
path.getBucket());
+
+ ServiceCallMetric serviceCallMetric =
+ new ServiceCallMetric(MonitoringInfoConstants.Urns.API_REQUEST_COUNT,
baseLabels);
+ try {
+ SeekableByteChannel channel =
+ googleCloudStorage.open(
+ new StorageResourceId(path.getBucket(), path.getObject()),
readOptions);
+ serviceCallMetric.call("ok");
+ return channel;
+ } catch (IOException e) {
+ if (e.getCause() instanceof GoogleJsonResponseException) {
+ serviceCallMetric.call(((GoogleJsonResponseException)
e.getCause()).getDetails().getCode());
Review comment:
Has this been verified to be the proper way to catch this using an
integration test?
##########
File path:
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
##########
@@ -487,9 +514,35 @@ public WritableByteChannel create(GcsPath path, String
type, Integer uploadBuffe
GoogleCloudStorage gcpStorage =
new GoogleCloudStorageImpl(
newGoogleCloudStorageOptions, this.storageClient,
this.credentials);
- return gcpStorage.create(
- new StorageResourceId(path.getBucket(), path.getObject()),
-
CreateObjectOptions.builder().setOverwriteExisting(true).setContentType(type).build());
+ HashMap<String, String> baseLabels = new HashMap<>();
+ baseLabels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "");
+ baseLabels.put(MonitoringInfoConstants.Labels.SERVICE, "Storage");
+ baseLabels.put(MonitoringInfoConstants.Labels.METHOD, "Objects.insert");
Review comment:
Lets call this something more general, incase the underlying API changes.
Then upstream code handling the MonitoringInfo won't need to change, as the
Method label will not need to change.
METHOD = "GcsInsert"
##########
File path:
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
##########
@@ -454,8 +458,31 @@ public SeekableByteChannel open(GcsPath path) throws
IOException {
@VisibleForTesting
SeekableByteChannel open(GcsPath path, GoogleCloudStorageReadOptions
readOptions)
throws IOException {
- return googleCloudStorage.open(
- new StorageResourceId(path.getBucket(), path.getObject()),
readOptions);
+ HashMap<String, String> baseLabels = new HashMap<>();
+ baseLabels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "");
+ baseLabels.put(MonitoringInfoConstants.Labels.SERVICE, "Storage");
+ baseLabels.put(MonitoringInfoConstants.Labels.METHOD, "Objects.get");
Review comment:
Lets call this something more general, incase the underlying API changes.
Then upstream code handling the MonitoringInfo won't need to change, as the
Method label will not need to change.
METHOD = "GcsGet"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 643048)
Time Spent: 0.5h (was: 20m)
> Java GCS - Implement IO Request Count metrics
> ---------------------------------------------
>
> Key: BEAM-11980
> URL: https://issues.apache.org/jira/browse/BEAM-11980
> Project: Beam
> Issue Type: Test
> Components: io-java-gcp
> Reporter: Alex Amato
> Assignee: Benjamin Gonzalez
> Priority: P3
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Reference PRs (See BigQuery IO example) and detailed explanation of what's
> needed to instrument this IO with Request Count metrics is found in this
> handoff doc:
> https://docs.google.com/document/d/1lrz2wE5Dl4zlUfPAenjXIQyleZvqevqoxhyE85aj4sc/edit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
