[
https://issues.apache.org/jira/browse/BEAM-13602?focusedWorklogId=705471&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705471
]
ASF GitHub Bot logged work on BEAM-13602:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Jan/22 10:10
Start Date: 08/Jan/22 10:10
Worklog Time Spent: 10m
Work Description: KevinGG commented on a change in pull request #16449:
URL: https://github.com/apache/beam/pull/16449#discussion_r780473158
##########
File path: sdks/python/apache_beam/internal/metrics/metric.py
##########
@@ -262,4 +262,8 @@ def
bigtable_error_code_to_grpc_status_string(grpc_status_code):
if (grpc_status_code is not None and
grpc_status_code in grpc_to_canonical_gcp_status):
return grpc_to_canonical_gcp_status[grpc_status_code]
+ if grpc_status_code is None:
+ # Bigtable indicates this can be retried but itself has exhausted retry
+ # timeout or there is no retry policy set for bigtable.
+ return grpc_to_canonical_gcp_status[4]
Review comment:
The behavior of the bigtable client is retrying until it exhausts all
the retry budget. There isn't a canonical status to notify the client to retry.
"UNAVAILABLE" indicates the request is retryable but shouldn't apply here. So I
think "DEADLINE_EXCEEDED" is a better fit here.
##########
File path: sdks/python/apache_beam/io/gcp/bigtableio.py
##########
@@ -116,11 +116,13 @@ def __setstate__(self, options):
self.service_call_metric = None
self.written = Metrics.counter(self.__class__, 'Written Row')
- def write_mutate_metrics(self, rows):
- for status in rows:
+ def write_mutate_metrics(self, response):
Review comment:
This is to prevent the cases where the callback tries to get a status
code from a potential None value returned from the bigtable client.
This PR doesn't handle failures of Metrics.counter itself as it's a common
module like any other Python code and it shouldn't fail for no reason.
##########
File path: sdks/python/apache_beam/internal/metrics/metric.py
##########
@@ -262,4 +262,8 @@ def
bigtable_error_code_to_grpc_status_string(grpc_status_code):
if (grpc_status_code is not None and
grpc_status_code in grpc_to_canonical_gcp_status):
return grpc_to_canonical_gcp_status[grpc_status_code]
+ if grpc_status_code is None:
Review comment:
Acked, will send an update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 705471)
Time Spent: 2h 20m (was: 2h 10m)
> Metrics reporting failed bigtable io
> ------------------------------------
>
> Key: BEAM-13602
> URL: https://issues.apache.org/jira/browse/BEAM-13602
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
> Reporter: Ning
> Assignee: Ning
> Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Details see comment in [https://github.com/apache/beam/pull/15562].
> {{{}We are seeing errors which possibly originate from this code:{}}}{{{}def
> write_mutate_metrics(self, rows): for status in rows: grpc_status_string = (
> ServiceCallMetric.bigtable_error_code_to_grpc_status_string( status.code))
> self.service_call_metric.call(grpc_status_string){}}}{{{}Error message:{}}}
> {{{}File
> "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigtableio.py",
> line 123, in write_mutate_metrics status.code)) AttributeError: 'NoneType'
> object has no attribute 'code' [while running 'Write event Bigtable
> row/ParDo(_BigTableWriteFn)-ptransform-58134']{}}}{{{}Would you have any idea
> about this issue?{}}}{{{}We are running{}}}{{{}Python 3.6{}}}
> {{Beam 2.35}}
> {{GCP DataflowRunner}}
> {{In addition, the choice of parameter name (rows) is confusing. Perhaps
> something like responses or response_list ?}}
> {{[https://cloud.google.com/bigtable/docs/writing-data#batch]}}
> {{[https://googleapis.dev/python/bigtable/1.3.0/_modules/google/cloud/bigtable/table.html#Table.mutate_rows]}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
