[google-appengine] Re: Python 3: how to deserialize App Engine application logs from StackDriver Logging API?

2019-11-07 Thread 'dustin...@google.com' via Google App Engine 
Not sure why but my original response in this thread did not make it to the 
list. Here's what I wrote:

You can use the LogEntry.to_api_repr() 

 
function to get a JSON version of the LogEntry.

You'll need to use the latest version of the client as well. Here's a 
snippet that might help:

>>> from google.cloud.logging import Client
>>> entries = Client().list_entries(
... ('projects/brid-gy',),
... filter_='resource.type="gae_app" AND protoPayload.@type="
type.googleapis.com/google.appengine.logging.v1.RequestLog"')) 
>>> entry = next(iter(entries))
>>> entry.to_api_repr()


On Thursday, November 7, 2019 at 11:44:25 AM UTC-6 Ryan B wrote:

Interesting. Dustin Ingram (Google DevRel) copied this question to 
StackOverflow 
,
 
and answered it there, but not here. Is that standard for this group?

In any case, thanks Dustin! Except your answer 
 to switch to the v1 
StackDriver Logging API and use LogEntry.to_api_repr() doesn't work for me. 
I get:

Traceback (most recent call last):
  File "", line 1, in 
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/cloud/logging/entries.py"
, line 357, in to_api_repr
info["protoPayload"] = MessageToDict(self.payload)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 168, in MessageToDict
return printer._MessageToJsonObject(message)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 202, in _MessageToJsonObject
return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 314, in _AnyMessageToJsonObject
sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 389, in _CreateMessageFromTypeUrl
'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/
google.appengine.logging.v1.RequestLog.

Searching turns up a few GitHub issues showing that people maintaining 
Google Cloud Python libraries have struggled with this for years. I got 
lost in them and haven't yet managed to find a conclusion:

https://github.com/googleapis/google-cloud-python/issues/2572
https://github.com/googleapis/google-cloud-python/issues/2674
etc.

In case it helps, here are the relevant packages I have installed in my 
virtualenv:

google-api-core1.14.3
google-api-python-client   1.7.11
google-auth1.6.3
google-auth-httplib2   0.0.3
google-cloud-core  1.0.3
google-cloud-logging   1.14.0
googleapis-common-protos   1.6.0
oauth2client   4.1.3


On Thursday, November 7, 2019 at 7:34:42 AM UTC-8, Ryan B wrote:
>
> Hi again! As part of migrating to Python 3, I need to migrate from 
> logservice 
> 
>  
> to the StackDriver Logging API 
> . I have 
> google-cloud-logging  
> installed, and I can successfully fetch GAE application logs with eg:
>
> >>> from google.cloud.logging_v2 import LoggingServiceV2Client
> >>> entries = LoggingServiceV2Client().list_log_entries((
> 'projects/brid-gy',),
>  filter_='resource.type="gae_app" AND protoPayload.@type="
> type.googleapis.com/google.appengine.logging.v1.RequestLog"')
> >>> print(next(iter(entries)))
> proto_payload {
>   type_url: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
>   value: "\n\ts~brid-gy\022\0018\032R5d..."
> }
> ...
>
> This gets me a LogEntry with text application logs in the 
> proto_payload.value field. How do I deserialize that field? I've found 
> lots of related mentions in the docs - links below - but nothing pointing 
> me to a google.appengine.logging.v1.RequestLog protobuf generated class 
> anywhere that I can use, if that's even the right idea. Has anyone done 
> this?
>
>
> https://googleapis.dev/python/logging/latest/gapic/v2/types.html#google.cloud.logging_v2.types.LogEntry.proto_payload
>
> https://cloud.google.com/logging/docs/reference/v2/rpc/google.appengine.logging.v1#requestlog
>
> https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.proto_payload
> https://cloud.google.com/logging/docs/api/v2/resource-list#resource-indexes
>

[google-appengine] Re: Python 3: how to deserialize App Engine application logs from StackDriver Logging API?

2019-11-07 Thread 'George (Cloud Platform Support)' via Google App Engine 
Hello Ryan, 

You are right, posting to stackoverflow is the right thing to do for such a 
subject. You are now able to follow replies and suggestions in the new 
stackoverflow thread. This discussion group is oriented more towards 
general opinions, trends, and issues of general nature touching the app 
engine. For coding and program architecture, as well as de-serializing App 
Engine application logs from StackDriver Logging API, you may be better 
served in dedicated forums such as stackoverflow, where experienced 
programmers are within reach and ready to help.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e9d6a56f-318c-40f0-a8c2-87531c5fab2c%40googlegroups.com.

[google-appengine] Re: Python 3: how to deserialize App Engine application logs from StackDriver Logging API?

2019-11-07 Thread Ryan B 
Interesting. Dustin Ingram (Google DevRel) copied this question to 
StackOverflow 
,
 
and answered it there, but not here. Is that standard for this group?

In any case, thanks Dustin! Except your answer 
 to switch to the v1 
StackDriver Logging API and use LogEntry.to_api_repr() doesn't work for me. 
I get:

Traceback (most recent call last):
  File "", line 1, in 
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/cloud/logging/entries.py"
, line 357, in to_api_repr
info["protoPayload"] = MessageToDict(self.payload)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 168, in MessageToDict
return printer._MessageToJsonObject(message)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 202, in _MessageToJsonObject
return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 314, in _AnyMessageToJsonObject
sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File 
"/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py"
, line 389, in _CreateMessageFromTypeUrl
'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/
google.appengine.logging.v1.RequestLog.

Searching turns up a few GitHub issues showing that people maintaining 
Google Cloud Python libraries have struggled with this for years. I got 
lost in them and haven't yet managed to find a conclusion:

https://github.com/googleapis/google-cloud-python/issues/2572
https://github.com/googleapis/google-cloud-python/issues/2674
etc.

In case it helps, here are the relevant packages I have installed in my 
virtualenv:

google-api-core1.14.3
google-api-python-client   1.7.11
google-auth1.6.3
google-auth-httplib2   0.0.3
google-cloud-core  1.0.3
google-cloud-logging   1.14.0
googleapis-common-protos   1.6.0
oauth2client   4.1.3


On Thursday, November 7, 2019 at 7:34:42 AM UTC-8, Ryan B wrote:
>
> Hi again! As part of migrating to Python 3, I need to migrate from 
> logservice 
> 
>  
> to the StackDriver Logging API 
> . I have 
> google-cloud-logging  
> installed, and I can successfully fetch GAE application logs with eg:
>
> >>> from google.cloud.logging_v2 import LoggingServiceV2Client
> >>> entries = LoggingServiceV2Client().list_log_entries((
> 'projects/brid-gy',),
>  filter_='resource.type="gae_app" AND protoPayload.@type="
> type.googleapis.com/google.appengine.logging.v1.RequestLog"')
> >>> print(next(iter(entries)))
> proto_payload {
>   type_url: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
>   value: "\n\ts~brid-gy\022\0018\032R5d..."
> }
> ...
>
> This gets me a LogEntry with text application logs in the 
> proto_payload.value field. How do I deserialize that field? I've found 
> lots of related mentions in the docs - links below - but nothing pointing 
> me to a google.appengine.logging.v1.RequestLog protobuf generated class 
> anywhere that I can use, if that's even the right idea. Has anyone done 
> this?
>
>
> https://googleapis.dev/python/logging/latest/gapic/v2/types.html#google.cloud.logging_v2.types.LogEntry.proto_payload
>
> https://cloud.google.com/logging/docs/reference/v2/rpc/google.appengine.logging.v1#requestlog
>
> https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.proto_payload
> https://cloud.google.com/logging/docs/api/v2/resource-list#resource-indexes
> https://groups.google.com/d/topic/google-cloud-dev/Xr6tZxOIaJY/discussion
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c28a02dc-b15a-4ff6-9771-023bae2b4e80%40googlegroups.com.