ASF GitHub Bot logged work on BEAM-7819:

                Author: ASF GitHub Bot
            Created on: 17/Aug/19 09:23
            Start Date: 17/Aug/19 09:23
    Worklog Time Spent: 10m 
      Work Description: matt-darwin commented on issue #9232: [BEAM-7819] 
Python - parse PubSub message_id into attributes property
URL: https://github.com/apache/beam/pull/9232#issuecomment-522214690
   Back to the isort error now: 
   Running isort...
 Imports are incorrectly sorted.
   2019-08-17 06:59:14.752504
    2019-08-17 07:01:17.704126
   @@ -26,6 +26,7 @@
   import hamcrest as hc
   import mock
   +from google.protobuf.timestamp_pb2 import Timestamp
   import apache_beam as beam
   from apache_beam.io.gcp.pubsub import PubsubMessage
   @@ -52,7 +53,6 @@
   from apache_beam.transforms.display_test import DisplayDataItemMatcher
   from apache_beam.utils import timestamp
   -from google.protobuf.timestamp_pb2 import Timestamp
   # Protect against environments where the PubSub library is not available.
   from google.cloud import pubsub
   Command exited with non-zero status 1
   @aaltay are you happy for the file to be excluded?  If I make the isort 
recommended change, it then errors due to grouping on the google side.
   The issue is if I leave the import in the try block, I get None exceptions 
all over the place for the timestamp object. Or is this simply because the try 
block for imports is triggered by a requirement to import the first module in 
the block and I have them in the wrong order?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 296755)
    Time Spent: 7.5h  (was: 7h 20m)

> PubsubMessage message parsing is lacking non-attribute fields
> -------------------------------------------------------------
>                 Key: BEAM-7819
>                 URL: https://issues.apache.org/jira/browse/BEAM-7819
>             Project: Beam
>          Issue Type: Bug
>          Components: io-py-gcp
>            Reporter: Ahmet Altay
>            Assignee: Udi Meiri
>            Priority: Major
>          Time Spent: 7.5h
>  Remaining Estimate: 0h
> User reported issue: 
> https://lists.apache.org/thread.html/139b0c15abc6471a2e2202d76d915c645a529a23ecc32cd9cfecd315@%3Cuser.beam.apache.org%3E
> """
> Looking at the source code, with my untrained python eyes, I think if the 
> intention is to include the message id and the publish time in the attributes 
> attribute of the PubSubMessage type, then the protobuf mapping is missing 
> something:-
> @staticmethod
> def _from_proto_str(proto_msg):
> """Construct from serialized form of ``PubsubMessage``.
> Args:
> proto_msg: String containing a serialized protobuf of type
> https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
> Returns:
> A new PubsubMessage object.
> """
> msg = pubsub.types.pubsub_pb2.PubsubMessage()
> msg.ParseFromString(proto_msg)
> # Convert ScalarMapContainer to dict.
> attributes = dict((key, msg.attributes[key]) for key in msg.attributes)
> return PubsubMessage(msg.data, attributes)
> The protobuf definition is here:-
> https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.PubsubMessage
> and so it looks as if the message_id and publish_time are not being parsed as 
> they are seperate from the attributes. Perhaps the PubsubMessage class needs 
> expanding to include these as attributes, or they would need adding to the 
> dictionary for attributes. This would only need doing for the _from_proto_str 
> as obviously they would not need to be populated when transmitting a message 
> to PubSub.
> My python is not great, I'm assuming the latter option would need to look 
> something like this?
> attributes = dict((key, msg.attributes[key]) for key in msg.attributes)
> attributes.update({'message_id': msg.message_id, 'publish_time': 
> msg.publish_time})
> return PubsubMessage(msg.data, attributes)
> """

This message was sent by Atlassian JIRA

Reply via email to