[
https://issues.apache.org/jira/browse/BEAM-11865?focusedWorklogId=558541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-558541
]
ASF GitHub Bot logged work on BEAM-11865:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Feb/21 12:16
Start Date: 26/Feb/21 12:16
Worklog Time Spent: 10m
Work Description: Fokko commented on pull request #14081:
URL: https://github.com/apache/beam/pull/14081#issuecomment-786612636
Thanks for looking at this @dpcollins-google @pabloem
> I don't have decision power on PubsubIO, but I'd really rather have fewer
configuration options here to maintain, not more.
I agree here, but the builder is private, so I need to this is the only way
to make this possible.
> Is there any reason you can't use
`readMessagesWithAttributesAndMessageId()` followed by a MapElements?
I currently use `readMessagesWithAttributes()` with a `ParDo` to do this.
However, I'm running into the limits of dataflow: The job graph is too large.
Please try again with a smaller job graph, or split your job into two or more
smaller jobs.
A `ParDo` is actually quite overkill since the parsing of the messages is
rather simple, based on the header I simply parse the message. Something that
would perfectly fit in the SimpleFunction, and reducing the size of the job
graph.
I need the headers since it contains the message type and the version, so I
know how to parse the message.
Let me know if this is enough context, and that we can get this in. Let me
know if there are any further questions or concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 558541)
Time Spent: 40m (was: 0.5h)
> Add readMessagesWithAttributesWithCoderAndParseFn to the PubSubIO
> -----------------------------------------------------------------
>
> Key: BEAM-11865
> URL: https://issues.apache.org/jira/browse/BEAM-11865
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.28.0
> Reporter: Fokko Driesprong
> Assignee: Fokko Driesprong
> Priority: P2
> Fix For: 2.29.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
