[
https://issues.apache.org/jira/browse/BEAM-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797983#comment-16797983
]
Jan Lukavský commented on BEAM-6873:
------------------------------------
Hm, this might be more complicated than I thought. This issue is, that the
option {{---}}{{project}} cannot be specified when using FlinkRunner, because
{{FlinkPipelineOptions}} are not {{GcpOptions}}. When using flink to read from
Pubsub, the project has to be taken from the topic string
{{projects/<project>/topics/<topic>}}. But, for Dataflow, the project might
(and probably will) be specified using {{---}}{{project}}, which should take
precedence. The problem is that there is no way (that I'm aware of) how to
determine if default value or explicit value has been provided.
A workaround could be that if {{options intanceof GcpOptions}} then the project
from {{GcpOptions}} would be used, otherwise it would use project from the
topic.
Any ideas?
> Simplify PubsubIO projectId management
> --------------------------------------
>
> Key: BEAM-6873
> URL: https://issues.apache.org/jira/browse/BEAM-6873
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.11.0
> Reporter: Jan Lukavský
> Priority: Major
> Fix For: 2.12.0
>
>
> PubsubIO uses some (probably) legacy code around projectId management, which
> among several other suboptimalities causes pipelines without
> {{GcpOptions.project}} specified to fail. This is not necessary, because
> projectId is already contained in {{projects/<project>/topics/<topic>}} when
> reading specified topic. The projectId should be read from there and
> (possibly to maintain backwards compatibility) be overridden by GcpOptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
