To clarify a bit further, this is all happening before pipeline.run() is called – the pipeline is a streaming pipeline intended to run in perpetuity, and we want to do some basic checking of projects, topic IDs, and such before the pipeline starts, but there are several parameter points that differ between the environments and pipeline runs.
From: Nathan Deren Reply-To: "[email protected]<mailto:[email protected]>" Date: Tuesday, July 25, 2017 at 3:06 PM To: "[email protected]<mailto:[email protected]>" Subject: Trouble with ValueProvider/NestedValueProvider Hi, I’m using Apache Beam via Google Cloud Dataflow, and I’m working on making a template for my job. My team is working to set up QA and prod environments within separate Google projects, so I’d like to be able to pass the project ID as a parameter at runtime. I’ve been playing around with the ValueProvider<String> type but this causes Java to complain at compile time “java.lang.RuntimeException: Not called from a runtime context.” I did some googling around and found a few links, and I’ve been working from the official docs as well. https://stackoverflow.com/questions/43992120/valueprovider-issue https://stackoverflow.com/questions/42843495/google-dataflow-create-templates-with-runtime-parameters/42843682 https://cloud.google.com/dataflow/docs/templates/creating-templates#using-nestedvalueprovider The solution seems to be to use the NestedValueProvider type, but I’m still getting a runtime exception if I call .get() on it. I’ve got the following in my pipeline options (according to the Google Cloud Dataflow docs, I’m still supposed to have (Runtime)ValueProviders in my options file): ValueProvider<String> getDataflowProjectId(); void setDataflowProjectId(ValueProvider<String> value); And I’m composing it into a NestedValueProvider in the following manner: NestedValueProvider<String, String> pid = NestedValueProvider.of( pipelineOptions.getDataflowProjectId(), (SerializableFunction<String, String>) s -> s ); If I try following Google’s example and try to assign the output of NestedValueProvider.of directly to a String, I get the following compilation error: incompatible types: no instance(s) of type variable(s) T,X exist so that org.apache.beam.sdk.options.ValueProvider.NestedValueProvider<T,X> conforms to java.lang.String Java’s not really my language but I’ve been getting along pretty well so far. This however has me completely stumped. What am I doing wrong? Thanks, —Nathan Deren
