kennknowles opened a new issue, #18446:
URL: https://github.com/apache/beam/issues/18446
0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters
are not persisted in the template file. At runtime, if i try to pass a value
for this parameter, i take a 400 error
I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
My parameters are the following :
```
trait MyParameters extends PipelineOptions {
def getInput: ValueProvider[String]
def setInput(value:
ValueProvider[String]): Unit
}
```
They are registred with this code
```
val options = PipelineOptionsFactory.fromArgs(cmdlineArgs:
_*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit
val (sc, args) = ContextAndArgs(cmdlineArgs)
```
To create the template i call sbt with this parameters :
```
run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject
--templateLocation=gs://myBucket/XmlImportTemplate
--tempLocation=gs://myBucket/staging --instance=myInstance
```
If i pass explicitly \--input, it becomes a StaticValue instead of
RuntimeValue, and this time, i can see it in the template file.
The template is called from a google function watching a bucket storage
(inspired from
https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/)
:
```
...
dataflow.projects.templates.create({
projectId: projectId,
resource:
{
parameters: {
input: `gs://${file.bucket}/${file.name}`
},
jobName: jobs[job].name,
gcsPath: 'gs://MyBucket/MyTemplate'
}
}
...
```
The 400 error :
```
problem running dataflow template, error was: { Error: (109c1c52dc52fec7):
The workflow could not be
created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: ['input'
(perhaps you meant 'runner')]
at Request._callback
(/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
at Request.self.callback
(/user_code/node_modules/googleapis/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at
Request.<anonymous(/user_code/node_modules/googleapis/node_modules/request/request.js:1171:10)
at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at
IncomingMessage.<anonymous>
(/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code:
400, errors: [ { message:
'(109c1c52dc52fec7): The workflow could not be created. Causes:
(109c1c52dc52fb8e): Found unexpected
parameters: [\'input\' (perhaps you meant \'runner\')]', domain: 'global',
reason: 'badRequest' } ]
}
```
Imported from Jira
[BEAM-2470](https://issues.apache.org/jira/browse/BEAM-2470). Original Jira may
contain additional context.
Reported by: dgouyette.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
