[
https://issues.apache.org/jira/browse/BEAM-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien GOUYETTE updated BEAM-2470:
----------------------------------
Description:
0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters
are not persisted in the template file. At runtime, if i try to pass a value
for this parameter, i take a 400 error
I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
My parameters are the following :
{code:java}
trait MyParameters extends PipelineOptions {
def getInput: ValueProvider[String]
def setInput(value: ValueProvider[String]): Unit
}
{code}
They are registred with this code
{code:java}
val options = PipelineOptionsFactory.fromArgs(cmdlineArgs:
_*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
{code}
To create the template i call sbt with this parameters :
{code:java}
run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject
--templateLocation=gs://myBucket/XmlImportTemplate
--tempLocation=gs://myBucket/staging --instance=myInstance
{code}
If i pass explicitly --input, it becomes a StaticValue instead of RuntimeValue,
and this time, i can see it in the template file.
was:
0
down vote
favorite
When i create a dataflow template, the characteristics of Runtime parameters
are not persisted in the template file. At runtime, if i try to pass a value
for this parameter, i take a 400 error
I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
My parameters are the following :
trait MyParameters extends PipelineOptions {
def getInput: ValueProvider[String]
def setInput(value: ValueProvider[String]): Unit
}
They are registred with this code
val options = PipelineOptionsFactory.fromArgs(cmdlineArgs:
_*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
To create the template i call sbt with this parameters :
run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject
--templateLocation=gs://myBucket/XmlImportTemplate
--tempLocation=gs://myBucket/staging --instance=myInstance
If i pass explicitly --input, it becomes a StaticValue instead of RuntimeValue,
and this time, i can see it in the template file.
> Inconsistent behavior on the functioning of the dataflow templates?
> -------------------------------------------------------------------
>
> Key: BEAM-2470
> URL: https://issues.apache.org/jira/browse/BEAM-2470
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Affects Versions: 0.6.0
> Reporter: Damien GOUYETTE
> Assignee: Kenneth Knowles
>
> 0
> down vote
> favorite
> When i create a dataflow template, the characteristics of Runtime parameters
> are not persisted in the template file. At runtime, if i try to pass a value
> for this parameter, i take a 400 error
> I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
> My parameters are the following :
> {code:java}
> trait MyParameters extends PipelineOptions {
> def getInput: ValueProvider[String]
> def setInput(value: ValueProvider[String]): Unit
> }
> {code}
> They are registred with this code
> {code:java}
> val options = PipelineOptionsFactory.fromArgs(cmdlineArgs:
> _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
> PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
> implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
> {code}
> To create the template i call sbt with this parameters :
> {code:java}
> run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject
> --templateLocation=gs://myBucket/XmlImportTemplate
> --tempLocation=gs://myBucket/staging --instance=myInstance
> {code}
> If i pass explicitly --input, it becomes a StaticValue instead of
> RuntimeValue, and this time, i can see it in the template file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
