Re: DoFn @Setup with PipelineOptions
I created a ticket to track this: https://issues.apache.org/jira/browse/BEAM-11914. Thanks everyone for the comments! Thanks, Xinyu On Mon, Mar 1, 2021 at 4:45 PM Xinyu Liu wrote: > The reason for not passing it in directly is that we have a large amount > of configs here at LinkedIn so we use an internal config management > framework to hide the actual detailed configs needed to construct these > resources. Internally we put a config map inside the PipelineOptions and > then during @Setup, we would like to init the config framework with the > configs inside PipelineOptions. The user does not need to be aware of how > the configus are populated. They can use something like > > ConfigFramework.create(PipelineOptions).getInstance(SomeResourceFactory) > > to create a resource instance they need. > > On the other hand, even without this kind of use case, it seems still > simpler for the users to use parameters in PipelineOptions if we provide it > directly with @setup. > > Thanks, > Xinyu > > On Mon, Mar 1, 2021 at 4:14 PM Kenneth Knowles wrote: > >> Why not just pass in the arguments to the DoFn constructor or as a >> variable in the containing scope? Do you only know the option after the >> pipeline is completely constructed so you need to make the switch at >> runtime? Makes sense. I think passing options to @Setup is useful and >> harmless. >> >> Kenn >> >> On Mon, Mar 1, 2021 at 3:42 PM Xinyu Liu wrote: >> >>> Hi, all, >>> >>> Currently the @Setup method signature in DoFn does not support any >>> arguments. This is a bit cumbersome to use for use cases such as creating a >>> db connection, rest client or fetch some resources, where we would like to >>> read the configs from the PipelineOptions during setup. Shall we support >>> adding a DoFn SetupContext that can let the users specify the >>> PipelineOptions in the arguments, similar to @StartBundle? Seems the >>> PipelineOptions should always be available when the DoFnRunner is created. >>> Anyone seeing the downside of it? >>> >>> Thanks, >>> Xinyu >>> >>
Re: DoFn @Setup with PipelineOptions
The reason for not passing it in directly is that we have a large amount of configs here at LinkedIn so we use an internal config management framework to hide the actual detailed configs needed to construct these resources. Internally we put a config map inside the PipelineOptions and then during @Setup, we would like to init the config framework with the configs inside PipelineOptions. The user does not need to be aware of how the configus are populated. They can use something like ConfigFramework.create(PipelineOptions).getInstance(SomeResourceFactory) to create a resource instance they need. On the other hand, even without this kind of use case, it seems still simpler for the users to use parameters in PipelineOptions if we provide it directly with @setup. Thanks, Xinyu On Mon, Mar 1, 2021 at 4:14 PM Kenneth Knowles wrote: > Why not just pass in the arguments to the DoFn constructor or as a > variable in the containing scope? Do you only know the option after the > pipeline is completely constructed so you need to make the switch at > runtime? Makes sense. I think passing options to @Setup is useful and > harmless. > > Kenn > > On Mon, Mar 1, 2021 at 3:42 PM Xinyu Liu wrote: > >> Hi, all, >> >> Currently the @Setup method signature in DoFn does not support any >> arguments. This is a bit cumbersome to use for use cases such as creating a >> db connection, rest client or fetch some resources, where we would like to >> read the configs from the PipelineOptions during setup. Shall we support >> adding a DoFn SetupContext that can let the users specify the >> PipelineOptions in the arguments, similar to @StartBundle? Seems the >> PipelineOptions should always be available when the DoFnRunner is created. >> Anyone seeing the downside of it? >> >> Thanks, >> Xinyu >> >
Re: DoFn @Setup with PipelineOptions
Why not just pass in the arguments to the DoFn constructor or as a variable in the containing scope? Do you only know the option after the pipeline is completely constructed so you need to make the switch at runtime? Makes sense. I think passing options to @Setup is useful and harmless. Kenn On Mon, Mar 1, 2021 at 3:42 PM Xinyu Liu wrote: > Hi, all, > > Currently the @Setup method signature in DoFn does not support any > arguments. This is a bit cumbersome to use for use cases such as creating a > db connection, rest client or fetch some resources, where we would like to > read the configs from the PipelineOptions during setup. Shall we support > adding a DoFn SetupContext that can let the users specify the > PipelineOptions in the arguments, similar to @StartBundle? Seems the > PipelineOptions should always be available when the DoFnRunner is created. > Anyone seeing the downside of it? > > Thanks, > Xinyu >
Re: DoFn @Setup with PipelineOptions
Any reason not to simply pass these parameters into the DoFn constructor? On Mon, Mar 1, 2021 at 3:42 PM Xinyu Liu wrote: > Hi, all, > > Currently the @Setup method signature in DoFn does not support any > arguments. This is a bit cumbersome to use for use cases such as creating a > db connection, rest client or fetch some resources, where we would like to > read the configs from the PipelineOptions during setup. Shall we support > adding a DoFn SetupContext that can let the users specify the > PipelineOptions in the arguments, similar to @StartBundle? Seems the > PipelineOptions should always be available when the DoFnRunner is created. > Anyone seeing the downside of it? > > Thanks, > Xinyu >
DoFn @Setup with PipelineOptions
Hi, all, Currently the @Setup method signature in DoFn does not support any arguments. This is a bit cumbersome to use for use cases such as creating a db connection, rest client or fetch some resources, where we would like to read the configs from the PipelineOptions during setup. Shall we support adding a DoFn SetupContext that can let the users specify the PipelineOptions in the arguments, similar to @StartBundle? Seems the PipelineOptions should always be available when the DoFnRunner is created. Anyone seeing the downside of it? Thanks, Xinyu
