[
https://issues.apache.org/jira/browse/BEAM-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983967#comment-15983967
]
Stephen Sisk commented on BEAM-2031:
------------------------------------
[~lcwik] also mentioned that it's definitely possible to serialize/deserialize
Configurations from PipelineOptions, that roughly looks like:
* Use a mixin to override Configuration's jackson annotations
* Use that to add @JsonSerialize(using = Serializer.class)
@JsonDeserialize(using = Deserializer.class) to override serialize/deserialize
To support this, we would need to add a serviceloader (probably in
ProxyInvocationHandler.java) so that users (including the HadoopFileSystem
code) can register arbitrary jackson modules.
For now, I'd like to use a simple Map<String,String> as the type in
PipelineOptions (and not use the above method) so we can get it working this
week, but it should be very do-able to implement the "Serialize Configuration"
option in the near future and have minimal change from a user's perspective
(the command line would be the same, there'd be a programmatic change since
they'd no longer be passing a string, but a configuration)
> Hadoop FileSystem needs to receive Hadoop Configuration
> -------------------------------------------------------
>
> Key: BEAM-2031
> URL: https://issues.apache.org/jira/browse/BEAM-2031
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-extensions
> Reporter: Stephen Sisk
> Assignee: Stephen Sisk
> Fix For: First stable release
>
>
> Since Beam FileSystem objects are configured via PipelineOptions, we need to
> pass a Hadoop Configuration through PipelineOptions. I think that's very
> solvable, but it does seem semi-complicated.
> cc [[email protected]] I believe you mentioned in the past that you had an
> answer to this - is that written down anywhere?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
