[
https://issues.apache.org/jira/browse/BEAM-6663?focusedWorklogId=199258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199258
]
ASF GitHub Bot logged work on BEAM-6663:
----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/19 14:31
Start Date: 15/Feb/19 14:31
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #7831: [BEAM-6663]
Add the ability to transform to/from json SerializablePipelineOptions
URL: https://github.com/apache/beam/pull/7831#discussion_r257254972
##########
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SerializablePipelineOptions.java
##########
@@ -17,35 +17,68 @@
*/
package org.apache.beam.runners.core.construction;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.Serializable;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
/**
* Holds a {@link PipelineOptions} in JSON serialized form and calls {@link
* FileSystems#setDefaultPipelineOptions(PipelineOptions)} on construction or
on deserialization.
*/
public class SerializablePipelineOptions implements Serializable {
+ private static final ObjectMapper MAPPER =
+ new ObjectMapper()
+
.registerModules(ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
private final String serializedPipelineOptions;
private transient PipelineOptions options;
public SerializablePipelineOptions(PipelineOptions options) {
- this.serializedPipelineOptions =
PipelineOptionsSerializationUtils.serializeToJson(options);
+ this.serializedPipelineOptions = serializeToJson(options);
this.options = options;
FileSystems.setDefaultPipelineOptions(options);
}
+ public SerializablePipelineOptions(String json) {
Review comment:
```
PipelineOptions options = PipelineOptionsFactory.create();
PipelineOptions options2 = PipelineOptionsFactory.create();
assertTrue(options.equals(options2));
assertTrue(Objects.equals(options, options2));
```
both assert fail ! That is because pipelineOptions is a proxy, its equal
impl compares the references. So in my SerializablePipelineOptions.equals()
impl (usefull to test objects created with the 2 constructors), I compare only
the json strings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 199258)
Time Spent: 40m (was: 0.5h)
> Add the ability to transform to/from json SerializablePipelineOptions
> ---------------------------------------------------------------------
>
> Key: BEAM-6663
> URL: https://issues.apache.org/jira/browse/BEAM-6663
> Project: Beam
> Issue Type: Improvement
> Components: runner-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Many runners, store both PipelineOptions and SerializablePipelineOptions in
> their context whereas SerializablePipelineOptions stores both Options and
> json serialized Options. Options can be obtain by
> SerializablePipelineOptions.get() but json form cannot be obtained.
> Similarly, we cannot create a SerializablePipelineOptions from json. It is
> needed in some cases (like spark structured streaming) where we cannot inject
> anything else than Strings into the framework.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
