steveniemitz commented on a change in pull request #15338:
URL: https://github.com/apache/beam/pull/15338#discussion_r690488023
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the
fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
As a point of conversation, I'm curious what people think of using the
built-in jackson `@JsonDeserialize` vs making our own attribute? The downside
I see of the built-in attribute is that other things may parse pipeline options
(dataflow for example writes it out as json), and this will affect that
deserialization as well.
We could have a `@ParseFunction` attribute or similar to map String -> T
instead possibly, which would only be used from PipelineOptionsFactory.
Curious what people's thoughts are.
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a
string it needs to be double quoted. You can see it in a test just by removing
these catch block and running the test, it'll fail with something like:
```com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test':
was expecting (JSON String, Number, Array, Object or token 'null', 'true' or
'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the
JsonDeserialize attribute support).
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the
fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
I'll double check, but I think it should already be supported. The
places I've seen pass the options into the mapper directly, which will use the
annotations. We only need this workaround here since the PipelineOptions class
isn't directly deserialized.
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
+ } catch (JsonParseException inner) {
+ // rethrow the original exception rather the one thrown from the
fallback attempt
+ throw e;
+ }
+ } else {
+ throw e;
+ }
+ }
+
+ JsonParser parser = new TreeTraversingParser(tree, MAPPER);
+ parser.nextToken();
+ return DESERIALIZATION_CONTEXT.readPropertyValue(parser, prop, type);
Review comment:
> But be aware that you'll want to ensure that all instances used via
the "as" transform use the same annotation
ack, I had saw this but didn't implement it yet pending agreement on using
the jackson annotation. I'll add it in.
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
jackson expects the input to be well formed json, which means if it's a
string it needs to be double quoted. You can see it in a test just by removing
these catch block and running the test, it'll fail with something like:
```
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'test':
was expecting (JSON String, Number, Array, Object or token 'null', 'true' or
'false')
at [Source: (String)"test"; line: 1, column: 5]
```
The deserialization is a different problem (which is addressed by the
JsonDeserialize attribute support).
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
##########
@@ -1572,6 +1588,63 @@ public int compare(Method o1, Method o2) {
return builder.build();
}
+ /**
+ * Attempt to parse an input string into an instance of `type` using an
{@link ObjectMapper}.
+ *
+ * <p>If the getter method is annotated with {@link
+ * com.fasterxml.jackson.databind.annotation.JsonDeserialize} the specified
deserializer will be
+ * used, otherwise the default ObjectMapper deserialization strategy.
+ *
+ * <p>Parsing is attempted twice, once with the raw string value. If that
attempt fails, another
+ * attempt is made by wrapping the value in quotes so that it is interpreted
as a JSON string.
+ */
+ private static Object tryParseObject(String value, JavaType type, Method
method)
+ throws IOException {
+ AnnotationCollector ac = AnnotationCollector.emptyCollector();
+ for (Annotation ann : method.getAnnotations()) {
+ ac = ac.addOrOverride(ann);
+ }
+
+ AnnotatedMethod annotatedMethod =
+ new AnnotatedMethod(
+ new TypeResolutionContext.Empty(MAPPER.getTypeFactory()),
+ method,
+ ac.asAnnotationMap(),
+ null);
+
+ BeanPropertyDefinition propDef =
+
SimpleBeanPropertyDefinition.construct(MAPPER.getDeserializationConfig(),
annotatedMethod);
+
+ BeanProperty prop =
+ new MethodProperty(
+ propDef,
+ type,
+ MAPPER.getDeserializationConfig().findTypeDeserializer(type),
+ ac.asAnnotations(),
+ annotatedMethod);
+
+ JsonNode tree;
+ try {
+ tree = MAPPER.readTree(value);
+ } catch (JsonParseException e) {
+ // try again, quoting the input string if it wasn't already
+ if (!(value.startsWith("\"") && value.endsWith("\""))) {
+ try {
+ tree = MAPPER.readTree("\"" + value + "\"");
Review comment:
ah yeah `convertValue` vs `readValue` use different code paths. afaik
convert will serialize `from` to json first, then read it back, which is
slightly different than the complex object case here, since the input string
potentially is an actual json value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
