[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/727 ---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362926
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
+return (HashMap) JSONUtils.INSTANCE.load((String)
strings.get(0), Object.class);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+ }
+ return new ParseException("Unable to parse JSON string");
+}
+ }
+
+ @Stellar(name = "JSON_TO_LIST"
+ , description = "Returns a List object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a List object containing the parsed JSON string"
+ )
+ public static class JsonToList extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_LIST] incorrect
arguments. Usage: JSON_TO_LIST ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
--- End diff --
Sure, done!
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362754
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
--- End diff --
Added check.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362883
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
--- End diff --
Sure, done!
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362796
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -506,29 +506,27 @@ public Object apply(List strings) {
}
}
- @Stellar( name = "PARSE_JSON_STRING"
+ @Stellar(name = "JSON_PARSE"
, description = "Returns a JSON object for the specified JSON
string"
, params = {
"str - the JSON String to convert, may be null"
}
, returns = "an Object containing the parsed JSON string"
)
- public static class ParseJsonString extends BaseStellarFunction {
+ public static class JsonParse extends BaseStellarFunction {
@Override
public Object apply(List strings) {
if (strings == null || strings.size() == 0) {
-throw new IllegalArgumentException("[PARSE_JSON_STRING] incorrect
arguments. Usage: PARSE_JSON_STRING ");
+throw new IllegalArgumentException("[JSON_PARSE] incorrect
arguments. Usage: JSON_PARSE ");
}
--- End diff --
Added check
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138362605
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
--- End diff --
Yup, that makes sense @ottobackwards . Cleaned up now.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138315063
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
--- End diff --
I have a question here. If you just do the load, and the string is
invalid, would you not get the same exception? What is the difference. If the
string is always valid, and that is the most common case, then we are always
doing 2* the processing. That doesn't seem right in a high volume stream system
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138315393
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
--- End diff --
variables = JSONUtils.INSTANCE.load(String, new TypeReference>()
Use TypeReference to determine the type returned, not Object for this case
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138314713
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
--- End diff --
instanceof string
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138314605
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -506,29 +506,27 @@ public Object apply(List strings) {
}
}
- @Stellar( name = "PARSE_JSON_STRING"
+ @Stellar(name = "JSON_PARSE"
, description = "Returns a JSON object for the specified JSON
string"
, params = {
"str - the JSON String to convert, may be null"
}
, returns = "an Object containing the parsed JSON string"
)
- public static class ParseJsonString extends BaseStellarFunction {
+ public static class JsonParse extends BaseStellarFunction {
@Override
public Object apply(List strings) {
if (strings == null || strings.size() == 0) {
-throw new IllegalArgumentException("[PARSE_JSON_STRING] incorrect
arguments. Usage: PARSE_JSON_STRING ");
+throw new IllegalArgumentException("[JSON_PARSE] incorrect
arguments. Usage: JSON_PARSE ");
}
--- End diff --
Should we make sure instanceof string?
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r138315823
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -550,4 +548,90 @@ else if(var.length() == 0) {
return new ParseException("Unable to parse JSON string");
}
}
+
+ @Stellar(name = "JSON_TO_MAP"
+ , description = "Returns a MAP object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a MAP object containing the parsed JSON string"
+ )
+ public static class JsonToMap extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_MAP] incorrect
arguments. Usage: JSON_TO_MAP ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
+// Return parsed JSON Object as a HashMap
+try {
+return (HashMap) JSONUtils.INSTANCE.load((String)
strings.get(0), Object.class);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+ }
+ return new ParseException("Unable to parse JSON string");
+}
+ }
+
+ @Stellar(name = "JSON_TO_LIST"
+ , description = "Returns a List object for the specified JSON
string"
+ , params = {
+ "str - the JSON String to convert, may be null"
+ }
+ , returns = "a List object containing the parsed JSON string"
+ )
+ public static class JsonToList extends BaseStellarFunction {
+
+@Override
+public Object apply(List strings) {
+
+ if (strings == null || strings.size() == 0) {
+throw new IllegalArgumentException("[JSON_TO_LIST] incorrect
arguments. Usage: JSON_TO_LIST ");
+ }
+ String var = (strings.get(0) == null) ? null : (String)
strings.get(0);
+ if (var == null) {
+return null;
+ } else if (var.length() == 0) {
+return var;
+ } else {
+ObjectMapper objectMapper = new ObjectMapper();
+
+// First parse and check if input is valid JSON string
+try {
+ objectMapper.readTree((String) strings.get(0));
+} catch (JsonProcessingException ex) {
+ throw new ParseException("Valid JSON string not supplied", ex);
+} catch (IOException e) {
+ e.printStackTrace();
+}
+
--- End diff --
Ok, I see from your tests it is list object:
SO variables = JSONUtils.INSTANCE.load(new
FileInputStream(variablesFile.get()), new TypeReference
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r137270218
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
I think it will make more sense to authors, and have less of a surprise
factor.
- you need to understand your data
- use the right function
- get good error checking and verification
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r137266659
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
I don't mind having them as convenience functions in addition to the
generic one.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136962617
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
What do you guys think of having PARSE_TO_MAP, PARSE_TO_ functions to
have **some** type safety?
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136954740
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/StringFunctionsTest.java
---
@@ -449,4 +450,68 @@ public void testCountMatches() throws Exception {
Assert.assertTrue(thrown);
}
+
--- End diff --
Thanks @ottobackwards . Have now used multiline strings.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user anandsubbu commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136954647
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
Updated README
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136594535
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
So, the usecase is having an enrichment which stores a JSON blob in HBase
(or some other store) that you want to marshall into a Java object to interact
with via stellar. Some people would prefer to have their data in a readable
format rather than serialized via kryo (like we do in the profiler) or
restricted to a certain format (like we do in enrichments). This opens up
those use-cases.
That JSON blob may be a map or a list or a primitive; the idea is to let
jackson handle the parsing of it, so the bounds are what JSON can represent.
The knowledge of what is returned would be necessary for the caller, though, as
you say.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136436430
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/StringFunctionsTest.java
---
@@ -449,4 +450,68 @@ public void testCountMatches() throws Exception {
Assert.assertTrue(thrown);
}
+
--- End diff --
Would it be possible for you to use the
```java
@multiline
```
Annotation for defining your json strings instead of inline concatenation
and escaping? It makes it more readable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #727: METRON-1146: Add ability to parse JSON string into...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/727#discussion_r136438190
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
---
@@ -450,4 +454,40 @@ public Object apply(List strings) {
}
}
+ @Stellar( name = "PARSE_JSON_STRING"
+ , description = "Returns a JSON object for the specified JSON
string"
+ , params = {
+"str - the JSON String to convert, may be null"
+ }
+ , returns = "an Object containing the parsed JSON string"
+ )
+ public static class ParseJsonString extends BaseStellarFunction {
+
--- End diff --
So we are just returning whatever we get back from Jackson to the caller.
Meaning that they have to understand the content of the json in order to write
the expression correctly. This needs to be documented if it stays like this.
I am not sure I like that. Can you give the use cases or circumstances
that where you imagine this being used? What are the bounds?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
