[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=318161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318161
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 25/Sep/19 09:10
Start Date: 25/Sep/19 09:10
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534928885
 
 
   I'm merging since it was already approved earlier. Thanks for review, 
@jklukas !
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 318161)
Time Spent: 4h 40m  (was: 4.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=318160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318160
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 25/Sep/19 09:10
Start Date: 25/Sep/19 09:10
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 318160)
Time Spent: 4.5h  (was: 4h 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=318159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318159
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 25/Sep/19 09:09
Start Date: 25/Sep/19 09:09
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534928885
 
 
   I'm merging since it was already approved earlier.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 318159)
Time Spent: 4h 20m  (was: 4h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317681
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:09
Start Date: 24/Sep/19 17:09
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534655270
 
 
   @jklukas I added javadoc and examples there
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317681)
Time Spent: 4h 10m  (was: 4h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317680
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:08
Start Date: 24/Sep/19 17:08
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327731980
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -73,4 +139,66 @@ public OutputT apply(String input) {
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link ParseJsons}. 
*/
+  public class ParseJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+ParseJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public ParseJsonsWithFailures exceptionsVia(
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317680)
Time Spent: 4h  (was: 3h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317678
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:08
Start Date: 24/Sep/19 17:08
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327731739
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
+ProcessFunction, FailureT> 
exceptionHandler) {
+  return new AsJsonsWithFailures<>(
+  new InferableFunction, 
FailureT>(
+  exceptionHandler) {},
+  failureType);
+}
+
+@Override
+public WithFailures.Result, FailureT> 
expand(PCollection input) {
+  return input.apply(
+  MapElements.into(TypeDescriptors.strings())
+  .via(
+  Contextful.fn(
+  (Contextful.Fn) (input1, c) -> 
writeValue(input1),
+  Requirements.empty()))
+  .exceptionsInto(failureType)
+  .exceptionsVia(exceptionHandler));
+}
+  }
+
+  /**
+   * A default handler that extracts information from an exception to a {@code 
Map}
 
 Review comment:
   I added a link there to this javadoc to avoid copy-paste
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317678)
Time Spent: 3h 40m  (was: 3.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317679
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:08
Start Date: 24/Sep/19 17:08
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327731915
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -55,6 +67,61 @@ private ParseJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, with the given type descriptor used for the failure 
collection but the
+   * exception handler yet to be specified using {@link
+   * ParseJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new ParseJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, passing the raised exception instance and the input 
element being processed
+   * through the given {@code exceptionHandler} and emitting the result to a 
failure collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * json.apply(
+   * ParseJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid POJOs
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new ParseJsonsWithFailures<>(
+exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public ParseJsonsWithFailures>> 
exceptionsVia() {
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317679)
Time Spent: 3h 50m  (was: 3h 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317674
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:06
Start Date: 24/Sep/19 17:06
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327730941
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +70,60 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, with the given type descriptor used for the 
failure collection but the
+   * exception handler yet to be specified using {@link
+   * AsJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new AsJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new AsJsonsWithFailures<>(exceptionHandler, 
exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317674)
Time Spent: 3h 20m  (was: 3h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=317676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-317676
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 24/Sep/19 17:06
Start Date: 24/Sep/19 17:06
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327731249
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
 
 Review comment:
   Done
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 317676)
Time Spent: 3.5h  (was: 3h 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316840
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327231280
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +70,60 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, with the given type descriptor used for the 
failure collection but the
+   * exception handler yet to be specified using {@link
+   * AsJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new AsJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new AsJsonsWithFailures<>(exceptionHandler, 
exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
 
 Review comment:
   Need Javadoc with example usage here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316840)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316841
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327231959
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
 
 Review comment:
   Needs Javadoc.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316841)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316842=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316842
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232550
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
+ProcessFunction, FailureT> 
exceptionHandler) {
+  return new AsJsonsWithFailures<>(
+  new InferableFunction, 
FailureT>(
+  exceptionHandler) {},
+  failureType);
+}
+
+@Override
+public WithFailures.Result, FailureT> 
expand(PCollection input) {
+  return input.apply(
+  MapElements.into(TypeDescriptors.strings())
+  .via(
+  Contextful.fn(
+  (Contextful.Fn) (input1, c) -> 
writeValue(input1),
+  Requirements.empty()))
+  .exceptionsInto(failureType)
+  .exceptionsVia(exceptionHandler));
+}
+  }
+
+  /**
+   * A default handler that extracts information from an exception to a {@code 
Map}
 
 Review comment:
   The info from this class doc would be perfect for filling out Javadoc for 
`exceptionsVia` that instantiates this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316842)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316844
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232852
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -73,4 +139,66 @@ public OutputT apply(String input) {
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link ParseJsons}. 
*/
+  public class ParseJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+ParseJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public ParseJsonsWithFailures exceptionsVia(
 
 Review comment:
   Javadoc needed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316844)
Time Spent: 3h 10m  (was: 3h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316843
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232693
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -55,6 +67,61 @@ private ParseJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, with the given type descriptor used for the failure 
collection but the
+   * exception handler yet to be specified using {@link
+   * ParseJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new ParseJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, passing the raised exception instance and the input 
element being processed
+   * through the given {@code exceptionHandler} and emitting the result to a 
failure collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * json.apply(
+   * ParseJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid POJOs
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new ParseJsonsWithFailures<>(
+exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public ParseJsonsWithFailures>> 
exceptionsVia() {
 
 Review comment:
   Javadoc needed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316843)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316818
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:56
Start Date: 23/Sep/19 16:56
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534185517
 
 
   I like the idea of simplifying the needed boiler-plate code but I tend to 
agree with @lukecwik that, from user perspective, it will be lees evident. So, 
"wrapping" does this job. 
   
   Not sure that this feature (exception handling) is requested by other 
transforms as some asked for Json ones, but it would be great to have simple 
and unified way how to add it into transform API and how to use it. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316818)
Time Spent: 2h 50m  (was: 2h 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316809
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:49
Start Date: 23/Sep/19 16:49
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534185517
 
 
   I like the idea of simplifying the needed boiler code but I tend to agree 
with @lukecwik that, from user perspective, it will be lees evident. So, 
"wrapping" does this job. 
   
   Not sure that this feature (exception handling) is requested by other 
transforms as some asked for Json ones, but it would be great to have simple 
and unified way how to add it into transform API and how to use it. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316809)
Time Spent: 2h 40m  (was: 2.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316788
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:23
Start Date: 23/Sep/19 16:23
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534175290
 
 
   It is easier to maintain but I do think it would be easy for us to have a
   copy of all the methods call the underlying MapElements without needing for
   users to know to invoke unwrapped. Some people discover the API via code
   completion and unwrapped is an unlikely place users will look.
   
   On Mon, Sep 23, 2019 at 8:42 AM Jeff Klukas 
   wrote:
   
   > I'm having a realization that we could probably avoid most of the
   > complexity here if ParseJsons simply returned the MapElements transform
   > it builds rather than wrapping it. I don't think that's particularly an
   > option since users already have code that expects an object of type
   > ParseJsons, but what if we exposed a method to return the underlying
   > MapElements so that users could call the existing error handling methods?
   > It might look like:
   >
   > PCollection p = ...
   > p.apply(ParseJsons
   >.of(MyClass.class)
   >.unwrapped()
   >.exceptionsVia(...))
   >
   > This is slightly less convenient for the user, as they have to know about
   > and invoke the unwrapped method (we can probably come up with a better
   > name), but it avoids us having to re-implement all the logic from
   > MapElements and it would establish a pattern by which other simple
   > transforms that wrap a MapElements invocation could provide error
   > handling.
   >
   > What do you think of that approach?
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316788)
Time Spent: 2.5h  (was: 2h 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316764
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:06
Start Date: 23/Sep/19 16:06
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9499: [BEAM-8100] Exception 
handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534158013
 
 
   I'm having a realization that we could probably avoid most of the complexity 
here if `ParseJsons` simply returned the `MapElements` transform it builds 
rather than wrapping it. I don't think that's particularly an option since 
users already have code that expects an object of type `ParseJsons`, but what 
if we exposed a method to return the underlying `MapElements` so that users 
could call the existing error handling methods? It might look like:
   
   ```
   PCollection p = ...
   p.apply(ParseJsons
  .of(MyClass.class)
  .unwrapped()
  .exceptionsVia(...))
   ```
   
   This is slightly less convenient for the user, as they have to know about 
and invoke the `unwrapped` method (we can probably come up with a better name, 
(`toMap`, `asMap`, `asMapElements`)?), but it avoids us having to re-implement 
all the logic from `MapElements` and it would establish a pattern by which 
other simple transforms that wrap a `MapElements` invocation could provide 
error handling.
   
   What do you think of that approach?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316764)
Time Spent: 2h 20m  (was: 2h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316755
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:42
Start Date: 23/Sep/19 15:42
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9499: [BEAM-8100] Exception 
handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534158013
 
 
   I'm having a realization that we could probably avoid most of the complexity 
here if `ParseJsons` simply returned the `MapElements` transform it builds 
rather than wrapping it. I don't think that's particularly an option since 
users already have code that expects an object of type `ParseJsons`, but what 
if we exposed a method to return the underlying `MapElements` so that users 
could call the existing error handling methods? It might look like:
   
   ```
   PCollection p = ...
   p.apply(ParseJsons
  .of(MyClass.class)
  .unwrapped()
  .exceptionsVia(...))
   ```
   
   This is slightly less convenient for the user, as they have to know about 
and invoke the `unwrapped` method (we can probably come up with a better name), 
but it avoids us having to re-implement all the logic from `MapElements` and it 
would establish a pattern by which other simple transforms that wrap a 
`MapElements` invocation could provide error handling.
   
   What do you think of that approach?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316755)
Time Spent: 2h 10m  (was: 2h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316671
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 13:40
Start Date: 23/Sep/19 13:40
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534103254
 
 
   @jklukas I addressed your review comments and added support of lambda and 
default handlers for `AsJsons` and `ParseJasons` transforms. Could you take a 
look again?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316671)
Time Spent: 2h  (was: 1h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316670
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 13:39
Start Date: 23/Sep/19 13:39
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534103254
 
 
   @jklukas I addressed your review comments and added support of lambda and 
default handlers for `AsJsons` and `ParseJasons` transforms.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316670)
Time Spent: 1h 50m  (was: 1h 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307987
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 16:08
Start Date: 06/Sep/19 16:08
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321806969
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures withFailures(
 
 Review comment:
   > Agree to use the same API names for all transforms. Perhaps, it would make 
sense to add an interface for that?
   
   Yes, if it doesn't end cluttering the API too much. We could potentially put 
this at `WithFailures.ExceptionsVia` and it would have to be templated on 
`` so we would end up with something like:
   
   implements WithFailures.ExceptionsVia>
   
   > Also, do we need to have separate method `exceptionsInto` only for lambdas?
   
   Yes, I think we need a single `exceptionsInto` method. If the handler is 
passed as a lambda, Beam likely won't be able to infer a coder for the failure 
output collection without explicitly giving type information via 
`exceptionsInto`.
   
   > And finally, yes, I also was thinking to have default exception handler. 
So, I think it will be a good idea to add this, thanks.
   
    
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307987)
Time Spent: 1h 40m  (was: 1.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307972
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 15:54
Start Date: 06/Sep/19 15:54
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321801038
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures withFailures(
 
 Review comment:
   Agree to use the same API names for all transforms. Perhaps, it would make 
sense to add an interface for that?
   
   Also, do we need to have separate method `exceptionsInto` only for lambdas?
   
   And finally, yes, I also was thinking to have default exception handler. So, 
I think it  will be a good idea to add this, thanks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307972)
Time Spent: 1.5h  (was: 1h 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307858
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:37
Start Date: 06/Sep/19 13:37
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321733747
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -55,6 +61,34 @@ private ParseJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, passing the raised exception instance and the input 
element being processed
+   * through the given {@code exceptionHandler} and emitting the result to a 
failure collection.
+   *
+   * Example usage:
 
 Review comment:
   ```suggestion
  * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
  * WithFailures.Result}.
  *
  * Example usage:
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307858)
Time Spent: 1h 20m  (was: 1h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307859
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:37
Start Date: 06/Sep/19 13:37
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321730735
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures withFailures(
 
 Review comment:
   My bias would be towards following the API in `MapElements` as closely as 
possible, which would make this method `exceptionsVia` and would suggest that 
we should also have `exceptionsInto` and `exceptionsVia(ProcessFunction)` to 
allow users to define exception processing inline. I would love to see this 
factored out somehow so it's easier to add exception handling to different 
classes with a stable API and without repeating all this boilerplate, but I 
haven't found a suitable alternative.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307859)
Time Spent: 1h 20m  (was: 1h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307856
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:37
Start Date: 06/Sep/19 13:37
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321736061
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures withFailures(
 
 Review comment:
   And perhaps these JSON cases are constrained enough that we could provide a 
default exceptionHandler so that the user doesn't need to specify one. Perhaps 
it would catch only the exceptions we expect to be raised 
(`JsonProcessingException` for AsJsons and `IOException` for ParseJsons), 
reraising an other exceptions, and return a KV that contains the input element 
along with a string containing the exception message.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307856)
Time Spent: 1h  (was: 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307855
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:37
Start Date: 06/Sep/19 13:37
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321737681
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
 
 Review comment:
   Perhaps we should provide specific default exception handlers for AsJsons 
and ParseJsons that are identical to ExceptionAsMapHandler except that they 
catch only the exceptions we expect to be raised (`JsonProcessingException` for 
AsJsons and `IOException` for ParseJsons), reraising any other exceptions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307855)
Time Spent: 50m  (was: 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307857
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:37
Start Date: 06/Sep/19 13:37
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321733602
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
 
 Review comment:
   ```suggestion
  * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
  * WithFailures.Result}.
  *
  * Example usage:
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307857)
Time Spent: 1h 10m  (was: 1h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307847
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:27
Start Date: 06/Sep/19 13:27
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-528854072
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307847)
Time Spent: 40m  (was: 0.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307816
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:04
Start Date: 06/Sep/19 13:04
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-528846455
 
 
   R: @jklukas 
   R: @reuvenlax 
   Could you take a look?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307816)
Time Spent: 0.5h  (was: 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307815
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 13:02
Start Date: 06/Sep/19 13:02
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r321723002
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +63,35 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .withFailures(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures withFailures(
 
 Review comment:
   I'm not sure that `withFailures` is a good name in this case and I couldn't 
find out if any agreement about that exists. So, any advices are welcomed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 307815)
Time Spent: 20m  (was: 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=307718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307718
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 06/Sep/19 09:33
Start Date: 06/Sep/19 09:33
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9499: 
[BEAM-8100] Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499
 
 
   This PR is based on work done in #7736 and it adds a possibility to handle 
exceptions for AsJsons and ParseJsons transforms.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build