[
https://issues.apache.org/jira/browse/NIFI-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985511#comment-15985511
]
ASF GitHub Bot commented on NIFI-3415:
--------------------------------------
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/1658
With PutHiveQL, I sent one valid statement, then one bad one, then another
valid one. The first one was processed correctly and successfully, the second
one caused and error and remained in the queue, but the third one was not
processed nor was it in the incoming or outgoing (failure/retry) queue.
Provenance shows a SEND event for the first one, DROP events for the first and
third, and only the CREATE event for the second (aka "bad") one. I was thinking
the incoming queue would possibly have the second one (if the error is a
ProcessException vs something that routes the flow file) and the third one, or
failure/retry (if a failed flow file is routed) would have the second one.
Here's a stack trace showing the processing of the bad record (in case it
helps):
```
org.apache.nifi.processor.exception.ProcessException: Failed to process
org.apache.nifi.processor.util.pattern.ExceptionHandler$$Lambda$397/1081958194@17605416
due to org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: ParseException line 1:0 cannot recognize input near 'INERT'
'INTO' 'TABLE'
at
org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnGroupError$12(ExceptionHandler.java:210)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnError$11(ExceptionHandler.java:178)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$9(ExceptionHandler.java:53)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$9(ExceptionHandler.java:53)
at
org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:147)
at
org.apache.nifi.processors.hive.PutHiveQL.lambda$new$44(PutHiveQL.java:199)
at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:58)
at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:100)
at
org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$46(PutHiveQL.java:255)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at
org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at
org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1115)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
> Add "Rollback on Failure" property to PutHiveStreaming, PutHiveQL, and PutSQL
> -----------------------------------------------------------------------------
>
> Key: NIFI-3415
> URL: https://issues.apache.org/jira/browse/NIFI-3415
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Matt Burgess
> Assignee: Koji Kawamura
>
> Many Put processors (such as PutHiveStreaming, PutHiveQL, and PutSQL) offer
> "failure" and "retry" relationships for flow files that cannot be processed,
> perhaps due to issues with the external system or other errors.
> However there are use cases where if a Put fails, then no other flow files
> should be processed until the issue(s) have been resolved. This should be
> configurable for said processors, to enable both the current behavior and a
> "stop on failure" type of behavior.
> I propose a property be added to the Put processors (at a minimum the
> PutHiveStreaming, PutHiveQL, and PutSQL processors) called "Rollback on
> Failure", which offers true or false values. If set to true, then the
> "failure" and "retry" relationships should be removed from the processor
> instance, and if set to false, those relationships should be offered.
> If Rollback on Failure is false, then the processor should continue to behave
> as it has. If set to true, then if any error occurs while processing a flow
> file, the session should be rolled back rather than transferring the flow
> file to some error-handling relationship.
> It may also be the case that if Rollback on Failure is true, then the
> incoming connection must use a FIFO Prioritizer, but I'm not positive. The
> documentation should be updated to include any such requirements.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
